[vcpkg manifest] Manifest Implementation (microsoft#11757)

==== Changes Related to manifests ====

* Add the `manifests` feature flag
  * This only says whether we look for a `vcpkg.json` in the cwd, not
    whether we support parsing manifests (for ports, for example)
* Changes to the manifests RFC
  * `"authors"` -> `"maintainers"`
  * `--x-classic-mode` -> `-manifests` \in `vcpkg_feature_flags`
  * reserve `"core"` in addition to `"default"`, since that's already
    reserved for features
  * Add a small helper note about what identifiers must look like
  * `<license-string>`: SPDX v3.8 -> v3.9
  * `"feature"."description"` is allowed to be an array of strings as well
  * `"version"` -> `"version-string"` for forward-compat with versions
    RFC
* Add the `--feature-flags` option
* Add the ability to turn off feature flags via passing
  `-<feature-flag>` to `VCPKG_FEATURE_FLAGS` or `--feature-flags`
* Add CMake toolchain support for manifests
  * Requires either:
    * a feature flag of `manifests` in either `Env{VCPKG_FEATURE_FLAGS}`
      or `VCPKG_FEATURE_FLAGS`
    * Passing the `VCPKG_ENABLE_MANIFESTS` option
  * The toolchain will install your packages to
    `${VCPKG_MANIFEST_DIR}/vcpkg_installed`.
* Add MSBuild `vcpkg integrate install` support for manifests
  * Requires `VcpkgEnableManifest` to be true
* `vcpkg create` creates a port that has a `vcpkg.json` instead of a
  `CONTROL`
* argparse, abseil, 3fd, and avisynthplus ports switched to manifest
  from CONTROL
* Add support for `--x-manifest-root`, as well as code for finding it if
  not passed
* Add support for parsing manifests!
* Add a filesystem lock!

==== Important Changes which are somewhat unrelated to manifests ====

* Rename `logicexpression.{h,cpp}` to `platform-expression.{h,cpp}`
* Add `PlatformExpression` type which takes the place of the old logic
  expression
  * Split the parsing of platform expressions from checking whether
    they're true or not
  * Eagerly parse PlatformExpressions as opposed to leaving them as
    strings
* Add checking for feature flag consistency
  * i.e., if `-binarycaching` is passed, you shouldn't be passing
    `--binarysource`
* Add the `Json::Reader` type which, with the help of user-defined
  visitors, converts JSON to your internal type
* VcpkgArgParser: place the switch names into a constant as opposed to
  using magic constants
  * In general update the parsing code so that this ^ works
* Add `Port-Version` fields to CONTROL files
  * This replaces the existing practice of
    `Version: <my-version>-<port-version>`

==== Smaller changes ====
* small drive-by cleanups to some CMake
  * `${_VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}` ->
    `${CURRENT_INSTALLED_DIR}`
  * Remove `-analyze` when compiling with clang-cl, since that's not a
    supported flag (vcpkg's build system)
  * Add a message about which compiler is detected by vcpkg's build
    system machinery
* Fix `Expected::then`
* Convert `""` to `{}` for `std::string` and `fs::path`, to avoid a
  `strlen` (additionally, `.empty()` instead of `== ""`, and `.clear()`)
* Add `Strings::strto` which converts strings to numeric types
* Support built-in arrays and `StringView` for `Strings::join`
* Add `operator<` and friends to `StringView`
* Add `substr` to `StringView`
* SourceParagraphParser gets some new errors

Author: strega-nil

.gitignore

@@ -13,6 +13,8 @@
 
 toolsrc/out*
 toolsrc/CMakeSettings.json
+# fuzzing
+sync_dir*
 
 # User-specific files (MonoDevelop/Xamarin Studio)
 *.userprefs
@@ -331,4 +333,4 @@ __pycache__/
 ############################################################
 archives
 .DS_Store
-prefab/
+prefab/

docs/about/privacy.md

@@ -41,7 +41,8 @@ We collect various telemetry events such as the command line used, the time of i
 
 You can see the telemetry events any command by appending `--printmetrics` after the vcpkg command line.
 
-In the source code (included in `toolsrc\`), you can search for calls to the functions `track_property()` and `track_metric()` to see every specific data point we collect.
+In the source code (included in `toolsrc\`), you can search for calls to the functions `track_property()`, `track_feature()`, `track_metric()`, and `track_buildtime()`
+to see every specific data point we collect.
 
 ## Avoid inadvertent disclosure information
 

docs/specifications/manifests.md

@@ -34,12 +34,6 @@ rather than JSON5 or JSON with comments because JSON is the everywhere-supported
 standard. That is not necessarily true of JSON with comments. Additionally, if one needs
 to write a comment, they can do so via `"$reason"` or `"$comment"` fields.
 
-### Why are `<platform-specification>`s so verbose?
-
-In the initial implementation, we didn't want to do more parsing than is strictly necessary,
-especially parsing languages which aren't defined anywhere. We may add a shorter way of
-defining platform specifications in the future (more similar to those in control files).
-
 ## Specification
 
 A manifest file shall have the name `vcpkg.json`, and shall be in the root directory of a package.
@@ -54,26 +48,28 @@ to specify the shape of a value. Note that any object may contain any directives
 a field key that starts with a `$`; these directive shall be ignored by `vcpkg`. Common
 directives may include `"$schema"`, `"$comment"`, `"$reason"`.
 
-A manifest must be a top-level object, and must have at least the following properties:
+A manifest must be a top-level object, and must have at least:
 
 * `"name"`: a `<package-name>`
-* `"version"`: A `string`. This will be defined further later.
-  * [Semver](https://semver.org) is recommended but not required.
+* One (and only one) of the following version fields:
+  * `"version-string"`: A `string`. Has no semantic meaning.
+    Equivalent to `CONTROL`'s `Version:` field.
+  * Other version fields will be defined by the Versions RFC
 
 The simplest vcpkg.json looks like this:
 
 ```json
 {
   "name": "mypackage",
-  "version": "0.1.0-dev"
+  "version-string": "0.1.0-dev"
 }
 ```
 
 Additionally, it may contain the following properties:
 * `"port-version"`: A non-negative integer. If this field doesn't exist, it's assumed to be `0`.
   * Note that this is a change from existing CONTROL files, where versions were a part of the version string
-* `"authors"`: An array of `string`s which contain the authors of a package
-  * `"authors": [ "Nicole Mazzuca <[email protected]>", "שלום עליכם <[email protected]>" ]`
+* `"maintainers"`: An array of `string`s which contain the authors of a package
+  * `"maintainers": [ "Nicole Mazzuca <[email protected]>", "שלום עליכם <[email protected]>" ]`
 * `"description"`: A string or array of strings containing the description of a package
   * `"description": "mypackage is a package of mine"`
 * `"homepage"`: A url which points to the homepage of a package
@@ -86,8 +82,8 @@ Additionally, it may contain the following properties:
 * `"dev-dependencies"`: An array of `<dependency>`s which are required only for developers (testing and the like)
 * `"features"`: An array of `<feature>`s that the package supports
 * `"default-features"`: An array of `<identifier>`s that correspond to features, which will be used by default.
-* `"supports"`: A `<platform-specification>`
-  * `"supports": { "and": [ "win", { "not": "arm" } ] }`
+* `"supports"`: A `<platform-expression>`
+  * `"supports": "windows & !arm"`
 
 Any properties which are not listed, and which do not start with a `$`,
 will be warned against and are reserved for future use.
@@ -105,7 +101,7 @@ Build-Depends: glib, gettext, cairo, fontconfig, freetype, harfbuzz[glib] (!(win
 ```json
 {
   "name": "pango",
-  "version": "1.40.11",
+  "version-string": "1.40.11",
   "port-version": 6,
   "homepage": "https://ftp.gnome.org/pub/GNOME/sources/pango/",
   "description": "Text and font handling library.",
@@ -118,22 +114,15 @@ Build-Depends: glib, gettext, cairo, fontconfig, freetype, harfbuzz[glib] (!(win
     {
       "name": "harfbuzz",
       "features": [ "glib" ],
-      "platform": {
-        "and": [
-          { "not": { "and": [ "windows", "static" ] } },
-          { "not": "osx" }
-        ]
-      }
+      "platform": "!(windows & static) & !osx"
     }
   ]
 }
 ```
 
-You may notice that the platform specification is fairly wordy. See [reasoning](#why-are-platform-specifications-so-verbose) for why.
-
 ## Behavior of the Tool
 
-There will be two "modes" for vcpkg from this point forward: "classic", and "modern".
+There will be two "modes" for vcpkg from this point forward: "classic", and "manifest".
 The former will act exactly like the existing vcpkg workflow, so as to avoid breaking
 anyone. The latter will be the mode only when the user either:
 
@@ -146,61 +135,49 @@ anyone. The latter will be the mode only when the user either:
     * The environment variable `VCPKG_FEATURE_FLAGS`
     * The option `--feature-flags`
       * (e.g., `--feature-flags=binarycaching,manifests`)
+    * If someone wants to use classic mode and silence the warning, they can add the
+      `-manifests` feature flag to disable the mode.
 
-Additionally, we'll add the `--x-classic-mode` flag to allow someone to force classic
-mode.
-
-When in "modern" mode, the `installed` directory will be changed to
+When in "manifest" mode, the `installed` directory will be changed to
 `<manifest-root>/vcpkg_installed` (name up for bikeshedding).
 The following commands will change behavior:
 
 * `vcpkg install` without any port arguments will install the dependencies listed in
   the manifest file, and will remove any dependencies
   which are no longer in the dependency tree implied by the manifest file.
 * `vcpkg install` with port arguments will give an error.
-* `vcpkg x-clean` will be added, and will delete your `vcpkg_installed` directory.
 
-The following commands will not work in modern mode, at least initially:
+The following commands will not work in manifest mode, at least initially:
 
 * `vcpkg x-set-installed`: `vcpkg install` serves the same function
 * `vcpkg remove`
 * `vcpkg export`
-* `vcpkg import`
-* `vcpkg create`
 
-We may add these features back for modern mode once we understand how best to
+We may add these features back for manifest mode once we understand how best to
 implement them.
 
 ### Behavior of the Toolchain
 
-Mostly, the toolchain file stays the same; however, we shall add one public cache variable:
-
-```cmake
-VCPKG_MANIFEST_ROOT:PATH=<path to the directory containing the vcpkg.json file>
-```
-
-and one function:
+Mostly, the toolchain file stays the same; however, we shall add
+two public options:
 
 ```cmake
-vcpkg_acquire_dependencies(
-  [TRIPLET <triplet>]
-  [MANIFEST <path to manifest>]
-  [INSTALL_DIRECTORY <install directory>])
+VCPKG_MANIFEST_MODE:BOOL=<we found a manifest>
+VCPKG_MANIFEST_INSTALL:BOOL=ON
 ```
 
-which installs the dependencies required by the manifest file.
-
-The default for `TRIPLET` is `VCPKG_TARGET_TRIPLET`
-(which is the default triplet for the configured system).
-For example, on x64 Windows, it defaults to `x64-windows`.
+The first option either explicitly turns on, or off, manifest mode;
+otherwise, we default to looking for a manifest file in the directory
+tree upwards from the source directory.
 
-The default for `INSTALL_DIRECTORY` is `${CMAKE_BINARY_DIR}/vcpkg_installed`.
+The `VCPKG_MANIFEST_INSTALL` option tells the toolchain whether to
+install the packages or not -- if you wish to install the manifest
+dependencies manually, you can set this to off, and we also turn it
+off for packages installed by vcpkg.
 
-Additionally, in the course of implementation, we would like to
-look at adding the following function, but may not be able to:
-
-It is almost certain that one should guard any use of this function
-by `if(EXISTS CACHE{VCPKG_MANIFEST_FILE})`.
+Additionally, if `-manifests` is set in the feature flags environment
+variable, we turn off manifest mode in the toolchain, and we act like
+the classic toolchain.
 
 ### Example - CMake Integration
 
@@ -232,7 +209,7 @@ Therefore, in `vcpkg.json`, we'll need to depend on `fmt`:
 ```json
 {
   "name": "example",
-  "version": "0.0.1",
+  "version-string": "0.0.1",
   "dependencies": [
     "fmt"
   ]
@@ -246,11 +223,6 @@ cmake_minimum_required(VERSION 3.14)
 
 project(example CXX)
 
-if(EXISTS CACHE{VCPKG_MANIFEST_FILE})
-  vcpkg_acquire_dependencies()
-endif()
-
-
 add_executable(example src/main.cxx)
 
 find_package(fmt REQUIRED)
@@ -285,7 +257,9 @@ Hello, world!
   * Does not have multiple consecutive hyphens
   * Does not begin nor end with a hyphen
   * Is not a Windows filesystem reserved name
-  * Is not a vcpkg reserved name: "default".
+  * Is not a vcpkg reserved name: "default" or "core".
+  * In other words, it must follow the regex `[a-z0-9]+(-[a-z0-9]+)*`, and must not be any of:
+    * `{ prn, aux, nul, con, lpt[1-9], com[1-9], core, default }`
 * `<package-name>`: A `string` consisting of a non-zero number of `<identifier>`s, separated by `.`.
   * `a.b.c` is valid
   * `a` is valid
@@ -296,15 +270,29 @@ Hello, world!
     * `"name"`: A `<package-name>`
     * Optionally, `"features"`: an array of `<identifier>`s corresponding to features in the package.
     * Optionally, `"default-features"`: a `boolean`. If this is false, then don't use the default features of the package; equivalent to core in existing CONTROL files. If this is true, do the default thing of including the default features.
-    * Optionally, `"platform"`: a `<platform-specification>`
+    * Optionally, `"platform"`: a `<platform-expression>`
   * `<dependency.port>`: No extra fields are required.
-* `<license-string>`: An SPDX license expression at version 3.8.
-* `<platform-specification>`: A specification of a set of platforms; used in platform-specific dependencies and supports fields. One of:
-  * `<platform-specification.exact>`: A string denoting a triplet tag like “windows”, “osx”, etc.
-  * `<platform-specification.not>`: An object containing a member with key "not" and value `<platform-specification>`.
-  * `<platform-specification.and>`: An object containing a member with key "and" and value array of `<platform-specification>`s.
-  * `<platform-specification.or>`: An object containing a member with key "or" and value array of `<platform-specification>`s.
+* `<license-string>`: An SPDX license expression at version 3.9.
+* `<platform-expression>`: A specification of a set of platforms; used in platform-specific dependencies and supports fields. A string that is parsed as follows:
+  * `<platform-expression>`:
+    * `<platform-expression.not>`
+    * `<platform-expression.and>`
+    * `<platform-expression.or>`
+  * `<platform-expression.simple>`:
+    * `( <platform-expression> )`
+    * `<platform-expression.identifier>`
+  * `<platform-expression.identifier>`:
+    * regex: `/^[a-z0-9]+$/`
+  * `<platform-expression.not>`:
+    * `<platform-expression.simple>`
+    * `! <platform-expression.simple>`
+  * `<platform-expression.and>`
+    * `<platform-expression.not>`
+    * `<platform-expression.and> & <platform-expression.not>`
+  * `<platform-expression.or>`
+    * `<platform-expression.not>`
+    * `<platform-expression.or> | <platform-expression.not>`
 * `<feature>`: An object containing the following:
   * `"name"`: An `<identifier>`, the name of the feature
-  * `"description"`: A `string`, the description of the feature
+  * `"description"`: A `string` or array of `string`s, the description of the feature
   * Optionally, `"dependencies"`: An array of `<dependency>`s, the dependencies used by this feature

docs/users/config-environment.md

@@ -15,6 +15,7 @@ subject to change without notice and should be considered highly unstable.
 Non-exhaustive list of off-by-default features:
 
 - `binarycaching`
+- `manifest`
 
 #### EDITOR
 

ports/3fd/CONTROL

@@ -0,0 +1,22 @@
+{
+  "name": "3fd",
+  "version-string": "2.6.2",
+  "port-version": 3,
+  "description": "C++ Framework For Fast Development",
+  "dependencies": [
+    {
+      "name": "boost-lockfree",
+      "platform": "windows"
+    },
+    {
+      "name": "boost-regex",
+      "platform": "windows"
+    },
+    {
+      "name": "poco",
+      "platform": "windows"
+    },
+    "sqlite3",
+    "rapidxml"
+  ]
+}

ports/3fd/vcpkg.json

@@ -0,0 +1,18 @@
+{
+  "name": "abseil",
+  "version-string": "2020-03-03",
+  "port-version": 7,
+  "homepage": "https://github.com/abseil/abseil-cpp",
+  "description": [
+    "an open-source collection designed to augment the C++ standard library.",
+    "Abseil is an open-source collection of C++ library code designed to augment the C++ standard library. The Abseil library code is collected from Google's own C++ code base, has been extensively tested and used in production, and is the same code we depend on in our daily coding lives.",
+    "In some cases, Abseil provides pieces missing from the C++ standard; in others, Abseil provides alternatives to the standard for special needs we've found through usage in the Google code base. We denote those cases clearly within the library code we provide you.",
+    "Abseil is not meant to be a competitor to the standard library; we've just found that many of these utilities serve a purpose within our code base, and we now want to provide those resources to the C++ community as a whole."
+  ],
+  "features": [
+    {
+      "name": "cxx17",
+      "description": "Enable compiler C++17."
+    }
+  ]
+}

ports/abseil/CONTROL

@@ -0,0 +1,7 @@
+{
+  "name": "argparse",
+  "version-string": "2.1",
+  "description": "Argument parser for modern C++",
+  "license": "MIT",
+  "homepage": "https://github.com/p-ranav/argparse"
+}

ports/abseil/vcpkg.json

@@ -0,0 +1,7 @@
+{
+  "name": "avisynthplus",
+  "version-string": "3.6.0",
+  "homepage": "http://avs-plus.net/",
+  "description": "An improved version of the AviSynth frameserver, with improved features and developer friendliness",
+  "supports": "!(uwp | arm | static)"
+}

ports/argparse/CONTROL

@@ -1,3 +1,4 @@
 Source: cmocka
-Version: 1.1.5-1
+Version: 1.1.5
+Port-Version: 2
 Description: An elegant unit testing framework for C with support for mock objects

ports/argparse/vcpkg.json

@@ -3,8 +3,8 @@ _find_package(${ARGS})
 get_filename_component(_cmocka_lib_name ${CMOCKA_LIBRARY} NAME)
 
 set(CMOCKA_LIBRARY
-    debug ${_VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/debug/lib/${_cmocka_lib_name}
-    optimized ${_VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/lib/${_cmocka_lib_name}
+    debug ${CURRENT_INSTALLED_DIR}/debug/lib/${_cmocka_lib_name}
+    optimized ${CURRENT_INSTALLED_DIR}/lib/${_cmocka_lib_name}
 )
 
 set(CMOCKA_LIBRARIES ${CMOCKA_LIBRARY})

ports/avisynthplus/CONTROL

@@ -1,5 +1,6 @@
 Source: libarchive
-Version: 3.4.1-3
+Version: 3.4.1
+Port-Version: 4
 Homepage: https://github.com/libarchive/libarchive
 Description: Library for reading and writing streaming archives
 Build-Depends: zlib