From f34d02017339623f492dbcdd5b955cbc93787ad8 Mon Sep 17 00:00:00 2001 From: memsharded Date: Tue, 4 Feb 2020 13:05:13 +0100 Subject: [PATCH 01/13] configuration section and download cache docs --- configuration.rst | 12 ++++ configuration/download_cache.rst | 82 +++++++++++++++++++++++++++ index.rst | 1 + reference/config_files/conan.conf.rst | 8 +++ 4 files changed, 103 insertions(+) create mode 100644 configuration.rst create mode 100644 configuration/download_cache.rst diff --git a/configuration.rst b/configuration.rst new file mode 100644 index 000000000000..15d48d349563 --- /dev/null +++ b/configuration.rst @@ -0,0 +1,12 @@ +.. _configuration: + +Configuration +--------------- + +The Conan client can be configured to behave differently. Most of the configuration can be found in the :ref:`conan.conf reference `, +but this section aims to be an introduction to the configuration based on different use cases. + +.. toctree:: + :maxdepth: 1 + + configuration/download_cache diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst new file mode 100644 index 000000000000..46e42966ad9f --- /dev/null +++ b/configuration/download_cache.rst @@ -0,0 +1,82 @@ +.. _download_cache: + +Download cache +============== + +.. warning:: + + This is an **experimental** feature subject to breaking changes in future releases. + +.. warning:: + + This is an **advanced** feature, use it only if you fully understand this document. Do not use it if you are mixing api V1 (without revisions) + and api V2 (with revisions) clients. + + + +Conan implements a shared download cache that can be used to reduce the time needed to populate the Conan package cache +with commands like :command:`install`, :command:`create`. + +This cache is purely an optimization mechanism. It is completely different to the Conan package cache, (typically the ``/.conan`` folder). + +This cache will store in a given path, which can be configured in **conan.conf** file, the following: + +- All files that are downloaded from a Conan server (conan_server, Artifactory), both in the api V1 (without revisions) and V2 (with revisions). + This include files like *conanfile.py*, but also the zipped artifacts like *conan_package.tgz* or *conan_sources.tgz* +- The downloads done by users with the ``tools.download()`` or ``tools.get()`` helpers, as long as they provide a checksum (md5, sha1, etc.). If + a checksum is not provided, even if the download cache is enabled, the download will be always executed. + +The cache uses the URL of the download, appending the checksum when provided. In the api V2, it is not necessary to append the checksum, because +the URL itself already encodes the recipe revision and/or the package revisions, which are already checksums of the recipe and package respectively. + +.. warning:: + + The download cache will not be able to correctly cache artifacts with revisions enabled if those artifacts are created and uploaded repeatedly + with a client without revisions, because that will keep overwriting the revision "0". + + +Activating the download cache +----------------------------- +The download cache is activated and configured in the *conan.conf* file (see :ref:`conan.conf reference `) + +.. code-block:: text + + [storage] + download_cache=/path/to/my/cache + +You can also define it with a Conan comand + +.. code-block:: bash + + $ conan config set storage.download_cache="/path/to/my/cache" + # Display it + $ conan config get storage.download_cache + + +And, as the *conan.conf* is part of the configuration, you can also put a common *conan.conf* file in a git repo or zip file and use +the :ref:`conan_config_install` command to automatically install it in Conan clients. + + +Concurrency, multiple caches and CI +----------------------------------- + +The downloads cache implements exclusive locks for concurrency, so it can be shared among different concurrent Conan instances. +This is a typical scenario in CI servers, in which each job uses a different Conan package cache (defined by ``CONAN_USER_HOME`` environment +variable). Every different Conan instance could configure its download cache to share the same storage. The download cache implements interprocess +exclusive locks, so only 1 process will access at a time to a given cached artifact. If other processes needs the same artifact, they will wait +until it is released, avoiding multiple downloads of the same file, even if they were requested almost simultaneously. + +For Continuous Integration processes, it is recommended to have a different Conan package cache (``CONAN_USER_HOME``) for each job, in most of the cases, +because the Conan package cache is not concurrent, and it might also have old dependencies, stale packages, etc. It is better to run CI jobs in a clean +environment. + + +Removing cached files +--------------------- + +The download cache will store a lot of artifacts, for all recipes, packages, versions and configurations that are used. This can grow and consume +a lot of storage. If you are using this feature, provide for a sufficiently large and fast download cache folder. + +At the moment, it is only a folder. You can clean the cached artifacts just by removing that folder and its contents. You might also be able to +run scripts and jobs that remove old artifacts only. If you do such operations, please make sure that there are not other Conan processes using +it simultaneously, or they might fail. diff --git a/index.rst b/index.rst index 3405e439e363..4c53a6a87bfc 100644 --- a/index.rst +++ b/index.rst @@ -24,6 +24,7 @@ Contents: systems_cross_building extending integrations + configuration howtos reference videos diff --git a/reference/config_files/conan.conf.rst b/reference/config_files/conan.conf.rst index 73ae698d9f61..e7d6dbcfc10a 100644 --- a/reference/config_files/conan.conf.rst +++ b/reference/config_files/conan.conf.rst @@ -66,6 +66,7 @@ The typical location of the **conan.conf** file is the directory ``~/.conan/``: # path beginning with "~" (if the environment var CONAN_USER_HOME is specified, this directory, even # with "~/", will be relative to the conan user home, not to the system user home) path = ./data + # download_cache = /path/to/my/cache [proxies] # Empty (or missing) section will try to use system proxies. @@ -218,6 +219,13 @@ On Windows: If you want to change the default "conan home" (directory where ``conan.conf`` file is) you can adjust the environment variable ``CONAN_USER_HOME``. + +The ``storage.download_cache`` variable defines the path to a folder that can be used to cache the different file downloads from Conan servers but +also from user downloads via the ``tools.get()`` and ``tools.download()`` methods that provide a checksum. Defining this variable will both configure +the path and activate the download cache. If it is not defined, the download cache will not be used. + +Read more about the :ref:`download cache here `. + .. _proxys: Proxies From 51d5a985fb050b3ff7eeae341658b2ad2796c410 Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:32:05 +0100 Subject: [PATCH 02/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 46e42966ad9f..7a13783c360f 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -19,7 +19,7 @@ with commands like :command:`install`, :command:`create`. This cache is purely an optimization mechanism. It is completely different to the Conan package cache, (typically the ``/.conan`` folder). -This cache will store in a given path, which can be configured in **conan.conf** file, the following: +This cache (whose path can be configured in the *conan.conf* file) will store the following items: - All files that are downloaded from a Conan server (conan_server, Artifactory), both in the api V1 (without revisions) and V2 (with revisions). This include files like *conanfile.py*, but also the zipped artifacts like *conan_package.tgz* or *conan_sources.tgz* From 52566dbc0c7e80a898e467a4706cca8311b9edfa Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:32:17 +0100 Subject: [PATCH 03/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 7a13783c360f..5c54dca7808f 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -22,7 +22,7 @@ This cache is purely an optimization mechanism. It is completely different to th This cache (whose path can be configured in the *conan.conf* file) will store the following items: - All files that are downloaded from a Conan server (conan_server, Artifactory), both in the api V1 (without revisions) and V2 (with revisions). - This include files like *conanfile.py*, but also the zipped artifacts like *conan_package.tgz* or *conan_sources.tgz* + This includes files like *conanfile.py*, but also the zipped artifacts like *conan_package.tgz* or *conan_sources.tgz*. - The downloads done by users with the ``tools.download()`` or ``tools.get()`` helpers, as long as they provide a checksum (md5, sha1, etc.). If a checksum is not provided, even if the download cache is enabled, the download will be always executed. From e88559f1551dc9267b5aab1be9cb193f6f7f2279 Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:32:30 +0100 Subject: [PATCH 04/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 5c54dca7808f..396828c2f5aa 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -24,7 +24,7 @@ This cache (whose path can be configured in the *conan.conf* file) will store th - All files that are downloaded from a Conan server (conan_server, Artifactory), both in the api V1 (without revisions) and V2 (with revisions). This includes files like *conanfile.py*, but also the zipped artifacts like *conan_package.tgz* or *conan_sources.tgz*. - The downloads done by users with the ``tools.download()`` or ``tools.get()`` helpers, as long as they provide a checksum (md5, sha1, etc.). If - a checksum is not provided, even if the download cache is enabled, the download will be always executed. + a checksum is not provided, even if the download cache is enabled, the download will be always executed and the files will not be cached. The cache uses the URL of the download, appending the checksum when provided. In the api V2, it is not necessary to append the checksum, because the URL itself already encodes the recipe revision and/or the package revisions, which are already checksums of the recipe and package respectively. From a100f94b88883ee4e43b4140cff32efc08bfd065 Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:32:43 +0100 Subject: [PATCH 05/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 396828c2f5aa..52d87f5da1ff 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -32,7 +32,7 @@ the URL itself already encodes the recipe revision and/or the package revisions, .. warning:: The download cache will not be able to correctly cache artifacts with revisions enabled if those artifacts are created and uploaded repeatedly - with a client without revisions, because that will keep overwriting the revision "0". + in a client without revisions, because that will keep overwriting the revision "0". Activating the download cache From cbf4d1a294e0be5a20e250ad16a53aba39ff5519 Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:33:08 +0100 Subject: [PATCH 06/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 52d87f5da1ff..4b646d9a1703 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -34,7 +34,6 @@ the URL itself already encodes the recipe revision and/or the package revisions, The download cache will not be able to correctly cache artifacts with revisions enabled if those artifacts are created and uploaded repeatedly in a client without revisions, because that will keep overwriting the revision "0". - Activating the download cache ----------------------------- The download cache is activated and configured in the *conan.conf* file (see :ref:`conan.conf reference `) From 7b085e1b2741989c2477b08c1792de26b660e53d Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:33:15 +0100 Subject: [PATCH 07/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 4b646d9a1703..e818bdf9e996 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -36,6 +36,7 @@ the URL itself already encodes the recipe revision and/or the package revisions, Activating the download cache ----------------------------- + The download cache is activated and configured in the *conan.conf* file (see :ref:`conan.conf reference `) .. code-block:: text From f4e50de2597507a765d0b120fc80bb43d8dbefea Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:33:26 +0100 Subject: [PATCH 08/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index e818bdf9e996..43fadb43e448 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -44,7 +44,7 @@ The download cache is activated and configured in the *conan.conf* file (see :re [storage] download_cache=/path/to/my/cache -You can also define it with a Conan comand +It can be defined from the command like as well: .. code-block:: bash From 3e0e14df264f9918dc1251e8cac4f7a967715256 Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 14:33:55 +0100 Subject: [PATCH 09/13] Update configuration/download_cache.rst Co-Authored-By: Daniel --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 43fadb43e448..d8f5c5521fac 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -37,7 +37,7 @@ the URL itself already encodes the recipe revision and/or the package revisions, Activating the download cache ----------------------------- -The download cache is activated and configured in the *conan.conf* file (see :ref:`conan.conf reference `) +The download cache is activated and configured in the :ref:`conan_conf` like this: .. code-block:: text From 25208567c0fa3d5821e47ef73f9aafb35c114d2c Mon Sep 17 00:00:00 2001 From: memsharded Date: Tue, 4 Feb 2020 14:58:26 +0100 Subject: [PATCH 10/13] review --- configuration/download_cache.rst | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index d8f5c5521fac..607499da3580 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -17,7 +17,10 @@ Download cache Conan implements a shared download cache that can be used to reduce the time needed to populate the Conan package cache with commands like :command:`install`, :command:`create`. -This cache is purely an optimization mechanism. It is completely different to the Conan package cache, (typically the ``/.conan`` folder). +This cache is purely an optimization mechanism. It is completely different to the :ref:`Conan package cache `, (typically the ``/.conan`` folder). +It is not related to the ``short_paths`` mechanism for long path in Windows, nor to the ``short_paths`` cache folder. The cache will contain a copy +of the artifacts, it is not a new location of files. Those files will still be copied to the Conan package cache, which will not change anything, +its behavior, layout or location of any file. This cache (whose path can be configured in the *conan.conf* file) will store the following items: @@ -34,8 +37,8 @@ the URL itself already encodes the recipe revision and/or the package revisions, The download cache will not be able to correctly cache artifacts with revisions enabled if those artifacts are created and uploaded repeatedly in a client without revisions, because that will keep overwriting the revision "0". -Activating the download cache ------------------------------ +Activating/deactivating the download cache +------------------------------------------ The download cache is activated and configured in the :ref:`conan_conf` like this: @@ -57,6 +60,13 @@ And, as the *conan.conf* is part of the configuration, you can also put a common the :ref:`conan_config_install` command to automatically install it in Conan clients. +To deactivate the download cache, you can remove the entry ``download_cache`` from the *conan.conf* file directly editing it, or with the command: + +.. code-block:: bash + + $ conan config rm storage.download_cache + + Concurrency, multiple caches and CI ----------------------------------- From b829a7189b9d71a930a1ba593b15fe4565c5069a Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 19:00:21 +0100 Subject: [PATCH 11/13] Update configuration/download_cache.rst Co-Authored-By: Javier G. Sogo --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 607499da3580..9b57e8b86add 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -47,7 +47,7 @@ The download cache is activated and configured in the :ref:`conan_conf` like thi [storage] download_cache=/path/to/my/cache -It can be defined from the command like as well: +It can be defined from the command line: .. code-block:: bash From b1369d6c13a93cf91448422669dfbc1f01a84761 Mon Sep 17 00:00:00 2001 From: James Date: Tue, 4 Feb 2020 19:00:46 +0100 Subject: [PATCH 12/13] Update configuration/download_cache.rst Co-Authored-By: Javier G. Sogo --- configuration/download_cache.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index 9b57e8b86add..fa64ae11ce63 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -60,7 +60,7 @@ And, as the *conan.conf* is part of the configuration, you can also put a common the :ref:`conan_config_install` command to automatically install it in Conan clients. -To deactivate the download cache, you can remove the entry ``download_cache`` from the *conan.conf* file directly editing it, or with the command: +To deactivate the download cache, you can remove the entry ``download_cache`` from the *conan.conf* with the command: .. code-block:: bash From f2f300eb8bd281ee4878ea02aa5e2559b22b429b Mon Sep 17 00:00:00 2001 From: memsharded Date: Tue, 4 Feb 2020 19:14:46 +0100 Subject: [PATCH 13/13] review 2 --- configuration/download_cache.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/configuration/download_cache.rst b/configuration/download_cache.rst index fa64ae11ce63..007bbcda9794 100644 --- a/configuration/download_cache.rst +++ b/configuration/download_cache.rst @@ -29,13 +29,15 @@ This cache (whose path can be configured in the *conan.conf* file) will store th - The downloads done by users with the ``tools.download()`` or ``tools.get()`` helpers, as long as they provide a checksum (md5, sha1, etc.). If a checksum is not provided, even if the download cache is enabled, the download will be always executed and the files will not be cached. -The cache uses the URL of the download, appending the checksum when provided. In the api V2, it is not necessary to append the checksum, because -the URL itself already encodes the recipe revision and/or the package revisions, which are already checksums of the recipe and package respectively. +The cache computes a sha256 checksum of the download URL and the file checksum whenever is available. .. warning:: - The download cache will not be able to correctly cache artifacts with revisions enabled if those artifacts are created and uploaded repeatedly - in a client without revisions, because that will keep overwriting the revision "0". + As not always the file checksums are available, the download cache will not be able to correctly cache artifacts with revisions enabled + if those artifacts are created and uploaded repeatedly in a client without revisions, because that will keep overwriting the revision "0". + Similarly, the download cache will fail if a proxy suddenly and transparently changes a existing server and moves it to a new location, + without the clients changing the URL too. + Activating/deactivating the download cache ------------------------------------------