Conan Blog Introducing the Local-Recipes-Index: A New Repository Type in Conan

ConanCenter, the central open-source repository for C and C++ libraries, is a success story, containing a vast collection of C and C++ open-source packages. It processes hundreds of thousands of download requests daily, building its packages from a central GitHub repository, which is organized in a specific folder layout and received nearly 6,000 pull requests from contributors in 2023 alone. However, ConanCenter does not fit all use cases. For instance, it cannot include closed-source libraries or tools that might still be beneficial to the community. Additionally, some organizations, particularly large enterprises, prefer not to use binaries downloaded from the internet. Instead, they build their own binaries in-house using the conan-center-index recipes. These organizations often need to customize these recipes to meet unique requirements that are not applicable to the broader community, making such contributions unsuitable for the upstream repository. The Conan documentation acknowledges this practice, recommending working from a fork of conan-center-index for such needs. For all those reasons in version 2.2.0, Conan introduced a new repository type called local-recipes-index, designed to offer more flexibility by allowing a Conan remote to point to a local copy of Conan recipes with a specific layout. The local-recipes-index allows users to maintain a local folder with the same structure as the conan-center-index GitHub repository, using it as a source for package recipes. This new type of repository is recipes-only, necessitating the construction of package binaries from source on each machine where the package is used. For sharing binaries across teams, we continue to recommend using a Conan remote server like Artifactory for production purposes. In this post, we will explore how this feature facilitates the following: This feature enables contributors to share package recipes with the community for libraries that might not be suitable for ConanCenter due to various reasons, such as licensing constraints or binary distribution policies. An example of this could be CUDA or other proprietary libraries, which are distributed as precompiled closed-source binaries. It simplifies the adoption of best practices outlined in the Conan documentation for organizations requiring custom-built binaries or modified ConanCenter recipes to meet unique requirements. This approach grants users complete control over their third-party dependencies, ensuring they are both robust and fully customizable. Next, we’ll delve into practical examples to demonstrate these two use cases of the local-recipes-index repository. Using a local-recipes-index repository with your own recipes In this section, we will illustrate how to use the local-recipes-index feature for scenarios where certain libraries or tools, due to licensing restrictions or proprietary nature, are not suitable for ConanCenter. For demonstration purposes, let’s create a local-recipes-index repository for a hypothetical hello closed-source library using the local_recipes_index template for the conan new command: $ mkdir repo && cd repo $ conan new local_recipes_index -d name=hello -d version=0.1 \ -d url=https://github.com/conan-io/libhello/archive/refs/tags/0.0.1.zip \ -d sha256=1dfb66cfd1e2fb7640c88cc4798fe25853a51b628ed9372ffc0ca285fe5be16b $ cd .. The conan new local_recipes_index command creates a template that assumes CMake as the build system alongside other heavy assumptions. In practice, it will require customizing it, but for this demo, it works as-is. It will create a folder layout equal to the conan-center-index GitHub repository: . └── repo └── recipes └── hello ├── all │ ├── conandata.yml │ ├── conanfile.py │ └── test_package │ ├── CMakeLists.txt │ ├── conanfile.py │ └── src │ └── example.cpp └── config.yml After setting up the repository, we add it as a local remote to Conan: $ conan remote add mylocalrepo ./repo --allowed-packages="hello/*" Please pay special attention to the --allowed-packages argument. This argument ensures that all packages other than hello are discarded by Conan. This can be used to minimize the surface area for a potential supply chain attack. Now you can list and install packages from this new repository: $ conan list "*" -r=mylocalrepo $ conan install --requires=hello/0.1 -r=mylocalrepo --build=missing At this point, you could push this repository to your GitHub account and share it with the community. Please be aware that, as we commented earlier, this feature is specifically tailored for scenarios where certain libraries are not suitable for ConanCenter. Remember, a “local-recipes-index” repository has limitations: it is not fully reproducible as it models only versions and not revisions, and it does not provide binaries. Therefore, outside of these cases, it is advised to use a remote package server such as Artifactory. Now, users simply need to clone the GitHub repository and add the cloned folder as a local repository themselves. Building Binaries from a private conan-center-index fork As outlined in the Conan DevOps Guide, there are many cases where organizations need to operate independently of ConanCenter by building their own binaries. Being decoupled from the public upstream ConanCenter server and building your own binaries from a fork of conan-center-index as suggested in the linked documentation page can have many advantages, including absolute control and possibility to customize recipes, giving us the ability for the repository to act as a snapshot of versions, be completely robust against possible continuous changes and new releases in upstream ConanCenter, etc. The local-recipes-index repository allows you to easily build binaries from a fork of conan-center-index, and then hosting them on a Conan remote repository like Artifactory. The main difference with the process explained in the Conan DevOps guide is the ability to immediately test multiple local changes without the need to export each time a recipe is modified. Note that in this case, mixing binaries from ConanCenter with locally built binaries is not recommended for several reasons: Binary compatibility: There may be small differences in setup between the ConanCenter CI and the user’s CI. Maintaining a consistent setup for all binaries can mitigate some issues. Full control over builds: Building all binaries yourself ensures you have complete control over the compilation environment and dependency versions. Instead, it’s recommended to build all your direct and transitive dependencies from the fork. To begin, remove the upstream ConanCenter as it will not be used, everything will come from our own fork: $ conan remote remove conancenter Then we will clone our fork (in this case, we are cloning directly the upstream for demo purposes, but you would be cloning your fork instead): $ git clone https://github.com/conan-io/conan-center-index Add this as our mycenter remote: # Add the mycenter remote pointing to the local folder $ conan remote add mycenter ./conan-center-index And that’s all! Now you’re set to list and use packages from your conan-center-index local folder: $ conan list "zlib/*" -r=mycenter mycenter zlib zlib/1.2.11 zlib/1.2.12 zlib/1.2.13 zlib/1.3 zlib/1.3.1 We can also install packages from this repo, for example we can do: $ conan install --requires=zlib/1.3 ... ======== Computing dependency graph ======== zlib/1.3: Not found in local cache, looking in remotes... zlib/1.3: Checking remote: mycenter zlib/1.3: Downloaded recipe revision 5c0f3a1a222eebb6bff34980bcd3e024 Graph root cli Requirements zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024 - Downloaded (mycenter) ======== Computing necessary packages ======== Requirements zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a - Missing ERROR: Missing binary: zlib/1.3:72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a As we can see, Conan managed to get the recipe for zlib/1.3 from mycenter, but then it failed because there is no binary. This is expected, the repository only contains the recipes, but not the binaries. We can build the binary from source with --build=missing argument: $ conan install --requires=zlib/1.3 --build=missing ... zlib/1.3: package(): Packaged 2 '.h' files: zconf.h, zlib.h zlib/1.3: package(): Packaged 1 file: LICENSE zlib/1.3: package(): Packaged 1 '.a' file: libz.a zlib/1.3: Created package revision 0466b3475bcac5c2ce37bb5deda835c3 zlib/1.3: Package '72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a' created zlib/1.3: Full package reference: zlib/1.3#5c0f3a1a222eebb6bff34980bcd3e024:72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a#0466b3475bcac5c2ce37bb5deda835c3 zlib/1.3: Package folder /home/conan/.conan2/p/b/zlib1ed9fe13537a2/p WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: WARN: deprecated: 'cpp_info.names' used in: zlib/1.3 ======== Finalizing install (deploy, generators) ======== cli: Generating aggregated env files cli: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] Install finished successfully We can see now the binary package in our local cache: $ conan list zlib:* Local Cache zlib zlib/1.3 revisions 5c0f3a1a222eebb6bff34980bcd3e024 (2024-04-10 11:50:34 UTC) packages 72c852c5f0ae27ca0b1741e5fd7c8b8be91a590a info settings arch: x86_64 build_type: Release compiler: gcc compiler.version: 9 os: Linux options fPIC: True shared: False Finally, upload the binary package to our Artifactory repository to make it avai