Skip to content

Commit 67ce4ff

Browse files
GMNGeoffreyjpienaar
authored andcommitted
Use GCS for Windows ccache (#13183)
We have found the GitHub actions built-in caching mechanism to be extremely limiting: slow, small, and buggy. Switch instead to using our own remote ccache hosted on GCS. This matches our Linux builds on our self-hosted runners except that we have to do GCS auth through service account keys, unfortunately, which means that access is restricted to postsubmit runs. Luckily, for these builds we're generally doing everything in one job and just want caching (which we only write on postsubmit anyway) and don't need artifact storage (which we'd need on presubmit too). Tested: Ran on this PR (hacked the workflow a bit). An [initial run](https://github.com/openxla/iree/actions/runs/4750257226/jobs/8438272681) with an empty cache took 28m total, 15.5m of which was in the build step. This includes writing the remote cache (minor overhead). A [rerun](https://github.com/openxla/iree/actions/runs/4750257226/jobs/8438619413) with a now populated cache took 14m total, 6.5m of which was in the build step. 79% of compiler calls were cacheable and of those 99% were remote cache hits. Contrast with a [recent post-submit run](https://github.com/openxla/iree/actions/runs/4748717136/jobs/8435229260) that ran on a docs-only change (so should've had a maximally populated cache), which took 20m, 7m of which was the build step, 2m of which was fetching the cache, and 1m of which was saving the cache. That's setting aside [runs like this one](https://github.com/openxla/iree/actions/runs/4741863995/jobs/8419465087) where fetching the cache just times out entirely (with no alerting other than if you happen to look at the UI). Tragically, most of the time in all of these jobs is spent just checking out the repository and submodules (see actions/checkout#1186). Overall this seems like a marked improvement. The main wins are in avoiding tons of complexity futzing with cache compression levels and restoring and saving the cache (actual cached build time is ~unchanged). Part of #13028 skip-ci: Windows builds don't run on presubmit
1 parent 515c6fd commit 67ce4ff

File tree

1 file changed

+12
-33
lines changed

1 file changed

+12
-33
lines changed

Diff for: .github/workflows/ci.yml

+12-33
Original file line numberDiff line numberDiff line change
@@ -83,19 +83,15 @@ jobs:
8383
BUILD_DIR: build-windows
8484
IREE_VULKAN_DISABLE: 1
8585
steps:
86+
- id: "gcp-auth"
87+
name: "Authenticating to Google Cloud"
88+
uses: "google-github-actions/auth@v1"
89+
with:
90+
token_format: "access_token"
91+
credentials_json: "${{ secrets.IREE_OSS_GITHUB_RUNNER_BASIC_TRUST_SERVICE_ACCOUNT_KEY }}"
92+
create_credentials_file: false
8693
- name: "Checking out repository"
8794
uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # v3.3.0
88-
# Attempt to restore from cache unconditionally.
89-
# Note: this will first try to grab a cache entry for this exact commit
90-
# then it will fall back to the latest for any commit.
91-
- name: "Fetching cache (CMake/ccache)"
92-
uses: actions/cache/restore@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1
93-
with:
94-
path: ${{ github.workspace }}/.ccache
95-
key: ccache_all_windows_${{ github.sha }}
96-
restore-keys: ccache_all_windows
97-
# Fetch dependencies.
98-
# TODO(scotttodd): Move some of these into a Docker image / add to PATH.
9995
- name: "Updating git submodules"
10096
run: git submodule update --init --jobs 8 --depth 1
10197
- name: "Setting up Python"
@@ -114,30 +110,13 @@ jobs:
114110
# Finally: build and run tests.
115111
- name: "Building IREE"
116112
env:
117-
IREE_READ_REMOTE_CCACHE: 0
118-
IREE_WRITE_REMOTE_CCACHE: 0
119-
IREE_READ_LOCAL_CCACHE: 1
120-
IREE_WRITE_LOCAL_CCACHE: ${{ needs.setup.outputs.write-caches }}
121-
CCACHE_DIR: ${{ github.workspace }}/.ccache
122-
# Cache size and compression level settings are a delicate balance.
123-
# * A full build cache is around 2-5GB depending on compression level
124-
# * Upload/download is slow (double compression may or may not help)
125-
# * We have a limit of 10GB across all cached files per repository
126-
# * Cache misses are quite costly:
127-
# * 99% cache hits -> ~5 minutes to build
128-
# * 20% cache hits -> ~15-20 minutes to build
129-
CCACHE_MAXSIZE: 4G
130-
CCACHE_COMPRESSLEVEL: 5
131-
run: ./build_tools/cmake/build_all.sh "${BUILD_DIR}"
113+
IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }}
114+
IREE_CCACHE_GCP_TOKEN: ${{ steps.gcp-auth.outputs.access_token }}
115+
CCACHE_NAMESPACE: github-windows-2022-64core
116+
run: |
117+
./build_tools/cmake/build_all.sh "${BUILD_DIR}"
132118
- name: "Testing IREE"
133119
run: ./build_tools/cmake/ctest_all.sh "${BUILD_DIR}"
134-
# Write cache (if configured to) after all other steps are finished.
135-
- name: "Saving cache (CMake/ccache)"
136-
if: needs.setup.outputs.write-caches == '1'
137-
uses: actions/cache/save@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1
138-
with:
139-
path: ${{ github.workspace }}/.ccache
140-
key: ccache_all_windows_${{ github.sha }}
141120

142121
build_test_all_macos_arm64:
143122
needs: setup

0 commit comments

Comments
 (0)