CI/CD Artifact Management: Caching, Storage & Build Outputs

Every CI/CD run produces outputs: compiled binaries, test reports, Docker images, coverage data. Managing these outputs efficiently is the difference between a pipeline that takes 2 minutes and one that takes 15. Understanding when to cache, when to artifact, and how to share data between jobs is essential for fast, reliable pipelines.

Caches vs Artifacts

These two concepts are often confused but serve different purposes:

Caches store data to speed up future runs. They are best-effort: a cache miss does not fail your build, it just makes it slower. Dependencies, compiled toolchains, and build caches are good candidates.

Artifacts are the outputs of your build that you need to keep: test reports, compiled binaries, deployment packages, coverage reports. They are uploaded explicitly and retained for a configurable period.

Dependency Caching

Built-in Setup Action Caching

Most setup actions support caching natively. This is the simplest approach:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: 'npm'   # Caches ~/.npm based on package-lock.json
      - run: npm ci
      - run: npm run build

Behind the scenes, this:

1. Generates a cache key from package-lock.json.
2. Checks if a cache exists for that key.
3. On hit, restores ~/.npm before npm ci runs.
4. On miss, saves ~/.npm after the job completes.

The npm ci command still runs (it installs from the cache), but downloading packages from the registry is skipped.

Python with pip Caching

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: 'pip'
          cache-dependency-path: 'requirements.txt'
      - run: pip install -r requirements.txt

Manual Cache Control

For more complex scenarios, use actions/cache directly:

      - uses: actions/cache@v4
        id: cache-deps
        with:
          path: |
            node_modules
            ~/.cache/Cypress
          key: deps-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            deps-${{ runner.os }}-

      - if: steps.cache-deps.outputs.cache-hit != 'true'
        run: npm ci

Key design decisions:

• Cache key includes runner.os because node_modules are platform-specific.
• hashFiles ensures the cache invalidates when dependencies change.
• restore-keys provides fallback: if the exact key misses, restore the most recent cache for this OS. A stale cache plus npm ci is still faster than a clean install.
• Conditional install: Skip npm ci entirely on cache hit (only safe if caching node_modules directly, not ~/.npm).

Caching Docker Layers

Docker builds benefit enormously from layer caching:

      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: myapp:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

The type=gha cache backend stores Docker layers in GitHub Actions cache. The mode=max option caches all layers, not just the final image layers. This can reduce Docker build times from minutes to seconds for incremental changes.

Turborepo / Nx Build Caching

For monorepos, Turborepo and Nx provide build-level caching. Cache their output directories:

      - uses: actions/cache@v4
        with:
          path: |
            node_modules/.cache/turbo
          key: turbo-${{ runner.os }}-${{ hashFiles('turbo.json', '**/package-lock.json') }}
          restore-keys: |
            turbo-${{ runner.os }}-
      - run: npx turbo run build test --cache-dir=node_modules/.cache/turbo

Build Artifacts

Uploading Artifacts

Use actions/upload-artifact to save build outputs:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: 'npm'
      - run: npm ci
      - run: npm run build

      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/
          retention-days: 7

      - uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: |
            coverage/
            test-results.xml
          retention-days: 30
          if-no-files-found: warn

Options explained:

• retention-days: Override the default retention (90 days for public repos, 400 for private). Shorter retention saves storage.
• if-no-files-found: warn logs a warning but does not fail. Use error for critical artifacts.

Downloading Artifacts in Another Job

Share build outputs between jobs using download:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: build-output
          path: dist/

  test-e2e:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: build-output
          path: dist/
      - run: npm ci
      - run: npx playwright test

  deploy:
    needs: [build, test-e2e]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: build-output
          path: dist/
      - run: aws s3 sync dist/ s3://my-bucket/ --delete

The build runs once, and both test-e2e and deploy consume the same artifact. This avoids rebuilding and ensures deployment uses the exact same binary that was tested.

Downloading All Artifacts

When you have multiple artifacts to collect:

      - uses: actions/download-artifact@v4
        with:
          path: all-artifacts/
          # No 'name' specified downloads everything

This creates a directory structure like:

all-artifacts/
  build-output/
    index.js
    styles.css
  test-results/
    coverage/
    test-results.xml

Artifact Patterns for Common Scenarios

Test Report Aggregation

Collect test results from matrix builds:

jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shard }}/4 --reporters=default --reporters=jest-junit
        env:
          JEST_JUNIT_OUTPUT_DIR: ./results
      - uses: actions/upload-artifact@v4
        with:
          name: test-results-${{ matrix.shard }}
          path: results/

  report:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: test-results-*
          merge-multiple: true
          path: all-results/
      - uses: dorny/test-reporter@v1
        with:
          name: Test Results
          path: 'all-results/**/*.xml'
          reporter: jest-junit

The merge-multiple: true option merges all matching artifacts into a single directory.

Release Artifacts

Attach build outputs to GitHub Releases:

  release:
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/v')
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - run: tar -czf app-${{ github.ref_name }}.tar.gz -C dist .

      - uses: softprops/action-gh-release@v2
        with:
          files: app-${{ github.ref_name }}.tar.gz
          generate_release_notes: true

Container Image as Artifact

For container-based deployments, the “artifact” is the Docker image pushed to a registry:

  build-image:
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.meta.outputs.tags }}
      image-digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=sha,prefix=
            type=ref,event=branch
      - id: build
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

The image digest in the output serves as an immutable artifact reference for downstream deployment jobs.

Retention and Cost Management

GitHub Actions storage is metered. Manage costs with:

1. Set retention policies per artifact. Test reports might need 7 days; release binaries might need 90.
2. Delete old artifacts programmatically:

# Delete artifacts older than 7 days
gh api repos/your-org/your-repo/actions/artifacts --paginate \
  --jq '.artifacts[] | select(.expired == false) | select(.created_at < (now - 604800 | todate)) | .id' \
  | xargs -I {} gh api repos/your-org/your-repo/actions/artifacts/{} -X DELETE

3. Avoid uploading node_modules as artifacts. Cache them instead.
4. Compress before uploading. Artifacts are zipped automatically, but pre-compressing large directories with tar can reduce upload time.

Cache Limits and Eviction

GitHub Actions provides 10 GB of cache storage per repository. When the limit is reached, the least recently used caches are evicted. Design your cache keys to avoid unbounded growth:

# Bad: unique key per run, fills cache quickly
key: deps-${{ github.run_id }}

# Good: key based on content hash, shared across runs
key: deps-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

Caches that have not been accessed in 7 days are automatically evicted regardless of the storage limit.

Summary

Effective artifact management makes CI/CD pipelines fast and reliable. Cache dependencies to avoid redundant downloads, upload build artifacts to share outputs between jobs, and set retention policies to control storage costs. Use content-based cache keys for high hit rates, the merge-multiple option for aggregating matrix outputs, and Docker layer caching for container builds. The goal is to build once, test the exact artifact that was built, and deploy the exact artifact that was tested.