Managing Caching

1. Understanding Cache Purpose

Cache Type	Description	Use Case
Dependencies Cache	Caches installed dependencies to avoid re-downloading them on each run.	Speeds up builds by reusing previously installed packages.
Workspace Cache	Stores files between jobs in a workflow to share data.	Enables job-to-job data sharing in multi-job workflows.
Local Cache	Stores cache on the executor for faster access during builds.	Reduces build times by avoiding network overhead for cache operations.

Example: Caching Dependencies

steps:
  - checkout
  - restore_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
  - run:
      name: Install Dependencies
      command: npm install
  - save_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
      paths:
        - node_modules

Note: Cache keys must be unique per dependency set to avoid conflicts.

2. Creating Cache Keys

Property	Value	Description
key	`cache-v1-{{ checksum "package-lock.json" }}`	Unique identifier for the cache. Uses checksum of lock file to invalidate cache on dependency changes.
paths	`["node_modules"]`	Directories or files to cache. Must be relative to the working directory.
restore-keys	`["cache-v1-"]`	Fallback keys to restore from if the primary key doesn't exist. Supports prefix matching.

Example: Cache Key with Fallbacks

steps:
  - save_cache:
      key: cache-v1-{{ checksum "package-lock.json" }}
      paths: 
        - node_modules
      restore-keys:
        - cache-v1-

Note: Cache keys are case-sensitive and must be unique per project. Using {{ checksum }} ensures cache invalidation when dependencies change.

Warning: Avoid using {{ epoch }} in cache keys as it makes caching ineffective. Always use deterministic values like file checksums or version tags.

3. Using Template Variables

Variable	Description	Usage
CIRCLE_BUILD_NUM	Unique build number for each build	`echo "Build number: $CIRCLE_BUILD_NUM"`
CIRCLE_SHA1	Git commit SHA	`echo "Commit: $CIRCLE_SHA1"`
CIRCLE_BRANCH	Branch name	`echo "Branch: $CIRCLE_BRANCH"`
CIRCLE_TAG	Tag name if build was triggered by a tag	`echo "Tag: $CIRCLE_TAG"`
CIRCLE_PROJECT_REPONAME	Repository name	`echo "Repo: $CIRCLE_PROJECT_REPONAME"`
CIRCLE_USERNAME	Username of the person who triggered the build	`echo "Triggered by: $CIRCLE_USERNAME"`

Example: Using Variables in Config

jobs:
  build:
    steps:
      - run:
          name: "Print Build Info"
          command: |
            echo "Building commit $CIRCLE_SHA1 on branch $CIRCLE_BRANCH"

Note: Template variables are automatically available in all jobs and can be used in commands, environment variables, and configuration keys.

4. Saving Cache

Example: Saving Cache with CircleCI

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package-lock.json" }}
            - v1-dependencies-
      - run: npm install
      - save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{{ checksum "package-lock.json" }}

Property	Description	Values
save_cache	Step to save cache	`save_cache`
paths	Directories or files to cache	Array of paths
key	Unique identifier for cache	String with optional checksums

Note: Cache keys must be unique to avoid overwriting existing caches.

Warning: Cache size is limited to 5GB per project.

5. Restoring Cache

Option	Values	Description
restore_cache	`restore_cache`	Step to restore cache from a previous build

Example: Restoring Cache

jobs:
  build:
    steps:
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "package-lock.json" }}
            - v1-dependencies-

Note: Cache keys must match the pattern used when saving the cache. The checksum is used to ensure cache consistency.

Warning: If no matching cache is found, the restore step will silently succeed without restoring anything.

6. Setting Cache Paths

Property	Values	Description
paths	`string[]`	List of paths to cache. Supports wildcards like `**/node_modules`.
key	`string`	Cache key to use for storing and retrieving cache. Must be unique per cacheable item.
restore_keys	`string[]`	Alternative keys to try if the primary key does not exist. Used for cache fallbacks.

Example: Basic Cache Configuration

steps:
  - restore_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
  - run:
      name: Install Dependencies
      command: npm install
  - save_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
      paths:
        - node_modules

Example: Cache with Fallback Keys

steps:
  - restore_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
      restore_keys:
        - v1-dependencies-
        - v1-

7. Caching Dependencies

Cache Key	Value	Description
key	`dependencies-{{ checksum "package-lock.json" }}`	Unique identifier for the cache. Uses checksum of lock file to detect changes.
paths	`node_modules`	Directory to cache. Typically `node_modules` for Node.js projects.
restore-keys	`dependencies-`	Fallback keys to restore cache if primary key doesn't match.

Example: Caching Node.js Dependencies

steps:
  - restore_cache:
      key: dependencies-{{ checksum "package-lock.json" }}
  - run:
      name: Install Dependencies
      command: npm install
  - save_cache:
      key: dependencies-{{ checksum "package-lock.json" }}
      paths:
        - node_modules

Note: Cache keys are case-sensitive and must match exactly to restore a cache.

Warning: If package-lock.json changes, the cache will not be restored.

8. Using Multiple Restore Keys

Key	Description	Usage
restore-keys	Array of keys to try when restoring cache	`restore-keys: - v1-dependencies-{{ checksum "package-lock.json" }} - v1-dependencies-`

Example: Multiple Restore Keys Configuration

steps:
  - restore_cache:
      keys:
        - v1-dependencies-{{ checksum "package-lock.json" }}
        - v1-dependencies-
        - v1-

Note: Keys are checked in order, and the first match is used to restore the cache.

9. Implementing Partial Cache Restore

Option	Values	Notes
restore_cache	`key: cache-key-v1`	Restore cache using specified key pattern
paths	`vendor/bundle`	Directories to restore from cache

Example: Restore cache with key pattern

restore_cache:
  key: "v1-dependencies-{{ checksum "Gemfile.lock" }}"
  paths:
    - vendor/bundle

Note: Cache keys support {{ checksum }} and {{ epoch }} for dynamic key generation.

Warning: Cache keys must be unique per project to avoid conflicts.

10. Understanding Cache Expiration

Cache Type	Expiration Policy	Notes
Workspace	`1 week`	Default expiration for workspace artifacts
Local Caching	`30 days`	Default for local cache in CircleCI
Remote Caching	`90 days`	Remote cache retention period

Example: Setting Cache TTL in Config

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - restore_cache:
          key: v1-dependencies-{{ checksum "package.json" }}
      - run: npm install
      - save_cache:
          key: v1-dependencies-{{ checksum "package.json" }}
          paths:
            - node_modules

Note: Cache keys are case-sensitive and must match exactly for cache hits.

Warning: Expired caches will not be restored, leading to re-computation of steps.

11. Optimizing Cache Size

Cache Key	Size (MB)	Notes
dependencies	`150`	Includes node_modules and lock files
build-artifacts	`200`	Compiled binaries and build outputs
custom-cache	`50`	Manually defined cache paths

Example: Cache Size Configuration

steps:
  - restore_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
  - run:
      name: Install Dependencies
      command: npm ci
  - save_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
      paths:
        - node_modules

Note: Cache size is limited to 5GB per project. Exceeding this limit will cause builds to fail.

Warning: Large cache sizes can slow down build times. Consider using save_cache with specific paths to reduce size.

Cache Optimization Tips

Use {{ checksum }} to invalidate cache only when dependencies change
Save only necessary paths to avoid bloating cache
Regularly review and clean up unused cache keys

12. Debugging Cache Misses

Example: Cache miss debug output

circleci local execute --config /path/to/config.yml --job build

Cache Key	Match Result	Notes
circleci/cache/v1/checkout	Miss	Cache key does not match any existing cache
circleci/cache/v1/dependencies	Hit	Cache key matches existing cache

Note: Cache misses can occur due to changes in dependencies or environment variables.

Warning: Large cache sizes can lead to slower job execution times.

Cache Debugging Steps

Check cache key patterns in configuration
Verify environment variables affecting cache keys
Review dependency changes that might cause misses