Debugging Build Failures

1. Reading Build Logs

Log Type	Description	Access Method	Notes
Build Logs	Standard output and error streams from the build process	CircleCI Web UI or API	Includes all steps in the workflow, including setup, tests, and deployment
Job Logs	Logs specific to a single job within a workflow	CircleCI Web UI or API	Useful for isolating failures to a specific job
Checkout Logs	Logs from the repository checkout step	CircleCI Web UI or API	Helps identify issues with cloning or checking out code

Example: Filtering Logs in CircleCI Web UI

circleci build --job <job-name> --verbose

Note: Logs are available for 30 days after a build completes.

Warning: Sensitive information (e.g., API keys) may be exposed in logs if not properly masked.

2. Validating Config Syntax

Validation Method	Description	Command	Notes
circleci config validate	Validates the syntax of a CircleCI configuration file	`circleci config validate [.circleci/config.yml]`	Checks for YAML syntax errors and configuration structure issues
circleci config process	Processes and outputs the expanded configuration	`circleci config process [.circleci/config.yml]`	Useful for debugging complex configurations or template expansions

Example: Validating a config file

circleci config validate .circleci/config.yml

Example: Processing a config file

circleci config process .circleci/config.yml

3. Using SSH Debugging

Example: Enable SSH access to a job

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - run: echo "Debugging session started"
      - ssh  # Enable SSH access to the job

Option	Description	Notes
ssh	Enables SSH access to the job container	Must be used as a step in the workflow. Requires CircleCI CLI to connect.

Note: SSH debugging is only available in CircleCI's paid plans.

Steps to use SSH debugging:

Enable SSH in your job configuration
Run the job with SSH enabled
Use circleci debug CLI command to connect to the job
Access the container via SSH once connected

SSH Debugging Overview

Allows direct shell access to job containers
Useful for troubleshooting build failures
Only available in CircleCI Enterprise and Pro plans
Requires circleci debug CLI tool to connect

4. Checking Environment Variables

Variable Name	Value	Description
CIRCLE_BUILD_NUM	`12345`	Unique build number for each build
CIRCLE_PROJECT_REPONAME	`my-app`	Name of the repository
CIRCLE_SHA1	`abc123def456...`	Git commit SHA
CIRCLE_USERNAME	`john_doe`	Username of the user who triggered the build
CIRCLE_WORKFLOW_ID	`workflow-uuid`	Unique ID for the workflow

Example: Print All Environment Variables

echo "Build Number: $CIRCLE_BUILD_NUM"
echo "Repository Name: $CIRCLE_PROJECT_REPONAME"
echo "Commit SHA: $CIRCLE_SHA1"

Note: Environment variables can be set in the CircleCI project settings or via configuration files like .circleci/config.yml.

5. Inspecting Cache Behavior

Cache Key	Description	Example
paths	List of directories to cache	`paths: [ "node_modules", "vendor" ]`
key	Unique identifier for cache	`key: "v1-dependencies-{{ checksum "package-lock.json" }}"`
restore_keys	Alternative keys to restore from if primary key missing	`restore_keys: [ "v1-dependencies-" ]`

Example: Cache Configuration

steps:
  - restore_cache:
      keys:
        - v1-dependencies-{{ checksum "package-lock.json" }}
        - v1-dependencies-
  - run:
      name: Install Dependencies
      command: npm install
  - save_cache:
      key: v1-dependencies-{{ checksum "package-lock.json" }}
      paths:
        - node_modules

Note: Cache keys are case-sensitive and must match exactly.

Warning: If cache key changes, the previous cache is not used.

Cache Key Pattern	Description	Best Practice
checksum	Hash of file content	Use to invalidate cache on dependency changes
branch	Current branch name	Use for branch-specific caching
build_number	Build number in CircleCI	Use to version cache per build

6. Debugging Workspace Issues

Issue	Description	Solution
Workspace not populated	Files are not available in the workspace when expected	Verify that the `save` step is executed before `restore`
Files missing after restore	Workspace files are not found in subsequent jobs	Check that paths match exactly between `save` and `restore`
Large workspace size	Workspace exceeds limits causing build failures	Use `persist_to_workspace` with minimal necessary files

Example: Saving and Restoring Workspace

jobs:
  build:
    steps:
      - run: |
          mkdir -p /tmp/workspace
          echo "Hello World" > /tmp/workspace/message.txt
      - persist_to_workspace:
          root: /tmp/workspace
          paths:
            - message.txt

  test:
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - run: cat /tmp/workspace/message.txt

Note: Workspace paths must be absolute and match exactly between save and restore steps.

Warning: Large workspace sizes can cause build timeouts or failures.

7. Testing Steps Locally

Example: Running a CircleCI Job Locally

circleci local execute --job build-and-test

Option	Values	Description
circleci local execute	`circleci local execute --job <job-name>`	Executes a job from your config locally
--job	`build-and-test`	Name of the job to run locally
--config	`.circleci/config.yml`	Path to the CircleCI config file

Note: Ensure circleci-cli is installed and configured to run local tests.

Warning: Local execution does not replicate all environment variables or secrets from the CircleCI platform.

8. Using on_fail Steps

The on_fail step in CircleCI allows you to define actions that run when a job fails. This is useful for debugging or cleanup tasks.

Property	Description	Example
on_fail	Defines steps to run when a job fails.	`on_fail: [run: echo "Job failed, running cleanup"]`

Example: on_fail with cleanup

jobs:
  build:
    steps:
      - run: echo "Running build"
      - run: exit 1
    on_fail:
      - run: echo "Cleanup task executed"
      - run: rm -rf /tmp/build

Note: The on_fail steps run after the failed job's steps, but before any subsequent steps in the workflow.

9. Checking Resource Limits

Resource Type	Limit	Notes
CPU	`4 vCPUs`	Maximum CPU allocation per job
Memory	`8 GB`	Maximum memory per job
Timeout	`10 minutes`	Maximum job execution time (default)
Build Time	`10 hours`	Maximum total build time per workflow

Note: Resource limits can be adjusted in project settings for paid plans.

Example: Setting Resource Limits in Config

jobs:
  build:
    resource_class: xlarge
    steps:
      - checkout
      - run: echo "Running with more resources"

10. Analyzing Test Failures

Aspect	Description	Notes
Test Failure Logs	`circleci-cli debug logs`	Retrieve detailed logs for test runs to identify failures.
Test Output	`circleci-cli debug output`	View test command outputs to understand runtime behavior.
Build Artifacts	`circleci-cli debug artifacts`	Access build artifacts to inspect test environments and data.

Example: Viewing Test Logs

circleci-cli debug logs --build-id <build-id>

Example: Accessing Build Artifacts

circleci-cli debug artifacts --build-id <build-id>

Note: Use circleci-cli debug to get detailed insights into test failures and build environments.