Skip to main content
Mapagam

Handling Timeouts and Retries

1. Setting No Output Timeout

Property Values Description
no_output_timeout 10m Maximum time a job can run without producing any output before being terminated
no_output_timeout 30m NEW in CircleCI 2.1, allows longer timeouts for long-running processes

Example: Configuring no_output_timeout

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - run:
          name: Run long process
          command: |
            echo "Starting long-running task"
            sleep 1200
          no_output_timeout: 30m
Note: Setting no_output_timeout to a value higher than the default (10 minutes) is useful for jobs that perform long-running operations like database migrations or large file processing.

2. Setting Step Timeout

Property Values Description
steps.*.timeout 300 Timeout in seconds for a specific step
steps.*.timeout 600 Maximum allowed timeout value (10 minutes)

Example: Setting a 5-minute timeout for a build step

steps:
  - run:
      name: "Run long task"
      command: "./long-running-script.sh"
      timeout: 300
Note: Setting a timeout prevents a step from running indefinitely and consuming resources.

3. Understanding Default Timeouts

Resource Default Timeout Notes
Job 10 minutes Maximum time a job can run before being terminated
Build 10 minutes Maximum time a build can run before being terminated
Checkout 5 minutes Time allowed for checking out code from repository
Command 10 minutes Maximum time for a single command to complete
Note: Timeouts can be customized in the configuration file using timeout key under steps.

4. Configuring Long-Running Jobs

Property Values Description
jobs.<job-name>.timeout 3600 Maximum runtime in seconds before job is cancelled. Default is 10 minutes (600s).
jobs.<job-name>.resource_class medium Resource class to use for job execution. Options: small, medium, large, xlarge.

Example: Setting a 1-hour timeout for a long-running job

jobs:
  build-and-test:
    docker:
      - image: circleci/node:18
    steps:
      - checkout
      - run:
          name: Run long test suite
          command: npm run test:long
    timeout: 3600
Note: The maximum allowed timeout is 120 minutes (7200 seconds). Jobs exceeding this will be terminated.

5. Implementing Retry Logic

Property Description Default Notes
retry Number of retry attempts for a job 0 Can be set to 1 or 2
retry.on Conditions under which to retry ["timeout", "failure"] Can include timeout, failure, or error

Example: Retry Configuration

jobs:
  build:
    steps:
      - run:
          name: "Run Test Suite"
          command: npm test
          retry:
            count: 2
            on:
              - timeout
              - failure
Note: Retry logic is applied at the job level, not step level.
Warning: Excessive retries can lead to increased build times and resource usage.
Condition Description Behavior
timeout Job exceeds its timeout limit Retries the job up to configured count
failure Job exits with non-zero status Retries the job up to configured count
error Job encounters an internal error Retries the job up to configured count

6. Using Retry with Backoff

Property Description Default Notes
retry Configures retry behavior for jobs None Can be set to 1 or 2 for up to 2 retries
backoff Enables exponential backoff for retries false When true, retry delays increase exponentially

Example: Retry with Backoff Configuration

jobs:
  build:
    steps:
      - run:
          name: "Run with retry and backoff"
          command: |
            echo "Running job..."
            exit 1
          retry:
            count: 2
            backoff: true
Note: Backoff delays are calculated as base_delay * (2 ^ attempt_number)

7. Handling Flaky Tests

Property Description Default Notes
fail-fast Stops the build if any job fails false Useful for catching issues early in CI
max-failures Maximum number of failures before build stops Unlimited Can be used to prevent long-running builds due to flaky tests
retry Number of times to retry a job 0 Can be set to 1 or more to handle transient failures

Example: Retry Configuration

jobs:
  test:
    docker:
      - image: circleci/node:14
    steps:
      - checkout
      - run:
          name: Run Tests
          command: npm test
          retry: 2
Note: Retrying jobs can mask underlying issues. It's best to address root causes of flaky tests rather than relying on retries.

Best Practices for Flaky Tests

  • Identify and fix the root cause of flakiness
  • Use retry logic sparingly to avoid masking real problems
  • Implement test isolation to prevent interference between tests
  • Use deterministic data in tests where possible

8. Setting API Request Timeouts

Property Value Description
api.timeout 30s Default timeout for API requests to CircleCI services
api.retry.count 3 Number of retry attempts for failed API requests
api.retry.delay 5s Delay between retry attempts (exponential backoff)

Example: Configuring API timeouts in config.yml

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/python:3.9
    steps:
      - checkout
      - run:
          name: "Run API call with custom timeout"
          command: |
            curl --max-time 60 https://api.circleci.com/v2/project/github/owner/repo
Note: The --max-time option in curl sets the maximum time in seconds that the request is allowed to take. This can be used to enforce timeouts on external API calls made within CircleCI jobs.

9. Using Background Steps

Background steps in CircleCI allow you to run processes in the background while continuing with other steps in your workflow. This is useful for long-running tasks like starting a database or web server.

Property Description Default
background Set to true to run the step in the background false

Example: Running a Background Service

steps:
  - run:
      name: Start Database
      command: |
        docker run -d --name mydb postgres:13
      background: true
  - run:
      name: Run Tests
      command: |
        # Wait for DB to be ready
        sleep 5
        npm test
Note: Background steps do not block the execution of subsequent steps. Make sure to handle dependencies properly, such as waiting for a service to be ready before proceeding.

10. Debugging Timeout Causes

Timeout Type Common Causes Debugging Steps
Job Timeout circleci/cli job exceeds 10 minutes
build or test steps taking too long
  1. Check job logs for longest-running steps
  2. Use circleci debug to inspect job runtime
  3. Review config.yml for inefficient steps
Resource Timeout Excessive memory usage
Long-running processes without cleanup
  1. Monitor resource usage in job logs
  2. Use circleci env to check resource limits
  3. Implement cleanup steps in workflows
Network Timeout Slow network or DNS resolution
External service unavailability
  1. Use curl -v to test external endpoints
  2. Check network connectivity in job logs
  3. Implement retry logic for external calls

Example: Debugging a Timeout in CircleCI

circleci debug
Note: Use circleci env to inspect resource limits and job runtime.

Timeout Debugging Summary

  • Identify timeout type (job, resource, network)
  • Check logs for longest-running steps
  • Use circleci debug and circleci env
  • Implement resource cleanup and retry logic

Was this article helpful?

Related Cheat Sheets