Skip to main content
Mapagam

Working with Parallel Jobs

1. Controlling Job Parallelism

Option Values Description
parallel 3 Number of parallel jobs to run
parallel.matrix ["os: linux", "os: windows"] Matrix of parallel jobs to run

Example: Using parallel jobs

test:
  script:
    - echo "Running test"
  parallel: 3

Example: Using parallel matrix

test:
  script:
    - echo "Running test"
  parallel:
    matrix:
      - os: linux
        image: ubuntu:latest
      - os: windows
        image: mcr.microsoft.com/windows/servercore:ltsc2019

2. Using Parallel with Index

GitLab CI/CD allows you to run jobs in parallel using the parallel keyword. When using parallel, you can also use the index variable to differentiate between jobs in a parallel group.

Property Description Values
parallel Defines the number of parallel jobs to run 3, 5, etc.
index Represents the index of the current job in the parallel group (0-based) 0, 1, etc.

Example: Using parallel with index

job_name:
  parallel:
    matrix:
      - index: [0, 1, 2]
  script:
    - echo "Running job with index: $CI_NODE_INDEX"
Note: The CI_NODE_INDEX variable is automatically set by GitLab CI/CD for each parallel job.

Key Points

  • The parallel keyword enables running jobs in parallel.
  • The index variable allows differentiation between jobs in a parallel group.
  • Each job in the parallel group gets a unique CI_NODE_INDEX.

3. Distributing Tests Across Parallel Jobs

Option Values Description
parallel 3 Number of parallel jobs to run
strategy matrix Strategy to use for parallelization

Example: Distributing tests across parallel jobs

test:
  stage: test
  script:
    - npm run test
  parallel:
    matrix:
      - TEST_SUITE: unit
      - TEST_SUITE: integration
Note: The parallel.matrix strategy allows you to run different test suites in parallel, improving overall test execution time.

Key Points

  • Use parallel.matrix to distribute tests across multiple jobs
  • Each job runs a specific test suite
  • Improves overall test execution time by running tests in parallel

4. Using Parallel for Build Distribution

Property Description Values
parallel Defines the number of parallel jobs to run 2, 4, 8
matrix Allows defining a matrix of job configurations to run in parallel ['os: linux', 'os: windows']

Example: Parallel Build Distribution

build:
  stage: build
  script:
    - echo "Building..."
  parallel: 4

Example: Matrix Parallel Jobs

test:
  stage: test
  script:
    - echo "Testing on $OS"
  parallel:
    matrix:
      - os: linux
        image: ubuntu:latest
      - os: windows
        image: mcr.microsoft.com/windows/servercore:ltsc2019
Note: Parallel jobs can significantly reduce build time by distributing workloads across multiple runners.

5. Implementing Parallel Deployments

Example: Parallel Deployment Strategy

deploy:
  stage: deploy
  parallel:
    matrix:
      - ENV: production
        REGION: us-east-1
      - ENV: production
        REGION: us-west-2
  script:
    - echo "Deploying to $ENV in $REGION"
    - deploy-to-region.sh $ENV $REGION
Property Description Values Notes
parallel Defines parallel job configuration matrix, jobs Used to run multiple jobs in parallel
matrix Defines a matrix of job configurations array of variables Each job runs with a unique set of variables
jobs Defines a list of parallel jobs array of job definitions Each job runs independently in parallel
Note: Parallel jobs are useful for deploying to multiple regions or environments simultaneously.
Warning: Ensure that parallel jobs do not interfere with each other, especially when deploying to shared resources.

Key Concepts

  • Parallel jobs allow running multiple instances of a job with different configurations.
  • Use matrix to define variable combinations for parallel execution.
  • Each job in a matrix runs independently and can be configured with different parameters.

6. Using needs with Parallel Jobs

The needs keyword in GitLab CI/CD allows you to define dependencies between jobs, ensuring that a job only runs after its required dependencies have completed successfully. When working with parallel jobs, needs can be used to coordinate between jobs running in parallel and jobs that depend on them.

Property Description Values
needs Specifies the jobs that this job depends on. needs: [job1, job2]
needs.job Name of the job that this job depends on. needs.job: build
needs.artifacts Whether to inherit artifacts from the needed job. true or false

Example: Using needs with parallel jobs

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - echo "Building..."
  artifacts:
    paths:
      - build/

parallel_test_jobs:
  stage: test
  needs:
    - job: build_job
  script:
    - echo "Running test $CI_NODE_INDEX"
  parallel:
    matrix:
      - TEST_SUITE: unit
      - TEST_SUITE: integration

deploy_job:
  stage: deploy
  needs:
    - job: parallel_test_jobs
  script:
    - echo "Deploying..."
Note: When using needs with parallel jobs, the job that depends on the parallel jobs will wait for all instances of the parallel jobs to complete before starting.

7. Optimizing Parallel Execution

Aspect Description Best Practice
Job Dependencies Define dependencies to ensure jobs run in correct order Use needs: to explicitly define job dependencies
Resource Constraints Limit concurrent jobs to prevent resource exhaustion Set max_concurrent: in workflow
Parallelism Control Control how many jobs run simultaneously Use parallel: with integer or matrix values

Example: Using needs: for Dependency Control

build_job:
  stage: build
  script:
    - echo "Building..."

test_job:
  stage: test
  needs: [build_job]
  script:
    - echo "Testing..."

deploy_job:
  stage: deploy
  needs: [test_job]
  script:
    - echo "Deploying..."

Example: Limiting Concurrent Jobs

workflow:
  max_concurrent: 5

job1:
  script:
    - echo "Job 1"

job2:
  script:
    - echo "Job 2"

Example: Parallel Matrix Jobs

test:
  parallel: 4
  script:
    - echo "Running test on $CI_NODE_INDEX"
Note: Parallel jobs are ideal for testing different configurations or running tests in isolation. Ensure that each job has sufficient resources to complete without contention.

8. Debugging Parallel Job Issues

Example: Identifying Job Failures in Parallel Execution

test:
  stage: test
  parallel: 3
  script:
    - echo "Running test job $CI_NODE_INDEX"
    - npm test
  artifacts:
    reports:
      junit: junit.xml
Issue Possible Cause Solution
Job timeout Long-running tests or resource constraints Adjust timeout setting in job or global config
Resource contention Parallel jobs competing for CPU/memory Reduce parallel count or use max_concurrent
Artifacts upload failure Large artifacts or network issues Use artifacts:expire_in and artifacts:paths to manage size
Inconsistent test results Shared state or race conditions Use needs: to enforce job dependencies

Debugging Steps for Parallel Jobs

  1. Check job logs individually using CI_NODE_INDEX to identify failing jobs
  2. Verify that each job has sufficient resources (CPU, memory)
  3. Review artifacts size and usage to prevent upload timeouts
  4. Use needs: or dependencies: to ensure correct job ordering
  5. Enable debug mode in GitLab CI to get more verbose logs
Note: Use CI_NODE_INDEX to differentiate between parallel job instances in logs and artifacts.
Warning: Avoid setting parallel too high, as it can lead to resource exhaustion and job failures.

9. Setting Parallel Limits

Option Values Description
parallel 2 Number of parallel jobs to run. Can be an integer or a hash with matrix.
parallel.matrix ['os:linux', 'os:windows'] Define matrix dimensions for parallel jobs. Supported keys: os, arch, ruby.
parallel.max 5 Maximum number of parallel jobs allowed in a pipeline. Default is 10.

Example: Using parallel with matrix

test:
  parallel:
    matrix:
      - os: linux
        arch: x86_64
      - os: windows
        arch: x86_64
  script:
    - echo "Running on $os with $arch"
Note: The parallel keyword is supported in GitLab CI/CD version 15.0 and later.

10. Using Parallel with Matrix

GitLab CI/CD supports running jobs in parallel using matrix expansion, which allows you to define a set of variables and run multiple instances of a job with different values.

Property Description Values
parallel Defines the number of parallel jobs to run 2, 4, etc.
matrix Defines the variables to expand into parallel jobs variables: object or include: array

Example: Parallel Jobs with Matrix

test:
  parallel:
    matrix:
      - RUBY_VERSION: "3.0"
        DATABASE: "postgres"
      - RUBY_VERSION: "3.1"
        DATABASE: "mysql"
  script:
    - echo "Testing with Ruby $RUBY_VERSION and $DATABASE"
Note: The parallel.matrix feature is available in GitLab CI/CD version 15.0 and later.

Was this article helpful?

Related Cheat Sheets