Skip to main content
Mapagam

Using Job Dependencies and Artifacts Passing

1. Setting Job Dependencies

Property Description Values
needs Defines jobs that must complete before this job runs job_name or { job: 'job_name', artifacts: true }
dependencies Deprecated in favor of needs job_name

Example: Basic needs usage

job1:
  script:
    - echo "Building..."
job2:
  needs: [job1]
  script:
    - echo "Testing after build"

Example: Needs with artifacts

build:
  script:
    - make
  artifacts:
    paths: [bin/]
test:
  needs:
    - job: build
      artifacts: true
  script:
    - ./bin/test.sh
Note: needs is preferred over dependencies as it provides more control and clarity.

2. Using needs for DAG Pipelines

The needs keyword in GitLab CI/CD enables you to define dependencies between jobs, allowing you to create Directed Acyclic Graph (DAG) pipelines. This feature is particularly useful for optimizing pipeline execution by running jobs in parallel while ensuring that dependencies are satisfied.

Property Description Values Notes
needs Defines the jobs that this job depends on job, pipeline, or artifacts Can be a string or array of strings
needs.job Name of the job to depend on String Must match a job name in the same pipeline
needs.pipeline Reference to a pipeline from another project or branch project: <path>, ref: <branch> Requires project and ref to be specified
needs.artifacts Indicates that the job requires artifacts from a previous job true or false Defaults to false

Example: Basic needs usage

job1:
  script:
    - echo "Running job 1"
  artifacts:
    paths:
      - build/

job2:
  needs: [job1]
  script:
    - echo "Running job 2 after job 1"

Example: Using needs with pipeline reference

job3:
  needs:
    - project: "my-group/my-project"
      ref: "main"
  script:
    - echo "Running job 3 with pipeline from another project"
Note: The needs keyword allows you to define complex DAGs, enabling jobs to run in parallel as long as their dependencies are satisfied. This is particularly useful for optimizing pipeline execution time.
Warning: If a job in the pipeline depends on another job that does not exist, GitLab will fail the pipeline. Ensure all job names referenced in needs are valid.

Key Points

  • needs allows defining dependencies between jobs in a DAG pipeline.
  • Jobs can depend on other jobs, pipelines, or artifacts from previous jobs.
  • Artifacts are automatically downloaded when a job needs them.

3. Creating Directed Acyclic Graph

Example: Basic DAG Configuration

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - echo "Building..."
  artifacts:
    paths:
      - build/

test_job:
  stage: test
  script:
    - echo "Testing..."
  dependencies:
    - build_job

deploy_job:
  stage: deploy
  script:
    - echo "Deploying..."
  dependencies:
    - test_job
Property Description Values
stages Defines the stages in the pipeline build, test, deploy
stage Assigns a job to a specific stage Any string defined in stages
dependencies Specifies jobs whose artifacts should be passed to this job List of job names
artifacts Defines files to be passed between jobs paths, excludes, untracked
Note: Artifacts are only available to jobs in subsequent stages.
Warning: Circular dependencies will cause pipeline failures.

4. Passing Artifacts Between Jobs

Property Description Default
artifacts Defines the artifacts to be passed between jobs none
artifacts.paths List of files or directories to include in the artifact none
artifacts.exclude List of files or directories to exclude from the artifact none
artifacts.name Name of the artifact archive artifact
artifacts.expire_in Time after which the artifact will expire 1 week
artifacts.reports Reports to be included in the artifact (e.g., test reports) none

Example: Artifact Configuration

job1:
  script:
    - echo "Hello World" > output.txt
  artifacts:
    paths:
      - output.txt
    name: "job1-artifacts"
    expire_in: 1 day

job2:
  script:
    - cat output.txt
  dependencies:
    - job1
Note: Artifacts are stored in GitLab and can be downloaded from the UI or API.
Warning: Artifacts are not automatically cleaned up and can consume storage space.

5. Using needs:artifacts Configuration

Property Description Values Notes
needs Defines job dependencies artifacts Used to pass artifacts from a parent job to a child job
artifacts Defines the artifacts to be passed paths, exclude Specify which files to include or exclude from artifacts

Example: Using needs:artifacts

job1:
  artifacts:
    paths:
      - build/
  script:
    - echo "Building..."
job2:
  needs:
    - job: job1
      artifacts: true
  script:
    - echo "Using artifacts from job1"
Note: The artifacts property in needs must be set to true to pass artifacts between jobs.
Warning: Artifacts are only available in the same pipeline and job scope.

6. Implementing Optional Dependencies

Property Description Values Notes
needs.optional Defines whether a job's dependency is optional true or false Defaults to false. If set to true, the job will run even if the dependency fails or is skipped.

Example: Using optional in job dependencies

job1:
  script:
    - echo "Running job 1"
job2:
  needs:
    - job: job1
      optional: true
  script:
    - echo "Running job 2 even if job1 fails"

7. Using Cross-Stage Dependencies

Example: Defining a Job with Cross-Stage Dependency

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - echo "Building..."
  artifacts:
    paths:
      - build/

test_job:
  stage: test
  script:
    - echo "Testing..."
  dependencies:
    - build_job

deploy_job:
  stage: deploy
  script:
    - echo "Deploying..."
  dependencies:
    - test_job
Property Description Values
dependencies Specifies jobs that must complete before this job runs job_name
stage Defines the stage this job belongs to build, test, deploy
artifacts Defines files to pass between jobs paths, excludes
Note: Artifacts are only available to jobs in subsequent stages.
Warning: Dependencies must be in previous stages or same stage.

Key Points

  • Cross-stage dependencies ensure jobs run in correct order
  • Artifacts are used to pass data between stages
  • Dependencies must be defined in previous stages

8. Understanding Job Ordering

Aspect Description Notes
Job Execution Jobs run in parallel by default unless dependencies are defined. Parallel execution improves pipeline speed but requires careful dependency management.
Dependency Definition Use needs: to define job dependencies. Jobs listed in needs must complete before the current job starts.
Artifacts Sharing Artifacts created by a job can be shared with downstream jobs using artifacts:. Artifacts are passed through the needs chain, enabling efficient data flow.
Job Status Jobs can depend on the success or failure of previous jobs. Use needs: with artifacts: { when: 'on_failure' } to handle errors.

Example: Basic Job Dependency

job1:
  script:
    - echo "Building..."
    - mkdir -p build
  artifacts:
    paths:
      - build/

job2:
  needs: [job1]
  script:
    - echo "Running tests..."
    - ls build/
Note: The job2 will only start after job1 completes successfully.

Key Concepts

  • Parallel Execution: Jobs run concurrently unless explicitly ordered.
  • Artifacts: Data shared between jobs via artifacts.
  • Dependencies: Defined using needs: to control job execution order.

9. Implementing Fan-In Patterns

Fan-in patterns in GitLab CI/CD allow you to combine outputs from multiple jobs into a single downstream job. This is useful for aggregating artifacts, running tests in parallel, and combining results.

Job Artifacts Description
build-job build-output.tar.gz Builds the application and archives artifacts
test-job-1 test-results.xml Runs unit tests and stores results
test-job-2 test-results.xml Runs integration tests and stores results
aggregate-job all-test-results.xml Collects and aggregates test results from all jobs

Example: Fan-In Pattern with Artifacts

build-job:
  stage: build
  script:
    - echo "Building application..."
    - tar -czf build-output.tar.gz build/
  artifacts:
    paths:
      - build-output.tar.gz

test-job-1:
  stage: test
  script:
    - echo "Running unit tests..."
    - mkdir -p test-results
    - echo "<testsuite name='unit' tests='1'><testcase name='test1'/></testsuite>" > test-results/unit.xml
  artifacts:
    paths:
      - test-results/

test-job-2:
  stage: test
  script:
    - echo "Running integration tests..."
    - mkdir -p test-results
    - echo "<testsuite name='integration' tests='1'><testcase name='test2'/></testsuite>" > test-results/integration.xml
  artifacts:
    paths:
      - test-results/

aggregate-job:
  stage: deploy
  script:
    - echo "Aggregating test results..."
    - cat test-results/*.xml > all-test-results.xml
  dependencies:
    - test-job-1
    - test-job-2
Note: The dependencies keyword ensures that artifacts from upstream jobs are downloaded before the downstream job runs.

Key Points

  • Fan-in patterns are useful for combining outputs from multiple jobs.
  • Use artifacts to define what files are passed between jobs.
  • Use dependencies to specify which jobs to pull artifacts from.
  • Artifacts are downloaded in the dependencies stage, not during job execution.

10. Implementing Fan-Out Patterns

Example: Fan-Out with Dependencies

job1:
  script:
    - echo "Running job 1"
  artifacts:
    paths:
      - output1.txt

job2:
  script:
    - echo "Running job 2"
  dependencies:
    - job1
  artifacts:
    paths:
      - output2.txt

job3:
  script:
    - echo "Running job 3"
  dependencies:
    - job1
  artifacts:
    paths:
      - output3.txt

job4:
  script:
    - echo "Running job 4"
  dependencies:
    - job2
    - job3
Job Dependencies Artifacts Description
job1 none output1.txt Initial job that produces output
job2 job1 output2.txt Processes output from job1
job3 job1 output3.txt Processes output from job1
job4 job2, job3 none Aggregates results from job2 and job3
Note: Fan-out patterns allow multiple jobs to consume the same artifact, enabling parallel processing.
Warning: Ensure that dependencies are correctly specified to avoid artifact access errors.

Was this article helpful?

Related Cheat Sheets