Skip to main content

Using Advanced Pipeline Patterns

1. Using Bridge Jobs

Example: Basic Bridge Job Configuration

job1:
  script:
    - echo "Running initial setup"
  artifacts:
    paths:
      - build/
job2:
  needs:
    - job1
  script:
    - echo "Processing artifacts from job1"
Property Description Values
needs Specifies jobs that must complete before this job runs job1, [job1, job2]
artifacts Defines files to pass between jobs paths, reports
dependencies Deprecated alias for needs DEPRECATED in GitLab 15.0+
Note: Bridge jobs are useful for decoupling stages in a pipeline, allowing parallel execution of dependent tasks.

2. Implementing Fan-In Fan-Out Patterns

The fan-in/fan-out pattern in GitLab CI/CD allows you to parallelize jobs and then aggregate results, useful for scenarios like running tests across multiple environments or processing data in parallel before combining results.

Pattern Description Use Case
Fan-Out Multiple jobs run in parallel, each processing a subset of data or tasks. Running tests across different environments or platforms
Fan-In A single job waits for all parallel jobs to complete and aggregates their results. Collecting test results or combining processed data

Example: Fan-Out Fan-In with Parallel Jobs

stages:
  - build
  - test
  - aggregate

build_job:
  stage: build
  script:
    - echo "Building artifacts..."

parallel_test_jobs:
  stage: test
  parallel: 3
  script:
    - echo "Running test on environment $CI_NODE_INDEX"
    - sleep 2
    - echo "Test result for node $CI_NODE_INDEX: PASSED"

aggregate_job:
  stage: aggregate
  script:
    - echo "Aggregating test results..."
    - echo "All tests completed successfully"
  dependencies:
    - parallel_test_jobs
Note: The parallel keyword allows you to run multiple instances of a job with different values for CI_NODE_INDEX. This is useful for distributing workloads.

Key Concepts

  • Parallel Jobs: Use parallel to run multiple instances of a job.
  • Dependencies: Use dependencies to ensure the fan-in job waits for all fan-out jobs.
  • Environment Variables: CI_NODE_INDEX is available to distinguish between parallel job instances.

3. Using Pipeline Generators

Example: Basic Pipeline Generator

include:
  - local: "pipeline-generators.yml"

variables:
  GENERATOR_SCRIPT: "generate-pipeline.sh"

job_generator:
  script:
    - "./$GENERATOR_SCRIPT"
  artifacts:
    reports:
      junit: "junit.xml"
Property Description Values
include Specifies the pipeline generator file to use local: "pipeline-generators.yml"
variables Variables used in the pipeline generator GENERATOR_SCRIPT: "generate-pipeline.sh"
job_generator Job that runs the pipeline generator script script: "./$GENERATOR_SCRIPT"
Note: Pipeline generators are useful for dynamically creating jobs based on project structure or configuration files.
Warning: Ensure that the generator script is secure and does not introduce vulnerabilities through dynamic job creation.

Key Points

  • Pipeline generators allow for dynamic job creation in GitLab CI/CD
  • Use include to reference generator files
  • Generator scripts must be executable and secure
  • Artifacts can be used to report results from generated jobs

4. Creating Conditional Stages

Example: Basic Conditional Stage

deploy:
  stage: deploy
  script:
    - echo "Deploying to production"
  only:
    - main
Property Values Description
only main, branch, tag Runs the job only on specified branches or tags
except develop, feature/* Skips the job on specified branches or tags
when on_success, on_failure, always, manual Determines when to run the job
Note: The only and except keywords support regex patterns for flexible matching.

Example: Conditional Stage with Variables

build:
  stage: build
  script:
    - echo "Building project"
  only:
    - branches
  except:
    - /^release\/.*$/

5. Implementing Dynamic Pipeline Creation

Example: Using include for Dynamic Job Definitions

include:
  - local: "./jobs/build.yml"
    inputs:
      environment: "staging"
  - local: "./jobs/deploy.yml"
    inputs:
      environment: "production"

stages:
  - build
  - deploy

build_job:
  stage: build
  script:
    - echo "Building for $environment"

deploy_job:
  stage: deploy
  script:
    - echo "Deploying to $environment"
Property Description Values Notes
include Specifies external files or templates to include in the pipeline local, project, file, template Supports dynamic job definitions via variables and templates
inputs Passes variables to included templates Object with key-value pairs Allows runtime customization of included jobs
Note: The include keyword supports dynamic pipeline creation by allowing templates to be parameterized and reused across stages.

Example: Conditional Pipeline Jobs Based on Branch

stages:
  - test
  - deploy

test_job:
  stage: test
  script:
    - echo "Running tests"
  only:
    refs:
      - main
      - develop

deploy_job:
  stage: deploy
  script:
    - echo "Deploying to production"
  only:
    refs:
      - main
  except:
    variables:
      CI_COMMIT_BRANCH: "feature/*"

Dynamic Pipeline Creation Patterns

  • Template Reuse: Use include to define reusable job templates with parameters.
  • Conditional Execution: Use only and except to dynamically control job execution based on branch or environment.
  • Variable Substitution: Pass variables to templates using inputs for dynamic behavior.

6. Using Cross-Project Pipeline Dependencies

GitLab CI/CD allows you to define pipeline dependencies across different projects using the trigger keyword. This enables you to orchestrate complex workflows where one project's pipeline can trigger another project's pipeline.

Property Description Example
trigger Defines a cross-project pipeline trigger trigger: my-project/pipeline
project Specifies the target project path project: my-group/my-project
ref Branch, tag, or commit to trigger the pipeline from ref: main
strategy Strategy for triggering pipelines (default: none) strategy: depend

Example: Triggering a Pipeline in Another Project

deploy:
  trigger:
    project: "my-group/my-project"
    ref: "main"
Note: The project must have a .gitlab-ci.yml file and the pipeline must be enabled for cross-project triggers.
Warning: Ensure that the trigger job is not part of a stage that runs in parallel with other jobs that depend on it, as this can lead to race conditions.

Key Considerations

  • Use project to specify the target project path.
  • Set ref to control which branch or tag is used to trigger the pipeline.
  • Ensure that the target project has proper permissions for cross-project pipeline access.

7. Creating Pipeline as Code Patterns

Example: Basic Pipeline Template

variables:
  IMAGE_TAG: "latest"

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - echo "Building image..."
    - docker build -t my-app:$IMAGE_TAG .
  artifacts:
    paths:
      - dist/

test_job:
  stage: test
  script:
    - echo "Running tests..."
    - npm test
  dependencies:
    - build_job

deploy_job:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - kubectl apply -f deployment.yaml
  only:
    - main
Pattern Description Usage
Template Jobs Reusable job definitions that can be extended or overridden Use extends: to inherit from templates
Variables Global or job-specific variables to parameterize pipelines Define in variables: or use CI/CD variables
Artifacts Files that are passed between jobs in a pipeline Use artifacts: to define and pass files
Dependencies Define job dependencies to control execution order Use dependencies: to specify upstream jobs
Note: Templates and variables can be defined in .gitlab-ci.yml or via GitLab's UI.

Example: Using Variables in Templates

variables:
  DOCKER_IMAGE: "myapp"
  DOCKER_TAG: "${CI_COMMIT_SHORT_SHA}"

build_job:
  stage: build
  script:
    - echo "Building $DOCKER_IMAGE:$DOCKER_TAG"
    - docker build -t $DOCKER_IMAGE:$DOCKER_TAG .
  artifacts:
    paths:
      - dist/

test_job:
  stage: test
  script:
    - echo "Testing image $DOCKER_IMAGE:$DOCKER_TAG"
    - docker run $DOCKER_IMAGE:$DOCKER_TAG npm test
  dependencies:
    - build_job

Best Practices for Pipeline Templates

  • Use extends: to define reusable job templates
  • Define common variables at the top level for consistency
  • Use artifacts: to pass data between jobs efficiently
  • Use dependencies: to control job execution order

8. Implementing Event-Driven Pipelines

Event-driven pipelines in GitLab CI/CD allow you to trigger jobs based on specific events or conditions, enabling more dynamic and responsive automation workflows.

Event Description Trigger Use Case
push Triggers on commits to a branch or tag push Automated builds and tests on code changes
merge_request Triggers on merge request creation or updates merge_request Testing merge requests before merging
schedule Triggers jobs at scheduled intervals schedule Regular maintenance or reporting jobs
webhook Triggers on external webhook events webhook Integrating with external services or CI systems

Example: Triggering a Pipeline on Merge Request

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request"
      when: always

test:
  stage: test
  script:
    - echo "Running tests for merge request"

Example: Using Scheduled Pipelines

schedule:
  - cron: "0 1 * * *"
    description: "Daily cleanup job"
    ref: "main"

cleanup:
  stage: cleanup
  script:
    - echo "Performing daily cleanup"
Note: GitLab CI/CD supports a wide range of event types, including push, merge_request, scheduled, and custom webhook triggers. These can be used to build complex, event-driven automation workflows.

Key Features of Event-Driven Pipelines

  • Conditional Execution: Jobs can be triggered only under specific conditions.
  • Integration with External Systems: Webhooks allow integration with external services.
  • Scheduling: Scheduled pipelines can automate regular tasks like backups or reporting.
  • Dynamic Workflows: Pipelines can adapt based on real-time events or changes in code.

9. Using Pipeline Composition Patterns

Example: Using include for Pipeline Composition

include:
  - local: "./pipelines/shared-job.yml"
  - project: "my-group/my-project"
    file: "templates/ci-template.yml"
    ref: "v1.0"

job1:
  extends:
    - "shared-job-template"
  script:
    - echo "Running job1"
Property Description Values
include Specifies external pipeline configurations to include local, project, template
local Path to a local YAML file String path (relative to root)
project Project to include pipeline from String (full project path)
file File path within the project String (relative to project root)
ref Branch, tag, or commit SHA to use String (branch name, tag, or SHA)
Note: The include directive supports local files, remote projects, and templates. It allows reusing pipeline configurations across multiple projects.

Example: Extending Jobs with Templates

include:
  - template: "Auto-DevOps.gitlab-ci.yml"

variables:
  CI_REGISTRY_IMAGE: "$CI_REGISTRY/image"

job1:
  extends:
    - "Auto-DevOps::build"
  script:
    - echo "Custom build logic"

Key Benefits of Pipeline Composition

  • Reusability across projects
  • Centralized configuration management
  • Template-based job definitions
  • Version control of pipeline components

10. Creating Self-Healing Pipelines

Self-healing pipelines in GitLab CI/CD automatically recover from failures or detect and respond to issues in the pipeline execution. These pipelines enhance reliability by implementing retry mechanisms, rollback strategies, and conditional logic to maintain system stability.

Feature Description Implementation
retry Automatically retries failed jobs up to a specified number of times. retry: 2
when Controls when a job runs based on conditions like failure or manual trigger. when: on_failure
allow_failure Allows a job to fail without stopping the pipeline. allow_failure: true

Example: Retry on Failure

deploy:
  script:
    - echo "Deploying application"
    - deploy.sh
  retry: 2
  when: on_failure

Example: Conditional Rollback

rollback:
  script:
    - echo "Rolling back to previous version"
    - rollback.sh
  when: on_failure
  only:
    - main
Note: Retry logic is useful for transient failures such as network issues or temporary resource unavailability.

Key Concepts

  • Retry logic ensures resilience against temporary failures.
  • Conditional execution allows jobs to run only under specific conditions.
  • Manual triggers can be used to initiate rollback or recovery steps.

Was this article helpful?