Using Advanced Pipeline Patterns
1. Using Bridge Jobs
Example: Basic Bridge Job Configuration
job1:
script:
- echo "Running initial setup"
artifacts:
paths:
- build/
job2:
needs:
- job1
script:
- echo "Processing artifacts from job1"
| Property | Description | Values |
|---|---|---|
| needs | Specifies jobs that must complete before this job runs | job1, [job1, job2] |
| artifacts | Defines files to pass between jobs | paths, reports |
| dependencies | Deprecated alias for needs |
DEPRECATED in GitLab 15.0+ |
2. Implementing Fan-In Fan-Out Patterns
The fan-in/fan-out pattern in GitLab CI/CD allows you to parallelize jobs and then aggregate results, useful for scenarios like running tests across multiple environments or processing data in parallel before combining results.
| Pattern | Description | Use Case |
|---|---|---|
| Fan-Out | Multiple jobs run in parallel, each processing a subset of data or tasks. | Running tests across different environments or platforms |
| Fan-In | A single job waits for all parallel jobs to complete and aggregates their results. | Collecting test results or combining processed data |
Example: Fan-Out Fan-In with Parallel Jobs
stages:
- build
- test
- aggregate
build_job:
stage: build
script:
- echo "Building artifacts..."
parallel_test_jobs:
stage: test
parallel: 3
script:
- echo "Running test on environment $CI_NODE_INDEX"
- sleep 2
- echo "Test result for node $CI_NODE_INDEX: PASSED"
aggregate_job:
stage: aggregate
script:
- echo "Aggregating test results..."
- echo "All tests completed successfully"
dependencies:
- parallel_test_jobs
parallel keyword allows you to run multiple instances of a job with different values for CI_NODE_INDEX. This is useful for distributing workloads.
Key Concepts
- Parallel Jobs: Use
parallelto run multiple instances of a job. - Dependencies: Use
dependenciesto ensure the fan-in job waits for all fan-out jobs. - Environment Variables:
CI_NODE_INDEXis available to distinguish between parallel job instances.
3. Using Pipeline Generators
Example: Basic Pipeline Generator
include:
- local: "pipeline-generators.yml"
variables:
GENERATOR_SCRIPT: "generate-pipeline.sh"
job_generator:
script:
- "./$GENERATOR_SCRIPT"
artifacts:
reports:
junit: "junit.xml"
| Property | Description | Values |
|---|---|---|
| include | Specifies the pipeline generator file to use | local: "pipeline-generators.yml" |
| variables | Variables used in the pipeline generator | GENERATOR_SCRIPT: "generate-pipeline.sh" |
| job_generator | Job that runs the pipeline generator script | script: "./$GENERATOR_SCRIPT" |
Key Points
- Pipeline generators allow for dynamic job creation in GitLab CI/CD
- Use
includeto reference generator files - Generator scripts must be executable and secure
- Artifacts can be used to report results from generated jobs
4. Creating Conditional Stages
Example: Basic Conditional Stage
deploy:
stage: deploy
script:
- echo "Deploying to production"
only:
- main
| Property | Values | Description |
|---|---|---|
| only | main, branch, tag |
Runs the job only on specified branches or tags |
| except | develop, feature/* |
Skips the job on specified branches or tags |
| when | on_success, on_failure, always, manual |
Determines when to run the job |
only and except keywords support regex patterns for flexible matching.
Example: Conditional Stage with Variables
build:
stage: build
script:
- echo "Building project"
only:
- branches
except:
- /^release\/.*$/
5. Implementing Dynamic Pipeline Creation
Example: Using include for Dynamic Job Definitions
include:
- local: "./jobs/build.yml"
inputs:
environment: "staging"
- local: "./jobs/deploy.yml"
inputs:
environment: "production"
stages:
- build
- deploy
build_job:
stage: build
script:
- echo "Building for $environment"
deploy_job:
stage: deploy
script:
- echo "Deploying to $environment"
| Property | Description | Values | Notes |
|---|---|---|---|
| include | Specifies external files or templates to include in the pipeline | local, project, file, template |
Supports dynamic job definitions via variables and templates |
| inputs | Passes variables to included templates | Object with key-value pairs | Allows runtime customization of included jobs |
include keyword supports dynamic pipeline creation by allowing templates to be parameterized and reused across stages.
Example: Conditional Pipeline Jobs Based on Branch
stages:
- test
- deploy
test_job:
stage: test
script:
- echo "Running tests"
only:
refs:
- main
- develop
deploy_job:
stage: deploy
script:
- echo "Deploying to production"
only:
refs:
- main
except:
variables:
CI_COMMIT_BRANCH: "feature/*"
Dynamic Pipeline Creation Patterns
- Template Reuse: Use
includeto define reusable job templates with parameters. - Conditional Execution: Use
onlyandexceptto dynamically control job execution based on branch or environment. - Variable Substitution: Pass variables to templates using
inputsfor dynamic behavior.
6. Using Cross-Project Pipeline Dependencies
GitLab CI/CD allows you to define pipeline dependencies across different projects using the trigger keyword. This enables you to orchestrate complex workflows where one project's pipeline can trigger another project's pipeline.
| Property | Description | Example |
|---|---|---|
| trigger | Defines a cross-project pipeline trigger | trigger: my-project/pipeline |
| project | Specifies the target project path | project: my-group/my-project |
| ref | Branch, tag, or commit to trigger the pipeline from | ref: main |
| strategy | Strategy for triggering pipelines (default: none) |
strategy: depend |
Example: Triggering a Pipeline in Another Project
deploy:
trigger:
project: "my-group/my-project"
ref: "main"
.gitlab-ci.yml file and the pipeline must be enabled for cross-project triggers.
trigger job is not part of a stage that runs in parallel with other jobs that depend on it, as this can lead to race conditions.
Key Considerations
- Use
projectto specify the target project path. - Set
refto control which branch or tag is used to trigger the pipeline. - Ensure that the target project has proper permissions for cross-project pipeline access.
7. Creating Pipeline as Code Patterns
Example: Basic Pipeline Template
variables:
IMAGE_TAG: "latest"
stages:
- build
- test
- deploy
build_job:
stage: build
script:
- echo "Building image..."
- docker build -t my-app:$IMAGE_TAG .
artifacts:
paths:
- dist/
test_job:
stage: test
script:
- echo "Running tests..."
- npm test
dependencies:
- build_job
deploy_job:
stage: deploy
script:
- echo "Deploying to production..."
- kubectl apply -f deployment.yaml
only:
- main
| Pattern | Description | Usage |
|---|---|---|
| Template Jobs | Reusable job definitions that can be extended or overridden | Use extends: to inherit from templates |
| Variables | Global or job-specific variables to parameterize pipelines | Define in variables: or use CI/CD variables |
| Artifacts | Files that are passed between jobs in a pipeline | Use artifacts: to define and pass files |
| Dependencies | Define job dependencies to control execution order | Use dependencies: to specify upstream jobs |
.gitlab-ci.yml or via GitLab's UI.
Example: Using Variables in Templates
variables:
DOCKER_IMAGE: "myapp"
DOCKER_TAG: "${CI_COMMIT_SHORT_SHA}"
build_job:
stage: build
script:
- echo "Building $DOCKER_IMAGE:$DOCKER_TAG"
- docker build -t $DOCKER_IMAGE:$DOCKER_TAG .
artifacts:
paths:
- dist/
test_job:
stage: test
script:
- echo "Testing image $DOCKER_IMAGE:$DOCKER_TAG"
- docker run $DOCKER_IMAGE:$DOCKER_TAG npm test
dependencies:
- build_job
Best Practices for Pipeline Templates
- Use
extends:to define reusable job templates - Define common variables at the top level for consistency
- Use
artifacts:to pass data between jobs efficiently - Use
dependencies:to control job execution order
8. Implementing Event-Driven Pipelines
Event-driven pipelines in GitLab CI/CD allow you to trigger jobs based on specific events or conditions, enabling more dynamic and responsive automation workflows.
| Event | Description | Trigger | Use Case |
|---|---|---|---|
| push | Triggers on commits to a branch or tag | push |
Automated builds and tests on code changes |
| merge_request | Triggers on merge request creation or updates | merge_request |
Testing merge requests before merging |
| schedule | Triggers jobs at scheduled intervals | schedule |
Regular maintenance or reporting jobs |
| webhook | Triggers on external webhook events | webhook |
Integrating with external services or CI systems |
Example: Triggering a Pipeline on Merge Request
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request"
when: always
test:
stage: test
script:
- echo "Running tests for merge request"
Example: Using Scheduled Pipelines
schedule:
- cron: "0 1 * * *"
description: "Daily cleanup job"
ref: "main"
cleanup:
stage: cleanup
script:
- echo "Performing daily cleanup"
push, merge_request, scheduled, and custom webhook triggers. These can be used to build complex, event-driven automation workflows.
Key Features of Event-Driven Pipelines
- Conditional Execution: Jobs can be triggered only under specific conditions.
- Integration with External Systems: Webhooks allow integration with external services.
- Scheduling: Scheduled pipelines can automate regular tasks like backups or reporting.
- Dynamic Workflows: Pipelines can adapt based on real-time events or changes in code.
9. Using Pipeline Composition Patterns
Example: Using include for Pipeline Composition
include:
- local: "./pipelines/shared-job.yml"
- project: "my-group/my-project"
file: "templates/ci-template.yml"
ref: "v1.0"
job1:
extends:
- "shared-job-template"
script:
- echo "Running job1"
| Property | Description | Values |
|---|---|---|
| include | Specifies external pipeline configurations to include | local, project, template |
| local | Path to a local YAML file | String path (relative to root) |
| project | Project to include pipeline from | String (full project path) |
| file | File path within the project | String (relative to project root) |
| ref | Branch, tag, or commit SHA to use | String (branch name, tag, or SHA) |
include directive supports local files, remote projects, and templates. It allows reusing pipeline configurations across multiple projects.
Example: Extending Jobs with Templates
include:
- template: "Auto-DevOps.gitlab-ci.yml"
variables:
CI_REGISTRY_IMAGE: "$CI_REGISTRY/image"
job1:
extends:
- "Auto-DevOps::build"
script:
- echo "Custom build logic"
Key Benefits of Pipeline Composition
- Reusability across projects
- Centralized configuration management
- Template-based job definitions
- Version control of pipeline components
10. Creating Self-Healing Pipelines
Self-healing pipelines in GitLab CI/CD automatically recover from failures or detect and respond to issues in the pipeline execution. These pipelines enhance reliability by implementing retry mechanisms, rollback strategies, and conditional logic to maintain system stability.
| Feature | Description | Implementation |
|---|---|---|
| retry | Automatically retries failed jobs up to a specified number of times. | retry: 2 |
| when | Controls when a job runs based on conditions like failure or manual trigger. | when: on_failure |
| allow_failure | Allows a job to fail without stopping the pipeline. | allow_failure: true |
Example: Retry on Failure
deploy:
script:
- echo "Deploying application"
- deploy.sh
retry: 2
when: on_failure
Example: Conditional Rollback
rollback:
script:
- echo "Rolling back to previous version"
- rollback.sh
when: on_failure
only:
- main