Working with Monorepo Pipelines

1. Using rules:changes for Selective Execution

Rule	Description	Usage
rules:changes	Triggers job only if files matching the specified paths have changed in the commit.	`rules:` `- changes:` `- src/**`

Example: Using rules:changes to run jobs only on relevant file changes

build:
  script:
    - echo "Building..."
  rules:
    - changes:
        - src/**/*.js
        - package.json

Note: rules:changes is evaluated at job runtime and compares the current commit with the previous one.

Warning: If changes is used with only or except, it will override the conditions.

2. Implementing Microservice Pipelines

Aspect	Description	Notes
pipeline structure	`include:` directive to reference service-specific CI files	Each microservice has its own `.gitlab-ci.yml`
shared jobs	`extends:` keyword to reuse job definitions	Centralized job templates in `.gitlab-ci.yml`
dependency management	Use `artifacts:` to pass build outputs between stages	Ensures services can depend on each other's builds

Example: Pipeline Configuration for Microservice

include:
  - local: "shared/jobs.yml"

services:
  - name: postgres:13
    alias: db

variables:
  POSTGRES_DB: test_db
  POSTGRES_USER: test_user
  POSTGRES_PASSWORD: test_pass

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - dist/

test_job:
  stage: test
  script:
    - npm run test
  dependencies:
    - build_job

deploy_job:
  stage: deploy
  script:
    - echo "Deploying to production"
  only:
    - main

Example: Shared Job Template

default:
  image: node:18
  before_script:
    - npm ci

build_job_template:
  extends: .default
  script:
    - npm run build
  artifacts:
    paths:
      - dist/

test_job_template:
  extends: .default
  script:
    - npm run test

Note: Artifacts are used to pass build outputs between stages or services in a monorepo.

Warning: Ensure that each microservice's pipeline is isolated to prevent unintended side effects.

Key Considerations for Microservice Pipelines

Each service has its own .gitlab-ci.yml file
Use include: to centralize shared configurations
Artifacts help pass data between stages or services
Isolate pipelines to avoid cross-service interference

3. Using needs for Complex Dependencies

The needs keyword in GitLab CI/CD allows you to define dependencies between jobs, ensuring that a job runs only after its specified dependencies have completed successfully. This is particularly useful in monorepo setups where jobs might depend on artifacts or outputs from other jobs across different packages or services.

Property	Description	Values
needs	Defines the jobs that this job depends on.	`job`, `pipeline`, or `artifacts`
job	Name of the job to depend on.	String
artifacts	Whether to inherit artifacts from the dependency job.	`true`, `false`

Example: Using needs with job dependency

build:
  stage: build
  script:
    - echo "Building..."
  artifacts:
    paths:
      - dist/

test:
  stage: test
  needs:
    - job: build
  script:
    - echo "Testing..."

Example: Using needs with artifacts

build:
  stage: build
  script:
    - echo "Building..."
  artifacts:
    paths:
      - dist/

deploy:
  stage: deploy
  needs:
    - job: build
      artifacts: true
  script:
    - echo "Deploying..."

Note: When using needs, the job will not run until all dependencies have completed successfully. If any dependency fails, the job will be skipped.

Warning: Avoid circular dependencies between jobs, as they can cause pipelines to hang or fail.

4. Creating Component-Specific Jobs

Job Name	Component	Trigger	Description
frontend-build	`frontend`	on push to `main`	Builds and tests the frontend component
backend-deploy	`backend`	on merge to `main`	Deploys the backend component to staging
shared-tests	`shared`	on any branch push	Runs tests for shared libraries used by both components

Example: Component-specific job configuration

frontend-build:
  stage: build
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - dist/
  only:
    - main

Example: Using component variables

backend-deploy:
  stage: deploy
  script:
    - echo "Deploying backend component"
    - ./deploy.sh $BACKEND_VERSION
  only:
    - main
  variables:
    BACKEND_VERSION: v1.2.3

5. Using Dynamic Child Pipelines

Dynamic child pipelines allow you to create pipelines that are generated at runtime based on conditions or variables. This is useful for complex workflows where the pipeline structure depends on the project state or configuration.

Property	Description	Values	Notes
trigger	Defines a child pipeline to be triggered dynamically	`include`	Must reference a YAML file or template
source	Specifies the location of the child pipeline YAML	`template`, `file`	Can be a local file or a template from a project
variables	Variables to pass to the child pipeline	Map of key-value pairs	Available in child pipeline as `$VAR_NAME`

Example: Triggering a Dynamic Pipeline

job_name:
  trigger:
    include:
      - artifact: pipeline.yml
        variables:
          ENVIRONMENT: "staging"

Note: The include directive supports both local files and templates. Templates must be defined in the .gitlab directory or a subdirectory.

Warning: Ensure that the child pipeline YAML file is valid and does not contain circular references or infinite loops.

Key Points

Child pipelines are defined using the trigger keyword.
Dynamic pipelines can be generated based on runtime conditions or variables.
Variables can be passed to child pipelines using the variables key.
Child pipeline files must be valid YAML and follow GitLab CI/CD syntax.

6. Implementing Cross-Project Dependencies

Dependency Type	Description	Configuration	Notes
project	Reference to another project in the same GitLab instance	`project: "group/project-name"`	Must be accessible to the runner
pipeline	Triggers pipeline in another project	`pipeline: "group/project-name"`	Can use variables from the triggering project
trigger	Triggers a pipeline in another project with specific variables	`trigger: "group/project-name"`	Supports passing variables and ref

Example: Triggering a Pipeline in Another Project

trigger_cross_project:
  trigger:
    project: "my-group/dependency-project"
    branch: "main"
    strategy: "none"

Example: Using Variables in Cross-Project Triggers

trigger_with_vars:
  trigger:
    project: "my-group/dependency-project"
    variables:
      DEPLOY_ENV: "$CI_ENVIRONMENT_SLUG"
      BUILD_ID: "$CI_PIPELINE_ID"

Note: Cross-project dependencies require proper permissions and visibility settings in GitLab to function correctly.

Warning: Ensure that the project containing the pipeline trigger has sufficient access to the target project's resources.

7. Using Path-Based Job Triggers

Path-based job triggers in GitLab CI/CD allow you to run jobs only when changes occur in specific paths within your repository. This is useful for monorepos where different parts of the codebase may require independent CI/CD pipelines.

Field	Description	Example
only.paths	Runs the job only when changes occur in specified paths.	`only:` `paths:` `- src/backend/**`
except.paths	Skips the job if changes occur in specified paths.	`except:` `paths:` `- docs/**`

Example: Path-Based Job Trigger

deploy_backend:
  only:
    paths:
      - src/backend/**
  script:
    - echo "Deploying backend"

Example: Excluding Paths

build_docs:
  except:
    paths:
      - src/**/*
  script:
    - echo "Building documentation"

Note: Path patterns use Git's pathspec syntax. They are evaluated relative to the root of the repository.

8. Optimizing Artifact Passing in Monorepos

Example: Using `artifacts:expire_in` to reduce storage costs

build:
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

Option	Values	Description
artifacts:expire_in	`1 week`, `30 days`	Time after which artifacts are automatically deleted to save storage
artifacts:when	`on_success`, `always`, `failure`	Determines when artifacts are uploaded (default: `on_success`)
artifacts:reports	`coverage: coverage.xml`	Uploads reports like code coverage to GitLab for display

Note: Artifacts are stored in GitLab's object storage and can be accessed by other jobs or pipelines.

Example: Using `artifacts:paths` to limit artifact size

test:
  script:
    - npm test
  artifacts:
    paths:
      - coverage/
      - test-results.xml

Example: Using `cache` to avoid re-downloading dependencies

build:
  script:
    - npm ci
  cache:
    key: "node-modules"
    paths:
      - node_modules/

9. Using Matrix for Multi-Component Testing

Example: Matrix Job Configuration

jobs:
  test-components:
    <span class="badge badge--new">NEW</span> matrix:
      component: [frontend, backend, shared]
      node-version: [16, 18, 20]
    script:
      - echo "Testing $component with Node.js $node-version"
      - npm run test -- --component=$component --node-version=$node-version

Property	Description	Values	Notes
matrix	Defines the matrix dimensions for job expansion	`component: [frontend, backend]`	Each combination creates a new job instance
matrix.component	Component name to test	`frontend`, `backend`	Used in scripts to target specific components
matrix.node-version	Node.js version to use for testing	`16`, `18`, `20`	Ensures compatibility across versions

Note: Matrix jobs are expanded at runtime, so each combination of matrix variables creates a new job instance.

Warning: Avoid excessive matrix dimensions as they can lead to a large number of job instances and increase pipeline duration.

10. Implementing Conditional Deployments

Condition	Description	Example
only	Specifies when a job should run based on branches or tags	`deploy: only: - main - /^release\/.*$/`
except	Excludes jobs from running on specified branches or tags	`deploy: except: - develop - /^feature\/.*$/`
changes	Runs job only if files matching a pattern have changed	`deploy: only: changes: - src/*/ - package.json`
rules	Advanced conditional logic using rule matching	`deploy: rules: - if: $CI_COMMIT_BRANCH == 'main' when: on_success - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/ when: manual`

Example: Using Rules for Conditional Deployments

deploy_to_staging:
  stage: deploy
  rules:
    - if: $CI_COMMIT_BRANCH == 'staging'
      when: on_success
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual
  script:
    - echo "Deploying to staging environment"

Example: Deploying Only on Specific File Changes

deploy_frontend:
  stage: deploy
  only:
    changes:
      - frontend/**/*
      - package.json
  script:
    - echo "Deploying frontend"

Note: The changes keyword can be used in combination with only or except to refine when jobs run based on file modifications.

Warning: Be careful with except as it can cause jobs to run unexpectedly if not properly configured.