Skip to main content

Monitoring and Analyzing Pipelines

1. Using Pipeline Analytics

Example: Accessing Pipeline Analytics in GitLab

curl --header "PRIVATE-TOKEN: <your_access_token>" \
  "https://gitlab.example.com/api/v4/projects/<project_id>/pipelines"
Analytics Metric Description Available in
Pipeline Duration Time taken for a pipeline to complete GitLab CI/CD
Success Rate Percentage of successful pipeline runs GitLab CI/CD
Failure Count Number of failed pipeline runs GitLab CI/CD
Job Count Total number of jobs in a pipeline GitLab CI/CD
Note: Pipeline analytics can be accessed via the GitLab UI or API. The data is updated in real-time.
Warning: Access to analytics data requires appropriate permissions. Ensure that the user has the necessary roles.

Key Features of Pipeline Analytics

  • Real-time monitoring of pipeline performance
  • Historical data tracking
  • Customizable dashboards for teams
  • Integration with GitLab's built-in reporting tools

2. Tracking Pipeline Duration

Metric Description Unit Notes
pipeline_duration Total time taken for a pipeline to complete from start to finish seconds Includes all stages and jobs
job_duration Time taken for a single job to complete seconds Can be used to identify slow jobs
stage_duration Time taken for a stage to complete seconds Helps in identifying bottlenecks at stage level

Example: Pipeline Duration in GitLab CI/CD

job_name:
  script:
    - echo "Job started"
    - sleep 10
    - echo "Job completed"
  after_script:
    - echo "Pipeline duration: $(( $(date +%s) - $START_TIME )) seconds"
Note: The after_script can be used to calculate and log the total pipeline duration.
Warning: Pipeline duration tracking may vary based on the GitLab runner's performance and resource availability.

3. Identifying Pipeline Bottlenecks

Bottleneck Type Description Common Indicators Tools/Commands
Resource Contention When jobs compete for limited resources like CPU, memory, or disk I/O
  • Jobs taking longer than expected
  • High system load during job execution
top, htop, iotop
Network Latency Delays caused by slow network connections between runners and GitLab
  • Jobs timing out or failing due to timeouts
  • Inconsistent job durations
ping, traceroute, curl -w
Job Dependencies Jobs that wait for previous jobs to complete before starting
  • Long build times due to sequential execution
  • Jobs blocked by upstream dependencies
gitlab-ci-lint, job duration logs
Slow Tests/Builds Individual jobs that take disproportionately long to run
  • Tests or builds consistently taking more time than others
  • High variance in job runtimes
gitlab-ci-lint, job logs

Example: Analyzing Job Duration Logs

gitlab-runner status
Note: Use gitlab-ci-lint to validate pipeline configuration and identify inefficient job dependencies.
Warning: High resource contention can cause jobs to be killed or delayed; consider increasing runner resources or using max_concurrent limits.

4. Using Value Stream Analytics

Example: Setting up Value Stream Analytics

stages:
  - analyze

value_stream_analytics:
  stage: analyze
  script:
    - echo "Running value stream analytics"
    - gitlab-value-stream-analytics --output report.json
  artifacts:
    reports:
      valuestream:
        - report.json
Field Description Values Notes
value_stream_analytics Defines the value stream analytics job job Must be defined under a stage
script Script to run value stream analytics string Can include shell commands or scripts
artifacts.reports.valuestream Reports to be included in value stream analytics array Must include the output file path
Note: Value Stream Analytics requires GitLab Premium or Ultimate.
Warning: Ensure the analytics report is generated in a supported format (e.g., JSON).

Key Features of Value Stream Analytics

  • Tracks pipeline execution times
  • Measures deployment frequency and lead time
  • Provides insights into release quality
  • Integrates with GitLab's built-in reporting tools

5. Implementing Pipeline Success Rate Tracking

Metric Description Formula Notes
Success Rate Percentage of successful pipeline runs over a time period (Successful Runs / Total Runs) * 100 Calculated per project or group
Failed Runs Number of pipeline runs that did not complete successfully Count of pipelines with status 'failed' Includes jobs that failed or were canceled
Total Runs Count of all pipeline executions in a given period Count of all pipelines Used as denominator in success rate calculation

Example: Pipeline Metrics in GitLab CI/CD

workflow:
  name: "pipeline-metrics"
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

variables:
  PIPELINE_SUCCESS_RATE: "$(curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/pipelines?status=success" | jq -r ".[] | .id" | wc -l) / $(curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/pipelines" | jq -r ".[] | .id" | wc -l) * 100"
Note: The success rate can be calculated using GitLab's API or through CI job logs. For accurate tracking, use the pipelines endpoint with status filter.

Tracking Pipeline Success Rate

  • Use GitLab API to fetch pipeline data
  • Calculate success rate using successful and total pipeline counts
  • Set up automated reporting or alerts based on thresholds

6. Monitoring Runner Queue Times

Property Description Values Notes
queue_time Time a job spends in the runner queue before execution begins duration Measured in seconds; includes time waiting for runner availability
runner_status Current status of the runner online, offline, busy Used to identify bottlenecks in runner availability
job_start_time Timestamp when the job started execution ISO 8601 Can be used to calculate queue duration from job submission

Example: Calculating Queue Time

curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.example.com/api/v4/projects/1/jobs/123"
Note: Queue times can vary based on runner type (shared vs. dedicated) and job priority.

7. Using DORA Metrics

Metric Definition Measurement Goal
Deployment Frequency How often deployments are successfully released to production Number of successful deployments per time period High frequency for faster feedback and reduced risk
Lead Time for Changes Time from code commit to successful deployment to production Average time in hours/days Reduce to improve agility and responsiveness
Mean Time to Recovery (MTTR) Time to restore service after a failure Average time in minutes/hours Reduce to minimize impact of incidents
Change Failure Rate Percentage of deployments that fail and require rollback Failure count / Total deployments Reduce to increase system reliability

Example: Setting up DORA Metrics in GitLab CI

variables:
  DEPLOYMENT_FREQUENCY: "10"
  LEAD_TIME: "2h"
  MTTR: "30m"
  CHANGE_FAILURE_RATE: "2%"

job_with_metrics:
  script:
    - echo "Deployment Frequency: $DEPLOYMENT_FREQUENCY"
    - echo "Lead Time for Changes: $LEAD_TIME"
    - echo "MTTR: $MTTR"
    - echo "Change Failure Rate: $CHANGE_FAILURE_RATE"
Note: DORA metrics can be visualized using GitLab's built-in dashboards or integrated with external tools like Grafana for advanced analytics.

8. Tracking Pipeline Costs

Cost Metric Description Tracking Method Notes
Pipeline Duration Time taken to complete a pipeline duration field in pipeline API response Used to calculate compute cost per run
Runner Usage Amount of time runners are active in a pipeline Runner metrics from GitLab Runner API Directly impacts cost in hosted environments
Resource Consumption CPU, memory, and disk usage during pipeline execution Runner metrics or external monitoring tools (e.g., Prometheus) Higher consumption increases cost in cloud runners
Pipeline Jobs Number of jobs executed in a pipeline Job count from pipeline API or job logs More jobs = more compute time and cost

Example: Pipeline Duration in API Response

{
  "id": 12345,
  "duration": 120,
  "status": "success",
  "created_at": "2025-04-01T10:00:00Z"
}
Note: Pipeline duration is measured in seconds and can be used to estimate resource consumption for cost tracking.

Cost Tracking Best Practices

  • Use GitLab's built-in metrics or external tools like Prometheus to track runner usage
  • Monitor job count and duration to identify cost-drivers in pipelines
  • Implement cost allocation tags for better visibility across projects

9. Optimizing Pipeline Schedules

Property Description Values Notes
schedule.cron Cron expression for scheduling 0 2 * * * Runs daily at 2 AM
schedule.description Description of the schedule Weekly cleanup job Helps identify purpose
schedule.ref Branch or tag to run on main Must exist in repo
schedule.active Whether schedule is active true Can be toggled off

Example: Schedule Configuration

job_name:
  trigger:
    schedule:
      cron: "0 2 * * *"
      description: "Daily backup job"
      ref: "main"
      active: true
Note: Cron expressions follow the standard format with 5 fields (minute, hour, day, month, weekday). GitLab supports 0 2 * * * for daily at 2 AM.
Warning: Schedules may run even if the pipeline is already running. Use only or except rules to prevent overlap.

Optimization Tips

  • Use only or except to avoid concurrent runs.
  • Set active: false for inactive schedules to reduce load.
  • Use descriptive description fields for better visibility in UI.

10. Using Pipeline Insights API

Example: Fetching Pipeline Metrics

const fetchPipelineMetrics = async () => {
  const response = await fetch(
    'https://gitlab.example.com/api/v4/projects/123/pipeline_insights',
    {
      headers: {
        'PRIVATE-TOKEN': 'your-access-token',
      },
    }
  );
  const data = await response.json();
  console.log(data);
};
Field Description Type Example Value
id Unique identifier for the pipeline integer 456789
status Current status of the pipeline string "success"
duration Total time taken to complete the pipeline integer (seconds) 120
started_at Timestamp when the pipeline started datetime "2025-04-01T10:00:00Z"
finished_at Timestamp when the pipeline finished datetime "2025-04-01T10:02:00Z"
Note: The Pipeline Insights API is available in GitLab Premium and Ultimate editions.
Warning: Ensure that the access token has sufficient permissions to read pipeline insights.

Key Features of Pipeline Insights API

  • Provides detailed metrics about pipeline execution
  • Supports filtering by project, status, and time range
  • Can be used to build custom dashboards or monitoring tools

Was this article helpful?