Monitoring and Analyzing Pipelines
1. Using Pipeline Analytics
Example: Accessing Pipeline Analytics in GitLab
curl --header "PRIVATE-TOKEN: <your_access_token>" \
"https://gitlab.example.com/api/v4/projects/<project_id>/pipelines"
| Analytics Metric | Description | Available in |
|---|---|---|
| Pipeline Duration | Time taken for a pipeline to complete | GitLab CI/CD |
| Success Rate | Percentage of successful pipeline runs | GitLab CI/CD |
| Failure Count | Number of failed pipeline runs | GitLab CI/CD |
| Job Count | Total number of jobs in a pipeline | GitLab CI/CD |
Note: Pipeline analytics can be accessed via the GitLab UI or API. The data is updated in real-time.
Warning: Access to analytics data requires appropriate permissions. Ensure that the user has the necessary roles.
Key Features of Pipeline Analytics
- Real-time monitoring of pipeline performance
- Historical data tracking
- Customizable dashboards for teams
- Integration with GitLab's built-in reporting tools
2. Tracking Pipeline Duration
| Metric | Description | Unit | Notes |
|---|---|---|---|
| pipeline_duration | Total time taken for a pipeline to complete from start to finish | seconds | Includes all stages and jobs |
| job_duration | Time taken for a single job to complete | seconds | Can be used to identify slow jobs |
| stage_duration | Time taken for a stage to complete | seconds | Helps in identifying bottlenecks at stage level |
Example: Pipeline Duration in GitLab CI/CD
job_name:
script:
- echo "Job started"
- sleep 10
- echo "Job completed"
after_script:
- echo "Pipeline duration: $(( $(date +%s) - $START_TIME )) seconds"
Note: The
after_script can be used to calculate and log the total pipeline duration.
Warning: Pipeline duration tracking may vary based on the GitLab runner's performance and resource availability.
3. Identifying Pipeline Bottlenecks
| Bottleneck Type | Description | Common Indicators | Tools/Commands |
|---|---|---|---|
| Resource Contention | When jobs compete for limited resources like CPU, memory, or disk I/O |
|
top, htop, iotop |
| Network Latency | Delays caused by slow network connections between runners and GitLab |
|
ping, traceroute, curl -w |
| Job Dependencies | Jobs that wait for previous jobs to complete before starting |
|
gitlab-ci-lint, job duration logs |
| Slow Tests/Builds | Individual jobs that take disproportionately long to run |
|
gitlab-ci-lint, job logs |
Note: Use
gitlab-ci-lint to validate pipeline configuration and identify inefficient job dependencies.
Warning: High resource contention can cause jobs to be killed or delayed; consider increasing runner resources or using
max_concurrent limits.
4. Using Value Stream Analytics
Example: Setting up Value Stream Analytics
stages:
- analyze
value_stream_analytics:
stage: analyze
script:
- echo "Running value stream analytics"
- gitlab-value-stream-analytics --output report.json
artifacts:
reports:
valuestream:
- report.json
| Field | Description | Values | Notes |
|---|---|---|---|
| value_stream_analytics | Defines the value stream analytics job | job |
Must be defined under a stage |
| script | Script to run value stream analytics | string |
Can include shell commands or scripts |
| artifacts.reports.valuestream | Reports to be included in value stream analytics | array |
Must include the output file path |
Note: Value Stream Analytics requires GitLab Premium or Ultimate.
Warning: Ensure the analytics report is generated in a supported format (e.g., JSON).
Key Features of Value Stream Analytics
- Tracks pipeline execution times
- Measures deployment frequency and lead time
- Provides insights into release quality
- Integrates with GitLab's built-in reporting tools
5. Implementing Pipeline Success Rate Tracking
| Metric | Description | Formula | Notes |
|---|---|---|---|
| Success Rate | Percentage of successful pipeline runs over a time period | (Successful Runs / Total Runs) * 100 |
Calculated per project or group |
| Failed Runs | Number of pipeline runs that did not complete successfully | Count of pipelines with status 'failed' |
Includes jobs that failed or were canceled |
| Total Runs | Count of all pipeline executions in a given period | Count of all pipelines |
Used as denominator in success rate calculation |
Example: Pipeline Metrics in GitLab CI/CD
workflow:
name: "pipeline-metrics"
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
variables:
PIPELINE_SUCCESS_RATE: "$(curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/pipelines?status=success" | jq -r ".[] | .id" | wc -l) / $(curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/pipelines" | jq -r ".[] | .id" | wc -l) * 100"
Note: The success rate can be calculated using GitLab's API or through CI job logs. For accurate tracking, use the
pipelines endpoint with status filter.
Tracking Pipeline Success Rate
- Use GitLab API to fetch pipeline data
- Calculate success rate using
successfulandtotalpipeline counts - Set up automated reporting or alerts based on thresholds
6. Monitoring Runner Queue Times
| Property | Description | Values | Notes |
|---|---|---|---|
| queue_time | Time a job spends in the runner queue before execution begins | duration |
Measured in seconds; includes time waiting for runner availability |
| runner_status | Current status of the runner | online, offline, busy |
Used to identify bottlenecks in runner availability |
| job_start_time | Timestamp when the job started execution | ISO 8601 |
Can be used to calculate queue duration from job submission |
Example: Calculating Queue Time
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
"https://gitlab.example.com/api/v4/projects/1/jobs/123"
Note: Queue times can vary based on runner type (shared vs. dedicated) and job priority.
7. Using DORA Metrics
| Metric | Definition | Measurement | Goal |
|---|---|---|---|
| Deployment Frequency | How often deployments are successfully released to production | Number of successful deployments per time period | High frequency for faster feedback and reduced risk |
| Lead Time for Changes | Time from code commit to successful deployment to production | Average time in hours/days | Reduce to improve agility and responsiveness |
| Mean Time to Recovery (MTTR) | Time to restore service after a failure | Average time in minutes/hours | Reduce to minimize impact of incidents |
| Change Failure Rate | Percentage of deployments that fail and require rollback | Failure count / Total deployments | Reduce to increase system reliability |
Example: Setting up DORA Metrics in GitLab CI
variables:
DEPLOYMENT_FREQUENCY: "10"
LEAD_TIME: "2h"
MTTR: "30m"
CHANGE_FAILURE_RATE: "2%"
job_with_metrics:
script:
- echo "Deployment Frequency: $DEPLOYMENT_FREQUENCY"
- echo "Lead Time for Changes: $LEAD_TIME"
- echo "MTTR: $MTTR"
- echo "Change Failure Rate: $CHANGE_FAILURE_RATE"
Note: DORA metrics can be visualized using GitLab's built-in dashboards or integrated with external tools like Grafana for advanced analytics.
8. Tracking Pipeline Costs
| Cost Metric | Description | Tracking Method | Notes |
|---|---|---|---|
| Pipeline Duration | Time taken to complete a pipeline | duration field in pipeline API response |
Used to calculate compute cost per run |
| Runner Usage | Amount of time runners are active in a pipeline | Runner metrics from GitLab Runner API | Directly impacts cost in hosted environments |
| Resource Consumption | CPU, memory, and disk usage during pipeline execution | Runner metrics or external monitoring tools (e.g., Prometheus) | Higher consumption increases cost in cloud runners |
| Pipeline Jobs | Number of jobs executed in a pipeline | Job count from pipeline API or job logs | More jobs = more compute time and cost |
Example: Pipeline Duration in API Response
{
"id": 12345,
"duration": 120,
"status": "success",
"created_at": "2025-04-01T10:00:00Z"
}
Note: Pipeline duration is measured in seconds and can be used to estimate resource consumption for cost tracking.
Cost Tracking Best Practices
- Use GitLab's built-in metrics or external tools like Prometheus to track runner usage
- Monitor job count and duration to identify cost-drivers in pipelines
- Implement cost allocation tags for better visibility across projects
9. Optimizing Pipeline Schedules
| Property | Description | Values | Notes |
|---|---|---|---|
| schedule.cron | Cron expression for scheduling | 0 2 * * * |
Runs daily at 2 AM |
| schedule.description | Description of the schedule | Weekly cleanup job |
Helps identify purpose |
| schedule.ref | Branch or tag to run on | main |
Must exist in repo |
| schedule.active | Whether schedule is active | true |
Can be toggled off |
Example: Schedule Configuration
job_name:
trigger:
schedule:
cron: "0 2 * * *"
description: "Daily backup job"
ref: "main"
active: true
Note: Cron expressions follow the standard format with 5 fields (minute, hour, day, month, weekday). GitLab supports
0 2 * * * for daily at 2 AM.
Warning: Schedules may run even if the pipeline is already running. Use
only or except rules to prevent overlap.
Optimization Tips
- Use
onlyorexceptto avoid concurrent runs. - Set
active: falsefor inactive schedules to reduce load. - Use descriptive
descriptionfields for better visibility in UI.
10. Using Pipeline Insights API
Example: Fetching Pipeline Metrics
const fetchPipelineMetrics = async () => {
const response = await fetch(
'https://gitlab.example.com/api/v4/projects/123/pipeline_insights',
{
headers: {
'PRIVATE-TOKEN': 'your-access-token',
},
}
);
const data = await response.json();
console.log(data);
};
| Field | Description | Type | Example Value |
|---|---|---|---|
| id | Unique identifier for the pipeline | integer | 456789 |
| status | Current status of the pipeline | string | "success" |
| duration | Total time taken to complete the pipeline | integer (seconds) | 120 |
| started_at | Timestamp when the pipeline started | datetime | "2025-04-01T10:00:00Z" |
| finished_at | Timestamp when the pipeline finished | datetime | "2025-04-01T10:02:00Z" |
Note: The Pipeline Insights API is available in GitLab Premium and Ultimate editions.
Warning: Ensure that the access token has sufficient permissions to read pipeline insights.