Troubleshooting Common Issues

1. Resolving Runner Connection Issues

Issue	Symptoms	Possible Causes	Solution
Runner not connecting	Runner shows as "offline" in GitLab UI Job logs show "Runner not responding"	Network issues Incorrect runner registration token Firewall blocking connections	Verify runner token matches GitLab Check network connectivity to GitLab instance Ensure port 443 is open
Runner timeout errors	Jobs hang or timeout Logs show "timeout waiting for response"	Slow network Runner resource constraints High job load	Optimize runner resources Reduce concurrent jobs Improve network performance
Authentication failure	Runner logs show "401 Unauthorized" Job fails with auth error	Expired registration token Incorrect runner configuration GitLab instance URL mismatch	Re-register runner with new token Verify GitLab URL in config Check token validity

Example: Re-registering a GitLab Runner

# Unregister existing runner
sudo gitlab-runner unregister --name "my-runner"

# Register new runner with updated token
sudo gitlab-runner register \
  --url "https://gitlab.example.com/" \
  --registration-token "new-registration-token" \
  --executor "shell" \
  --name "my-runner"

Note: Ensure the runner's registration token is valid and matches the one in GitLab's CI/CD settings.

Warning: If runners are behind a firewall or proxy, ensure proper configuration to allow outbound connections to GitLab.

2. Fixing Docker Authentication Errors

Issue	Solution	Notes
Unauthorized access to registry	`docker login <registry>`	Ensure credentials are correctly set in GitLab CI/CD variables
Invalid token or credentials	Re-generate and reconfigure registry credentials	Use `CI_REGISTRY_USER` and `CI_REGISTRY_PASSWORD` for GitLab-managed registry
Permission denied for image pull/push	Verify project access rights and group permissions	Check that the GitLab runner has appropriate roles to access the registry

Example: Setting up Docker registry credentials in GitLab CI

variables:
  DOCKER_REGISTRY: "registry.gitlab.com"
  DOCKER_IMAGE: "$DOCKER_REGISTRY/$CI_PROJECT_PATH"

before_script:
  - echo "Logging into Docker registry..."
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $DOCKER_REGISTRY

build_job:
  stage: build
  image: docker:latest
  script:
    - docker build -t $DOCKER_IMAGE .
    - docker push $DOCKER_IMAGE

Note: Ensure CI_REGISTRY_USER and CI_REGISTRY_PASSWORD are properly configured in GitLab CI/CD variables or protected secrets.

Warning: Avoid hardcoding credentials in .gitlab-ci.yml files. Use GitLab's built-in CI/CD variables or secret variables.

3. Debugging Cache Miss Issues

Issue	Description	Solution
Cache key mismatch	Cache is not hit due to different cache keys between jobs.	Ensure consistent cache key across jobs using `cache:key` with same values.
Cache path mismatch	Cache paths do not match between job runs.	Verify `cache:path` matches exactly in all jobs using the cache.
Cache TTL expiration	Cache is invalidated due to time-to-live (TTL) limits.	Adjust `cache:ttl` or use `cache:key` with versioning to prevent premature invalidation.

Example: Cache Key Configuration

job:
  cache:
    key: "${CI_COMMIT_REF_SLUG}"
    paths:
      - node_modules/

Note: Use CI_COMMIT_REF_SLUG to ensure cache keys are consistent across branches.

Warning: Avoid using $CI_COMMIT_SHA in cache keys if you want to reuse cache across different commits.

4. Resolving Artifact Download Failures

Issue	Cause	Solution
Artifacts not found	Artifacts were not uploaded correctly or path is incorrect	Verify `artifacts.path` in job configuration and ensure artifacts are uploaded before download
Permission denied	Job does not have access to artifacts from previous stages	Ensure `artifacts.expire_in` is set appropriately and job has correct permissions
Timeout during download	Large artifact size or slow network	Use `artifacts.when: on_failure` or split artifacts into smaller chunks

Example: Artifact Upload and Download Configuration

job1:
  script:
    - echo "Hello World" > output.txt
  artifacts:
    paths:
      - output.txt
    expire_in: 1 day

job2:
  script:
    - cat output.txt
  needs:
    - job: job1
      artifacts: true

Note: Artifacts are only available within the same pipeline unless artifacts.expire_in is set to a longer duration.

Warning: If artifacts.when is set to on_failure, artifacts will only be uploaded if the job fails.

5. Fixing YAML Syntax Errors

Issue	Description	Solution
Incorrect Indentation	YAML is sensitive to indentation. Misaligned spaces or tabs can cause parsing errors.	Ensure consistent use of 2 spaces for indentation. Avoid mixing tabs and spaces.
Missing Colon	YAML requires a colon after keys in key-value pairs.	Add the missing colon: `key: value`.
Unescaped Special Characters	Characters like `:`, `{`, `[`, `&` must be escaped or quoted.	Wrap problematic values in quotes: `key: "value:with:colon"`.
Unquoted Values with Special Characters	Values containing special characters without quotes can cause parsing errors.	Use double quotes for values with special characters: `key: "{value}"`.

Example: Correct YAML Syntax

stages:
  - build
  - test

build_job:
  stage: build
  script:
    - echo "Building..."
    - echo "Build completed"

test_job:
  stage: test
  script:
    - echo "Testing..."
    - echo "Test completed"

Example: Incorrect YAML Syntax

stages:
  - build
  - test

build_job:
  stage: build
  script:
    - echo "Building..."
    - echo "Build completed"

test_job:
  stage: test
  script:
    - echo "Testing..."
    - echo "Test completed"

Note: Use a YAML validator or GitLab CI Linter to catch syntax errors before committing.

6. Debugging Variable Expansion Issues

Variable Type	Expansion Behavior	Debugging Tips
CI/CD Variables	`$VAR_NAME` or `${VAR_NAME}`	Use `echo $VAR_NAME` to verify expansion in job logs
Predefined Variables	Automatically expanded by GitLab	Check `echo $CI_PIPELINE_ID` to confirm value
Custom Variables	Must be defined in project/group/runner settings	Verify variable visibility scope (CI/CD, protected, etc.)

Example: Variable Expansion in Script

script:
  - echo "Pipeline ID: $CI_PIPELINE_ID"
  - echo "Branch Name: $CI_COMMIT_REF_NAME"
  - echo "Custom Var: $CUSTOM_VAR"

Note: Variables defined in a job's variables block are expanded at runtime, not at parse time.

Warning: Using $VAR_NAME without braces can lead to unexpected expansion if the variable name is followed by alphanumeric characters.

Common Expansion Issues

Unexpanded Variables: Ensure variables are defined in the correct scope (e.g., project, group, or runner settings).
Incorrect Syntax: Use ${VAR_NAME} instead of $VAR_NAME to avoid ambiguity.
Variable Scope: Variables set in a job do not propagate to downstream jobs unless explicitly passed via artifacts or dependencies.

7. Resolving Permission Denied Errors

Error Message	Cause	Solution
Permission denied (publickey)	SSH key not configured or invalid	Ensure SSH key is added to GitLab and matches local key
Permission denied (access)	Insufficient access level to repository	Verify user has at least `Developer` access or higher
403 Forbidden	Token or credential invalid	Regenerate personal access token with correct scope

Example: Setting up SSH Keys for GitLab CI

ssh-keygen -t rsa -b 4096 -C "gitlab-ci@example.com"
ssh-add ~/.ssh/id_rsa

Note: Ensure the public key is added to your GitLab account under Settings > SSH Keys.

Warning: Never commit SSH private keys to repositories or expose them in job logs.

Common Permission Issues

SSH Key Mismatch - Ensure local and GitLab keys match
Access Level - Users must have appropriate project permissions
Token Expiry - Personal access tokens may expire and require regeneration

8. Fixing Timeout Issues

Issue	Solution	Notes
Job exceeds timeout	`timeout: 30 minutes` in job config	Default is 1 hour; adjust as needed
Runner hangs on startup	`timeout: 10 minutes` in `config.toml`	Adjust runner-specific timeout settings
Network request timeouts	Use `retry:` with `max_attempts: 3`	Retry logic can prevent transient network issues

Example: Setting Job Timeout

build_job:
  script:
    - echo "Building..."
  timeout: 30 minutes

Example: Configuring Runner Timeout

[runner]
  timeout = 600

Note: Use timeout in job config to set per-job limits. For runner-level timeouts, adjust config.toml.

Warning: Setting timeouts too low may cause jobs to fail prematurely. Ensure sufficient time for builds and tests.

9. Debugging Merge Conflict Failures

Issue	Description	Solution
Merge conflict in pipeline	Conflicts occur when GitLab attempts to merge branches and cannot automatically resolve differences.	Manually resolve conflicts in the codebase before re-running the pipeline or using `git merge` locally.
Pipeline fails due to unmerged changes	When a merge request has unmerged changes, GitLab may fail to run the pipeline.	Ensure all changes are merged into the target branch before triggering a pipeline or use `git rebase` to resolve conflicts.

Example: Resolving Merge Conflicts Locally

git checkout feature-branch
git rebase main
# Resolve conflicts in editor
git add .
git commit -m "Resolve merge conflict"
git push --force-with-lease

Note: Merge conflicts in CI/CD pipelines often stem from changes in the target branch that are not reflected in the feature branch. Always ensure your local branch is up to date with the latest changes from the target branch before merging.

10. Resolving Resource Exhaustion

Issue	Symptoms	Potential Causes	Solutions
High CPU Usage	`builds timing out`, `runner lagging`	`too many concurrent jobs`, `inefficient scripts`	Reduce concurrent jobs, optimize scripts
Memory Exhaustion	`builds failing with OOM errors`	`large artifacts`, `memory leaks in scripts`	Limit artifact size, use memory-efficient tools
Disk Space Full	`builds failing with disk space errors`	`unmanaged cache`, `large logs`	Set up cleanup jobs, configure cache limits

Example: Limiting Concurrent Jobs

concurrent: 2

Example: Setting Cache Limits

cache:
  key: "${CI_COMMIT_REF_SLUG}"
  paths:
    - node_modules/
  policy: pull-push

Note: Use git gc in cleanup scripts to reduce repository size.

Warning: Excessive cache usage can lead to disk space exhaustion.