Using Docker with CI/CD

1. Building Images in CI Pipelines

Step	Description	Action	Notes
1. Checkout Code	Retrieve source code from version control	`git clone $CI_PROJECT_URL`	Ensure proper permissions and submodules
2. Set Up Build Environment	Prepare environment for building Docker image	`docker version`	Verify Docker is installed and accessible
3. Build Image	Execute Docker build command	`docker build -t myapp:${CI_COMMIT_SHORT_SHA} .`	Use commit SHA for tagging to ensure uniqueness
4. Tag Image	Assign tags to the built image	`docker tag myapp:${CI_COMMIT_SHORT_SHA} registry.example.com/myapp:${CI_COMMIT_SHORT_SHA}`	Tag with registry and version for deployment
5. Push Image	Upload image to container registry	`docker push registry.example.com/myapp:${CI_COMMIT_SHORT_SHA}`	Ensure credentials are configured securely

Example: GitLab CI Pipeline Configuration

stages:
  - build
  - deploy

build_image:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

Note: Use docker:dind (Docker-in-Docker) service in CI environments that do not have Docker pre-installed.

Warning: Avoid pushing to public registries with sensitive data in image layers or build context.

2. Tagging Images with Build Numbers

Tag Format	Description	Example
build-number	Unique identifier for each build	`1234`
branch-name	Source branch name	`main`
commit-sha	Short commit hash	`abc123d`

Example: Tagging with Build Number

docker build -t myapp:1234 .
docker push myapp:1234

Example: Tagging with Branch and Commit

docker build -t myapp:main-abc123d .
docker push myapp:main-abc123d

Note: Use CI/CD variables to dynamically inject build numbers and commit hashes into image tags.

3. Pushing Images to Registry Automatically

Option	Values	Notes
docker push	`docker.io/username/repo:tag`	Pushes image to Docker Hub or custom registry
CI/CD Integration	`GitHub Actions, GitLab CI, Jenkins`	Automate image pushing on successful builds
Registry Auth	`DOCKER_USER, DOCKER_PASS`	Environment variables for registry login

Example: GitHub Actions Workflow

name: Push Docker Image
on:
  push:
    branches: [ main ]
jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build Image
        run: |
          docker build -t myapp .
      - name: Push to Registry
        run: |
          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
          docker push myapp:latest

Note: Ensure registry credentials are stored securely using CI/CD secrets.

Warning: Avoid pushing to public registries with sensitive data in image layers.

4. Running Tests in Containers

Aspect	Description	Notes
Test Environment	`docker run --rm -v $(pwd):/app test-container`	Mounts current directory to /app inside container
Test Command	`npm test`	Runs tests inside the containerized environment
Parallel Execution	`docker-compose run --parallel`	Run multiple test containers in parallel

Example: Running Tests in a Node.js Container

docker run --rm -v $(pwd):/app -w /app node:18 npm test

Example: Using Docker Compose for Test Execution

version: '3.8'
services:
  test:
    build: .
    command: npm test
    volumes:
      - .:/app

Note: Ensure that your test container has all required dependencies installed and that volumes are correctly mounted to access the source code.

5. Implementing Multi-Stage Builds for CI

Example: Multi-stage Dockerfile for CI

FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build

FROM node:18-alpine AS runtime
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["node", "dist/index.js"]

Stage	Description	Use Case	Benefits
builder	Builds application with all dependencies	Compilation, bundling, linting	Large image with dev tools and build artifacts
runtime	Minimal image with only runtime dependencies	Production deployment	Smaller size, faster startup, reduced attack surface

Best Practices for Multi-Stage Builds in CI

Use specific base image tags (e.g., node:18-alpine) to ensure reproducibility
Separate build and runtime stages to minimize final image size
Utilize --from=stage-name to copy artifacts between stages efficiently
Cache layers by copying package.json before installing dependencies

Note: Multi-stage builds are particularly effective in CI environments where build tools and intermediate files should not be included in the final deployment image.

6. Caching Layers in CI/CD

Note: Caching Docker layers in CI/CD pipelines significantly improves build times by reusing previously built image layers.

Cache Type	Description	Implementation	Benefits
Layer Caching	Reuses unchanged layers from previous builds	`docker build --cache-from myapp:latest -t myapp .`	Reduces build time by skipping unchanged steps
Registry Caching	Stores built layers in a registry for reuse across pipelines	`docker build --cache-from myregistry.com/myapp:latest -t myapp .`	Enables cross-pipeline layer reuse

Example: Using Docker BuildKit with Cache

DOCKER_BUILDKIT=1 docker build --cache-from myapp:latest -t myapp .

Best Practices for Layer Caching

Use --cache-from to reference previous builds or base images
Tag base images with specific versions for consistent caching
Structure Dockerfiles to place stable layers before volatile ones
Use multi-stage builds to reduce final image size and cache invalidation

7. Using Docker Compose for Testing

Feature	Description	Usage
services	List of services to test	`services:` in `docker-compose.yml`
build	Build context for service image	`build: .`
environment	Environment variables for testing	`environment: TEST_ENV: "true"`
volumes	Mount test data or logs	`volumes: ./logs:/app/logs`

Example: Test Service Setup

version: '3.8'
services:
  app:
    build: .
    environment:
      - NODE_ENV=test
    volumes:
      - ./logs:/app/logs

Note: Ensure test containers have proper network access to dependencies like databases or external APIs.

Best Practices for Testing with Docker Compose

Use NODE_ENV=test to isolate test configurations
Mount logs or artifacts for debugging failed tests
Use depends_on to ensure services start in order
Keep test environments clean with docker-compose down after tests

8. Implementing Security Scans in Pipeline

Example: Docker Security Scan with Trivy

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:latest image nginx:latest

Scan Type	Description	Tool	Integration
Image Vulnerability Scan	Checks for known vulnerabilities in container images	`Trivy`, `Clair`	CI/CD pipeline step
Secrets Scan	Detects exposed secrets in image layers or configs	`Trivy`, `Snyk`	Pre-commit hook, CI pipeline
Configuration Audit	Validates image configuration against best practices	`Conftest`, `Checkov`	Pre-deployment check

Note: Security scans should be run as part of the CI pipeline before pushing images to a registry.

Security Scanning Tools Comparison

Trivy: Fast, open-source, supports multiple formats
Snyk: Integrates with GitHub, supports SAST
Clair: Deep inspection, can be resource-heavy

9. Deploying Containers from CI/CD

Step	Description	Action	Notes
1	Build container image	`docker build -t myapp:latest .`	Ensure Dockerfile is in context
2	Push to registry	`docker push myapp:latest`	Use private or public registry (e.g., Docker Hub, AWS ECR)
3	Deploy to target environment	`kubectl set image deployment/myapp myapp=myapp:latest`	For Kubernetes deployments
4	Verify deployment	`kubectl get pods`	Check that new image is running

Example: GitHub Actions Deployment

name: Deploy to Production
on:
  push:
    branches: [ main ]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Build and push image
        run: |
          docker build -t myapp:latest .
          docker tag myapp:latest myregistry/myapp:latest
          docker push myregistry/myapp:latest
      - name: Deploy to Kubernetes
        run: |
          kubectl set image deployment/myapp myapp=myregistry/myapp:latest

Note: Ensure CI/CD pipeline has appropriate permissions to push to container registry and deploy to target infrastructure.

10. Using Docker API in Automation

Endpoint	Method	Description	Example
/containers/json	GET	List running containers	`curl --unix-socket /var/run/docker.sock http://localhost/containers/json`
/containers/create	POST	Create a new container	`curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/create -d '{"Image": "nginx", "HostConfig": {"PortBindings": {"80/tcp": [{"HostPort": "8080"}]}}}'`
/containers/{id}/start	POST	Start a container	`curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/1234567890/start`
/containers/{id}/stop	POST	Stop a container	`curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/1234567890/stop`
/images/create	POST	Pull an image from registry	`curl --unix-socket /var/run/docker.sock -X POST http://localhost/images/create?fromImage=nginx`

Example: Automate Container Lifecycle

#!/bin/bash
# Pull image
curl --unix-socket /var/run/docker.sock -X POST http://localhost/images/create?fromImage=redis

# Create container
CONTAINER_ID=$(curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/create -d '{"Image": "redis", "HostConfig": {"PortBindings": {"6379/tcp": [{"HostPort": "6379"}]}}}' | jq -r '.Id')

# Start container
curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/$CONTAINER_ID/start

# Stop container
curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/$CONTAINER_ID/stop

Note: Requires Docker daemon running with Unix socket access.

Warning: Ensure proper permissions and security when exposing Docker API.

11. Configuring Docker Daemon for CI

Example: Docker daemon configuration file

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "live-restore": true,
  "userland-proxy": false
}

Configuration Option	Description	Default Value	Notes
log-driver	Log driver to use for containers	`json-file`	Used for log aggregation in CI systems
log-opts	Log driver options	`{}`	Configures log rotation and retention
live-restore	Restore containers on daemon restart	`false`	Useful for CI systems with long-running containers
userland-proxy	Use userland proxy for port forwarding	`true`	Disable in CI to avoid performance overhead

Note: The Docker daemon configuration file is typically located at /etc/docker/daemon.json.

Warning: Modifying the daemon configuration requires restarting the Docker service.

12. Managing CI/CD Secrets with Docker

Secret Type	Description	Usage in Docker	Best Practices
Environment Variables	Key-value pairs injected into containers at runtime	`docker run -e SECRET_KEY=abc123 myapp`	Use `--env-file` for multiple vars; avoid hardcoding in Dockerfiles
Docker Configs	Securely store and manage sensitive data as config objects	`docker config create db_password <file>`	Configs are not mounted into containers by default; use `--config` to mount
Docker Secrets	Securely pass secrets to services in swarm mode	`echo "secret" \| docker secret create db_pass -`	Use `--secret` when deploying services; secrets are mounted as files in /run/secrets

Example: Using Docker Secrets in a Service

docker service create \
  --secret db_pass \
  --env DB_USER=admin \
  myapp

Note: Secrets are only available in swarm mode. For non-swarm environments, use environment variables or external secret managers like HashiCorp Vault.

Warning: Never store secrets in Dockerfile or base image layers. They are not secure and can be exposed via docker history.

Best Practices for CI/CD Secrets

Use environment variables or Docker configs/secrets for runtime secrets
Avoid hardcoding secrets in scripts or configuration files
Use CI/CD platform's built-in secret management (e.g., GitHub Actions, GitLab CI)
Rotate secrets regularly and audit access logs