Using Workspaces for Data Sharing

1. Understanding Workspace Purpose

Aspect	Description	Usage
Data Persistence	Stores data between jobs in a workflow	Use for sharing artifacts, logs, or build outputs
Scope	Available to all subsequent jobs in the same workflow	Not shared across different workflows or branches
Retention	Data persists until the end of the workflow	Automatically cleaned up after workflow completion
Access Control	Read-only access to workspace data	Jobs can only read from the workspace, not write

Example: Storing Data in Workspace

jobs:
  build:
    steps:
      - run: |
          echo "Building project..." && \
          mkdir -p /tmp/artifacts && \
          echo "Build successful" > /tmp/artifacts/build.log
      - persist_to_workspace:
          root: /tmp/artifacts
          paths:
            - build.log

Example: Using Data from Workspace

jobs:
  test:
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - run: |
          echo "Testing with build artifacts..." && \
          cat /tmp/workspace/build.log

Note: Workspaces are ideal for passing data between jobs in a single workflow. They do not support cross-workflow sharing.

2. Persisting Data to Workspace

Property	Description	Example
paths	List of paths to persist to the workspace	`paths: - /tmp/artifacts - /home/circleci/project/build`
output_directory	Directory to store the workspace data	`output_directory: /tmp/workspace`

Example: Persisting Build Artifacts

workflows:
  version: 2
  build-and-test:
    jobs:
      - build
      - test:
          requires:
            - build

jobs:
  build:
    steps:
      - run: |
          mkdir -p /tmp/artifacts
          echo "build output" > /tmp/artifacts/build.log
      - persist_to_workspace:
          root: /tmp/artifacts
          paths:
            - build.log

  test:
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - run: |
          echo "Testing with $(cat /tmp/workspace/build.log)"

3. Specifying Workspace Root

Property	Values	Description
workspaces	`root`	Defines the root directory for workspace data in a CircleCI job.

Example: Setting Workspace Root

workflows:
  version: 2
  build-and-test:
    jobs:
      - build:
          workspace:
            root: /home/circleci/project

Note: The root path must be an absolute path within the job's working directory.

4. Adding Paths to Workspace (paths key)

Key	Description	Values	Notes
paths	List of paths to be added to the workspace	`['path/to/file', 'another/path']`	Each path is relative to the working directory. Supports wildcards like `**` for recursive matching.

Example: Adding paths to workspace

workflows:
  version: 2
  build-and-test:
    jobs:
      - build
      - test:
          requires:
            - build

jobs:
  build:
    steps:
      - checkout
      - run: mkdir -p output
      - run: echo "result" > output/result.txt
      - persist_to_workspace:
          root: .
          paths:
            - output/
  test:
    steps:
      - attach_workspace:
          at: /tmp/workspace

5. Attaching Workspaces in Jobs

Property	Description	Example
attach_workspace	Step to attach a workspace from a previous job	`steps: - attach_workspace: at: /tmp/workspace`

Example: Attaching Workspace in a Job

jobs:
  build:
    steps:
      - checkout
      - run: echo "Building..."
      - attach_workspace:
          at: /tmp/workspace

Note: The attach_workspace step must be used after the job that created the workspace has completed.

6. Specifying Attachment Location

Property	Description	Default	Notes
path	Path to the directory or file to be attached	None	Required field. Can be a glob pattern like `./logs/*`
destination	Directory where the attachment will be placed in the workspace	None	Optional. If not specified, files are attached at the root of the workspace

Example: Attaching a Log File

workflows:
  version: 2
  build-and-test:
    jobs:
      - build
      - test:
          requires:
            - build
          artifacts:
            paths:
              - ./logs/

Note: The destination field allows you to organize files in the workspace under a specific directory structure.

Property	Description	Usage
paths	List of paths to save as artifacts	`paths: - /tmp/build-artifacts`
name	Unique name for the artifact	`name: build-artifacts-{{ .Branch }}`
destination	Destination path in the workspace	`destination: /artifacts`

Example: Saving Build Artifacts

jobs:
  build:
    steps:
      - run:
          name: Build Project
          command: |
            mkdir -p /tmp/build-artifacts
            echo "Build completed" > /tmp/build-artifacts/output.txt

      - persist_to_workspace:
          root: /tmp
          paths:
            - build-artifacts

Example: Using Artifacts in Another Job

jobs:
  build:
    steps:
      - persist_to_workspace:
          root: /tmp
          paths:
            - build-artifacts

  test:
    steps:
      - attach_workspace:
          at: /tmp/workspace

      - run:
          name: Run Tests
          command: |
            ls -R /tmp/workspace/build-artifacts

Note: Artifacts are persisted to the workspace and can be accessed in subsequent jobs using attach_workspace.

8. Understanding Workspace Lifecycle

Workspace Lifecycle Phases

Creation: Workspace is created when a job starts and is associated with a build.
Population: Data is added to the workspace using save_cache or persist_to_workspace.
Sharing: Workspace is shared with downstream jobs in the same build or subsequent builds.
Consumption: Downstream jobs can retrieve data using attach_workspace.
Cleanup: Workspace is automatically deleted after the build completes or when it expires.

Phase	Description	Duration	Access Control
Creation	Workspace is initialized at the start of a job.	Immediate	Read/Write access to current job
Population	Data is stored in the workspace using commands.	Job execution time	Read/Write access to current job
Sharing	Workspace is made available to downstream jobs.	Build duration	Read access to downstream jobs
Consumption	Downstream jobs attach and retrieve data from the workspace.	Job execution time	Read access to current job
Cleanup	Workspace is deleted after build completion or expiration.	Build end or expiration	No access

Note: Workspaces are scoped to a build and are not shared across different builds.

Key Points

Workspaces are ephemeral and tied to a specific build.
Data is persisted using persist_to_workspace or save_cache.
Downstream jobs can consume data using attach_workspace.
Workspaces are automatically cleaned up after the build completes.

9. Comparing Workspace vs Cache

Workspace vs Cache

Workspace: Stores data and files that are shared between jobs in a workflow. Data is persisted for the duration of the workflow and can be accessed by subsequent jobs.
Cache: Stores data that is shared between builds of the same branch or tag. Data is persisted across builds and can be restored to speed up subsequent builds.

Feature	Workspace	Cache
Persistence	Workflow duration	Across builds
Scope	Within workflow	Across builds
Data Sharing	Between jobs in a workflow	Between builds of the same branch/tag
Use Case	Passing data between jobs	Speed up builds by caching dependencies
Configuration	Using `save_workspace` and `attach_workspace`	Using `restore_cache` and `save_cache`

Note: Workspaces are ideal for passing data between jobs in a workflow, while caches are better for speeding up builds by reusing dependencies.

10. Optimizing Workspace Size

Workspace size can significantly impact build times and resource usage. Optimizing workspace size is crucial for efficient CI/CD pipelines.

Optimization Strategy	Description	Impact
Exclude unnecessary files	Use `.circleci/workspace/` to exclude large or irrelevant files from the workspace.	Reduces build time and resource consumption
Use `persist_to_workspace`	Only persist artifacts that are needed in downstream jobs.	Improves pipeline efficiency
Compress files before persisting	Compress large files or directories to reduce size.	Reduces workspace size

Example: Exclude large files from workspace

jobs:
  build:
    steps:
      - checkout
      - run: |
          mkdir -p /tmp/workspace
          cp -r ./large-data /tmp/workspace/
      - persist_to_workspace:
          root: /tmp/workspace
          paths:
            - large-data

Note: CircleCI has a default workspace size limit of 1GB. Exceeding this limit will cause builds to fail.

Warning: Persisting large workspace files can lead to build timeouts or failures. Always validate workspace size before committing changes.