Working with Test Splitting

1. Understanding Test Splitting

Test splitting is a technique used to distribute test execution across multiple machines or containers to reduce overall build times. CircleCI supports automatic test splitting for various testing frameworks.

Feature	Description	Support Status
Test Splitting	Automatically splits tests across multiple executors	NEW in CircleCI 2.1
Parallelism	Enables running tests in parallel on multiple machines	NEW in CircleCI 2.1
Test Distribution	Ensures tests are distributed evenly across executors	NEW in CircleCI 2.1

Example: Enabling Test Splitting in CircleCI

version: 2.1
jobs:
  test:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - run:
          name: Install Dependencies
          command: npm install
      - run:
          name: Run Tests with Splitting
          command: npm test -- --split=4

Note: Test splitting is supported for Node.js projects using Jest, Mocha, and other test runners that support parallel execution.

2. Using circleci tests split

The circleci tests split command is used to split test files across multiple CircleCI executors for parallel execution. This helps reduce the overall build time by distributing test loads.

Option	Values	Description
--split-by	`name`, `time`	Determines how tests are split. `name` splits by test file name, `time` splits by estimated execution time.
--pattern	`string`	Specifies the glob pattern to match test files. Example: `*/_test.go`
--parallelism	`integer`	Number of executors to split tests across. Default is determined by CircleCI configuration.
--output	`string`	Path to output the split test list. If not specified, results are printed to stdout.

Example: Splitting tests by name

circleci tests split --split-by=name --pattern="**/*_test.go" --parallelism=4

Example: Splitting tests by time

circleci tests split --split-by=time --pattern="**/test_*.js" --output="/tmp/split-tests.txt"

Note: The circleci tests split command requires the CircleCI CLI to be installed and configured in your project.

Warning: If --split-by=time is used, the command will attempt to read historical test execution data. If no such data exists, it defaults to splitting by name.

3. Splitting by Filename

Option	Values	Description
split_by	`filename`	Splits tests based on the filename of each test file.
test_files	`array`	List of test files to be split.

Example: Configuring test splitting by filename

version: 2.1
jobs:
  test:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - run:
          name: Run tests with filename splitting
          command: |
            npm install
            npm test -- --split-by=filename

Note: The --split-by=filename option is used to enable filename-based test splitting in the test runner.

4. Splitting by Timing

Option	Values	Notes
split_by	`timings`	Split tests based on historical execution times
split_by	`count`	Split tests evenly by count (default)

Example: Configuring Test Splitting by Timing

jobs:
  test:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - run:
          name: Run tests with timing-based splitting
          command: |
            npm install
            npm test -- --split-by=timings

Note: Timing-based splitting requires historical data to be available in CircleCI's test metadata.

Warning: If timing data is not available, CircleCI will fall back to count splitting.

5. Splitting by Filesize

Option	Values	Description
split_by	`filesize`	Splits tests based on the size of test files in bytes.
max_split	`5`	Maximum number of splits to create.
min_split	`1`	Minimum number of splits to create.

Example: Configuring Test Splitting by Filesize

version: 2.1
jobs:
  test:
    docker:
      - image: circleci/node:18
    steps:
      - checkout
      - run:
          name: Run tests with file size splitting
          command: |
            npm install
            npm test -- --split-by=filesize --max-split=5

6. Generating File Lists

Property	Description	Example
circleci test split	Command to split tests and generate file lists	`circleci test split`
--split-by	How to split tests (e.g., by file size, test count)	`circleci test split --split-by size`
--output	Output file path for generated list	`circleci test split --output test-list.txt`

Example: Splitting Tests by File Size

circleci test split --split-by size --output test-list.txt

Note: The circleci test split command requires the CircleCI CLI to be installed and configured.

7. Using Parallelism (parallelism key)

Property	Description	Values	Notes
parallelism	Number of parallel jobs to run	`1` to `20`	Default is `1`. Set to `0` to disable parallelism

Example: Setting parallelism in config

jobs:
  test:
    docker:
      - image: circleci/node:16
    parallelism: 4
    steps:
      - checkout
      - run: npm test

8. Collecting Timing Data

Field	Description	Example
timing_data	Collection of test execution times for each test suite or individual test	`{ "test_suite_1": 45.2, "test_suite_2": 67.8 }`

Example: Collecting Test Timing Data

version: 2.1
jobs:
  test:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - run:
          name: Run Tests with Timing
          command: |
            npm test -- --timing-data > timing.json
      - store_artifacts:
          path: timing.json

Note: The --timing-data flag is a hypothetical example for demonstration purposes. Actual implementation may vary depending on the testing framework used (e.g., Jest, Mocha).

9. Using Test Environment Variables

Variable	Description	Default Value	Usage
CIRCLE_TEST_REPORTS_DIR	Directory where test reports are stored	`/tmp/test-reports`	Used by test runners to output reports
CIRCLE_TEST_ENV	Environment name for test runs	`test`	Set to `test` or custom value
CIRCLE_TEST_TIMEOUT	Timeout in seconds for test execution	`300`	Configurable via config file or env var

Example: Setting Test Environment Variables

version: 2.1
jobs:
  test:
    docker:
      - image: circleci/node:18
    environment:
      CIRCLE_TEST_REPORTS_DIR: /tmp/test-reports
      CIRCLE_TEST_ENV: test
      CIRCLE_TEST_TIMEOUT: 600
    steps:
      - run:
          name: Run Tests
          command: npm test

Note: Environment variables set in the job level will override any project-level settings.

Warning: Be cautious with sensitive data in environment variables. Use CircleCI's encrypted environment variables feature for secrets.

10. Balancing Test Distribution

Strategy	Description	Benefits	Considerations
Test Splitting	Dividing tests into groups to run in parallel across multiple executors	Reduces overall test runtime by distributing load	Requires careful balancing to avoid uneven distribution
Dynamic Splitting	Tests are distributed dynamically based on runtime data	Improves load balancing by adapting to actual test durations	May introduce slight overhead due to runtime analysis
Fixed Splitting	Tests are pre-assigned to executors based on a static configuration	Simpler to implement and understand	Can lead to uneven test distribution if not carefully planned

Example: Configuring Test Splitting in CircleCI

version: 2.1
jobs:
  test:
    docker:
      - image: circleci/node:16
    steps:
      - checkout
      - run:
          name: Run Tests with Splitting
          command: |
            npm test -- --split=4 --shard=1

Note: The --split parameter defines how many shards to split tests into, and --shard specifies which shard to run.

Warning: Ensure that all test files are compatible with the splitting mechanism to avoid missing or duplicated tests.