Fixing Bug: Jest Ignore Patterns Dropped In CI Workflow

by Alex Johnson 56 views

Introduction

In this article, we'll dive into a tricky bug encountered in a CI workflow where Jest ignore patterns were being dropped, leading to unexpected test executions and increased CI times. We'll explore the root cause, the problematic behavior, and the proposed solution to ensure consistent test exclusions across local and CI environments. This issue was observed in the yukimura-ixa and school-timetable-senior-project, highlighting the importance of maintaining a single source of truth for test configurations.

Description of the Bug

The core issue lies in the CI workflow's repetitive use of the --testPathIgnorePatterns flag when invoking Jest. Specifically, the flag is repeated five times. Due to how yargs parses command-line arguments, Jest only considers the last value provided for duplicated flags. This means that the intended exclusions for /e2e/, __test__/stores/, __test__/integration/, and two teaching-assignment suites are not respected. Consequently, these suites are unexpectedly executed during CI runs, negating the purpose of the ignore patterns. Furthermore, the original two exclusions defined elsewhere are lost, exacerbating the problem by including even more heavy suites in the CI process.

Location of the Issue

The problem manifests in two key locations:

  • .github/workflows/ci.yml: This file defines the CI workflow and contains the erroneous Jest invocation with repeated ignore patterns.
  • jest.config.ts: This file should be the single source of truth for Jest configuration, including ignore patterns. However, the CI workflow's command-line arguments override these settings, leading to inconsistencies.

Current Behavior

The current behavior exhibits several undesirable outcomes:

  • Unit test job still loads __test__/stores and __test__/integration: Despite the presence of CLI flags intended to exclude these directories, they are still included in the test execution scope. This leads to longer test runs and potential for flaky tests.
  • The heavy teaching-assignment suites run twice (unit + heavysuite) and frequently time out: The teaching-assignment suites, which are known to be resource-intensive, are executed twice – once as part of the unit tests and again as part of a separate heavy suite. This duplication increases the overall CI time and frequently results in timeouts, causing the CI pipeline to fail.

Expected Behavior

The desired behavior is to have a consistent and reliable mechanism for excluding tests, ensuring that only the intended tests are executed during CI runs. This can be achieved by:

  • Ignore patterns should live in jest.config.ts so both local and CI runs share a single source of truth: This ensures that the same ignore patterns are applied regardless of whether the tests are run locally or in the CI environment. By centralizing the configuration, we eliminate the risk of discrepancies and ensure consistent test behavior.
  • The CI job should run pnpm exec jest once and let the config handle exclusions: Instead of overriding the configuration with command-line arguments, the CI job should simply invoke Jest using pnpm exec jest and allow the jest.config.ts file to dictate the test execution scope. This ensures that the ignore patterns defined in the configuration are respected.

Reproduction Steps

To reproduce this issue, follow these steps:

  1. Run pnpm exec jest --listTests locally or inspect CI logs.
  2. Observe the ignored suites still appear in the discovered test list. This confirms that the ignore patterns are not being applied correctly.

Proposed Solution

The proposed solution involves the following steps:

  • Move the testPathIgnorePatterns list into jest.config.ts (if not already there) and reference the config from CI: Ensure that all ignore patterns are defined in the jest.config.ts file. This centralizes the configuration and ensures that both local and CI runs use the same set of ignore patterns.
  • Replace the multiline CLI invocation with a plain pnpm exec jest so the shared config is respected: Modify the CI workflow to invoke Jest using pnpm exec jest without any additional command-line arguments that override the configuration. This allows Jest to read the ignore patterns from the jest.config.ts file and apply them correctly.

Technical Context

This issue has a significant impact on the CI pipeline:

  • Adds ~6 minutes to CI and causes flakes when Playwright also covers the same suites: The unnecessary execution of ignored tests adds approximately 6 minutes to the CI run time. Furthermore, when Playwright, an end-to-end testing framework, also covers the same suites, it can lead to flaky tests and inconsistent results.
  • Estimated effort: S: The estimated effort to implement the proposed solution is small, indicating that it is a relatively straightforward fix.

Related Issues/PRs

This issue is blocking progress on other tasks:

  • Blocks the "ship ASAP" goal by keeping CI red due to duplicated heavy suites: The frequent timeouts caused by the duplicated heavy suites are causing the CI pipeline to fail, which in turn is blocking the deployment of new features and bug fixes.

Conclusion

In conclusion, the bug related to dropped Jest ignore patterns in the CI workflow significantly impacts the efficiency and reliability of the CI pipeline. By centralizing the ignore patterns in jest.config.ts and simplifying the Jest invocation in the CI workflow, we can ensure consistent test exclusions, reduce CI time, and prevent flaky tests. This fix is crucial for maintaining a healthy and productive development environment.

For more information on Jest configuration and ignore patterns, refer to the official Jest documentation.