Pre-commit test policies

Definitions

• Pre-commit test - Any single test in a pre-commit test suite.
• Pre-commit test suite - A collection of pre-commit tests that have a common denominator. A test suite runs in a single Jenkins job. Currently, suites are grouped by SDK languages, e.g., Python, Java, and Go.

Policies

Pull Requests

• A PR must pass pre-commit tests before being committed to the main Beam repo.
  • The relevant pre-commit test suites are automatically launched according to PR contents.

Problems

Breakage

Breakage is when one or more tests in a pre-commit test suite fails or is flaky (occasionally fails).

• Breakages should be fixed within 8 hours.

Slowness

Slowness is when the total time to run a pre-commit suite exceeds 30 minutes*, including the time the job spends in the Jenkins queue.

• Slowness should be fixed within 24 hours.

* See the Pre-commit Slowness Triage Guide for a precise definition of slowness and for information on dealing with slowness.

Problem Resolution

For any problem, the options are, one of:

• Roll back the culprit PR.
• Roll out a fix within 24 hours.
• Disable the slow test or feature temporarily (make sure there’s a tracking issue to re-enable it).