Open ACE Test Structure and CI Strategy

This document is the single source of truth for the project's test classification, placement, and execution semantics. Before Issue #2429, tests/issues/<number>/ doubled as both the provenance record and the test taxonomy, which mixed tests with different runtime requirements together and excluded them wholesale from the default CI.

Design Conclusions

Tests are kept in exactly one copy: the directory answers "what environment it needs to run in", and the marker answers "why it exists and what its priority is". Therefore:

• No top-level tests/regression/ or tests/security/ is created. Regression and security are test attributes, not runtime environments.
• The same test is never copied into multiple directories such as unit/ or integration/. Copies drift, execute redundantly, and end up in divergent fix states.
• The tests/issues/ legacy quarantine was retired and deleted wholesale with the final #2429 batch and must not be recreated; regression tests go straight into the canonical directories with the regression/issue markers.
• GitHub issue tracking uses @pytest.mark.issue(number); defect regressions additionally use @pytest.mark.regression.

Canonical Directories

Directory	Runtime contract	Default timing
tests/unit/	In-process, fast, no network/real database/subprocesses/servers	Every PR, required
tests/integration/	Crosses database, filesystem, subprocess, or component boundaries; progressively subdivided into sqlite/, postgres/, filesystem/, subprocess/	Every PR; PostgreSQL on a dedicated lane
tests/e2e/	Requires a running Open ACE, a browser, or remote services	Critical subset by path/label; full set on a schedule
tests/performance/	Has time or resource thresholds; may be affected by runner noise	Separate non-blocking lane

Route-boundary tests and concurrency tests have been placed into tests/integration/routes/ and tests/integration/concurrency/ respectively. The historical inventory at the tests root and in the autonomous domain has been fully migrated (#3185); the grandfather inventory was retired with it, and the layout policy pins the end state (only conftest/__init__ at the root). Do not add new "per functional domain" top-level directories or test files in the tests root.

Test database isolation (#2869): the _isolated_unit_db autouse fixture in tests/unit/conftest.py points every unit test at its own throwaway sqlite database (DATABASE_URL=sqlite:///<tmp>/unit-test.db), and this holds unconditionally for any test not marked @pytest.mark.postgres. Tests that go through create_app therefore no longer share the workspace-level app.db — previously, a same-named table with an inconsistent column shape left behind by an earlier test could make a later create_app's schema replay crash at random (no such column); the fixture eliminates that shared state at the root and makes the default test-database choice explicit (no silent inheritance of the development machine's DATABASE_URL). tests/integration/conftest.py already used a per-test tmp_path database. Tests that need Postgres carry @pytest.mark.postgres and live under tests/integration/ (executed by the dedicated postgres lane).

Writing Regression Tests

New regression tests are written directly into their canonical directory, with the origin recorded on the module or the test function:

import pytest

pytestmark = [pytest.mark.security, pytest.mark.regression, pytest.mark.issue(2429)]

Common commands:

# 默认 required suite（与 GitHub Actions 共用定义，并隔离 HOME/数据库环境）
python scripts/ci.py run default-collection python-core

# 按 issue 运行已经迁移到任意规范目录的测试
pytest --issue=2429

# 扩展测试
python scripts/run_extended_tests.py --category e2e --isolated-home

CI Guarantees

Directories alone constitute no guarantee; what the CI consumes is the guarantee:

1. The default suite runs deterministic tests such as unit and integration on the production Python 3.11; the security marker is included in that same required suite.
2. The Python 3.11 required job runs pytest collection over tests/ and fails immediately on collection errors or an item count below .test-baseline.json. The issue directories that had historically entered the PR gate were long since fully migrated into the canonical layers (#2429 batches 1/2); the subsequent legacy-pr required suite, tests/issues/pr-gate-directories.txt, the issue-collection collection gate, and the tests/issues/ quarantine tree were retired wholesale with the final #2429 batch; those regressions are now executed automatically, by directory, by the python-core/python-min required lanes.
3. Critical E2E runs by changed path or label and is not a required check until stabilized; the full E2E runs nightly.
4. PostgreSQL and performance use dedicated lanes, so environment requirements are not hidden as skips.
5. tests/unit/test_test_layout_policy.py forbids adding numbered directories, top-level functional-domain directories, test files in the tests root, and tests/regression/ or tests/security/.

Commands, timeouts, and toolchain versions for all suites come solely from ci/suites.json; both local runs and GitHub Actions execute through python scripts/ci.py. The PR matrix divides work by version (#2868):

• 3.11 (production runtime): python-core — the full pytest tests/ (including integration) plus coverage, run on every non-documents change. This suite's budget was likewise raised from 600s to 1200s due to GitHub runner variance — healthy runs on the same branch have oscillated between 289s and 535s, and a slow runner's tail can blow straight through 600s (#3280). The pytest commands of both python-core and python-min carry --timeout 300 --timeout-method thread --durations 20: when a single test hangs, all thread stacks are dumped at the 300s mark and it fails loudly, instead of burning through the whole suite budget and leaving only an undiagnosable "Command exceeded"; --durations makes the shrinkage of budget headroom visible in the logs before the timeout.
• false-positive-scan: the test-code false-positive scan (Issue #2189, Scope #6), run on every non-documents change (ci.py select_pr_suites default set + consumed by PR Gate, #3186 Phase A), independent of python-core to avoid timeouts. Known debt is expressed as a precise-identity ledger (ci/false-positive-ledger.json: pattern + file + class-qualified function name) — any new or replaced finding goes red; fixes may only shrink it (--prune-ledger only deletes, never adds), and its size is pinned by a contract test (only ever decreasing). The old count-based baseline has been retired.
• 3.10 (minimum supported version): python-min — compileall + the full pytest tests/unit/, run on every non-documents change. Version-specific regressions almost always surface on the oldest interpreter first (for example, before 3.11 datetime.fromisoformat did not accept the Z suffix); the old matrix ran only a 7-file smoke on 3.10, letting such unit-level regressions slip through both PRs and post-merge main pushes — one red main went unnoticed for 75 minutes. Running only unit keeps it fast (~2min) and deterministic (no integration flake, no coverage overhead; the suite's budget was likewise raised from 600s to 1200s due to GitHub runner variance — the same commit has oscillated between 183s and 652s (#3240) — but running the full tests/ directly on the slower 3.10 would still stretch wall-clock time and amplify flake).
• 3.12, 3.14 (forward compatibility): compatibility-smoke — compileall + a small set of critical unit files, selected on dependency changes. Like the postgres lane it carries --timeout 300 --timeout-method thread --durations 20 (#3282): hang protection and slow-test visibility no longer belong to the unit lane alone; the performance lane deliberately carries no per-test timeout — slow wall-clock benchmarks are the design intent. There is also a non-fatal budget-erosion warning (#3282): when any suite finishes successfully having consumed more than 75% of its budget, scripts/ci.py prints ::warning::...completed in ...s, ...% of its ...s budget to the log (which also becomes a Checks UI annotation on GitHub Actions) and (when the nightly metrics stream is enabled) records a suite_budget_warning event — this restores the gradual-slowdown signal after #3281 retired the 600s hard tripwire, letting the shrinkage of budget headroom be seen before it turns into intermittent timeouts.

Both python-min and python-core apply to every code change, so changes to app/** always truly run the full unit suite on the minimum supported version. Python 3.13 remains a declared supported version but is not in the PR matrix. Scheduled workflows run the complete Python suite on 3.10, 3.11, and 3.14, and carry the E2E and the checks most susceptible to runner noise. tests/unit/test_ci_runner.py::test_min_supported_python_runs_the_full_unit_suite locks "the minimum supported version must run the full tests/unit" into the gate. Post-merge verification of main is carried by the existing push: [main] (ci.yml and schema-sync.yml); this change makes it genuinely effective for the minimum version — a red main is therefore an honest, visible red check on the commit. (Note: the current ruleset does not enable "require branches up to date", so stale baselines across PRs can still redden main; that is a separate ruleset setting.)

Before submitting you can run python scripts/ci.py doctor --strict to verify that the local Python/Node major versions match the PR, then use python scripts/ci.py pr --base origin/main to select suites under the same path rules. The PostgreSQL and E2E suites still require their declared services/browsers to be provided locally. .python-version and .nvmrc are pinned to 3.11 and 20 respectively, letting tools such as uv/pyenv/nvm automatically select the same major versions as Actions. requirements-ci.lock is universally resolved from the minimum supported Python 3.10, and every local and GitHub test job installs that file. Production dependencies stay in requirements.txt; the test, lint, build, and audit tooling needed for development and CI stays in requirements-ci.in and must not be added to the production-install requirements.txt; the corresponding dev extra must stay consistent with that input — policy tests automatically check both plus the production dependency boundary. After changing either input you must regenerate and commit the lock using the commands in CONTRIBUTING.md.

Successful collection only proves that the tests "exist and import"; it does not prove the assertions are green. Only tests in required lanes count as merge gates.

CI Health Metrics Timing Validity

ci-health-metrics.yml collects on a GitHub schedule and also supports manual dispatch on the default branch. The collector bounds the time window, sample size, and API request budget per ci/ci-health-policy.json; when the report has too few same-contract samples, the p95 must be flagged insufficient_data and must not be used to promote Critical E2E from advisory to required.

Report schema/timing-derivation version 2 (#3358) marks negative queue durations of GitHub workflows/attempts individually as unavailable: seconds=null, preserving the raw delta, both endpoint timestamps, and the reason. Such values are neither clamped to zero seconds, nor do they cause the whole run/attempt or its success/failure/cancelled verdict to be dropped. The JSON workflow_queue_anomalies and the Markdown provide run/attempt locating information; queue quantiles use only valid values, and the p95 minimum sample count is judged by the number of distinct runs with valid queue data — repeated reruns or already-excluded samples cannot pad it. Consumers reading the report must check schema_version first; queue statistics from versions 1 and 2 must not be mixed directly.

This tolerance applies only to chronology anomalies in the workflow queue's source data. Missing or malformed timestamps still error out; job queue, job execution, attempt wall, and the like keep the existing fail-closed validation and the small clock-skew allowance of at most 2 seconds, and the inherited/skipped job rules are unchanged.

Legacy Issue Test Migration (Completed and Retired)

#2429 migrated tests/issues/ into the canonical directories in batches (unit-like → integration → e2e/performance); the final batch (17) moved out the remaining e2e and deleted the entire quarantine tree, the legacy-directories.txt inventory, the issue-collection collection gate, the issue-tests nightly shards, ci/legacy-issue-{quarantine,failures}.json, and the scripts/legacy_issue_baseline.py comparator. Every promoted test satisfied:

• it can run independently in a clean checkout;
• it failed before the fix and passed after it, with assertions verifying behavior rather than source strings;
• it writes nothing to the developer HOME, production databases, or fixed remote resources;
• it is discovered automatically by its CI lane, with no per-issue workflow YAML edits.

Baseline

.test-baseline.json records item counts and file counts separately. The default CI checks the item count with a real pytest collection; the extended runner's shards can only be allocated per file, so the file baseline is checked proportionally by split_total.

The baseline is a lower bound against tests silently disappearing, not a coverage metric. Lowering the baseline requires the PR to explain why tests were migrated, deleted, or merged; after adding tests it should be tightened upward periodically.

Update procedure: first run python scripts/ci.py run default-collection to record the actual item/file counts; then update actual_* in .test-baseline.json. Lower min_* only when tests are deliberately deleted, merged, or migrated, and explain why in the PR; when adding tests, update only actual_* and periodically tighten min_* toward the actual values. Finally rerun the collection suite. The baseline_runbook field in ci/suites.json is pinned to this section, so the procedure is discoverable from the suite inventory.

E2E Governance Baseline (Issue #2491)

The file-level disposition of tests/e2e/, the nodeid-level debt/promotion state, and the mutually exclusive lane selection are governed by the pure-stdlib tool family in scripts/e2e/ (carrying over the same "one implementation for local and CI" pattern as the legacy_issue_baseline of the retired #2457 failure baseline); governance data lives in ci/e2e-*.json:

• inventory (ci/e2e-inventory.json): every .py under the managed roots (disk enumeration is authoritative, including helper/demo scripts) must have a unique disposition (pytest-automated | standalone-automated | manual-demo) and a home lane; manual entries have executor=none and do not count toward automated coverage. The collects flag declares whether the file currently produces pytest nodeids; a collection change is judged as manifest drift rather than silently absorbed.
• expected nodeids (ci/e2e-expected-nodeids.json): derived by python scripts/e2e/manifest.py snapshot from pytest --collect-only -q -o addopts= (no server/frontend build needed; -o addopts= cancels pytest.ini's -v, otherwise the output is an unparseable collection tree). Contract: test_e2e_inventory_manifest.py / test_e2e_governance.py in the required unit lane re-collect live and compare — if tests/e2e's conftest pulls in heavy dependencies or regresses collection, it reddens the PR lane directly; that gate is intentional, not a false positive.
• debt / promotion state (ci/e2e-state.json / ci/e2e-promotion.json): stored keyed by normalized nodeid/entry, defaulting to unclassified + observing when missing (the observation lane's starting point); every change must go through the write subcommands of python scripts/e2e/governance.py atomically (the only legitimate write path), with the command line attached to the PR description.
• selector: python scripts/e2e/selector.py --event {pr,nightly,weekly} outputs the four mutually exclusive and exhaustive sets normal/advisory/probe/invalid plus selection.json (--shadow is the record-only mode for P1–P3).
• attempt evidence: pytest -p pytest_attempts --e2e-attempts=<path> records every phase of every attempt (JUnit keeps only the final outcome; see docs/dev-notes/2491-rerunfailures-junit-probe.md).

A new E2E file must be registered via governance.py set-disposition before merging into the managed roots; otherwise the inventory completeness check (both the local and the tests/unit paths) exits non-zero.

Appendix: Local shell prerequisites

ci/suites.json declares Bash minimum major version 4; python-core and python-min include real subprocess tests of the fetch wrapper that need Bash associative arrays. python scripts/ci.py doctor reports the Bash path actually selected by PATH, its version, and its capabilities; doctor --strict returns non-zero for missing, incompatible, or failed probes. The strict doctor additionally checks the production Python 3.11 / Node 20; it is not a launch condition for the Python 3.10 minimum-version lane.

run / pr check the Bash prerequisites of the whole execution plan before any selected suite starts, avoiding spending minutes on collection or tests before an environment problem surfaces. Standalone scans, list, and detect do not require Bash; running pytest directly bypasses this entry-precondition check but does not change the tests' own requirements or passing criteria.

macOS /bin/bash 3.2 does not satisfy the requirement. You can install a standalone modern Bash and adjust PATH for the current terminal only; there is no need to replace the system Bash, change the default login shell, or touch wrapper shebangs:

brew install bash
# 先激活装有 requirements-ci.lock 的环境（重复激活会恢复旧 PATH）
source venv/bin/activate
# 保持 venv Python 第一、现代 Bash 随后，不丢弃其余 CLI 路径
export PATH="$VIRTUAL_ENV/bin:$(brew --prefix)/bin:$PATH"
command -v bash
bash --version
command -v python python3
python scripts/ci.py doctor --strict
python scripts/ci.py run default-collection python-core

Keep the PATH directories that existing dependencies such as the Docker Compose CLI live in; do not clobber them with a reduced PATH just to select Bash. The related script tests invoke python3 and docker compose version, so launching the entry point with the absolute path of the venv Python alone does not guarantee those subprocesses use the same dependency set.

Probes and suite subprocesses uniformly strip BASH_ENV / ENV, so developer startup scripts cannot inject into and affect CI; the parent process environment is unchanged. PATH keeps its original order, and relative/empty directory entries are converted to absolute paths against the working directory at invocation time, preventing a different Bash from being selected after a suite switches working directories. Do not rely on personal shell startup scripts to provide CI dependencies; prepare PATH and the locked dependency environment explicitly.

A modern Bash is a necessary condition, not an equivalent guarantee of the full Linux CI on macOS. The canonical execution environment remains the Ubuntu 24.04 of ci/suites.json, the lane's Python, the locked dependencies, and the required services. Cross-platform acceptance should separately record the OS, the actual Python/Bash versions, the identical suite commands, the test results, and the GitHub run link; a passing precondition check or a passing targeted test must not be written up as a full pass. This change adds no time-consuming macOS PR lane and skips no wrapper regression.