Testing Architecture

Tests are organized by scope and live under packages/**/tests. The core engine has unit, integration, and end-to-end tests.

Test layout

packages/nl2sql/tests/
  unit/             # Node-level tests, registries, stores
  integration/      # Pipeline components with real data
  e2e/              # End-to-end flows
  cli/              # CLI command tests
  adapters/         # Cross-dialect adapter contract tests
  sqlalchemy_base/  # tests of the SDK's SQLAlchemy base (nl2sql_sdk.sqlalchemy)
  realdb/           # Against real PostgreSQL, MySQL and SQL Server (the test bed)
packages/api/tests/ # FastAPI route and service tests (stubbed engine)

pytest.ini is the single source of truth for pytest configuration - test paths, markers, and addopts. Do not add a [tool.pytest.ini_options] table to pyproject.toml: pytest reads only one config file, and pytest.ini wins, so such a table is silently ignored. Dev/test tooling lives in the root pyproject.toml under [dependency-groups] dev (pip install --group dev).

Markers and what CI runs

Three independent things can make a test unrunnable in the fast job, so they are three markers. A test may carry more than one.

Everything under packages/nl2sql/tests/integration/ is marked integration automatically by packages/nl2sql/tests/conftest.py, and everything under packages/nl2sql/tests/realdb/ is marked realdb the same way. The llm marker is declared per module, next to a comment saying which call needs the key. Only four modules carry it - test_ast_planner_real_data.py, test_decomposer_real_data.py, test_logical_validator_real_data.py and test_answer_synthesizer_real_data.py - all of them nodes that call a chat model directly. Nothing else in the directory needs a key: indexing skips its optional enrichment pass when no LLM is available, so tests that index real data run key-free.

.github/workflows/test.yml has two test jobs over the source tree:

• test runs pytest -m "not integration and not realdb" across Python 3.12 and 3.13. This is the fast job and must stay fast.
• integration runs pytest -m "integration and not llm" on one Python version, after writing and indexing a demo project (nl2sql init --demo demo-ci and nl2sql --config demo-ci/nl2sql.yaml index), key-free with the local embedder. The ONNX model is cached across runs. That subset covers schema indexing, datasource resolution, schema retrieval, aggregation and the local embedder against the real Chinook database; run it with --collect-only -q for the current list.

A separate workflow, .github/workflows/realdb.yml, runs pytest -m realdb against the test bed on GitHub's runners: on pull requests that change database-facing code, nightly on main, and on demand before a release. Its realdb-gate job always runs and is the required check.

Three more jobs test the package rather than the source tree:

• api-image builds packages/api/Dockerfile from the repository root, the image the release pushes to GHCR, without pushing it, and imports nl2sql_api.main inside it. tests/unit/test_docker_images.py separately checks that every Dockerfile's FROM python:X.Y satisfies every package's requires-python.

• build builds the three wheels and sdists, installs them into a clean virtualenv and runs import nl2sql and nl2sql --help.

• fresh-install takes those wheels and installs them the way a user does, pip install "nl2sql-engine[demo]", into an empty virtualenv outside the checkout. It then boots nl2sql demo --hosted (the command the Hugging Face Space runs) with no API key, and passes only when the playground page, /api/meta, /api/schema and /api/index all answer from that install. It is what catches a wheel that imports but ships without the built playground, the sample databases or a dependency the demo extra forgot. scripts/fresh_install_check.py is the whole check, so it runs the same on a laptop:

  pip install build
  python scripts/fresh_install_check.py --build   # builds into dist/ first
  

Any *_API_KEY in your environment is dropped for the run, so a key you exported cannot make it pass where CI would fail. The decisions it makes are unit-tested in tests/unit/test_fresh_install_check.py. After a release the same script runs twice more, from publish_pypi.yaml: with --pypi X.Y.Z to install the published version from PyPI itself, and with --url https://nadeem4nk-nl2sql-demo.hf.space --expect-version X.Y.Z to wait for the live Space to serve it, then once more with its custom domain as --url (see Releasing).

Nothing selects llm. Those tests are run by hand with a key:

OPENAI_API_KEY=sk-... pytest -m integration

Collection is not enough to keep these honest. Every test in the directory imports cleanly even when it calls a method that no longer exists, so rot shows up only at call time - which is the point of running the key-free subset in CI rather than gating on --collect-only. The subset is worth keeping wide: every test that can run without a key should, because the ones that cannot are never run by CI at all.

tests/e2e builds one demo project per session (demo_project), indexes it, then moves its .nl2sql aside and leaves a file of that name, so no test can open the shared index or state store. Each test runs on its own copy: private_storage(demo_project, tmp_path / "store") as project_config's storage, or copy_demo(demo_project, tmp_path / "demo") for a whole project. When tests shared the one state.db, engines left open by one test, copies taken while it was open and CLI runs writing it all met in the same file, and CI failed with disk I/O error in whichever test pytest-randomly ran next.

Randomised test order

pytest-randomly shuffles the order of tests on every run, which turns order-dependent tests - shared class-level caches, leaked globals, fixtures that do not unwind - into loud local failures rather than intermittent CI ones. Each run prints the seed it used; reproduce a failure with pytest -p randomly --randomly-seed=<seed>.

Settings in tests

The engine reads its settings from its runtime, not from a process global (configuration v2, section 8), so a test sets them on the runtime it builds:

• One node or service: nl2sql.testing.runtime.fake_runtime(settings=..., policies=..., traces_dir=..., **attrs) builds a stand-in runtime with a real config, a local results store and a files trace store on traces_dir (pass traces= for another trace store). Settings are written as in YAML: settings={"traces": {"mode": "always"}}.
• A whole engine: Runtime.from_config(make_config(tmp_path, ...)). An Embeddings(instance=FakeEmbeddings(size=8)) keeps it free of any model.
• One run with different settings: ctx.derive({"settings.pipeline.plan_cache": True}) returns a run-scoped runtime with a derived config, its own curated answers and its own LLMSet, sharing the stores and the engine's LLM client cache. Evaluation and trace replay use it the same way; the runtime it came from never changes. tests/unit/test_run_scoped_runtime.py pins that down.

There is no process-global settings object to monkeypatch, and no module writes os.environ: a test sets what it needs in the config it builds. The architecture tests keep it that way (An Engine Owns Its Config).

flowchart TD
    Unit[Unit Tests] --> Core[Core Components]
    Integration[Integration Tests] --> Pipeline[Pipeline Nodes]
    E2E[End-to-end Tests] --> Orchestration[Full Orchestration]

Runtime coverage targets

• Pipeline nodes (test_node_*.py)
• Subgraph orchestration (test_sql_agent_subgraph.py)
• DAG layering (test_graph_layers.py)
• Registry and store behavior (test_*_registry.py, test_state_store.py)
• REST contract (packages/api/tests/) via FastAPI TestClient with the get_engine dependency overridden, so no datasource, LLM or network is needed

Deterministic validation

Tests enforce deterministic behavior by:

• using structured output schemas for planner/decomposer
• validating stable IDs and DAG layering
• running logical validator checks for expected schema alignment

Adapter compliance testing

The SDK includes testing utilities to validate schema introspection and result contracts for new adapters.

Source references

• Test configuration: pytest.ini
• Core tests: packages/nl2sql/tests/
• API tests: packages/api/tests/