Skip to content

NL2SQL 0.6.0

Phase 2: every built-in adapter is now tested against real PostgreSQL, MySQL and SQL Server servers, adapters are built on SDK v2 with a conformance kit, storage has four pluggable stores, LLM and embeddings providers take any OpenAI-compatible server, and policies gain allow and deny globs. Several config keys and interfaces change: read Upgrading before you install it.

Released 2026-10-15.

Highlights

  • Tested on real databases. A test bed runs PostgreSQL with Pagila, MySQL with Sakila and SQL Server with AdventureWorksLT, plus a generated schema of 1,200 tables. It runs in CI on every PR that touches database code, and on a GCP VM on demand. The VM run found three bugs that CI had hidden, all fixed in this release.
  • Adapter SDK v2 and a conformance kit. Each adapter has a typed connection model, so a misspelt key gets "did you mean". You can choose which schemas and tables to index (IndexScope). Row counts come from the database catalog. A table that cannot be read becomes an issue in the index build instead of a crash. Read-only sessions and statement timeouts are enforced where the database allows it and reported where it does not, and a failed connection test names the stage that failed. The SQLAlchemy base is public in nl2sql-sdk[sqlalchemy], and nl2sql-sdk[testing] is the kit that all five built-in adapters pass.
  • Four storage slots. storage has index, state, traces and results, each a built-in or a plugin store. The index's build pointer is now the commit, so a failed build leaves the previous index live. Each embedding model gets its own namespace. Traces and results are deleted after a set number of days.
  • Providers and policies. openai_compatible with a base_url works with any OpenAI-style server, with openrouter and ollama as presets of it. Each agent can set structured_output, max_tokens, timeout_s, max_retries and headers, and a malformed answer is retried with the error. Policies use allow and deny globs on datasource.schema.table, the planner never sees a denied table, and a PolicyCheck plugin can add your own access rules. engine.datasources.types(), test() and add() are the methods a Connect-a-database form needs.

Real-database results

  • The CI realdb job passed on every PR that touched database code.
  • The run on the GCP VM on 2026-10-14 (python scripts/testbed.py run -- pytest -m realdb): 110 passed, 4 skipped, 0 failed. The four skips are expected. SQL Server does not declare ENFORCES_READ_ONLY (2 tests), and the 60-second target for reading the wide schema is measured on PostgreSQL only (2 tests).
  • Runs on the real VM found and fixed three problems. MySQL 8 logins needed cryptography, and SQL Server used a hard-coded ODBC driver (#303). The test bed also had Windows problems (#299, #301, #302, #304).

What's included

Features

  • Embeddings providers v2: openai_compatible, base_url on the hosted providers, a known dimension for each model, and one probe call when the dimension is unknown (#281).
  • Store interfaces in nl2sql_sdk.stores, the four storage slots and the store registry kind (#283).
  • LLM providers v2: openai_compatible, with openrouter and ollama as its presets, and the agent fields structured_output, max_tokens, timeout_s, max_retries and headers. Header values are never printed (#284).
  • Policy allow and deny globs, one Scope wherever access is decided, and denied tables hidden from the planner (#285).
  • The real-database test bed: a Docker Compose file, the realdb pytest marker and a CI workflow (#286).
  • A malformed structured answer is sent back to the model with the error, up to parse_retries times. After that the run fails with MODEL_OUTPUT_INVALID (#287).
  • The PolicyCheck extension point. Its grants are intersected with the YAML roles, and it denies if it fails (#288).
  • The index store: one namespace per embedding model, the build pointer as the commit, and nl2sql index prune (#289).
  • scripts/testbed.py starts, uses and stops the GCP test-bed VM through IAP (#290).
  • Adapter SDK v2: DatasourceAdapter, strict connection models with "did you mean", and IndexScope (#291).
  • Trace and result stores with max_age_days retention, engine.storage.info() and purge(), and nl2sql storage purge (#292).
  • The sqlite state store: snapshots that are never changed, migrations that adopt a 0.5 state.db, and a lock during index builds (#293).
  • The SQLAlchemy base in nl2sql-sdk[sqlalchemy]. It reads the schema once, takes row estimates from the catalog, collects sample values only when asked, and reports index issues (#295).
  • engine.datasources.types(), test(Datasource) without registering it, and add() checking the index scope (#296).
  • Read-only sessions and statement timeouts on every adapter that can enforce them, and connection tests that name the failed stage (#297).
  • The adapter conformance kit, nl2sql-sdk[testing], run on all five built-in adapters (#298).

Fixes

  • The SDK docs page no longer has leftover merge-conflict markers, and CI now fails on any it finds (#294).
  • DuckDB schemas are read from DuckDB's own catalog, so indexing works on SQLAlchemy 2.1 (#300).
  • nl2sql-engine[mysql] installs cryptography for MySQL 8 logins. SQL Server uses the newest installed ODBC driver, with a clear message when none is installed (#303).
  • Fixes to the test bed on Windows and on the VM: passwords, Docker start-up, SQL Server start-up, and stopping cleanly (#299, #301, #302, #304).

Docs

Known limits

  • SQL Server does not make the session read-only. check() warns about it, so connect with a login that can only read.
  • The built-in stores are for a single replica. A PostgreSQL state store and a pgvector index, for several replicas, come later.
  • There is no built-in S3 or ADLS results store. Write one as a ResultStore plugin; the Artifact Store page has an S3 example.
  • Policies go down to tables. Column denies, row filters and user attributes come later.
  • engine.datasources.types(), test() and add() are Python only. The REST routes and the Connect-a-database form come in Phase 4.
  • There are no Snowflake or BigQuery adapters yet.
  • The 60-second target for reading the 1,200-table schema is measured on PostgreSQL only. Retrieval limits as settings and incremental re-indexing come after 0.6.

Upgrading

pip install --upgrade "nl2sql-engine[demo]==0.6.0"

Adapter, store and policy-check authors: pip install "nl2sql-sdk==0.6.0" (add [sqlalchemy] for the SQLAlchemy base, [testing] for the conformance kit).

Breaking changes: several. Each needs a look before you upgrade.

  • Storage sub-blocks and a new index namespace (#283, #289, #292). storage has index, state, traces and results sub-blocks. path on its own is still valid. The index now lives in one namespace per embedding model, and 0.6 never reads 0.5's nl2sql_store collection. Traces are deleted after 14 days and results after 7 by default. What to do: run nl2sql index after upgrading, then nl2sql index prune to drop the 0.5 collection. Set max_age_days on traces or results if you want them kept longer. A 0.5 state.db is adopted as it is.
  • Policy patterns (#285). allow and deny replace allowed_datasources and allowed_tables. A pattern is *, datasource or datasource.schema.table. A two-part pattern such as sales.orders is rejected when the config loads, and matching ignores case. What to do: rewrite each role, for example sales.orders as sales.<schema>.orders or sales.<schema>.*.
  • Denied tables are hidden from the planner (#285). A table a role cannot read never reaches the model. A user who asks about one sees "can't be answered from the connected data". A plan that still names a hidden table is refused by the validator. What to do: check that each role allows the tables its users need.
  • sample_values is off by default (#295). Column values are no longer collected into the index and prompts unless index.sample_values is on. The demo keeps them on. What to do: set index: {sample_values: true} on a datasource whose answers need its values, then run nl2sql index.
  • Strict connection keys (#291). An unknown key in a datasource's connection block is now an error with a "did you mean" hint. It used to be ignored. What to do: run nl2sql config check and fix what it reports.
  • Adapter interface version 2 (#291, #295). DatasourceAdapter replaces DatasourceAdapterProtocol, with no alias, and a version 1 adapter is refused when it loads. What to do: port your adapter to DatasourceAdapter, ideally on nl2sql_sdk.sqlalchemy.SQLAlchemyAdapter, and run the conformance kit. See Adapter development.
  • LLM and embeddings provider interface version 2 (#281, #284). A version 1 LLM or embeddings provider is refused when it loads. What to do: add structured_output to an LLM provider and describe() to an embeddings provider, and set interface_version = 2.
  • Deleted index arguments and setting (#289). index.build(full=..., switch_guard=...), nl2sql index --full and settings.schema.version_mismatch are gone. What to do: drop them. A failed build now leaves the previous build live, so there is nothing to guard.
  • engine.policies.allowed() is now engine.policies.scope() (#285). A flat table list cannot express a deny. What to do: call scope(user_context, datasource_id) and ask it allows(schema, table).
  • S3 and ADLS results only through a plugin (#292). What to do: implement the ResultStore interface and set it as the results store.
  • SQL Server always connects with Encrypt=yes (#303). ODBC Driver 17 did not encrypt by default. What to do: for a server with a self-signed certificate, set trust_server_certificate: true in its connection block.
  • nl2sql-engine[mysql] installs pymysql[rsa] (#303), which brings in cryptography. What to do: nothing, unless you pin dependencies, in which case allow cryptography.

What's next

Next: Phase 3, hooks, observers and evals.

Full changelog: v0.5.0...v0.6.0