Skip to content

NL2SQL 0.7.0

Wave A of the redesign: every run publishes typed events that observers log, audit or export, hooks can change or stop a run and ask for a person's approval, evals become engine.evals and nl2sql eval with a regression gate, routing votes across five signals, and answers are checked for join fan-out and empty columns. It also adds per-question cost caps, schema drift checks and an MCP server. Several APIs and config keys change: read Upgrading before you install it.

Released 2026-10-15.

Highlights

  • Events, observers and hooks. Every run publishes typed events on one bus: run, node, answer, LLM call, policy denial, index and drift events, each in a catalog. stream_query yields those events from a caller-safe allow-list, ready for SSE. The observers section sends them to a log, a JSON-lines audit log under storage.path, or OpenTelemetry. Six hook points let your code modify a run before the validator and deny it after, and a hook can stop a run as needs_approval until an approver re-runs it.
  • Evals as a product. engine.benchmark is now engine.evals, and nl2sql benchmark is now nl2sql eval. One case format, five scorers and three modes (full, plans, retrieval), with the Chinook gold set moved unchanged and pinned by its digest. A regression gate on execution accuracy runs in CI, a new routing suite of 78 cases covers the demo's three databases, and nl2sql eval seed drafts cases from your own database.
  • Better routing and safer answers. Each example question is its own index entry. Routing fuses five signals (description, examples, verified queries, tables, columns) by rank and drops a datasource far less similar than the best. A question that needs too many sub-queries is refused as QUESTION_TOO_BROAD. Join keys of different types are converted before a cross-database join. A count inflated by a join is caught and retried (JOIN_FAN_OUT), and a column that came back all null is flagged.
  • Cost caps, drift and MCP. settings.pipeline.max_llm_calls and max_cost_usd stop a question before the call that would cross them, with status stopped. nl2sql index --check reports schema drift and what it breaks, without re-indexing. nl2sql mcp serves the engine to MCP clients over stdio, as one role.

Real-database results

  • The CI realdb job passed on the PRs that touched database code.
  • The run on the GCP test bed against main at 5920d3f1: 111 passed, 9 skipped, 0 failed, in 9.5 minutes.

What's included

Features

  • The foundation: the event bus and catalog, the run scope, the observer, hook and scorer registry kinds, the new config sections and settings, the RunStatus values and nl2sql --log-format text|json (#313).
  • Run events on every run, stream_query yielding Events from a caller-safe allow-list, engine.events.subscribe() and catalog(), and the run's trace_id on every log line (#314).
  • Hooks: six points (run.before, datasources.after, plan.after, execute.before, execute.after, answer.after), run in config order, failing closed unless a hook sets on_error: open (#315).
  • Routing: one index entry per example question, five signals fused by rank, a similarity floor, and the settings.routing and settings.retrieval limits (#316).
  • engine.evals: the suite format, the Chinook suite, five scorers, modes full, plans and retrieval, and regenerate through the datasource's adapter (#317).
  • The log, audit and otel observers, with llm.call and llm.changed on the bus. The audit log moves under storage.path (#318).
  • Approval: a hook can ask for one, the result is needs_approval with a request, and an approver's re-run pins the same plan. run_query(plan=SuppliedPlan(...)) runs a plan you already hold (#319).
  • nl2sql eval run, regenerate and publish, and engine.evals.gate(), a regression gate on execution accuracy, with records kept by mode and suite (#320).
  • The Phase 2 events are published: index builds, failed datasource checks, parse retries, policy denials and purges (#321).
  • A limit on sub-queries per question (QUESTION_TOO_BROAD), join keys converted to one type before a cross-database join, and a validator check for join keys of incompatible types (#322).
  • The routing suite of 78 cases over the demo's three databases, and its gate on top-1 and shortlist recall in CI (#323).
  • Answer checks for join fan-out and all-null columns, closest values from the index's sample values, and typed RunWarnings (#324).
  • nl2sql mcp, an MCP server over stdio with ask, list_datasources and describe_table, all limited to the caller's role (the new mcp extra) (#325).
  • The dated price table in llm/prices.py, every LLM call priced, engine.llm.prices(), the per-question caps and a Budget you can use outside a question (#326).
  • Schema drift: engine.index.check(), versions() and diff(), nl2sql index --check, and the verified queries and policy rules a change affects (#327).
  • engine.evals.seed() and nl2sql eval seed: draft eval cases from your own datasource, as typed plans the engine validates and runs read-only, under a cost cap (#328).
  • SQL Server connections take encrypt: false for a local server without TLS. Encryption stays on by default (#307).

Docs

Known limits

  • Routing is not perfect. On the routing suite, top-1 is 76 of 78 and the shortlist 77 of 78. "Which customers complained most about refunds?" and "How many purchases were there per month in 2013?" go to Chinook first.
  • Approvals are not stored. The caller keeps the request and sends the approval back with the re-run. Stored approvals come with the REST API in Wave D.
  • needs_clarification is a reserved status. Nothing returns it until Wave B.
  • MCP runs over stdio only, as one role. MCP over HTTP with API keys comes in Wave C.
  • Schema drift is measured, not acted on. Nothing re-indexes or fixes a verified query by itself.
  • When a hook is attached at execute.before, the value and fan-out checks run no probe query. Only the all-null check runs.
  • While chromadb is a core dependency, it brings OpenTelemetry with it, so the otel extra installs nothing new yet.
  • Seeded drafts have no rows. Run nl2sql eval regenerate after you approve them.

Upgrading

pip install --upgrade "nl2sql-engine[demo]==0.7.0"

Add [otel] for the OpenTelemetry observer and [mcp] for nl2sql mcp. Third-party store authors: pip install "nl2sql-sdk==0.7.0".

Breaking changes: several. Each needs a look before you upgrade.

  • engine.benchmark is now engine.evals, and nl2sql benchmark is now nl2sql eval (#317, #320). There is no alias. The gold files are now suites/chinook.yaml with renamed keys (expected is outcome, gold_sql is sql, alt_gold_sql is alt_sql), and the gate reads execution accuracy. What to do: call engine.evals.run() and gate(), or nl2sql eval run. Move your own gold files to the case format. A 0.6 record starts a new series in the gate.
  • QueryEvent is gone; stream_query yields Events (#314). The stream carries run.started, node.*, answer.delta, result.manifest, sanity.flagged, run.limit_reached, then one run.finished (with the QueryResult in data.result) or run.failed. Policy, hook and LLM call events never reach it. Each SSE data line of POST /api/v1/query/stream is now the whole event: name, at, trace_id, tenant_id and data. What to do: read events by name, and take the result from run.finished. See Events.
  • QueryResult.warnings are typed RunWarnings (#324). Each is {code, node, message, sub_query_id, detail}. Over REST, error_code is now code, and severity is gone: an earlier attempt that a retry replaced has detail.recovered: true. What to do: read code instead of error_code, and detail.recovered instead of severity.
  • QueryResult.status is a RunStatus (#313, #319, #326). Beside success, plan_only and error, a run can now end needs_approval (with approvals_needed) or stopped (with stop, at a cost or call cap). What to do: handle the two new values.
  • engine.datasources.schema() requires user_context (#325), and drops the tables the caller may not read. engine.index.check(), versions() and diff() take it too (#327). What to do: pass the caller's UserContext.
  • settings.observability is gone; use the observers section (#313, #318). A config that still sets it fails to load. By default the engine runs the log and audit observers, and the audit log is now <storage.path>/audit/<date>.jsonl, not logs/audit_events.log. What to do: move exporter and otlp_endpoint to an otel observer. See Observers.
  • OpenTelemetry is the otel extra (#318). It has left the core dependencies, and an otel observer without it is a ConfigError. What to do: pip install "nl2sql-engine[otel]" if you export to a collector.
  • TABLE_K, PLANNING_K and MAX_DECOMPOSER_TABLES are settings (#313, #316): settings.retrieval.table_k, planning_k and decomposer_tables, with the same defaults. What to do: if you patched the constants, set the settings instead.
  • Routing changed (#316). settings.routing.candidates now defaults to 3, a datasource far less similar than the best is dropped (floor_ratio, 0.5), and Retriever.route() replaces retrieve_datasource_candidates. What to do: run nl2sql index after upgrading, so each example question gets its own entry. Until you do, routing has no example votes.
  • A question needs at most settings.routing.max_sub_queries sub-queries (#322), 6 by default. More ends the run with QUESTION_TOO_BROAD. What to do: raise the setting if your users ask wider questions.
  • Store interface version 2 (#327). StateStore gains put_schema_diff and get_schema_diff, and a version 1 store is refused when it loads. The built-in sqlite store migrates itself. What to do: third-party state stores add the two methods and set interface_version = 2.
  • The price table moved to nl2sql/llm/prices.py (#326). nl2sql.evaluation.prices is gone, with no alias. A model the table does not know now costs None, not 0, and a token count the provider did not report is None. What to do: import from nl2sql.llm.prices, price your own models with llm.prices, and handle None in usage and cost. With max_cost_usd set, an unpriced model is a ConfigError.
  • The mcp extra (#325). nl2sql mcp needs pip install "nl2sql-engine[mcp]".

New, not breaking: run_query(plan=SuppliedPlan(...)) runs a plan you hold, through plan.after and the validator, with no planning (#319).

What's next

Next: 0.8, Wave B: business knowledge, the schema profiler, clarifying questions and a cost guard.

Full changelog: v0.6.0...v0.7.0