Skip to content

Benchmark Results

Recorded eval runs of the engine on the Chinook and routing suites. Generated by nl2sql eval publish from the records in benchmarks/; do not edit by hand. Grouped by mode and database, newest first; the runs 0.6's tier 2 recorded are mode full runs. Accuracy is execution accuracy (result_match and outcome, the measure 0.6 published), with its 95% Wilson interval. "Δ vs previous" is the change against the previous run of the same config, and how many cases flipped each way with McNemar's exact p-value on them: with a suite this size a few points either way is noise, so read the flips and the p-value, not the headline. Runs on another suite (@digest), another schema or, for mode full, other roles or curated answers on are not comparable and start a new series. See the Chinook suite and the routing suite.

Mode full: chinook (sqlite, 11 tables / 64 columns)

nl2sql eval run --mode full: the configured model end to end. Accuracy is execution accuracy (result_match and outcome), the measure 0.6's tier 2 published, so the runs before and after 0.7 read the same; faithfulness is its own column.

Latest per config

Config Date (UTC) Commit Accuracy (strict, 95% CI) Accuracy (lenient, 95% CI) Faithfulness $/question Δ vs previous Note
gpt-5.4 2026-09-23 de42c40 79.1% [64.8-88.6] 79.1% [64.8-88.6] 86.8% $0.0194 new series (suite changed) new scoring

All runs

Date (UTC) Commit Note Config Models Dataset Schema Roles Passes Accuracy (strict, 95% CI) Accuracy (lenient, 95% CI) Faithfulness $/question Δ vs previous Answerability P / R Tokens/question (in / cached / out) p50 p95 Determinism Status
2026-09-23 01:30:50 de42c40 new scoring gpt-5.4 openai:gpt-5.4 (answersynthesizer, astplanner, datasourceresolver, decomposer, refiner) chinook_gold.yaml @dcb8c715 @ceed80fe admin 1 79.1% [64.8-88.6] 79.1% [64.8-88.6] 86.8% $0.0194 new series (suite changed) 100.0% / 75.0% 10,291 / 6,900 / 617 7.03s 9.15s - complete
2026-09-22 14:18:51 27cad30 after baseline fixes gpt-5.4 openai:gpt-5.4 (answersynthesizer, astplanner, datasourceresolver, decomposer, refiner) chinook_gold.yaml @7d104084 @ceed80fe admin 1 58.1% - 87.5% $0.0202 accuracy +16.3 pp, faithfulness -6.9 pp, $/question -$0.0034, 0 flipped to fail, 7 to pass (McNemar p=0.016) 100.0% / 75.0% 10,655 / 7,082 / 633 8.54s 10.99s - complete
2026-09-22 04:41:56 1a8101d first gpt-5.4 baseline run gpt-5.4 openai:gpt-5.4 (answersynthesizer, astplanner, datasourceresolver, decomposer, refiner) chinook_gold.yaml @7d104084 @ceed80fe admin 1 41.9% - 94.4% $0.0236 first run 100.0% / 75.0% 11,222 / 6,153 / 627 6.62s 8.96s - complete

Mode plans: chinook (sqlite, 11 tables / 64 columns)

nl2sql eval run --mode plans: each case's hand-written plan through the validator, generator and executor, on a fake LLM, once per role. No key; every case should pass. CI's integration job gates on the newest run here.

Date (UTC) Commit Note Dataset Schema Roles Cases Accuracy (execution, 95% CI) Δ vs previous Status
2026-10-15 16:26:08 1099c15 CI baseline: ubuntu-latest, Python 3.12, https://github.com/nadeem4/nl2sql/actions/runs/38767715412 chinook @dc1e8456 @ceed80fe all 141 100.0% [97.4-100.0] first run complete

Retrieval recall: chinook (sqlite, 11 tables / 64 columns)

nl2sql eval run --mode retrieval: the share of each answerable case's needed tables and columns that schema retrieval sends the planner, with the vector search forced on and no LLM. Mean over cases; tables and columns sent are means too. Routing top-1 is the share of cases routing ranks one of their datasources first, shortlist the share whose datasources are all among those it keeps. CI's integration job gates on the newest run of each suite here.

Date (UTC) Commit Note Dataset Schema Search Embedding Cases scored Table recall Column recall Routing top-1 / shortlist Δ vs previous Tables / columns sent
2026-10-15 17:44:47 a33c8ce CI baseline: ubuntu-latest, Python 3.12, https://github.com/nadeem4/nl2sql/actions/runs/38778183715 chinook @dc1e8456 @ceed80fe k 8 tables / 12 planning local/all-MiniLM-L6-v2 96 97.9% 76.5% 100.0% / 100.0% tables +0.0 pp, columns +0.0 pp 8.0 / 22.2
2026-10-15 16:27:15 1099c15 CI baseline: ubuntu-latest, Python 3.12, https://github.com/nadeem4/nl2sql/actions/runs/38767715412 chinook @dc1e8456 @ceed80fe k 8 tables / 12 planning local/all-MiniLM-L6-v2 96 97.9% 76.5% - new series (suite changed) 8.0 / 22.2
2026-10-15 13:05:04 653fa73 0.6 baseline before Phase R (R1) chinook_gold.yaml @8be35a10 @ceed80fe k 8 tables / 12 planning all-MiniLM-L6-v2 43 97.7% 71.9% - new series (suite changed) 8.2 / 24.1
2026-09-22 03:39:36 571cd16 first retrieval baseline chinook_gold.yaml @7d104084 @ceed80fe k 8 tables / 12 planning all-MiniLM-L6-v2 39 98.5% 72.6% - first run 8.3 / 24.2

Retrieval recall: chinook+support+webanalytics (sqlite, 20 tables / 121 columns)

nl2sql eval run --mode retrieval: the share of each answerable case's needed tables and columns that schema retrieval sends the planner, with the vector search forced on and no LLM. Mean over cases; tables and columns sent are means too. Routing top-1 is the share of cases routing ranks one of their datasources first, shortlist the share whose datasources are all among those it keeps. CI's integration job gates on the newest run of each suite here.

Date (UTC) Commit Note Dataset Schema Search Embedding Cases scored Table recall Column recall Routing top-1 / shortlist Δ vs previous Tables / columns sent
2026-10-15 19:01:25 baaae0c CI baseline: ubuntu-latest, Python 3.12, https://github.com/nadeem4/nl2sql/actions/runs/38788733052 routing @1f65922e @bffaf096 k 8 tables / 12 planning local/all-MiniLM-L6-v2 78 - - 97.4% / 98.7% new series (suite changed) -
2026-10-15 17:45:34 a33c8ce CI baseline: ubuntu-latest, Python 3.12, https://github.com/nadeem4/nl2sql/actions/runs/38778183715 routing @c87d2a8e @bffaf096 k 8 tables / 12 planning local/all-MiniLM-L6-v2 78 - - 97.4% / 98.7% first run -