Wave C: conversation, charts, dashboards, launchpad and MCP over HTTP (design)

Principles

The owner set them for the redesign, and they rank above everything below:

• The package is the product; the playground only demonstrates it. Every capability on this page is a public facade method or an extension point in nl2sql first. The CLI, nl2sql-api, the playground and the MCP server use the public API only. The web UI for conversations, charts and dashboards is Wave D, not here.
• No active users: design it correctly, not compatibly. No shims, no aliases, no deprecation paths.
• Never over-engineer. Ship what 0.9 needs; the cut list is part of the design.
• Configurable in nl2sql.yaml (and its section files) or in code with dependency injection, extensible through the registry, every new action published on the Wave A event bus, and hooks apply to every run this wave adds: a conversation turn, a dashboard tile.

Three older rules decide most of this wave: the model emits a typed plan, never SQL (dashboards build plans in code, and the model never writes a chart); anything the generator can reject, the validator rejects first (a dashboard filter is a predicate added to a plan before the validator, not text added to SQL); and Wave B's Only an Accept Writes a Person's File, which this wave amends, explicitly and narrowly, for the setup workflow (D123).

Decisions

The owner was asleep when this design was written and asked that its recommendations be used, each marked for confirmation. Wave A uses D9 to D35 and Wave B D36 to D59, so Wave C starts at D100. After an independent review, several were amended; those are marked so. Each decision is written into the section it affects.

Where 0.8 will leave us

What the code has today (0.6), plus what Waves A and B add, as their designs describe it.

1. Goals and non-goals

What ships in 0.9

Not in Wave C

2. Order of work

One small foundation PR, then four tracks that touch different files, then dashboards, which need the charts and the supplied-plan path.

flowchart LR
  C0["C0. Foundation<br/>models, config, codes, events, registry kind,<br/>store tables, namespace and command stubs"]
  C0 --> T1["Track 1: conversation<br/>C1 core, C2 chat tools"]
  C0 --> T2["Track 2: charts and diagrams<br/>K1 charts, K2 diagrams"]
  C0 --> T3["Track 3: launchpad and Connect<br/>L1 scan, L2 writer and REST, L3 launch"]
  C0 --> T4["Track 4: MCP over HTTP<br/>M1 API keys, M2 transport"]
  T2 --> D["Dashboards<br/>D1 generate and refresh, D2 filters and pin"]
  T1 --> D
  D --> REL["Release 0.9"]
  T1 --> REL
  T3 --> REL
  T4 --> REL

3. Conversation

The shape

# nl2sql/conversations/models.py (engine: a turn holds a QueryResult)
TurnKind = Literal["question", "choice", "tool", "retry"]
TurnStatus = Literal["answered", "needs_clarification", "needs_approval", "stopped", "error", "tool"]

class Turn(BaseModel):
    index: int                         # 1-based, in the order asked
    at: datetime                       # UTC
    kind: TurnKind
    text: str                          # what the user typed, or the chosen option's label
    question: str | None               # what ran; None for a tool turn
    rewritten: bool                    # question differs from text
    refers_to: list[int] = []          # the turns the rewrite drew on
    status: TurnStatus
    trace_id: str | None               # the run's id: its events, trace and result pages
    result: QueryResult | None         # stored without rows (D106)
    tool: ToolOutput | None = None     # a tool turn's output (section 4)
    error: dict[str, Any] | None = None   # code and message, for a tool turn or a turn with no result

class Conversation(BaseModel):
    id: str                            # random, 22 url-safe characters
    tenant_id: str | None
    user_id: str | None                # the owner (D105)
    title: str                         # the first question, cut at 80 characters
    datasource_id: str | None          # when set, every turn skips routing
    created_at: datetime
    updated_at: datetime
    turns: list[Turn]
    pending: ClarificationRequest | None   # Wave B's request the last turn ended with

class ConversationInfo(BaseModel):
    id: str; title: str; updated_at: datetime; turns: int

Conversation, Turn and ConversationInfo are exported from nl2sql. A turn's result.sub_queries[].rows is always None once stored; everything else of the QueryResult is kept, so a turn shows its SQL, plan, explanation, validation, warnings, source and usage without its trace.

The facade

Every method takes user_context keyword-only and required (D105, D130).

Every method raises ConversationNotFound (a LookupError) for an id that does not exist or is not the caller's. ask, stream, retry and tool on a conversation that is already running a turn raise ConversationBusy: the turn holds the state store's lock conversation:<id> (Phase 2's StateStore.lock) for the pipeline's timeout_s plus ten seconds.

Rewriting a follow-up

flowchart TD
  IN["ask(conversation_id, text or choice)"] --> SC["open the turn's RunScope"]
  SC --> K{"what is it?"}
  K -->|"first turn"| RUN["run the text"]
  K -->|"choice of the pending request"| RE["run_query(pending.question, choice=...)"]
  K -->|"a follow-up"| RW["conversation_rewriter: one call, typed Rewrite"]
  RW -->|"kind: question"| RUN2["run rewrite.question"]
  RW -->|"kind: choice"| RE
  RW -->|"kind: tool"| TOOL["the named chat tool"]
  RW -->|"invalid after parse retries"| FAIL["REWRITE_FAILED: scope closed, no graph run"]
  RUN --> ST["store the turn, publish conversation.turn"]
  RUN2 --> ST
  RE --> ST
  TOOL --> ST
  FAIL --> ST

# nl2sql/conversations/models.py
class ToolCall(BaseModel):
    name: str                          # an enabled chat tool
    args: dict[str, Any] = {}          # validated by that tool's args_model

class Rewrite(BaseModel):              # the conversation_rewriter's structured output
    kind: Literal["question", "choice", "tool"]
    question: str | None = None        # kind "question": answerable with no other context
    standalone: bool = False           # the text needed no context; question is the text verbatim
    refers_to: list[int] = []          # turn indexes it used
    choice: str | None = None          # kind "choice": an option id of the pending request
    tool: ToolCall | None = None       # kind "tool"

One scope per turn. Wave A opens a RunScope in QueryAPI and publishes run.started when it opens. A turn needs the scope before the rewrite, so the rewrite's model call carries the turn's trace_id, its usage lands in the turn's QueryResult.usage and it counts against max_llm_calls and max_cost_usd. So (a need from Wave A):

1. The conversation API calls an internal QueryAPI.open_scope(text, user_context, datasource_id, execute), which opens the scope and publishes run.started with question = the typed text.
2. The rewrite's structured() call is made with the scope's callbacks (RunScope.callbacks(): the usage collector and the budget), inside its context, so llm.call and llm.parse_retry carry the trace_id.
3. The run then goes through QueryAPI.run_query(question, ..., scope=scope) (or stream_query), which uses the open scope instead of opening one. conversation.rewritten carries the question that ran.
4. On REWRITE_FAILED, the conversation API builds a QueryResult with status error and the code without a graph run, and closes the scope with run.finished, so every turn's scope closes exactly once.

Hooks. A turn is a run, so all six of Wave A's points apply. run.before sees the rewritten question; HookContext gains conversation_id (set for a turn, None otherwise). Nothing else in the hook interface changes.

Clarifications and approvals in a conversation

• A clarification (D104). A run that ends needs_clarification stores a turn with that status and sets Conversation.pending to the result's clarification (ClarificationRequest). The next turn may pass choice="b", or text the rewriter maps to an option ("the second one", "by order date"). Either way the engine calls run_query(pending.question, choice=ClarificationChoice(request=pending, option_id=...)): Wave B refuses a choice unless the question equals the request's, so the request's own question is passed, not the typed text. The turn's kind is choice and its text is the option's label. CLARIFICATION_STALE (the schema or the glossary changed) ends the turn error with Wave B's message. Any next turn clears pending; a choice with nothing pending raises ValueError ("Nothing in this conversation is waiting for a choice.").
• An approval (D107). A run that ends needs_approval stores a turn with that status and the result's approvals_needed. An approver acts on each ApprovalRequest outside the conversation (an app shows it, as Wave A describes); the asker then calls retry(conversation_id, user_context=, approvals=[...]), which re-runs the last question turn's question with those approvals. Wave A checks that the request's question and asker are this run's and pins the approved decomposition and plan; different SQL asks again, as everywhere.

Ownership and retention

The config

# nl2sql.yaml
conversations: conversations.yaml

# conversations.yaml (every value shown is the default)
context_turns: 3        # turns the rewriter sees
max_turns: 50
max_age_days: 30        # null keeps them

The section is optional; absent, these defaults apply. In code: Config(..., conversations=Conversations(context_turns=5)).

Events

None carries rows (Wave A D15). conversations.stream yields Wave A's caller-safe allow-list (D17) plus these two, which carry only what the caller typed and was answered; the allow-list grows by them for the conversation stream only.

4. Chat tools

Invoked, not chosen (D108)

A chat tool turns something the conversation already has into something else: a chart, a diagram, a report, a dashboard tile. It never asks the database a new question. A tool runs only when the user asks for it:

1. By name. engine.conversations.tool(cid, "chart", {"turn": 3}, user_context=...). A UI button, an MCP client or a script does this.
2. In words. "Chart that", "draw the tables behind this", "write this up", "pin it to my sales dashboard". The rewrite call D101 already makes returns kind: "tool" with a typed ToolCall, so asking in words costs nothing extra.

The model never calls a tool on its own initiative, and nothing loops: one turn is one question run or one tool call.

The interface

# nl2sql/chat_tools/__init__.py (engine: a tool sees a Conversation)
class ToolContext(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True)
    conversation: Conversation          # a copy; the tool cannot change it
    user: UserContext
    engine: "NL2SQL"                    # the public facade only (D109)

class ToolOutput(BaseModel):
    kind: Literal["chart", "mermaid", "markdown", "json"]
    title: str
    body: str | dict[str, Any]          # chart: ChartSpec as JSON; mermaid and markdown: text
    refs: dict[str, str] = {}           # e.g. {"dashboard_id": ..., "tile_id": ...}

class ToolNotApplicable(Exception): ...   # the tool cannot apply to that turn; the message says why

class ChatTool(Protocol):
    interface_version: ClassVar[int]          # 1
    description: ClassVar[str]                # one line; the rewriter reads it
    args_model: ClassVar[type[BaseModel]]     # its JSON Schema is shown to the rewriter
    def __call__(self, ctx: ToolContext, args: BaseModel) -> ToolOutput: ...

class ChatToolInfo(BaseModel):
    name: str; description: str; args_schema: dict[str, Any]

Kept tiny: a tool has no config of its own in 0.9 (chat_tools.yaml lists names only), so there is no config_model or from_config; it is built with no arguments.

registry.add_chat_tool(name, cls) and @registry.chat_tool(name).

# nl2sql.yaml
chat_tools: [chart, diagram, report, pin]    # absent: these four; []: none; a plugin by its registered name

The four built-ins

Diagrams

Mermaid text, written to read the same on a light or a dark page (D113): no %%{init} line, no style, classDef or linkStyle, no colour. Emphasis is carried by shape and grouping.

• The answer view (nl2sql.diagram_for(result) -> str, a pure function exported from nl2sql): a flowchart LR of the tables each sub-query's plan read, one node per table (datasource.schema.table), one edge per join labelled with its columns, and one subgraph per datasource when there are several, with the combine step's join keys between them. It reads only the result the caller already holds. A verified query has no plan; its diagram lists the tables its SQL reads, unconnected.
• The schema view (engine.datasources.diagram(datasource_id, *, user_context, tables=None) -> str, caller required, D130): an erDiagram from the live snapshot, each table with its key columns, each foreign key and each accepted glossary join as a relationship. Only tables the caller's scope reaches, through datasources.schema(datasource_id, user_context=...) (Wave A, B29). With tables, those and their neighbours one relationship away; without, the whole datasource. More than max_tables (30) is cut to the 30 with the most relationships, and a %% 41 more tables not shown comment says so.

Reports

engine.conversations.report(cid, user_context=, turns=None, title=None) assembles Markdown from stored turns, with no model call (D114):

# <title, or the conversation's title>
Generated <date> · <n> questions · nl2sql <version>

## <question>
<final_answer.summary>
**Why this answer.** <each step's QueryResult.explanation.steps[].text (Wave B)>
<a Markdown table of the first 20 rows, read through engine.results.page; or "results expired">
<the takeaway of the chart engine.results.chart chooses, when it is not a table>
<details><summary>SQL</summary> ```sql ... ``` </details>
Trace: <trace_id>

5. Auto charts

The spec

# nl2sql/charts/spec.py
ColumnKind = Literal["number", "text", "temporal", "boolean", "other"]   # Wave A's D30 categories

class Encoding(BaseModel):
    field: str                          # a result column
    kind: ColumnKind
    title: str                          # the column, humanised: "total_amount" -> "Total amount"

class ChartData(BaseModel):
    trace_id: str
    result_id: str
    columns: list[str]
    rows: list[list[Any]]               # at most settings.charts.max_points
    truncated: bool                     # the result had more rows than were charted

class ChartSpec(BaseModel):
    kind: Literal["line", "bar", "big_number", "table"]
    title: str                          # the question that ran
    x: Encoding | None = None           # line: temporal; bar: the category
    y: list[Encoding] = []              # one or more measures
    series: Encoding | None = None      # line only: a category split into lines
    value: Encoding | None = None       # big_number
    data: ChartData
    takeaway: str | None = None
    reason: str                         # why this kind: "one date column and one number"

    def to_vega_lite(self) -> dict[str, Any] | None: ...   # None for a table

to_vega_lite() returns a Vega-Lite v5 spec with the rows inline (data.values), mark line, bar or (for a big number) a text mark, temporal or nominal x, quantitative y, several measures folded with a fold transform, and a color encoding only for series. It sets no colour scheme, font or size (D110). A table returns None; the renderer draws a table from data.

Choosing a chart

engine.results.chart(result, result_id=None, *, kind=None, tenant_id=None) -> ChartSpec. result is a QueryResult (its plan is used) or a trace_id (only the columns' kinds are). kind forces a kind the data can draw, else ValueError naming the ones it can. It reads results as engine.results.page does, by trace_id and tenant (B13 unchanged).

Column kinds (D112). The artifact writer records each column's kind when it writes the parquet file, from the Arrow type, into ResultSummary.column_kinds: integer, decimal and float are number; date, time and timestamp are temporal; string is text; bool is boolean; anything else other. When the run had one sub-query, its plan refines the kinds of the result's columns, matched by the select items' aliases:

• a select item that is DATE_TRUNC(unit, x) is temporal (the plan's portable DATE_TRUNC returns an ISO string);
• a select item that is DATE_PART('year', x) and is grouped on is temporal (a year axis);
• a select item that is an aggregate (SUM, COUNT, AVG, MIN, MAX) is a measure; a grouped column is a dimension.

With several sub-queries (an aggregated result) or no plan (a verified query, a trace_id), every number is a measure and everything else a dimension.

The rules (D111), first match wins:

The data is read up to max_points rows; when the result has more, data.truncated is true and a line chart becomes a table rather than drop points silently.

The takeaway. One sentence from a template, filled from the data, with no model call:

Numbers are written with thousands separators and at most three significant digits after the point; a null first or last value gives no change.

# settings.yaml
charts:
  max_points: 1000
  max_categories: 20
  max_series: 5

6. Auto dashboards

A dashboard is a saved list of tiles. A tile is one question, the typed plan (with its sub-query) that answers it, how to chart it and where it sits. A refresh runs every tile through the pipeline as the caller and returns the results with their charts.

The shape

# nl2sql/dashboards/models.py
class TileLayout(BaseModel):            # a 4-column grid
    x: int; y: int; w: int; h: int

class Tile(BaseModel):
    id: str
    title: str                          # short: "Revenue by month"
    question: str                       # the template's question, or the pinned turn's question
    datasource_id: str
    source: Literal["plan", "verified"]
    sub_query: dict[str, Any] | None    # the SubQuery as JSON (source "plan"): intent, expected schema
    plan: dict[str, Any] | None         # the PlanModel as JSON (source "plan")
    verified: VerifiedRef | None        # a verified query's id and bound params (source "verified")
    chart: Literal["auto", "line", "bar", "big_number", "table"] = "auto"
    layout: TileLayout

class DashboardFilter(BaseModel):
    id: str
    label: str
    column: str                         # "schema.table.column" in the dashboard's datasource
    kind: ColumnKind                    # its type category, for checking values
    op: Literal["in", "between"]        # values of a dimension, or a range of a temporal column

class Dashboard(BaseModel):
    id: str
    title: str
    datasource_id: str
    tenant_id: str | None
    user_id: str | None                 # the owner (D118)
    tiles: list[Tile]
    filters: list[DashboardFilter] = []
    created_at: datetime
    updated_at: datetime

class TileRun(BaseModel):
    tile_id: str
    status: Literal["ok", "empty", "stale", "denied", "needs_approval", "stopped", "error"]
    trace_id: str | None
    chart: ChartSpec | None
    explanation: str | None             # Wave B's explanation of the run, for "why"
    filtered: bool                      # every active filter applied to this tile
    error: dict[str, Any] | None        # the QueryResult's first error, client-safe

class SkippedTile(BaseModel):
    title: str; status: str; reason: str

class DashboardRun(BaseModel):
    dashboard_id: str
    at: datetime
    filters: dict[str, list[Any]]
    tiles: list[TileRun]
    skipped: list[SkippedTile] = []     # generate only: template tiles not kept
    usage: QuestionUsage                # summed over the tiles

class DashboardInfo(BaseModel):
    id: str; title: str; datasource_id: str; tiles: int; updated_at: datetime

The facade

Every method takes user_context keyword-only and required (D130).

DashboardNotFound (a LookupError) for a missing id or another owner's.

Column roles

generate needs an index, not a profile (D115). It computes each column's role in nl2sql/dashboards/roles.py:

1. Wave B's role rules (nl2sql/profiler/roles.py) over the datasource's live snapshot: its derived roles (identifier for a key column, date for a temporal type) and its proposed heuristics (enum, money, quantity, date on ISO text), applied directly. They read only the snapshot (types, keys, names, ColumnStatistics), so they cost no query. A heuristic role that nobody has accepted only picks a tile; it is never sent to a model, which keeps Wave B's D38.
2. A role accepted into the glossary for that column wins over the rule's (the merged knowledge the run pins).
3. The dashboard's own reading of them:

Only tables and columns the caller's scope reaches are considered. A datasource that was never indexed raises NotIndexed ("Index it first: nl2sql index --datasource sales").

Choosing the tiles

Each template builds a SubQuery and a PlanModel in code, and carries its own question text, filled from the columns' names humanised (or their glossary descriptions' first clause). The plan then takes the same path as a model's: plan.after hooks, the logical validator, the generator, execute.before hooks (Wave B's policy.cost among them), the adapter, execute.after.

Tiles and refresh

Filters

A filter value becomes a predicate added to a tile's plan before the validator (D117), never text added to a question or SQL:

1. Values are checked first. The validator does not compare a literal with its column's type, so refresh does, against the filter's kind: a number takes ints and floats, temporal an ISO date or timestamp string, boolean a bool, text a string; in takes 1 to 100 values, between exactly two. A failure is a ValueError naming the filter, and nothing runs.
2. Where it applies. For each active filter, find the tile plan's TableRef for the filter's schema.table. A plan that does not read that table runs unfiltered, and its TileRun.filtered is False, so the UI can say so.
3. The predicate. between [a, b] is col >= a AND col <= b; in [v1, v2] is (col = v1 OR col = v2 ...). Both use only the binary operators the plan and generator already have, with the values as plan literals. The predicate is joined to the plan's where with AND.
4. Then the usual path. plan.after, the validator (columns, scope) and the generator, which renders the literals as it renders any literal.

filters maps filter ids to values; an id the dashboard does not have is a ValueError.

Ownership

• Private (D118). A dashboard is seen and changed by its owner only (tenant_id and user_id, as a conversation); another caller gets DashboardNotFound.
• Runs as the caller. generate, refresh and filter_values run with the caller's user_context. A tile reading a table the caller's scope no longer reaches is denied, exactly as the question would be. Nothing in a saved dashboard carries a grant.

The config and events

# nl2sql.yaml
dashboards: dashboards.yaml

# dashboards.yaml (every value shown is the default)
max_tiles: 10

Each tile run is a run, with its own run.started and run.finished, and every hook point applies.

7. Launchpad: the setup workflow

The flow

flowchart TD
  S["nl2sql launch [DIR]"] --> E{"DIR has nl2sql.yaml?"}
  E -->|"no"| NEW["starter engine: no datasources"]
  E -->|"yes"| OLD["the project's engine"]
  NEW --> SC["engine.datasources.scan()"]
  OLD --> SC
  SC --> PICK["pick a candidate, or a type from types()"]
  PICK --> FORM["fill what is missing, from the connection model's schema"]
  FORM --> T["engine.datasources.test(Datasource)"]
  T -->|"failed: stage and fix"| FORM
  T -->|"ok"| KEY["model provider and key: the environment, or asked"]
  KEY --> W["write the project or the datasource file; secrets to .env"]
  W --> RL["load_env_file(DIR), rebuild the engine from the files"]
  RL --> IX["engine.index.build([id])"]
  IX --> PR["Wave B: engine.knowledge.profile([id], draft=...)"]
  PR --> SQ["starter questions filed as example proposals"]
  SQ --> RV["review: engine.proposals accept or reject, or export"]
  RV --> END["next steps: nl2sql run, nl2sql demo --dir DIR"]

Every box but the prompts is a package method, so the Wave D web form calls the same ones.

Finding a database: scan()

# nl2sql_sdk/discovery.py (pydantic and the standard library only)
class Found(BaseModel):
    kind: Literal["url", "file", "port"]
    value: str                           # the URL, the file path, or "127.0.0.1:5432"

@runtime_checkable
class DiscoverableAdapter(Protocol):     # optional: an adapter without it is never found, only chosen
    file_suffixes: ClassVar[tuple[str, ...]]      # (".sqlite", ".sqlite3", ".db")
    default_ports: ClassVar[tuple[int, ...]]      # (5432,)
    url_schemes: ClassVar[tuple[str, ...]]        # ("postgresql", "postgres")
    @classmethod
    def connection_for(cls, found: Found) -> dict[str, Any] | None:
        """The connection fields this finding fills in, or None when it is not this adapter's
        (a .db file that is not SQLite). Never connects."""

# engine.datasources
def scan(self, folder: str | Path = ".", *, url: str | None = None,
         ports: bool = True) -> list[Candidate]: ...

class Candidate(BaseModel):
    type: str                            # the adapter's registry name
    title: str
    found: Found
    connection: dict[str, Any]           # what was filled in; repr masks secret fields
    missing: list[str]                   # required fields the form still needs
    installed: bool
    install_hint: str

The five built-in adapters declare: SQLite (.sqlite, .sqlite3, .db, URL sqlite), DuckDB (.duckdb, URL duckdb), PostgreSQL (5432, postgresql, postgres), MySQL (3306, mysql, mariadb) and SQL Server (1433, mssql, sqlserver).

Writing the config

Three writers, all in the package, all named in the amended invariant (D123):

Secrets (D122). Every field the config marks secret is written the same way: a connection model's secret fields (Phase 2: format: password), an LLM's or an agent's api_key, the embeddings api_key, and a secret provider block's secret fields.

NAME is NL2SQL_ plus the field's path in upper case with every character that is not a letter or digit as _: NL2SQL_SALES_PASSWORD, NL2SQL_LLM_API_KEY, NL2SQL_AGENTS_PLANNER_API_KEY. A name that is already in .env for another field, or that two fields map to, is refused with both paths named, never overwritten.

nl2sql launch

nl2sql launch [DIR] [--provider NAME] [--as ROLE] [--draft] [--yes] [--no-ports], also run as uvx --from nl2sql-engine nl2sql launch (the PyPI distribution is nl2sql-engine; its script is nl2sql). --as (default: the first of policies.admin_roles) is the admin the launch acts as for by and every caller argument (D130); the person at the terminal is the operator, as with every CLI command.

The model key is not checked with a call (D125); the first nl2sql run is the check, and its error names the provider and the variable.

nl2sql demo --dir DIR serves any project folder, but today warns that a folder without a demo stamp was "created by an older version" (cli/demo/stamp.py, outdated_warning). L3 changes it to warn only for a folder that has a stamp from an older version: a folder with no stamp is not a demo folder.

Connect a database over REST

Thin routes in nl2sql-api over the facade, every one admin-only (auth.require_admin), as POST /api/v1/datasource is today. They import only the top-level nl2sql.

No scan, profile or review route (D121). The routes publish no events of their own; the facade methods they call do.

8. Needs from Waves A and B

Wave C uses these as the Wave A and Wave B designs state them now. Where they do not yet say something Wave C needs, it is listed with the PR it would fold into. Where a name differs when built, Wave C follows the built name.

From Wave A

Used as stated: the bus, the catalog and engine.events; approvals_needed, Approval(request, by) and run_query(approvals=); the needs_approval and stopped statuses and stop; typed RunWarnings; max_llm_calls, max_cost_usd and price_for; the D30 categories; datasources.schema(..., user_context) with the caller required (B29); D33's refusal to start without a role.

From Wave B

Used as stated: needs_clarification, QueryResult.clarification: ClarificationRequest, ClarificationChoice(request, option_id), CLARIFICATION_INVALID and CLARIFICATION_STALE; QueryResult.explanation; engine.knowledge.profile(datasource_ids, *, user_context, draft, max_cost_usd), draft=True being the drafting opt-in, under its cost cap; Proposal in nl2sql_sdk.stores; engine.proposals.list, accept(id, *, by, edit), reject, export(path); ReviewError (NOT_WRITABLE, FILE_CHANGED); the cost guard (policy.cost) at execute.before; the store at version 3.

9. MCP over HTTP with auth

The transport (D126)

nl2sql mcp --http [--host 127.0.0.1] [--port 8765] serves Wave A's MCP server over the MCP SDK's streamable HTTP transport at /mcp, in stateless mode with JSON responses: each request stands alone, which is all the tools need, because a follow-up names its conversation_id. stdio stays the default. The server is its own process, built on the same nl2sql/mcp/server.py; mounting it in nl2sql-api is Wave D.

# nl2sql.yaml
mcp: mcp.yaml

# mcp.yaml (every value shown is the default)
http:
  host: 127.0.0.1
  port: 8765
  path: /mcp
  allowed_hosts: ["127.0.0.1:*", "localhost:*"]   # Host headers accepted
  allowed_origins: []                              # browser Origins accepted
auth: api_keys            # api_keys | none (none: loopback only)

Authentication: API keys (D127)

# engine.api_keys (caller required on every method that manages keys, D130)
def create(self, *, user_id: str, roles: list[str], by: UserContext,
           tenant_id: str | None = None, label: str = "") -> tuple[ApiKey, str]: ...
def list(self, *, by: UserContext) -> list[ApiKey]: ...
def revoke(self, key_id: str, *, by: UserContext) -> None: ...
def verify(self, token: str) -> UserContext | None: ...

class ApiKey(BaseModel):              # never the secret
    id: str                           # 8 characters, shown in lists and logs
    label: str
    user_id: str
    tenant_id: str | None
    roles: list[str]
    created_at: datetime
    created_by: str | None
    revoked_at: datetime | None

engine.api_keys is the engine's and not MCP's, so Wave D's nl2sql-api can accept the same keys.

The tools

No chart tool over MCP: a chat client draws its own charts from the rows ask returns.

The hosted demo (D128)

Not exposed in 0.9. The hosted demo answers from recordings with no model key, so an MCP client could ask only the recorded questions, and exposing a live one raises whose key pays and how it is limited. Both wait for OAuth.

10. The registry and the state store after Wave C

DiscoverableAdapter is optional, found with isinstance, so the adapter interface stays at version 2. New on Registry: add_chat_tool and the chat_tool decorator.

StateStore gains, with JSON documents as add_feedback and Wave B's proposals take them (the SDK holds no engine model):

# nl2sql_sdk/stores.py
def put_conversation(self, doc: dict) -> None: ...
def get_conversation(self, conversation_id: str) -> dict | None: ...
def list_conversations(self, tenant_id: str | None, user_id: str | None, limit: int) -> list[dict]: ...
def delete_conversation(self, conversation_id: str) -> bool: ...
def purge_conversations(self, older_than: datetime) -> int: ...
def put_dashboard(self, doc: dict) -> None: ...
def get_dashboard(self, dashboard_id: str) -> dict | None: ...
def list_dashboards(self, tenant_id: str | None, user_id: str | None) -> list[dict]: ...
def delete_dashboard(self, dashboard_id: str) -> bool: ...
def put_api_key(self, row: dict) -> None: ...
def get_api_key(self, key_id: str) -> dict | None: ...
def list_api_keys(self) -> list[dict]: ...

The sqlite store adds migration 6, three tables: conversations (id, tenant, user, updated_at, the document), dashboards (id, tenant, user, the document) and api_keys (id, the hash, revoked_at, the row). A conversation is one document, rewritten per turn: 50 turns without rows is a few hundred kilobytes at most, and it is read whole.

11. Error codes and exceptions

Pipeline ErrorCodes, on a turn's QueryResult or Turn.error, or a tile's QueryResult:

Facade exceptions, each exported from nl2sql: ConversationNotFound, ConversationBusy, DashboardNotFound (all LookupError); NotIndexed, NoTiles, TileNotPinnable, SecretValueInConfig (all ValueError); ToolNotApplicable (raised by a tool, caught by the runner). save's changed file reuses Wave B's FILE_CHANGED code on a ValueError. On nl2sql-api, a LookupError is a 404 and a ValueError a 422.

12. Boundary and invariant changes

13. PR plan

Each PR is written test-first and updates the docs it affects in the same PR. At most four PRs are open at once. C0 lands first and alone; it front-loads every shared shape and stub (D129) so the tracks touch different files. Every PR uses fakes: no test calls a real model. Wave C starts once Waves A and B are released, including Wave A's supplied-plan entry (need A1) and Wave B's L2.

flowchart TD
  D["0. This design"] --> C0["C0. Foundation"]
  C0 --> C1["C1. Conversations core"]
  C0 --> K1["K1. Auto charts"]
  C0 --> L1["L1. scan and DiscoverableAdapter"]
  C0 --> M1["M1. API keys"]
  C1 --> C2["C2. Chat tools runner and report"]
  K1 --> C2
  C0 --> K2["K2. Diagrams"]
  K1 --> D1["D1. Dashboards: roles, generate, refresh"]
  C0 --> L2["L2. Config writers, save, Connect routes"]
  D1 --> D2["D2. Filters and pin"]
  C2 --> D2
  L1 --> L3["L3. nl2sql launch"]
  L2 --> L3
  D1 --> L3
  M1 --> M2["M2. MCP over HTTP"]
  C1 --> M2
  C2 --> REL["Release 0.9"]
  K2 --> REL
  D2 --> REL
  L3 --> REL
  M2 --> REL

The waves, with at most four PRs at once:

Files more than one PR touches, all in wave order or in different waves: api/datasource_api.py (K2's diagram in wave 3 and L2's save in wave 3 fill separate stub methods C0 wrote; L1's scan in wave 2), nl2sql/conversations/api.py (C1, then C2), nl2sql/dashboards/api.py (D1, then D2), nl2sql/mcp/server.py (Wave B's L2, then M2). C0's stubs mean no two PRs add to public_api.py, pyproject.toml, cli/main.py, mkdocs.yml, nl2sql/__init__.py or the boundary test's constants.

14. Bugs Wave C fixes

Bugs are numbered from B100, as decisions are from D100, so they never collide with Wave B's (B31 onwards).

15. Release 0.9

• release-please opens the release PR as usual. Releasing is the owner's act (see Releasing); no agent merges it.
• docs/releases/0.9.0.md is written with the release-notes process. Its breaking-changes table covers: the state store's interface version 4 (an 0.8 file is migrated in place by migration 6), ResultSummary.column_kinds, the new conversation_rewriter agent (a config that pins every agent's model should add it), the MCP ask tool's choice (an option id with conversation_id, replacing {request, option_id}), MCP's build_server(engine, identity), POST /api/v1/datasource answering 422 for a bad body, and init_project writing through write_project (same files).
• Before tagging, the owner runs nl2sql launch once against the test bed's PostgreSQL with a key, asks three follow-ups through engine.conversations in a short script (the chat UI is Wave D), generates a dashboard for Chinook, and connects one MCP client over HTTP with a key. The results go in the release notes.

16. Open questions

None blocking. Every decision above was made overnight and is marked for the owner to confirm. Four are worth a look first, because they set what a user sees or change an earlier wave: D108 (tools are asked for, never chosen by the model), D116 with need A1 (tiles run through a public supplied-plan entry that Wave A's approval path provides), D123 (the package writes a datasource file and .env, amending Wave B's writer invariant and Phase 1's rule) and D128 (no MCP on the hosted demo).