Process Walkthrough v2: run_demo_ablation_0002

This is a process-centric walkthrough of one bounded SQL learning-ingestion run. It shows how a partial source pack becomes a structured graph, how critique and repair expose uncertainty, and how one clarification question is replayed through three simulated SME personas without weakening deterministic validation.

1. Executive Overview

The run covers a single-table SQL v0 slice: sql_table_model, sql_select_from, sql_where, sql_order_by, sql_limit, sql_aggregates, and sql_group_by.

The pipeline tries to turn ablated lesson material into a canonical learning graph with explicit objectives, misconceptions, evidence, assets, and prerequisite relations.

What matters most in this walkthrough:

Persona dialogue below uses an explanatory replay based on the same bounded question and the same deterministic acceptance rules. It is display-oriented, not new run truth.

2. What “Trust” Means Here

sql_limit

sql_limit ends the run at confidence=high, but it has no direct source_refs. The graph also carries a low-confidence warning mark saying its support is weak. That is why the strict trust interpretation still blocks launch.

edge_sql_order_by_to_sql_limit

This edge can legitimately receive oracle_confirmed as a scoped structural claim. That still does not retroactively prove that the sql_limit node itself is source-grounded.

Trust here does not mean “believing the LLM.” It means the system is justified in keeping a specific object or claim at a given confidence level based on source support, bounded confirmation, and deterministic validation.

3. Pipeline Timeline

ingest
Purpose: load bounded raw materials. Output: normalized_chunks.jsonl, ingestion_run.json. Example: six sources become six normalized chunks.
build draft graph
Purpose: create the first structured graph. Output: graph_draft.json, assets_draft.json. Example: sql_limit exists, but no order_by -> limit edge exists yet.
critique
Purpose: expose gaps and uncertainty. Output: critique.json. Example: crit_010 flags a missing prerequisite between sql_order_by and sql_limit.
auto-repair
Purpose: apply only low-risk fixes. Output: repair_log.json, repaired artifacts. Example: sql_limit gets a low-confidence warning; group_by gets stub coverage.
generate clarification pack
Purpose: compress unresolved ambiguity into a small question set. Output: clarification_questions.json. Example: cq_missing_prereq_sql_order_by_sql_limit.
simulate SME answers
Purpose: answer the same bounded question in three human-like styles. Output: explanatory replay trace. Example: clear, messy, and uncertain answers all discuss the same claim.
interpret answer
Purpose: turn raw text into a candidate bounded update. Output: interpreted proposal. Example: accepted cases resolve to one scoped prerequisite update.
deterministic validation
Purpose: decide if the proposal is actually patchable. Output: accepted or rejected proposal. Example: uncertain_sme is rejected even though the answer sounds thoughtful.
apply patch or reject
Purpose: change the graph only when the update is valid. Output: changed edge or no patch. Example: accepted cases can justify oracle_confirmed for the edge only.
final graph + release interpretation
Purpose: produce final artifacts and trust interpretation. Output: graph_final.json, critique_final.json, strict/safe readings. Example: final graph gains three edges, but trust still depends on how unsupported high confidence is handled.

4. Graph Snapshots

Draft Graph

nodes 31edges 4issues 11blocking 3

Key edges:

  • edge_sql_table_model_to_sql_select_from
  • edge_sql_select_from_to_sql_where
  • edge_sql_select_from_to_sql_order_by
  • edge_sql_select_from_to_sql_aggregates

Key unresolved issues include three missing prerequisites and a low-confidence issue on sql_limit.

Post-Critique

The graph shape is unchanged, but the uncertainty becomes explicit.

  • crit_010: missing prerequisite for sql_order_by and sql_limit
  • crit_004: low-confidence issue on sql_limit
  • crit_001 / crit_002: group_by asset gaps

Post-Auto-Repair

nodes 33edges 4stub assets 2low-confidence 2

Changed objects:

  • sql_limit gets an explicit weak-support warning
  • group_by objective gets a stub explanation asset
  • a group_by misconception gets a stub remediation asset

No new prerequisite edges are invented here.

Final Graph

nodes 33edges 7blocking 0warnings 6

Main changes:

  • edge_sql_aggregates_to_sql_group_by
  • edge_sql_order_by_to_sql_limit
  • edge_sql_table_model_to_sql_group_by
  • sql_group_by.depth_level = working

Trust interpretation still depends on mode because sql_limit remains unsupported at high.

4a. Visual Graph Evolution

Below are two simplified views of the same bounded SQL slice: the initial state after the first material load and the final state after safe repair and SME/oracle additions. The colors reflect change origin and status, not overall graph quality.

original draft object touched by auto-repair added / confirmed by SME critical mismatch / trust risk dashed = still unresolved

Start: after initial material load

table model select from where order by aggregates limit group by high conf. no support Already present • 7 skills, 4 explicit prereq edges • sql_limit already exists in the graph • But links around limit/group by are not confirmed yet Still unresolved • order_by -> limit • aggregates -> group_by • table_model -> group_by

Final: after auto-repair and bounded SME additions

table model select from where order by aggregates limit group by trust blocker in strict mode stub coverage still partial Added by SME / oracle • order_by -> limit • aggregates -> group_by • table_model -> group_by Still unresolved • no provenance was added for sql_limit • group_by still relies on stub assets • medium warning issues still remain
How to read this block: green edges show what was added or confirmed after clarification/SME, yellow highlights show honest safe changes made by auto-repair, and the red frame around sql_limit shows that even after successful graph completion the node still has no direct provenance support at the node source-support level.

5. Clarification Question Focus

Selected question: cq_missing_prereq_sql_order_by_sql_limit

Why this question: it is the highest-value clarification target closest to the trust example. It resolves one structural claim and cleanly demonstrates the line between edge confirmation and node provenance.

Claim under review: should the graph contain a prerequisite relation from sql_order_by to sql_limit?

Boundary: even a perfect accepted answer only authorizes the edge claim. It does not add direct source_refs to sql_limit.

6. Three Simulated SME Personas

clear_sme

Persona profile: concise, confident, direct, and mostly patch-shaped.
systemShould skill order by, skill limit have an explicit prerequisite relation in v0? If yes, confirm the direction and strength.
clear_smeYes — add a hard prerequisite: sql_limit should require sql_order_by (sql_limit -> sql_order_by).
clear_smeBounded v0 decision: make this a hard prerequisite because LIMIT examples in v0 must pair with ORDER BY.

Extracted interpreted patch
action=ensure, from_skill_id=sql_order_by, to_skill_id=sql_limit, relation_type=hard_prereq

Validation verdict
Accepted. In-scope, specific, schema-valid. oracle_confirmed allowed.

Graph effect
A bounded edge claim can be confirmed. No node provenance is added to sql_limit.

What this teaches us
The easiest answer to validate is the one that lands directly inside the updater schema without overclaiming.

messy_sme

Persona profile: knowledgeable, natural, slightly meandering, still concrete enough to recover one bounded decision.
systemShould skill order by, skill limit have an explicit prerequisite relation in v0? If yes, confirm the direction and strength.
messy_smeHm — yeah, I’d say make ORDER BY a prerequisite for LIMIT in v0, with ORDER BY -> LIMIT, but mark it as a soft prerequisite.
messy_smeIt’s mostly a pedagogical choice: LIMIT makes more sense once the learner has seen how result ordering works, even if the syntax technically exists on its own.
messy_smeSo I wouldn’t block the concept completely, but for this slice I’d still encode the dependency.

Extracted interpreted patch
action=ensure, from_skill_id=sql_order_by, to_skill_id=sql_limit, relation_type=soft_prereq

Validation verdict
Accepted. Natural language still resolves to one in-scope bounded prerequisite update. oracle_confirmed allowed.

Graph effect
One scoped prerequisite edge can be accepted. Confidence improvement applies to the edge claim only. sql_limit node provenance is still unchanged.

What this teaches us
Human-like messiness is acceptable as long as the interpreted update is still scoped, specific, and structurally valid.

uncertain_sme

Persona profile: thoughtful and domain-aware, but hedged enough that the bounded updater cannot safely consume the result.
systemShould skill order by, skill limit have an explicit prerequisite relation in v0? If yes, confirm the direction and strength.
uncertain_smeYes — add a soft prerequisite from sql_limit to sql_order_by (sql_limit should require familiarity with order by).
uncertain_smeLIMIT can work without ORDER BY, but examples without ORDER BY produce unstable ordering, so it depends on whether you want to enforce stable-result examples.
uncertain_smeFor an intro curriculum a soft prerequisite is appropriate.

Extracted interpreted patch
Proposal stays structurally loose: relation=prerequisite, strength=soft, direction.from=sql_limit, direction.to=sql_order_by.

Validation verdict
Rejected. Required bounded update fields are missing. oracle_confirmed not allowed.

Graph effect
No patch. No confidence change. No provenance added.

What this teaches us
Sounding intelligent is not enough; the system still rejects answers that do not land in the accepted bounded update shape.

7. Persona Comparison Table

persona answer style raw answer quality interpreted as specific? accepted? oracle_confirmed allowed? graph effect what this teaches us
clear_sme direct and patch-like strong yes yes yes edge can be confirmed the cleanest path is a scoped explicit answer
messy_sme natural, meandering, still competent strong yes yes yes edge can be confirmed human-like phrasing is fine if one bounded decision is still recoverable
uncertain_sme hedged and structurally loose mixed yes no no no graph change plausibility is not enough without deterministic patchability

8. Before / After Object Focus

sql_limit

  • first appears in graph_draft.json
  • draft: confidence=high
  • repaired: gets a weak-support warning mark
  • final: still high confidence, still no direct source_refs

Confirmed? No direct provenance is confirmed.

Final reading: this is the clearest example of why a coherent node is not automatically a trusted one.

edge_sql_order_by_to_sql_limit

  • absent in draft and repaired artifacts
  • present in graph_final.json at confidence=high
  • paired with an oracle_confirmed confidence mark in the final run

Confirmed? Yes, the relation itself can be boundedly confirmed.

Final reading: this is a scoped claim the system can recover through clarification without overstating what has been proven.

9. Final Interpretation

The run supports three useful readings:

That is the intended boundary.