Skip to main content

Recipes

Fabrication-resistant Python agent

context-hub (current API docs) → pydantic-ai (schema-validated outputs) → e2b (sandboxed execution) → pytest post-check. Fails fast on hallucinated symbols before any change leaves the sandbox.

case studied python

Description

A four-stage Python-agent composition whose ordering is designed to push every failure class the `fabrication` and `logic_error` FailureModes cover into a different observable signal. Stage 1 (pre-generation): context-hub retrieves relevant in-repo and standard-library context so the model generates against real symbols. Stage 2 (in-generation): pydantic-ai enforces Pydantic schemas on every tool call and return value, rejecting malformed outputs at parse time. Stage 3 (post-generation): e2b runs the generated code inside a fresh sandboxed container. `ImportError` / `AttributeError` on a fabricated symbol surfaces here, not in the user's repo. Stage 4 (post-execution): a pytest run over the generated module — the test suite is the final correctness gate and also catches logic errors that parsed and imported successfully.

Stack

Ordered composition. Each stage consumes the output of the one before it.

  1. Stage 1
    Context Hub

    Up-to-date, curated API docs for coding agents.

  2. Stage 2
    PydanticAI

    Structured-output agent framework with Pydantic-typed tool and return schemas.

  3. Stage 3
    E2B

    Secure sandboxes for AI-generated code execution.

  4. Stage 4
    pytest

    Python test framework — discipline gate for "done means tests pass."

Targets failure modes

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Logic error — Correct API use, incorrect logic.

Notes

Changed on 2026-11-21: pytest, which the text has always named as stage 4, was added to `stack` (it has been a KG node since 2026-05-01, and it is the stack tool the KG tags for `logic_error`). A sentence that called this the reference composition for `graduated_hitl` workflows was removed; the KG labels these tools `fully_autonomous`. Of the stack tools the KG tags context-hub for fabrication and pytest for logic_error; pydantic-ai and e2b are tagged for other modes and are here for the mechanism the text describes. The basis given for `case_studied` (the composition is said to appear in context-hub and PydanticAI material) was not re-checked and no write-up is cited. The composition has not been benchmarked.

View raw YAML on GitHub →