Drift Management & Edge-Case Triage
Industrial agent evaluation requires more than static datasets; it needs the ability to learn from real-world production “drift” and systematically classify why agents fail.
🔄 Behavioral Drift Importers
Section titled “🔄 Behavioral Drift Importers”The Drift Importer (import-drift) allows you to convert production interaction logs into reusable evaluation scenarios. This enables the creation of a “regression suite” from actual edge cases encountered in the field.
agentv import-drift --input path/to/trace.json --industry telecomTrace Format
Section titled “Trace Format”The engine expects standard role-based interaction objects:
[ {"role": "user", "content": "I need help with my bill."}, {"role": "assistant", "content": "I can help with that. What is your account number?"}]Automated Scenario Generation
Section titled “Automated Scenario Generation”Upon import, the engine generates a new AES v1.4 scenario file in industries/[industry]/scenarios/drift-[hash].json. The original conversation is stored as ground_truth_history for N-shot comparison.
🔍 Edge-Case Triage Library
Section titled “🔍 Edge-Case Triage Library”The Triage Engine automatically inspects failed tasks and applies diagnostic tags based on the Failure Taxonomy.
Industrial Triage Tags
Section titled “Industrial Triage Tags”| Tag | Description |
|---|---|
CONNECTION_ERROR | Communication failure with the agent or LLM provider (e.g., 500 reset). |
POLICY_VIOLATION | The agent attempted an action forbidden by the Secure Sandbox. |
TOOL_ERROR | A World Shim returned an error status during tool execution. |
STALL | The agent hit the turn limit without reaching a terminal final_answer. |
How It Works
Section titled “How It Works”After each run, the engine executes a heuristic pass over the trajectory.
Task: refund_processing [FAILURE [CONNECTION_ERROR]] FAILED Metric: generic_accuracy | Score: 0.00 | Threshold: 0.80🧪 Forensic Diagnostics (explain)
Section titled “🧪 Forensic Diagnostics (explain)”While triage provides categorical tags, the explain command performs a deep forensic analysis of the execution trace to identify root causes.
agentv explain --run-id <id>Forensic Features
Section titled “Forensic Features”- Tiered Confidence Scoring: Distinguishes between explicit violations (100%), induced system errors (85%), and heuristic fallbacks (50%).
- Remediation Advice: Provides targeted prompts for refinement (e.g., “Refine sandbox policy for
read_fileto allow restricted access”). - Pinpoint Divergence: Identifies the exact turn index where the agent’s logic diverged from the Trust Protocol.