Skip to content

Execution trace enrichment: CoordinationResultWithAttribution wrapper + NodeType tagging on TurnRecord #1123

@Aureliolo

Description

@Aureliolo

Problem

Two related gaps in execution trace data that limit structural credit assignment and post-hoc analysis:

1. No per-agent attribution in coordination results

CoordinationResult is frozen (model_config = ConfigDict(frozen=True)). There is no way to attach per-agent contribution data without modifying the model. When a coordinated task fails, attribution is imprecise -- the failure is attributed to the executing agent, not to the upstream agent whose bad output caused the downstream failure.

2. No node type tagging on TurnRecord

TurnRecord does not record which node types (LLM call, tool invocation, quality check, budget check, stagnation check) executed in that turn. Execution trace analyzers and evaluation pipelines cannot distinguish a turn that failed during a tool call from one that failed during a quality check.

Source: docs/research/acg-formalism-evaluation.md, closes research #848.

Solution

CoordinationResultWithAttribution

Introduce a frozen wrapper (do not modify CoordinationResult):

@dataclass(frozen=True)
class AgentContribution:
    agent_id: str
    subtask_id: str
    outcome: Literal["success", "failure", "partial"]
    failure_category: str | None  # from infer_failure_category()
    token_usage: int

@dataclass(frozen=True)
class CoordinationResultWithAttribution:
    result: CoordinationResult
    contributions: tuple[AgentContribution, ...]

Populate AgentContribution in _post_execution_pipeline using the existing infer_failure_category() keyword heuristic applied per-agent subtask result. Feed into PerformanceTracker.record_task_metric() for trend detection. Surface in GET /tasks/{id} response metadata.

NodeType enum on TurnRecord

class NodeType(Enum):
    LLM_CALL = "llm_call"
    TOOL_INVOCATION = "tool_invocation"
    QUALITY_CHECK = "quality_check"
    BUDGET_CHECK = "budget_check"
    STAGNATION_CHECK = "stagnation_check"

Add node_types: tuple[NodeType, ...] = () to TurnRecord as an optional field with a default of () so existing persisted records remain valid without migration. Populate during turn execution. Consumers must treat an empty tuple as "unknown composition".

Files

  • src/synthorg/engine/coordination/models.py -- add AgentContribution, CoordinationResultWithAttribution
  • src/synthorg/engine/coordination/ -- populate attribution in post-execution pipeline
  • src/synthorg/engine/models.py (or wherever TurnRecord lives) -- add NodeType enum + field
  • src/synthorg/hr/performance/tracker.py -- accept AgentContribution input

Metadata

Metadata

Assignees

No one assigned

    Labels

    prio:mediumShould do, but not blockingscope:medium1-3 days of workspec:agent-systemDESIGN_SPEC Section 3 - Agent Systemtype:featureNew feature implementation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions