BigQueryAgentAnalyticsPlugin:A2A Remote Agent Events Have No Trace/Session Correlation in BigQuery

[evekhm]

GitHub Issue: A2A Remote Agent Events Have No Trace/Session Correlation in BigQuery

🔴 Required Information

Describe the Bug:

When a supervisor agent calls a RemoteA2aAgent via transfer_to_agent, the remote agent's BigQueryAgentAnalyticsPlugin creates an entirely independent session_id and trace_id. The supervisor's BQ plugin logs AGENT_STARTING and AGENT_COMPLETED for the remote agent with content: null — neither the remote agent's response nor a cross-reference to its session is captured.

This makes it impossible to reconstruct the full end-to-end execution trace within BigQuery. A user question that gets routed to a remote A2A agent appears to have no response in BQ analytics, even though the remote agent successfully processed and responded.

Steps to Reproduce:

1. Create a supervisor agent with a RemoteA2aAgent sub-agent:

from google.adk.agents import Agent
from google.adk.agents.remote_a2a_agent import RemoteA2aAgent
from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin, BigQueryLoggerConfig,
)

# Remote agent on Cloud Run (also has BQ plugin logging to the SAME table)
remote_agent = RemoteA2aAgent(
    name="pto_agent",
    description="Calculates PTO",
    agent_card="https://my-service.run.app/a2a/pto_agent/.well-known/agent-card.json",
)

supervisor = Agent(
    name="knowledge_supervisor",
    model="gemini-2.5-pro",
    instruction="Route PTO questions to pto_agent.",
    sub_agents=[remote_agent],
)

# Both supervisor AND remote agent use BQ plugin pointing to the SAME table
bq_plugin = BigQueryAgentAnalyticsPlugin(
    project_id="my-project", dataset_id="my-dataset",
    table_id="agent_events",
    config=BigQueryLoggerConfig(enabled=True, batch_size=1),
)

2. Send a query that gets routed to the remote agent
3. Query BQ to find the supervisor's session trace:

SELECT event_type, agent, session_id, trace_id,
  SUBSTR(TO_JSON_STRING(content), 0, 300) AS content
FROM `project.dataset.agent_events`
WHERE session_id = '<supervisor_session_id>'
ORDER BY timestamp;

4. Observe that AGENT_COMPLETED for the remote agent has content: null — no response captured
5. Search for the remote agent's own events — they exist but under a completely different session_id and trace_id with no cross-reference

Expected Behavior:

When a supervisor calls a RemoteA2aAgent, at least one of the following should be true:

1. The remote agent's response should be captured in the supervisor's AGENT_COMPLETED event content, OR
2. The remote agent's session_id/trace_id should be stored as a reference in the supervisor's trace (e.g., remote_session_id, remote_trace_id fields in the AGENT_COMPLETED or TOOL_COMPLETED event), OR
3. The supervisor's session_id/trace_id should be propagated to the remote agent (via A2A protocol headers or context) so the remote agent's BQ plugin logs events with the parent's identifiers

Any of these would allow users to reconstruct the complete execution trace from a single BQ query.

Observed Behavior:

The supervisor's trace and the remote agent's trace are completely disconnected in BQ. There is no shared key — no shared session_id, no shared trace_id, no parent reference.

Concrete example from a production trace:

SUPERVISOR SESSION: 3eae7ca9-0f3a-4409-b466-1ea1fdadbf8c
SUPERVISOR TRACE:   967791bf9cde65934134cc0969eda854

timestamp            event_type          agent                 content
──────────────────────────────────────────────────────────────────────────────────
20:54:07             USER_MSG_RECEIVED   knowledge_supervisor  "Can I take a two-week vacation..."
20:54:07             INVOCATION_STARTING knowledge_supervisor  null
20:54:07             AGENT_STARTING      knowledge_supervisor  (system instruction)
20:54:07             LLM_REQUEST         knowledge_supervisor  (prompt)
20:54:10             LLM_RESPONSE        knowledge_supervisor  "call: transfer_to_agent"
20:54:10             TOOL_STARTING       knowledge_supervisor  {tool: "transfer_to_agent", args: {agent_name: "pto_agent"}}
20:54:10             TOOL_COMPLETED      knowledge_supervisor  {result: null}  ← NULL result
20:54:10             AGENT_STARTING      pto_agent             ""              ← empty content
20:54:45             AGENT_COMPLETED     pto_agent             null            ← NULL! Response lost
20:54:45             AGENT_COMPLETED     knowledge_supervisor  null
20:54:45             INVOCATION_COMPL    knowledge_supervisor  null

The pto_agent successfully processed the request and returned a response, but:
- TOOL_COMPLETED result is null
- AGENT_COMPLETED content is null
- No reference to the remote agent's session


PTO_AGENT'S OWN SESSION: 13ba2fab-5c98-436e-b357-c45cdff6a1ef  ← DIFFERENT!
PTO_AGENT'S OWN TRACE:   419128b33598bb869a390ed3046a9a0f      ← DIFFERENT!

timestamp            event_type          agent       content
──────────────────────────────────────────────────────────────────────────────────
20:54:38             USER_MSG_RECEIVED   pto_agent   "Can I take a two-week vacation... | For context: ..."
20:54:38             INVOCATION_STARTING pto_agent   null
20:54:38             AGENT_STARTING      pto_agent   (system instruction)
20:54:38             LLM_REQUEST         pto_agent   (prompt with user question)
20:54:42             LLM_RESPONSE        pto_agent   "call: calculate_pto_details"
20:54:42             TOOL_STARTING       pto_agent   {tool: "calculate_pto_details"}
20:54:42             TOOL_COMPLETED      pto_agent   {result: "23.7 PTO days remaining..."}
20:54:42             LLM_REQUEST         pto_agent   (second turn with tool result)
20:54:45             LLM_RESPONSE        pto_agent   "text: 'Alright, drumroll please!...'"  ← ACTUAL RESPONSE
20:54:45             AGENT_COMPLETED     pto_agent   null
20:54:45             INVOCATION_COMPL    pto_agent   null

The response "Alright, drumroll please!..." exists in session 13ba2fab but there is no way to discover it from the supervisor's session 3eae7ca9 using BQ alone.

Consequences:

1. Session analytics show false nulls. Any query that joins user questions to agent responses via session_id reports no response for every A2A-routed session. In our dataset, ~30% of sessions are routed to the remote agent, so ~30% of sessions appear to have no response.

2. End-to-end latency is unmeasurable. The supervisor's INVOCATION_COMPLETED has no response content, so you can't measure time-to-answer for A2A sessions. The remote agent's latency is in a separate session with no link back.

3. Quality evaluation is broken. Any evaluation that checks response quality (usefulness, grounding, etc.) sees null responses for A2A sessions and either skips them or misclassifies them.

4. Execution tree reconstruction is incomplete. Building a supervisor → remote_agent → tool → LLM tree requires spanning two disconnected sessions. Users have to resort to heuristic time-window joins, which are fragile and can produce false matches under concurrent load.

5. Dashboards and reports are degraded. Any dashboard showing "response rate", "answer quality", or "agent utilization" undercounts A2A agent contributions.

Root Cause Analysis:

The issue occurs at two levels:

Level 1: Supervisor's BQ plugin doesn't capture the A2A response.

When RemoteA2aAgent completes, the supervisor's after_agent_callback receives the event but logs AGENT_COMPLETED with content: null. The actual response that was returned via the A2A protocol is not captured in the event data.

Similarly, TOOL_COMPLETED for transfer_to_agent logs result: null — the tool result doesn't include the remote agent's response.

Level 2: No trace context propagation across A2A boundary.

The A2A protocol call from the supervisor to the remote agent does not propagate:

• session_id — the remote agent creates a new session
• trace_id — the remote agent starts a new trace (even though OTel W3C trace context could be propagated via HTTP headers)
• Any parent reference — there's no parent_session_id or parent_trace_id field

This means even if both agents log to the same BQ table, their events are in completely isolated namespaces.

Proposed Fix:

Option A (Minimal — capture response in supervisor's trace):

In the supervisor's after_agent_callback, capture the remote agent's response text in the AGENT_COMPLETED event content. This would at least make the response visible in the supervisor's session trace.

Option B (Ideal — propagate trace context):

Propagate the supervisor's trace_id (and optionally session_id) to the remote agent via A2A protocol headers (e.g., W3C traceparent). The remote agent's BQ plugin would then log events with the parent's trace_id, allowing JOIN on trace_id.

Option C (Pragmatic — add cross-reference):

When the remote agent's session is created via A2A, store a parent_trace_id or parent_session_id in the BQ events. This wouldn't merge the traces but would allow them to be linked.

Workaround (SQL-level):

Until a fix is available, we use a time-window heuristic join — the supervisor logs AGENT_STARTING and AGENT_COMPLETED timestamps for the remote agent, and we match the remote agent's own session by finding events with the same agent name within that time window:

-- Find remote agent's time window from supervisor's trace
WITH remote_agent_windows AS (
  SELECT s.session_id,
         s.agent AS remote_agent,
         s.timestamp AS agent_start,
         c.timestamp AS agent_end
  FROM agent_events s
  JOIN agent_events c
    ON s.session_id = c.session_id
    AND s.agent = c.agent
    AND s.span_id = c.span_id
  WHERE s.event_type = 'AGENT_STARTING'
    AND c.event_type = 'AGENT_COMPLETED'
    AND s.agent != 'knowledge_supervisor'
),
-- Match remote agent's own LLM response within that window
remote_responses AS (
  SELECT r.agent,
         JSON_VALUE(r.content, '$.response') AS response,
         r.timestamp AS resp_time,
         w.session_id AS parent_session_id
  FROM agent_events r
  JOIN remote_agent_windows w
    ON r.agent = w.remote_agent
    AND r.timestamp BETWEEN w.agent_start AND w.agent_end
    AND r.session_id != w.session_id  -- different session = remote agent's own
  WHERE r.event_type = 'LLM_RESPONSE'
    AND JSON_VALUE(r.content, '$.response') LIKE 'text:%'
)
SELECT * FROM remote_responses;

This workaround is fragile under concurrent load (multiple requests to the same remote agent at the same time could produce false matches).

Environment Details:

• ADK Library Version: google-adk 1.28.1
• Desktop OS: Linux (Ubuntu 24.04)
• Python Version: 3.11

Model Information:

• Which model is being used: gemini-2.5-pro (supervisor), gemini-2.5-flash (remote agent)

---

🟡 Optional Information

Regression:

Unknown — this may have existed since A2A + BQ plugin support was introduced.

How often has this issue occurred?:

Always (100%) — every A2A call via RemoteA2aAgent produces disconnected sessions with no cross-reference in the BQ table. The remote agent's response is never captured in the supervisor's trace.

Related Issues:

This is related to the phantom parent_span_id issue (internal OTel spans not logged to BQ). Both issues stem from incomplete trace context management in the BQ plugin when crossing agent boundaries.

[zeel2104]

Is it open for contribution?