fix(genai-openai): capture multimodal content parts as semconv media parts

[FeelKyoun]

Description

Fixes #521.

When a chat message uses the OpenAI content-part array form for multimodal input, e.g.

{"role": "user", "content": [
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": "https://example.com/cat.png"}},
]}

_is_text_part rejects the array (it only accepts str or an iterable of str) and _prepare_input_messages had no fallback branch, so the message was captured with empty parts — the text part was silently dropped along with the non-text parts, even when the user explicitly opted in to content capture.

This PR routes message content through a single _content_to_parts / _convert_content_part path used by both the assistant and user/system/fallback branches of _prepare_input_messages, mapping each OpenAI content part to its semconv message part:

OpenAI part	Message part
str / {"type": "text", ...}	TextPart
image_url (external URL)	UriPart (via image_from_url)
image_url (data: URL)	BlobPart (via image_from_url)
input_audio (base64 or data URL)	audio BlobPart (mime type for wav/mp3, else None)
file with file_id	FilePart
file with inline file_data (base64 or data URL)	document BlobPart (mime type from filename)
any other typed part	GenericPart (provider-specific, no semconv mapping)

A bare string is a single text part and a bare mapping is a single content part. Only sequences (list/tuple) are walked as content-part arrays: the SDK accepts any iterable and materializes it itself, so iterating a generator here would drain the caller's input before the request is sent; such content is left untouched and not captured. Items without a type discriminator, text parts whose text is not a string, and media parts whose payload can't be decoded are skipped.

Content capture never raises into the caller: each part is converted under a guard and skipped on failure, and GenericPart.value is made JSON-safe (pydantic model_dump(mode="json"), json round-trip with default=str) so span export cannot fail on datetime/set/Enum payloads.

Behavior for plain string content, None content, tool messages, and tool calls is unchanged.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Conformance: new MultimodalScenario in tests/conformance/multimodal.py — a chat turn with text + an external image URL + an inline base64 PNG, cassette recorded against the real gpt-4o-mini API, validated with weaver live-check, and asserting text / uri (image) / blob (image) parts land on the input message. Full openai conformance suite (9 scenarios) passes in replay mode with weaver 0.25.1.
• Unit: tests/test_prepare_input_messages_unit.py covers plain string content (regression), image_url URL → UriPart, image_url data URL → BlobPart, input_audio (base64 / data URL / unknown format) → BlobPart, file (file_id / data-URL file_data / plain-base64 file_data / undecodable), unknown typed part → GenericPart with JSON-safe non-aliased value, untyped or non-string-text parts dropped, a part that raises during conversion is skipped without breaking the message, assistant history, None content, string lists, tuples, dict and non-dict mapping content, generator content left unconsumed, and JSON serializability via gen_ai_json_dumps — 24/24 pass.
• Existing openai unit suite passes unchanged (270 passed, 1 skipped).
• pre-commit (ruff / ruff-format) clean on changed files.

Does This PR Require a Core Repo Change?

• No.

Checklist

• Followed the style guidelines of this project
• Unit tests have been added
• Conformance test has been added
• Changelog fragment added (.changelog/522.fixed)

🤖 Generated with Claude Code

https://claude.ai/code/session_01QsXFvtQoFP6DwdBgyEA3xB

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: FeelKyoun / name: feelkyoun (4ca5aaa, a1fa5da, c6030bb, d7e6b9a)

[opentelemetry-pr-dashboard]

Pull request dashboard status

Waiting on reviewers · refreshed 2026-09-02 01:30 UTC

Review the latest changes.

Status above doesn't look right?

• Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
• Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

[Copilot]

Pull request overview

This PR fixes captured gen_ai.input.messages for opentelemetry-instrumentation-genai-openai when OpenAI chat messages use the multimodal “content-part array” form, ensuring text parts are preserved and non-text parts are retained via GenericPart instead of being dropped.

Changes:

• Add _extract_content_parts to map OpenAI content-part arrays into TextPart and GenericPart instances.
• Update _prepare_input_messages to use the new extraction logic for assistant and non-tool roles when content is iterable.
• Add focused unit tests and a changelog fragment for the multimodal capture behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
instrumentation/opentelemetry-instrumentation-genai-openai/src/opentelemetry/instrumentation/genai/openai/utils.py	Adds content-part extraction helpers and wires them into input message capture.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_prepare_input_messages_unit.py	Adds unit tests covering multimodal content-part arrays and JSON serializability.
instrumentation/opentelemetry-instrumentation-genai-openai/.changelog/522.fixed	Documents the bug fix in a towncrier fragment.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[FeelKyoun]

Good catch — addressed in the latest commit. content is now routed through a single _content_to_parts helper that materializes non-string, non-mapping iterables exactly once before branching, so single-pass iterables are no longer partially consumed, and mappings keep the previous _is_text_part behavior (string-keyed mappings captured as str(mapping)) instead of being iterated as content-part arrays. Added unit tests for both cases (generator content and mapping content).

[FeelKyoun]

Correction to my earlier reply here: "materialize once" was the wrong fix. The SDK accepts any iterable for content and materializes it itself, so consuming a generator inside the instrumentation (which runs before wrapped()) drained the caller's input and the SDK sent content: []. As of 7d339a8 only Sequence content is walked; other iterables are left untouched and simply not captured, and the unit test now asserts the generator is still unconsumed afterwards.

[lmolkova]

Thank for the contribution! Please use specialized parts for images, URLs and others.

Also please consider adding conformance test for new modalities - this is where we validate value schema.

[lmolkova]

image_url parts should map to semconv standard media parts (UriPart or BlobPart) rather than GenericPart.

opentelemetry-util-genai provides the image_from_url helper in opentelemetry.util.genai.utils for this exact purpose (it handles external URLs as UriPart and base64 data URLs as BlobPart). Please reuse image_from_url(url) here. GenericPart should only be used for provider-specific parts that have no standard semconv representation.

[FeelKyoun]

Done in d7e6b9a — image_url parts now go through image_from_url(url): an external URL becomes a UriPart and a data: URL becomes a BlobPart.

While at it I mapped the other standard OpenAI parts too, so GenericPart is only used for typed parts that have no semconv representation:

• input_audio → audio BlobPart (mime type derived from format)
• file → FilePart for file_id, or a document BlobPart for inline file_data

[lmolkova]

Stringifying mappings as TextPart(content=str(content)) turns arbitrary dicts into malformed text parts like TextPart(content="{'unexpected': 'shape'}").

Similarly, all(isinstance(item, str) for item in items): return [TextPart(content=str(items))] produces stringified Python list syntax (TextPart(content="['a', 'b']")). Iterables should route through _extract_content_parts so string items become individual TextPart instances.

[FeelKyoun]

Agreed — that was a bad carry-over from the old _is_text_part behaviour. Both stringifications are gone in d7e6b9a:

• a bare mapping is treated as a single content part, so {"type": "text", "text": ...} becomes a TextPart and an untyped dict is dropped instead of becoming a bogus text part;
• string items in a list each become their own TextPart.

Everything now routes through one _convert_content_part and the iterable is consumed exactly once.

[FeelKyoun]

Thanks for the review, @lmolkova — addressed in d7e6b9a:

• Standard media parts instead of GenericPart for image_url (UriPart / BlobPart via image_from_url), input_audio (audio BlobPart) and file (FilePart / document BlobPart); GenericPart is only used for typed parts with no semconv mapping. No more str(dict) / str(list) text parts (details in the threads).
• Conformance test: added MultimodalScenario to the openai conformance suite — a chat turn with text + an external image URL + an inline base64 PNG. The cassette is recorded against the real gpt-4o-mini API and the scenario validates with weaver live-check, asserting that text, uri (image) and blob (image) parts all land on the input message.

Verification: the full openai conformance suite (9 scenarios) passes locally in replay mode with weaver 0.25.1, the unit suite passes (260 passed), and pre-commit is clean.

input_audio and file are covered by unit tests but not by the conformance scenario, since they need an audio-capable model / an uploaded file. Happy to add those if you'd like them exercised there as well.

[FeelKyoun]

/easycla

[FeelKyoun]

Follow-up in 7d339a8 after a self-review pass on utils.py (no change to the part mapping you asked for, just hardening):

• No request mutation. Only Sequence content is walked as a content-part array. A generator (which the SDK accepts and materializes itself) was being drained before wrapped() ran, so the SDK sent empty content — that predates this PR for the first item, but my earlier "materialize once" change made it lose everything. Generator content is now left untouched and not captured (per the "No new side effects" rule in AGENTS.md).
• Never raises into the caller. Each part is converted under a guard and skipped on failure, and GenericPart.value is made JSON-safe (pydantic model_dump(mode="json") + json round-trip with default=str), so neither create_chat_invocation (which runs outside the try in patch.py) nor span export at invocation.stop() can fail on odd payloads.
• file_data is base64 per the SDK docs, not only a data URL: plain base64 is decoded into a document BlobPart (mime type from filename, also used for FilePart.mime_type); data URLs still work.
• input_audio.data given as a data URL is decoded instead of dropped; unmapped audio formats get mime_type=None rather than a synthesized audio/<format>.
• get_property_value accepts any Mapping; non-string text is dropped rather than stringified; the dispatch table is now an if-chain like the sibling instrumentations.

Unit tests: 24/24 (new cases for each of the above); full openai suite 270 passed; conformance suite (9 scenarios incl. MultimodalScenario) passes in replay mode; pre-commit clean.