Rename gen_ai.openai.request.response_format to gen_ai.request.response_format

[lmolkova]

Structured outputs and response format is used by multiple GenAI systems, so generalizing it.
Also fixing a tiny bug where we have different id and value for completion token types and other minor nits.

See Cohere, Azure AI Inference, Vertex AI docs

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• schema-next.yaml updated with changes to existing conventions.

[aabmass]

Vertex AI/Gemini also supports response schema. I'll take a look and see if it works with this PR

[karthikscale3]

LGTM

[aabmass]

I know it wasn't changed in this PR, but what is the difference between these? Neither one is an actual "format"

[lmolkova]

that's what OpenAI, Azure AI and Cohere do:

• OpenAI /Azure AI Inference
  • {"type": "json_object"} - json without schema
  • {"type": "json_schema", "json_schema": {...}} - json with schema
• Cohere
  • {"type": "json_object" } - json without schema
  • {"type": "json_object", "json_schema": {...}} - json with schema
• Vertex AI (correct me if I'm wrong, reading this)
  • responseMimeType: application/json | text/plain | text/x.enum
  • responseSchema

I.e. if we apply existing things to vertex, it would be

• check if responseMimeType is json
  • if responseSchema is set, then gen_ai.request.response_format=json_schema
  • otherwise gen_ai.request.response_format=json_object
• if responseMimeType is text, gen_ai.request.response_format=text
• in other cases, probably gen_ai.request.response_format={responseMimeType}

I guess the question is whether we should continue doing this?

[aabmass]

I think you're reading of Vertex API is correct. Looking ahead, there is a separate API for generating images which also supports more mime types https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/imagen-api#output-options. I'm guessing gen_ai.request.response_format attribute would be shared for multimodal as well–what do you think of changing this attribute to a mime type?

Regarding json schema, I opened #1760 to support capturing the schema if available.

[lmolkova]

I'm guessing gen_ai.request.response_format attribute would be shared for multimodal as well–what do you think of changing this attribute to a mime type?

great point!

Just did a basic research on types:

• OpenAI
  • Audio response format types: mp3, opus, aac, flac, wav, and pcm.
  • Realtime response format types: pcm16, g711_ulaw, or g711_alaw - this one is called output_audio_format
  • Image response format types: url or b64_json
• Vertex AI uses mime-types for images too

I think mime-type won't work in some cases (e.g. image URL), but also it requires to map everything to a mime type. It's not straightforward - e.g. b64_json DALL-E produces seems to be webp but it's not documented (!?) and internet suggests to decode and look at the image header - not something instrumentation should do.

Seeing output_audio_format makes me think that there is a future where you could specify multiple formats in one request (generate video and audio, etc) and we might need different attributes to capture text_format, audio_format, etc.

---

I think I see two options:

1. Use response_format as a union of all possible formats across modalities: (text | json | mp3 | url | .. )
   • we can define a few well-known types (e.g. json would be used for all json things - json_object, json_schema, application/json, etc)
   • for everything else it would match constant used by the specific system, Vertex can use audio/x-alaw-basic and openai would do g711_alaw, we don't really try to unify all
2. Break it down into format-per-modality
   • gen_ai.request.text.response.format = plain | json | bson | xml | python
   • gen_ai.request.text.response.schema = {schema type name} - this is only for text and only for structured output
   • gen_ai.request.image.response_format = url | png | webp | b64_json - we can do some normalization for a few things and record the rest as is
   • gen_ai.request.image.response.aspect_ratio = 4:3 - there are other properties that would be worth recording
   • ...

Either way we could/should add something like
gen_ai.request.response.type = text | image | video | audio | ... - this is the modality

(all attribute names need polishing, but I hope you get the idea)

I hope we can also make it symmetrical with request format and the actual response output type.
Let me do a bit more research, but I'm leaning towards option 2 since it provides typed way to record modality-specific options

[aabmass]

I hope we can also make it symmetrical with request format and the actual response output type. Let me do a bit more research, but I'm leaning towards option 2 since it provides typed way to record modality-specific options

Option 2 sounds good to me on cursory read through. It sounds like a bigger follow up task, so this PR LGTM if you want to address the rest later.

[lmolkova]

I introduced gen_ai.output.json.schema.name and gen_ai.output.type following option 2, let's discuss on the GenAI call tomorrow

[aabmass]

Did you push the updated code yet?

[lmolkova]

apparently no, did it now, thanks!

[trentm]

This is inaccurate now.

[trentm]

This is inaccurate now.
As well, the PR title should change.