ChatModels.ChatOpenAI doesn't work with LMStudio when response_format is set to text

[vkryukov]

Hi Mark,

as far as I understand, using ChatModels.ChatOpenAI with a custom endpoint is a recommended way to use LangChain with an LLM that provides OpenAI-compatible API, and indeed, it works well with XAI, DeepSeek, and Groq. However, it doesn't work with LM Studio, failing with the following error:

%{"error" => "'response_format.type' must be 'json_schema'"}

The issue seems to be response_format, which is always set by ChatModels.ChatOpenAI to either type: text, type: json_object, or type: json_schema. That seems to be in accordance with OpenAI Assistant API, and doesn't seem to bother other LLMs, however, LM Studio clearly barks on type: text.

Note that the issue is indeed with these extra parameter in the body of the request; I can e.g. send the following request and receive a proper response:

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "model": "deepseek-coder-v2-lite-instruct-mlx",
   "messages": [
       { "role": "system", "content": "Always answer in rhymes. Today is Thursday" },
       { "role": "user", "content": "What day is it today?" }
   ],
  "temperature": 0.7,
  "max_tokens": -1,
  "stream": false
}'

I'm not sure what the correct course of action is for LangChain. I can see three possible options:

1. Be more compatible with incomplete OpenAI-compatible implementation, and don't set response_format when type is text.

2. Consider this an upstream problem with LM Studio, and file a ticket there.

3. Do nothing; since it works for other providers, we shouldn't cover a (somewhat) minor edge case.

Thoughts?

[vkryukov]

Related LM Studio issue: lmstudio-ai/lmstudio-bug-tracker#189

[brainlid]

Thanks @vkryukov! I'm not sure what to do with this exactly. 🤔

A challenge is that these systems are all moving pieces. For instance, OpenAI's new o1 reasoning models break with the past and instead of a Message using a role: :system, it requires a role: :developer. But only on o1+ models. I addressed this here (#234). What makes this even harder is my OpenAI account level doesn't give me API access to o1 models! It's like they want to make it hard on purpose! 🙂

My point is, OpenAI can move fast and lots of external software projects adopted their API for portability. However, they don't always keep up.

In this case, I could see an argument for not explicitly setting the response_format to text, especially if it's the default with OpenAI when unspecified.

I'm open to a PR for that if it includes tests! 🙂

[vkryukov]

Hi Mark,

I agree that it's a tricky problem to solve (and yes, I saw the OpenAI-related discussion - their decision is pretty developer unfriendly if you ask me). Let us try to not explicitly set the response_format, I will work on the PR once I have a bit more time.