Bug in create_react_agent with response_format??

[Courvoisier13]

Hi,
I have been getting an issue with create_react_agent with response_format enabled. The issue occurs when I use create_react_agent with no tools and a prompt and a response_format:

        """Worker to route to next. If no workers needed, route to FINISH."""
        next: Literal["option 1", "option 2", "option 3", "option 4"]
agent = create_react_agent(
        llm,
        tools=[],
        prompt=system_prompt,
        response_format=Router
    )
response = agent.invoke({"messages": messages})

The prompt asks the agent to pick one of ENUMs in next.
I am getting a response with a message say equal to "option 1", by virtue of the prompt, it restricts the answer to one of the ENUMs in the Router.
However, in the response, I get a structured_response equal to "option 4". I thought it was weird that the llm generate an exact answer equal to one of the ENUMs but the parsed one is something else.

I debugged and found that two calls were happening in chat_agent_executor.py in langgraph/prebuilt. The first to call_model which gives the right answer and the other to generate_structured_response which gives the structured_response field. After following the trail of generate_structured_response I reach _create_chat_result in langchain_openai/chat_models/base.py. However, There is no prompt anymore. The llm is simply getting the messages, with no prompt, and it's just picking that makes no sense.

Adding the system prompt to the front of the messages solves the issue. But somewhere along the way the system prompt given to the llm when using response_format disappear from the final llm call.

This seems like a bug, doesn't it?

[vbarda]

it's not a bug, but i can see how this can be confusing - you can pass dedicated prompt for the structured output LLM call by passing a tuple (prompt, Schema) to response_format -- see API reference https://langchain-ai.github.io/langgraph/reference/prebuilt/#langgraph.prebuilt.chat_agent_executor.create_react_agent

[Courvoisier13]

If I already have a prompt set in create_react_agent why would I need to add another one in the response_format? This is not explained at all in the doc.
it says "The prompt will be used together with the model that is being used to generate the structured response." but shouldn't the prompt argument be part of the model config already? It makes it sound like if I added a prompt in the response_format I would just be duplicating.

[vbarda]

they are separate because the LLM calls in the react loop are different from the LLM call for structured output - it's a separate LLM call after the react loop is done

for example, you might want to provide specific instructions for structured output that make no sense for the main system prompt for the agent

[Courvoisier13]

Thanks, it would be great to update the documentary to make it clear that the main prompt is not used at all in the structured output. because currently this is not clear at all.