[Bug]: Flux image generation tool compatibility with Claude 3.7 on AWS Bedrock

[ogipogi]

What happened?

Bug Report: Image Generation Fails with Claude 3.7 on AWS Bedrock

Issue Description

When using the Flux image generation tool with Claude 3.7 on AWS Bedrock, generated images fail to display in the chat interface. The following error is returned in the API response:

The format of the value at messages.2.content.0.toolResult.content.0.json is invalid.

Environment

• LibreChat version: 0.7.7
• LLM: Claude 3.7 Sonnet
• Provider: AWS Bedrock
• Tool: Flux image generation
• Browser: Chrome
• Operating System: OSX

Expected Behavior

The generated image should be displayed in the chat interface after successful generation.

Actual Behavior

No image is displayed. Looking at the developer console or logs reveals the format validation error message.

Root Cause Analysis

The issue occurs because the response format returned by the FluxAPI tool doesn't match the schema expected by Claude 3.7 on AWS Bedrock. The current implementation in FluxAPI.js uses a nested array structure that triggers format validation errors with the AWS Bedrock implementation.

Specifically, the current code returns:

[response, { content }]  // response is itself an array

which creates a nested array structure that fails validation.

Solution

This issue has been fixed in PR #6253 by updating the response format to use a cleaner structure:

[
  { type: ContentTypes.TEXT, text: displayMessage },
  { content: [{ type: ContentTypes.IMAGE_URL, image_url: { url: "..." } }] }
]

This flat structure with properly typed content objects satisfies the validation requirements of AWS Bedrock's Claude 3.7 implementation.

Additional Context

This issue appears to be specific to Claude 3.7 on AWS Bedrock, which has stricter validation requirements for tool response formats compared to some other implementations.

Version Information

35da0859ce14bf0fbdf63507ced93a2cf61301fd

Steps to Reproduce

Steps to Reproduce

1. Connect to Claude 3.7 via AWS Bedrock in LibreChat
2. Use the Flux tool to generate an image with a prompt like: "Generate an image of a mountain landscape"
3. Observe that instead of displaying an image, an error occurs and no image is shown

What browsers are you seeing the problem on?

Chrome

Relevant log output

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

I will address this soon. Also commented on the PR: #6253 (comment)

This needs to be solved differently outside of the tool call. Your solution takes away the vision capability for the image response for OpenAI/Google/Anthropic compatible API's.

• First, what formats Bedrock accepts vs. others. The current tool code works with OpenAI/Anthropic/Google.
• Second, if the model has vision capabilities, whether to exclude the image in the result.

references:
https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview#example-of-tool-result-with-images

One additional way to handle it is for the user to determine if the agent should receive image results from tools, into the context, as that lets the user decide and have more control rather than the system based on what is understood as a vision model.

AWS Bedrock is the least flexible with tool results (OpenAI, Google, Anthropic, all allow some way of returning images from tools), so with Bedrock specifically, it's either the "display message" or the image to be shown from the potential agent option.

[danny-avila]

FWIW merging an unrelated improvement that will still display the image in case this happens:

[ogipogi]

Thanks for the info.

I updated the fix to be bedrock-specific and used it in my fork until the bug get fixed.

#6253

ogipogi@3194ab2