Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multimodal support for CodeAgent #875

Draft
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

codegen-sh[bot]
Copy link
Contributor

@codegen-sh codegen-sh bot commented Mar 17, 2025

Description

This PR adds multimodal support to the CodeAgent, allowing it to process images in Slack messages. This is the second part of the implementation, building on PR #11444 in the cloud repository which adds the ability to capture image information from Slack messages.

Changes

  1. Updated the LLM class to support multimodal content:

    • Added a _process_messages_for_multimodal method that converts image URLs in the format [Image: filename](URL) to the appropriate multimodal format for Claude 3 models
    • Modified the _generate method to process messages for multimodal content when using Claude 3 models
  2. Updated the CodeAgent class to use multimodal models by default:

    • Added a multimodal parameter (default: True) to the constructor
    • When multimodal is enabled, ensures a Claude 3 model is used even if a non-multimodal model was specified

How to Test

  1. Send a message with an image in a Slack thread
  2. Mention the CodeAgent bot
  3. Verify that the bot can see and process the image in its response

Related PRs

This PR builds on PR #11444 in the cloud repository, which adds the ability to capture image information from Slack messages.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


codegen-bot seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant