Robots do not respond to messages with "!v"

[mwnu]

"gpt_vision_api_endpoint": "https://xxxx/v1/chat/completions",
"gpt_vision_model": "gpt-4-turbo",

"content": {
    "body": "!v Please explain the above diagram.",
    "format": "org.matrix.custom.html",
    "formatted_body": "!v Please explain the above diagram.",
    "msgtype": "m.text"
  }

No response, no logs.

[hibobmaster]

You should quote a image. Besides, gpt vision won't work in E2EE room.

[mwnu]

You should quote a image. Besides, gpt vision won't work in E2EE room.

I use @+!v, and it responds twice; the first time, it indicates that it doesn't know.

[hibobmaster]

Can you provide a screenshot?

[mwnu]

Can you provide a screenshot?

[hibobmaster]

I know what's wrong. When we mention the bot, it trigger thread chat as the same time.

[mwnu]

I know what's wrong. When we mention the bot, it trigger thread chat as the same time.

Perhaps the command "!v" can be omitted. Instead, different models could be invoked based on the event's mimetype, as some models, such as gpt-4-turbo and claude-3, support vision.

[hibobmaster]

With 81543d5, mention bot

• with image will trigger gpt vision
• with plain text will trigger thread level chatting

Try image: hibobmaster/matrixchatgptbot:sha-81543d561b46df4158892324172b5145e44f0e32

[mwnu]

sha-81543d561b46df4158892324172b5145e44f0e32

Robot can now recognize images in rooms without !v, but they cannot perform this function within threads. Additionally, commands like !pic, !help, and !lc are also unusable in threads, indicating that the two interaction modes are not well integrated. Of course, commands such as !gpt, !chat, and !new are unnecessary in threads. However, due to the poor compatibility of "io.element.thread" (Element PC's implementation for servers that do not support the Matrix standard threads) on client devices (it does not display on mobile phones), retaining these commands is still essential.

[mwnu]

With 81543d5, mention bot

• with image will trigger gpt vision
• with plain text will trigger thread level chatting

Try image: hibobmaster/matrixchatgptbot:sha-81543d561b46df4158892324172b5145e44f0e32

Some of the robot's responses are displayed entirely in red font, while others are not. It appears that the <mx-reply> <blockquote> tags have been added in the HTML. Is this another way of implementing the reply function in Matrix? Very strange!

[hibobmaster]

c5834db
Try image: hibobmaster/matrixchatgptbot:sha-c5834db9b270181a9987aff05c311d6c698a3d49

[mwnu]

sha-c5834db9b270181a9987aff05c311d6c698a3d49

I tested it, and these commands execute successfully within threads. They seem to be independent of the thread context. For example, after uploading a picture, the bot cannot describe it directly. Instead, it requires a separate reply to reference it, and the bot's response is not considered part of the thread context. Of course, the current method is okay, almost like a thread within a thread 😀. This can reduce unnecessary context, which can be referenced separately when needed.

[hibobmaster]

the bot cannot describe it directly

GPT Vision has a prompt which should be provided by user since matrix doesn't support sending a image with description.

and the bot's response is not considered part of the thread context

Try image: hibobmaster/matrixchatgptbot:v1.7.2

[mwnu]

the bot cannot describe it directly

GPT Vision has a prompt which should be provided by user since matrix doesn't support sending a image with description.

and the bot's response is not considered part of the thread context

Try image: hibobmaster/matrixchatgptbot:v1.7.2

done👍