Replace litellm with native API implementations.

[blkt]

Refactors client architecture to use native implementations instead of litellm dependency. Adds support for OpenAPI, Ollama, OpenRouter, and fixes multiple issues with Anthropic and Copilot providers. Improves message handling and streaming responses.

Commit message brought you by Anthropic Claude 3.7.

[JAORMX]

Omaigá

[blkt]

This pull request is humongous, sorry about that, but it turns out that almost every piece of code with the exception of the database layer was relying on litellm, implicitly or otherwise, and I could not find another way to refactor the code in steps without doubling the amount of code. All the pain this brings is my fault, apologies for that.

To make it easier to review, you can think of this refactoring as three big chunks of work

• adding a new representation: this is fully encapsulated in codegate.types module and its submodules, that's logically ~100% new code
• porting the pipeline to the new representation: this is all the work required to change event["messages"] to event.get_messages(); the interface should be simple enough and is based on generators
• muxing: muxing required changing how the wiring is done, we can discuss this further on code, but it only required adding a few routines to map openai->muxed-llm and back, plus optional parameters to some functions used within providers

Feel free to pull me in a call if necessary, but my last suggestion would be to just try it out locally and see if it works.

[lukehinds]

I spent quite some time testing this over the weekend, mostly continue and some copilot (albeit factoring around know issues), seems good to me and I think we should move towards merging this after a rebase , is there anything in particular you wanted testing / checking over @blkt @jhrozek ?

[blkt]

Hey @lukehinds , thank you so much for spending time testing and reviewing this! 🙏

I would be inclined to merge this, although we have some integration tests failures due to an undesired (or unexpected) compatibility of muxing with Ollama-specific APIs.

TL;DR is that, by design, muxing was expecting to talk to OpenAI endpoints, but litellm allowed also Ollama endpoints to work because it internally maps everything to OpenAI types. We don't have that capability in this new implementation, so this would be a breaking change, but being "bug-compatible" would add more code, and I'd rather avoid that.

My proposal is to

• rebase
• remove the failing integration tests (or rewrite them to use OpenAI endpoints if removing them lowers coverage)
• merge

Please folks, chime in if you don't agree. Meanwhile, I'll start making the described changes.