-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add OpenRouter as a supported provider #29
base: main
Are you sure you want to change the base?
Conversation
1c1bfbe
to
399c9eb
Compare
This is an excellent PR! You've clearly understood RubyLLM's design philosophy and implemented OpenRouter support in a way that fits seamlessly with the rest of the library. I'll test it out and do a comprehensive code review soon. Your concern about models.json is valid - we need a better long-term solution. Requiring users to refresh models has its own issues (filesystem access, rate limits), so maybe we need something like a CDN that we update via GitHub Actions. For multi-provider model selection, we might want to introduce a provider: parameter to .chat and a .with_provider(:sym) method to let users explicitly choose, with defaults to the original provider. This would let us cleanly handle cases where multiple providers offer the same model. |
Models.json issueShould I propose some provider options? If that's the case should I do it in another PR or extend this one? I think Ollama PR already has some progress by creating a concept of enabled/disabled providers and fetching models on demand, but of course we can do that CDN option as well :) Python's AI ecosystem often downloads config/model files during first run (and some Ruby AI ecosystem follow this pattern, like for example Tokenizers gem), so it wouldn't be unusual to fetch models.json during initialization. We can also leverage github or huggingface for hosting that models.json. Rate limits and depending on 3rd party APIsOne thing that I've noticed about openrouter is that the models API is actually public and doesn't require any keys which simplifies the implementation a bit. However it's also somewhat undocumented, as the supported capabilities aren't mentioned in the docs, but they are used by the openrouter website. |
6a18236
to
7ccdea5
Compare
Rebased to resolve a conflict in README.md |
Addresses #8
This adds support for OpenRouter.
You can use it by configuring the provider with:
Then calling models by their OpenRouter ids (it might be wise to prefix those with openrouter or something else if there is a possibility there will be more multi-model providers in the future)
OpenRouter doesn't support embedding models and image generation so those are not implemented here.
Getting model details requires multiple calls to OpenRouter API, since they don't expose function calling and structured output support in the responses. However they do allow filtering models by those capabilities. I memoized those calls for subsequent lookups, so the model fetching goes faster.
Code is based on OpenAI provider, since they share most of the API, but I've kept it separate since they might differ in future.
A small note - this adds a lot of data to models.json, it might be wise to generate that file on first use instead of bundling it with the gem or add refreshing to docs just like Ollama PR does (#10).