Text Splitting / Chunking for Embedding

[mikemanyoung]

Langchain Python / Js (and even rb, to some extent), have the concept (and facilities) to split / chunk / overlap content before embedding. Does RubyLLM have support for this, or do we need to roll our own?

[crmne]

Hi @mikemanyoung! RubyLLM does not support for text splitting, by design.

I embrace the UNIX philosophy of doing one thing well: RubyLLM is a library to talk with LLMs.

There are plenty of fine libraries and techniques to do chunking for embedding. I recommend you check them out!