Update `embedders` settings, hybrid search, and add tests for AI search methods #1087

Strift · 2025-03-08T09:36:27Z

Pull Request

Related issue

Requires #1086

What does this PR do?

Update settings to handle embedders

Docs: https://www.meilisearch.com/docs/reference/api/settings#embedders

Add embedders setting: Update methods getEmbedders, updateEmbedders, resetEmbedders. Also, the method updateSettings should be able to accept the new embedders field. Here is the list of the acceptable sub fields:

Review comment:

Add a test to check if the format of each embedder type has the fields you need them to have

Update search to handle vector search and hybrid search

Docs: https://www.meilisearch.com/docs/reference/api/search

Update search method:

hybrid search parameter, with sub fields semanticRatio and embedder. embedder is mandatory if hybrid is set.
vector parameter is available
retrieveVectors parameter available
semanticHitCount in search response
Accept _semanticScore in the search response (optional)
_vectors are returned in the hit objects, when retrieveVectors is true
_vectors NOT present in the search response

Add similar documents endpoint

Docs: https://www.meilisearch.com/docs/reference/api/similar

Implement searchSimilarDocuments associated with the POST /indexes/:uid/similar. Do NOT implement with GET.

PR checklist

Please check if your PR fulfills the following requirements:

Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
Have you read the contributing guidelines?
Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

brunoocasali · 2025-03-18T13:36:01Z

meilisearch/index.py

+            Union[
+                OpenAiEmbedder,
+                HuggingFaceEmbedder,
+                OllamaEmbedder,
+                RestEmbedder,
+                UserProvidedEmbedder,
+            ],
+        ] = {}
        for k, v in response.items():
-            if v.get("source") == "openAi":
+            source = v.get("source")
+            if source == "openAi":
                embedders[k] = OpenAiEmbedder(**v)
-            elif v.get("source") == "huggingFace":
+            elif source == "huggingFace":
                embedders[k] = HuggingFaceEmbedder(**v)
+            elif source == "ollama":
+                embedders[k] = OllamaEmbedder(**v)
+            elif source == "rest":
+                embedders[k] = RestEmbedder(**v)
+            elif source == "userProvided":
+                embedders[k] = UserProvidedEmbedder(**v)
            else:
+                # Default to UserProvidedEmbedder for unknown sources
                embedders[k] = UserProvidedEmbedder(**v)


At first sight I think this could be a hash map so you can map the keys to the specific classes something like this:

EMBEDDER_MAP = { "openAi": OpenAiEmbedder, "huggingFace": HuggingFaceEmbedder, "ollama": OllamaEmbedder, "rest": RestEmbedder, "userProvided": UserProvidedEmbedder, } for k, v in response.items(): source = v.get("source", "userProvided") # Default to "userProvided" if missing embedder_class = EMBEDDER_MAP.get(source, UserProvidedEmbedder) embedders[k] = embedder_class(**v)

brunoocasali · 2025-03-18T13:36:46Z

meilisearch/index.py

+            Supported embedder sources:
+            - 'openAi': OpenAI embedder
+            - 'huggingFace': HuggingFace embedder
+            - 'ollama': Ollama embedder
+            - 'rest': REST API embedder
+            - 'userProvided': User-provided embedder
+
+            Required fields depend on the embedder source:
+            - 'rest' requires 'request' and 'response' fields
+            - 'userProvided' requires 'dimensions' field
+
+            Optional fields (availability depends on source):
+            - 'url': The URL Meilisearch contacts when querying the embedder
+            - 'apiKey': Authentication token for the embedder
+            - 'model': The model used for generating vectors
+            - 'documentTemplate': Template defining the data sent to the embedder
+            - 'documentTemplateMaxBytes': Maximum size of rendered document template
+            - 'dimensions': Number of dimensions in the chosen model
+            - 'revision': Model revision hash (only for 'huggingFace')
+            - 'distribution': Object with 'mean' and 'sigma' fields
+            - 'binaryQuantized': Boolean to convert vector dimensions to 1-bit values


I wonder if this docs follow any pattern to describe the fields and if this is the correct definition, can you confirm?

It's AI-generated based on my input (the meilisearch docs). It does not follow any particular conventions.

I think it might be better to have less information here, and let the user refer to the documentation. I will remove it.

brunoocasali · 2025-03-18T13:38:00Z

meilisearch/index.py

-
-        if body:
-            for _, v in body.items():
-                if "documentTemplateMaxBytes" in v and v["documentTemplateMaxBytes"] is None:


Is this handling done by Meili now?

Removing it did not trigger any test failure but it might simply be untested, so I added it back to avoid any unwanted side effects

brunoocasali · 2025-03-18T13:39:54Z

meilisearch/index.py

@@ -1881,24 +1913,68 @@ def get_embedders(self) -> Embedders | None:
        if not response:
            return None

-        embedders: dict[str, OpenAiEmbedder | HuggingFaceEmbedder | UserProvidedEmbedder] = {}
+        embedders: dict[


If you defined this in the embedders file (line 203) can't you just use that definition here?

meilisearch/models/embedders.py

brunoocasali · 2025-03-18T13:44:06Z

tests/index/test_index_search_meilisearch.py

+    assert "default" in response["hits"][0]["_vectors"]
+
+
+def test_get_similar_documents_with_identical_vectors(empty_index):


If I understood it correctly, you're manually creating the vector for that given document, so you don't need to define any model in your test instance before, right?

If you're referring to the test_get_similar_documents_with_identical_vectors test, that's correct.

I'm only creating an embedder so Meilisearch knows which embedder to use to compute the vector similarity:

# Configure the embedder settings_update_task = index.update_embedders( { "default": { "source": "userProvided", "dimensions": 2, } } )

Strift

Ty for the review @brunoocasali, I answered your comments and made the changes

Strift · 2025-03-20T06:47:59Z

meilisearch/index.py

+            Supported embedder sources:
+            - 'openAi': OpenAI embedder
+            - 'huggingFace': HuggingFace embedder
+            - 'ollama': Ollama embedder
+            - 'rest': REST API embedder
+            - 'userProvided': User-provided embedder
+
+            Required fields depend on the embedder source:
+            - 'rest' requires 'request' and 'response' fields
+            - 'userProvided' requires 'dimensions' field
+
+            Optional fields (availability depends on source):
+            - 'url': The URL Meilisearch contacts when querying the embedder
+            - 'apiKey': Authentication token for the embedder
+            - 'model': The model used for generating vectors
+            - 'documentTemplate': Template defining the data sent to the embedder
+            - 'documentTemplateMaxBytes': Maximum size of rendered document template
+            - 'dimensions': Number of dimensions in the chosen model
+            - 'revision': Model revision hash (only for 'huggingFace')
+            - 'distribution': Object with 'mean' and 'sigma' fields
+            - 'binaryQuantized': Boolean to convert vector dimensions to 1-bit values


It's AI-generated based on my input (the meilisearch docs). It does not follow any particular conventions.

I think it might be better to have less information here, and let the user refer to the documentation. I will remove it.

Strift · 2025-03-20T06:55:30Z

meilisearch/index.py

-
-        if body:
-            for _, v in body.items():
-                if "documentTemplateMaxBytes" in v and v["documentTemplateMaxBytes"] is None:


Removing it did not trigger any test failure but it might simply be untested, so I added it back to avoid any unwanted side effects

Co-authored-by: Bruno Casali <[email protected]>

sanders41

Sorry, I've been busy and only had a chance to take a really quick look at this. Can we test this against some older v1 meilisearch? I'm thinking there may be some changes here that make the package incompatible with older versions, but haven't had time to test myself.

curquiza · 2025-03-30T16:05:23Z

@Strift I merged the PR of Ellnix first: #1075 and now there are conflicts on your PR

Maybe I shouldn't have merged it
It's either his PR or your PR with conflict anyway...

Sorry again!

brunoocasali · 2025-03-30T19:54:20Z

I'm thinking there may be some changes here that make the package incompatible with older versions, but haven't had time to test myself.

This is expected @sanders41 since this version introduced the stabilization of the AI capabilities. @curquiza it looks good to me, but I would wait for @sanders41 review before merging :)

Strift marked this pull request as draft March 8, 2025 09:36

Strift marked this pull request as ready for review March 9, 2025 05:58

Strift changed the title ~~Feat/add embedders settings~~ Update embedders settings, hybrid search, and add tests for AI search methods Mar 9, 2025

brunoocasali reviewed Mar 18, 2025

View reviewed changes

Strift commented Mar 20, 2025

View reviewed changes

Strift requested a review from brunoocasali March 21, 2025 08:50

Strift and others added 20 commits March 26, 2025 15:37

Update embedders

c125d64

Update embedders models

7f61d2c

Add docs

93e8f69

Allow updating embedders via update_settings

d3aa65b

Refactor config validation to avoid duplicate code

2fcbc47

Update validation code

f9258e9

Remove validation to let meilisearch handle it

8a4369d

Remove unused parameters

742ef5e

Add hybrid search

c4e26d7

Add test for retrieving vectors

5e954ac

Add semanticHitCount test

05291f4

Update comment

b064b0b

Add test for similar documents

d5d928e

Fix linters errors

b49cb42

Sort imports

ef1b771

Update meilisearch/models/embedders.py

297b3e4

Co-authored-by: Bruno Casali <[email protected]>

Avoid repeating embedder type

c7c1700

Remove docs

b1258c7

Add unintentionally removed

8960bc2

Fix mypy issues

268aa4c

Strift force-pushed the feat/add-embedders-settings branch from bf4e7cb to 268aa4c Compare March 26, 2025 07:39

Strift added 3 commits March 26, 2025 16:09

Add test for embedders fields

d8825aa

Add tests for fields presence

b324323

Split tests

057377b

sanders41 reviewed Mar 26, 2025

View reviewed changes

Strift added 2 commits April 2, 2025 16:17

Merge branch 'main' into feat/add-embedders-settings

8082344

Fix missing imports

e515f29

curquiza requested a review from sanders41 April 3, 2025 17:56

Merge branch 'main' into feat/add-embedders-settings

dc98a1e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update `embedders` settings, hybrid search, and add tests for AI search methods #1087

Update `embedders` settings, hybrid search, and add tests for AI search methods #1087

Strift commented Mar 8, 2025 •

edited

Loading

brunoocasali Mar 18, 2025

brunoocasali Mar 18, 2025

Strift Mar 20, 2025

brunoocasali Mar 18, 2025

Strift Mar 20, 2025

brunoocasali Mar 18, 2025

brunoocasali Mar 18, 2025

Strift Mar 20, 2025 •

edited

Loading

Strift left a comment •

edited

Loading

Strift Mar 20, 2025

Strift Mar 20, 2025

sanders41 left a comment

curquiza commented Mar 30, 2025 •

edited

Loading

brunoocasali commented Mar 30, 2025

		assert "default" in response["hits"][0]["_vectors"]


		def test_get_similar_documents_with_identical_vectors(empty_index):

Update embedders settings, hybrid search, and add tests for AI search methods #1087

Are you sure you want to change the base?

Update embedders settings, hybrid search, and add tests for AI search methods #1087

Conversation

Strift commented Mar 8, 2025 • edited Loading

Pull Request

Related issue

What does this PR do?

Update settings to handle embedders

Update search to handle vector search and hybrid search

Add similar documents endpoint

PR checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Strift Mar 20, 2025 • edited Loading

Choose a reason for hiding this comment

Strift left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanders41 left a comment

Choose a reason for hiding this comment

curquiza commented Mar 30, 2025 • edited Loading

brunoocasali commented Mar 30, 2025

Update `embedders` settings, hybrid search, and add tests for AI search methods #1087

Update `embedders` settings, hybrid search, and add tests for AI search methods #1087

Strift commented Mar 8, 2025 •

edited

Loading

Strift Mar 20, 2025 •

edited

Loading

Strift left a comment •

edited

Loading

curquiza commented Mar 30, 2025 •

edited

Loading