feat: Add "latest" and "related" search. #2055

jimsynz · 2025-01-21T23:00:36Z

Here's my first stab at adding latest and related search as per #2013

I'm keen for early feedback, thus this draft PR.

I have a few questions:

Is there a standard function in the JS (I couldn't find one) to generate a link to hexdocs for a specific package name, version and ref? I feel iffy about just having string interpolation in the code.
It looks like it might be difficult (but not impossible) to correctly generate the metadata (and thus the excerpts) from the response payload. Each hit contains a list of highlight snippets however it treats underscores as token separators so searching for to_ast is returns results for to ast.
It seems to return the same hits over and over again. Each returned hit has a different document ID so I think it's actually an index problem not a search problem. I can dedup them by ref but I'd rather that the search index returned better results.

I haven't yet changed any of the markup as I wanted to get this step right before I go adding things like package names/versions to results, etc.

josevalim · 2025-01-22T11:04:55Z

Is there a standard function in the JS (I couldn't find one) to generate a link to hexdocs for a specific package name, version and ref? I feel iffy about just having string interpolation in the code.

There isn't because this is the first time we do something like this. However, since it comes from hexdocs, I believe it is fair to assume a naming scheme.

Each hit contains a list of highlight snippets however it treats underscores as token separators so searching for to_ast is returns results for to ast.

@ruslandoga, is this something we can change in the indexer?

It seems to return the same hits over and over again. Each returned hit has a different document ID so I think it's actually an index problem not a search problem. I can dedup them by ref but I'd rather that the search index returned better results.

The reason it returns the same results over and over again is because we break the documentation of a single function into multiple entries, one per h2 to be more precise. The benefit is that we can provide more precise links too. For now, I don't think we dedup them for regular results right? So I would not dedup them here, but we can dedup them later if we want to (we can even render it like Google results, where we show the main entry and below we refer to specific sections in the result).

This is fantastic progress. :)

garazdawi · 2025-01-22T11:41:42Z

FYI, in the algolia for https://erlang.org/doc/search we disable _ as punctuation just as proposed here for the same reasons.

ruslandoga · 2025-01-22T12:21:53Z

👋

I couldn't find any options to make highlighting ignore the token separators. I'll try asking Typesense people if it's supported or planned to be supported.

For now, I think we might need to roll our own highlighter, possibly similar to the one used in autocomplete:

ex_doc/assets/js/autocomplete/suggestions.js

Lines 289 to 311 in 51cd422

    
           /** 
        
            * Returns an HTML string highlighting the individual tokens from the query string. 
        
            */ 
        
           function highlightMatches (text, query) { 
        
             // Sort terms length, so that the longest are highlighted first. 
        
             const terms = tokenize(query).sort((term1, term2) => term2.length - term1.length) 
        
             return highlightTerms(text, terms) 
        
           } 
        
           function highlightTerms (text, terms) { 
        
             if (terms.length === 0) return text 
        
             const [firstTerm, ...otherTerms] = terms 
        
             const match = text.match(new RegExp(`(.*)(${escapeRegexModifiers(firstTerm)})(.*)`, 'i')) 
        
             if (match) { 
        
               const [, before, matching, after] = match 
        
               // Note: this has exponential complexity, but we expect just a few terms, so that's fine. 
        
               return highlightTerms(before, terms) + '<em>' + escapeHtmlEntities(matching) + '</em>' + highlightTerms(after, terms) 
        
             } else { 
        
               return highlightTerms(text, otherTerms) 
        
             } 
        
           }

Removing _ from the current token_separators somewhat breaks non-prefix search. For example, searching upload stops returning allow_upload. Typesense does have a way around it with infix query option but I couldn't make it work well. We can try it again :)

ruslandoga · 2025-01-22T12:33:23Z

assets/js/search-page.js

+      const searchNodes = getSearchNodes()
+
+      if (['related', 'latest'].includes(queryType) && searchNodes.length > 0) {
+        results = await remoteSearch(value, queryType, searchNodes)


Just a couple nitpicks :)

Can we have a race condition here? When the previous request returns after the current request and updates the items to stale results. I think it's possible with multiple HTTP/1.1 connections, but not sure about multiple streams on the same HTTP/2 connection, are they guaranteed to be ordered? Or maybe JS runtime resolves it in some way?

Also, do we need to debounce on remote search or check for response.ok and results.length > 0?

For some reason I decided to do these things in ruslandoga#1 but I don't remember if I actually had these problems or was just playing it safe ...

Thanks. I'm sure you're right. As you can probably tell it's been almost a decade since I wrote any JavaScript so I'm still getting the hang of the new idioms.

So looking at the code more carefully, it appears that the search function is only called on page load, so should only be run once in the page's lifecycle.

Additionally, the search result handlebars template takes care of whether any results were actually returned.

assets/js/search-page.js

jimsynz · 2025-01-30T20:46:16Z

Okay so I've pushed some updates:

Uses the shorter package filter syntax in the API query.
Repurpose the existing highlighter code to highlight search terms in the API search results.

jimsynz · 2025-01-30T20:49:07Z

I'll be moving on to adding the search type dropdown in the search bar next.

jimsynz · 2025-02-16T21:13:47Z

Okay, so I've added the search type to the UI, although you can probably tell that CSS is not my strong suit. It enables and disables the "latest" and "related" search based on the contents of searchNodes and versionNodes. It also completely disables the select box when the browser is not online.

josevalim · 2025-02-22T19:48:45Z

Thank you @jimsynz. I have been a bit busy this week but I will try to add review and add polish to this feature this week.

jimsynz · 2025-02-23T21:22:04Z

Thanks @josevalim 🙌

…var.

josevalim · 2025-03-02T09:48:48Z

assets/js/search-page.js

+  const index = await getIndex()
+
+  // We cannot match on atoms :foo because that would be considered
+  // a filter. So we escape all colons not preceded by a word.


Is this considered a filter by typesense?

josevalim · 2025-03-02T09:50:48Z

lib/ex_doc/formatter/html/search_data.ex

  end

+  defp maybe_append_related_packages(search_nodes, _related_packages), do: search_nodes


We should revert these changes for now, I believe the related packages will come from hex.pm itself. You can emulate it though by calling mix docs and then customizing the docs_config.js in doc yourself. :)

Actually, we added our own for testing:

ex_doc/mix.exs

Line 127 in 8a5217e

File.write!("doc/docs_config.js", """

So I think you can just change the searchNodes there, perhaps by adding Elixir itself or earmark_parser to searchNodes, so you can test it too.

josevalim · 2025-03-02T09:51:08Z

lib/ex_doc/formatter/html/templates/head_template.eex

@@ -21,7 +21,7 @@
      <link rel="canonical" href="<%= config.canonical %>" />
    <% end %>
    <script defer src="<%= Assets.rev config.output, "dist/sidebar_items-*.js" %>"></script>
-    <script defer src="docs_config.js"></script>
+    <script defer src="<%= Assets.rev config.output, "dist/search_data-*.js" %>"></script>


This change should not be necessary either!

josevalim · 2025-03-02T09:59:13Z

assets/js/search-bar.js

+  if (versionNodes.length > 0) {
+    const latest = versionNodes[0]
+    const searchNodes = getSearchNodes()
+    const match = searchNodes.some(node => `v${node.version}` === latest.version)


I don't think we need to check if latest or not. It is easier if the user doesn't have to think about it. Regardless if it we are in the latest package or not, we should allow latest to be chosen (unless there is no searchNodes). So basically, we will have this:

If searchNodes.length > 1, the related option is enabled, otherwise it is disabled

If searchNodes.length > 0, the latest option is enabled, otherwise it is disabled

Otherwise it is only current/local

If online, the default should be related > latest > current. If offline, it is the current one.

In other words, this should return return versionNodes.length > 0.

josevalim · 2025-03-02T14:48:44Z

Thank you @jimsynz, I have dropped some comments. About the online vs offline, maybe instead of checking it upfront, we should try to run the search and, if the search fails for any reason, we fallback to regular search with an error message? I believe there is a toast we could use.

jimsynz · 2025-03-02T20:53:34Z

Thanks for the review @josevalim - I'll get onto it.

ruslandoga reviewed Jan 22, 2025

View reviewed changes

assets/js/search-page.js Outdated Show resolved Hide resolved

jimsynz added 4 commits February 13, 2025 10:51

wip: First draft of latest/related typesense search function.

4745a6e

improvement: shorten package filter.

c24dbc7

improvement: Use existing match highlighter for search matches.

26f7232

improvement: Allow the search type to be selected in the UI.

655d6af

jimsynz force-pushed the feat/typesense-search branch from 5ff3096 to 655d6af Compare February 13, 2025 04:15

jimsynz marked this pull request as ready for review February 16, 2025 22:18

improvement: Publish related_packages as part of the searchNodes …

b22465b

…var.

josevalim reviewed Mar 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add "latest" and "related" search. #2055

feat: Add "latest" and "related" search. #2055

jimsynz commented Jan 21, 2025 •

edited

Loading

josevalim commented Jan 22, 2025

garazdawi commented Jan 22, 2025

ruslandoga commented Jan 22, 2025 •

edited

Loading

ruslandoga Jan 22, 2025 •

edited

Loading

jimsynz Jan 22, 2025

jimsynz Jan 30, 2025

jimsynz Jan 30, 2025

jimsynz commented Jan 30, 2025

jimsynz commented Jan 30, 2025

jimsynz commented Feb 16, 2025

josevalim commented Feb 22, 2025

jimsynz commented Feb 23, 2025

josevalim Mar 2, 2025

josevalim Mar 2, 2025

josevalim Mar 2, 2025

josevalim Mar 2, 2025

josevalim Mar 2, 2025

josevalim commented Mar 2, 2025

jimsynz commented Mar 2, 2025

		end

		defp maybe_append_related_packages(search_nodes, _related_packages), do: search_nodes

feat: Add "latest" and "related" search. #2055

Are you sure you want to change the base?

feat: Add "latest" and "related" search. #2055

Conversation

jimsynz commented Jan 21, 2025 • edited Loading

josevalim commented Jan 22, 2025

garazdawi commented Jan 22, 2025

ruslandoga commented Jan 22, 2025 • edited Loading

ruslandoga Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimsynz commented Jan 30, 2025

jimsynz commented Jan 30, 2025

jimsynz commented Feb 16, 2025

josevalim commented Feb 22, 2025

jimsynz commented Feb 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josevalim commented Mar 2, 2025

jimsynz commented Mar 2, 2025

jimsynz commented Jan 21, 2025 •

edited

Loading

ruslandoga commented Jan 22, 2025 •

edited

Loading

ruslandoga Jan 22, 2025 •

edited

Loading