Limiting ElasticSearch results when working with Pagy

[pnchkirr]

I have a problem with limiting the total number of search results in my web application.

The web app is written in Ruby, the search functionality is powered by ElasticSearch, and the pagination is handled by the Pagy ruby gem (documentation is here).

First, to give you a big picture, the main issue I figured out so far is in the pagy_elasticsearch_rails method that I use to run and paginate the ElasticSearch query. It overrides the size parameter of the query (which, in turn, is used to limit the total number of results returned) when I pass the items parameter to it. The reason I need to also use items is to limit the number of results per page (this is the right parameter to use with Pagy, according to the documentation). pagy_elasticsearch_rails drops the size parameter that comes from the ElasticSearch query, and places items as size inside of the query. It limits the results per page number indeed but it also falls back to the default total number of results (which is 10,000).

Now, to be more specific, I have the controller file containing the search method - it's the starting point for running a search.
The first thing that happens here is the pagy_search method being called:

@pagy_cars, cars = pagy_search(params[:query], "car")
@pagy_bikes, bikes = pagy_search(params[:query], "bike")

I am calling pagy_search twice because I have two types of results (and two different ElasticSearch queries and indices correspondingly: cars and bikes).
pagy_search method is located in the same file:

def pagy_search(search_term, results_type)
search_query = Matches::SearchQuery::Elasticsearch.call(search_term)
query_by_type = search_query[results_type.to_sym] # To treat car and bike queries differently
pagy_elasticsearch_rails(query_by_type, items: 24)

search_query variable contains the results of the Elasticsearch call - specifically the elasticsearch.rb file that holds the logic for building the search query itself. That's where I also set up the limit for the total number of results. Here are some of its contents:

module Matches
  module SearchQuery
    class Elasticsearch < Base

      def initialize(term)
        @search_term   = term
        car_query = { min_score: 0.5, query: function_score_wrapper("car"), size: 120 } # I use function_score_wrapper method here because I apply a custom function to refine _score
        bike_query = { min_score: 0.5, query: function_score_wrapper("bike"), size: 120 }
        @car_response = Car.pagy_search(car_query)
        @course_response = Bike.pagy_search(bike_query)
      end

      def results
        {
          car: @car_response,
          bike: @bike_response
        }
      end

      private

      def function_score_wrapper(type)
        {
          function_score: {
            max_boost: 1.6,
            boost_mode: "multiply",
            script_score: {
              script: {
                lang: "painless",
                source:
                  "some_custom_function_here_with_logic_based_on_type_parameter"
              }
            },
            query: general_query(type)
          }
        }
      end
    
      def general_query(type)
        {
          bool: {
            should: multi_match_queries(type),
            filter: filters(type)
          }
        }
      end

      def multi_match_queries(type)
        if type == "car"
          [
            nested_text_match_query_clause(type, nested: true)
          ]
        else
          [
            fuzzy_text_match_query_clause(type, nested: false),
            text_match_query_clause(type, nested: false),
            nested_text_match_query_clause(type, nested: true)
          ]
        end
      end

etc. - I don't think it's worth copy-pasting all the methods as my point here is to show you the structure in general terms. Otherwise, please let me know :)

If we print the result of pagy_elasticsearch_rails(query_by_type, items: 24), we'll get this structure:

[
  #<Pagy object
      pagy_object_parameters
  >,
  <Elasticsearch::Model::Response::Response object
      Car(car table fields),
      @search = #<Elasticsearch::Model::Searching::SearchRequest object
      Car(car table fields),
      @options = {},
      @definition = {search query}
      >,
  @raw_response = {search query results}
  >
]

Inside the @definition variable I can see that the size was overwritten: :size=>24, so the value I set in elasticsearch.rb was replaced by items value. And it's then being used to limit the number of results per page which I also need.

Have you experienced a similar problem before? Do you have any thoughts on how can I navigate around it?

P.S. I am using a pagy version 3.7.0
P.P.S I can confirm that the size parameter limits the ElasticSearch results correctly as I verified it in Kibana

[ddnexus]

I am using a pagy version 3.7.0

Why not 6.0.4? The 3, is super old and is missing a lot of fixes, refactoring and features, besides there is no time to support it anymore.

The pagy_search is a pagy method and you are overriding it, besides you only post part of your override, so it's impossible to tell what you are doing with it.

More importantly, you should formulate your questions in a way that does not require to go through a lot of your logic, or I doubt that anybody would have the time to get through it.

[pnchkirr]

Hi @ddnexus, thank you for taking the time to go through my question.

You are right, I apologize for overloading it with so much detail - I just wanted to include every important piece to make following my logic easier. Seems I did quite the opposite 😬

I tried updating to the latest (6.0.4) version but it doesn't seem to resolve the issue.

Let me rephrase my question to make it more concise and simple:

In my ElasticSearch query, I use the size parameter to limit the number of results. The limit works fine when I execute the query in Kibana. However, once I pass it to the Pagy gem method (pagy_elasticsearch_rails, in particular), the size value is overwritten by the items one (which I pass to set the results per page limit). As a result, the total limit I set in the ElasticSearch (120 records) query is lost.

Here's how I pass the query to the pagy_elasticsearch_rails method:

es_query = { min_score: 0.5, query: <the actual query body>, size: 120 }
collection = MyModel.pagy_search(es_query)  # pagy_search here is the native Pagy method
@pagy, @response = pagy_elasticsearch_rails(collection, items: 24)  # pagy_elasticsearch_rails here is the native Pagy method

In my code, I do override pagy_search method indeed but the only reason I do that is to separate two different ElasticSearch queries. Apart from that, I do not change any other logic.

Do you know how can I keep the total limit I set by the size parameter, and make it work with the items one?

[ddnexus]

It looks that you use the "active" mode, which controls the search, including the size, so you should NOT pass the size to ES which - as you noticed - gets ignored (i.e overridden). Just set the :items variable instead.

I don't get how you think to get 120 items from the ES size, and at the same time want only 24 in the page. Size and items are the same concept.

In my code, I do override pagy_search method indeed but the only reason I do that is to separate two different ElasticSearch queries. Apart from that, I do not change any other logic.

Don't override it! I doubt that you even know why that method exists and what it does. Besides, you don't need to "separate" 2 queries: they have never been "joined" or whatever you think they are before "separating" them. Just use the same pagination code with each different query, and you should be done.

[pnchkirr]

Thanks for the quick reply @ddnexus !

I don't get how you think to get 120 items from the ES size, and at the same time want only 24 in the page. Size and items are the same concept.

I am not sure I am following this logic 🤔
I've seen it all across the documentation but I don't quite get it: ElasticSearch default total results limit is 10k. I don't need that many - instead, I need less, only 120 results in my case. The reason is that I will be applying some extra logic to the resulting data set - and if I retrieve the default 10k amount, it will significantly decrease the performance.

Don't override it! I doubt that you even know why that method exists and what it does. Besides, you don't need to "separate" 2 queries: they have never been "joined" or whatever you think they are before "separating" them. Just repeat your pagination code with the 2 different queries, and you should be done.

Thanks for pointing it out 👍
You are right - I know this method was sitting in its own file pagy_search.rb in version 3.7.0 but I am not aware of what's exactly happening inside in any case. I will make sure I stop overriding it without a solid reason to do so

[ddnexus]

I am not sure I am following this logic 🤔
I've seen it all across the documentation but I don't quite get it: ElasticSearch default total results limit is 10k

If you need 24 records out of ES to display in your page.l, THAT is the size, (I.e. :items in pagy). AFAIK and unless they changed it recently the default is 10, not 10k. You are confusing it with something else.

[pnchkirr]

@ddnexus Pardon me, I just verified it - you're right, ES searches return only top 10 results by default.

I guess the thing that I'd like to change lies in this line:

By default, you cannot use from and size to page through more than 10,000 hits.

In order to modify that max hit number, they suggest changing the max_result_window variable:

This limit is a safeguard set by the index.max_result_window index setting

I tried it - I set max_result_window on the index level, and I reindexed, but it didn't prevent all 10k results from still being paginated.

[ddnexus]

All that has nothing to do with pagination!

[pnchkirr]

By the way, a question on your first part:

It looks that you use the "active" mode, which controls the search, including the size, so you should NOT pass the size to ES which - as you noticed - gets ignored (i.e overridden). Just set the :items variable instead.

Could switching to the "passive" mode help me control the total number of results?
If yes, according to the doc, I need to retrieve the @pagy and @response objects separately:

# standard response (already paginated)
@response = Article.search('*', from: 0, size: 10, ...)
# get the pagy object out of it
@pagy = Pagy.new_from_elasticsearch_rails(@response, ...)

How would I make sure I get the same output format as when I use the active mode?

[ddnexus]

Could switching to the "passive" mode help me control the total number of results?

The mode has nothing to do with your problem. The "the total number of results" of the query has nothing to do with the size/items concept, that is the chunk of the total results that you want to display in one page.

It looks like you are very confused about the most basic concept of pagination and/or ElasticSearch and ruby itself, and in that case I am not the most effective tool at your disposal. I am sure that some course, tutorial or online research will serve you better.