Allowing non-rollup and rollup indices to be searched together #1268
Conversation
tandonks
requested review from
bowenlan-amzn,
Hailong-am and
vikasvb90
as code owners
tandonks
force-pushed
the
source_rollup_indices_search
branch
from
4ea022d to
5e9c1c5
Compare
|
thanks @tandonks for taking this change up! LGTM |
|
Thanks. What would be the condition to use this? Better to provide the instructions for normal users on when they can enable this and search source and target. |
| @@ -1092,12 +1092,12 @@ class RollupInterceptorIT : RollupRestTestCase() { | |||
| }, | |||
| "aggs": { | |||
| "sum_passenger_count": { "sum": { "field": "passenger_count" } }, | |||
| "max_passenger_count": { "max": { "field": "passenger_count" } }, | |||
| "value_count_passenger_count": { "value_count": { "field": "passenger_count" } } | |||
There was a problem hiding this comment.
why removing value_count?
It's better to check all this values are right in rollup search
There was a problem hiding this comment.
It was initially also not being used in the test earlier since we were testing the validation (where we used to throw the exception) on rollup and source index searches together
I was actually trying with that too but was getting an Internal Server Error for the same
{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"","phase":"fetch","grouped":true,"failed_shards":[],"caused_by":{"type":"class_cast_exception","reason":"class org.opensearch.search.aggregations.metrics.InternalScriptedMetric cannot be cast to class org.opensearch.search.aggregations.metrics.InternalValueCount (org.opensearch.search.aggregations.metrics.InternalScriptedMetric and org.opensearch.search.aggregations.metrics.InternalValueCount are in unnamed module of loader 'app')"}},"status":500}
| @@ -347,6 +353,9 @@ class RollupInterceptor( | |||
| if (searchAllJobs) { | |||
| request.source(request.source().rewriteSearchSourceBuilder(matchingRollupJobs.keys, fieldNameMappingTypeMap, concreteSourceIndex)) | |||
| } else { | |||
| if (matchingRollupJobs.keys.size > 1) { | |||
| logger.warn("Trying search with search across multiple rollup jobs disabled so will give result with largest rollup window") | |||
There was a problem hiding this comment.
Let move this to trace since multiple such search queries can lead to significant logging.
Signed-off-by: Kshitij Tandon <tandonks@amazon.com>
Signed-off-by: Kshitij Tandon <tandonks@amazon.com>
Signed-off-by: Kshitij Tandon <tandonks@amazon.com>
tandonks
force-pushed
the
source_rollup_indices_search
branch
from
89a044e to
c7f003e
Compare
|
The backport to To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/index-management/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/index-management/backport-2.x
# Create a new branch
git switch --create backport/backport-1268-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 dd710e6367cfea34fe1f6a179bf2e1eb742ee8cb
# Push it to GitHub
git push --set-upstream origin backport/backport-1268-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/index-management/backport-2.xThen, create a pull request where the |
…earch-project#1268) * Allowing non-rollup and rollup indices to be searched together Signed-off-by: Kshitij Tandon <tandonks@amazon.com> * Fixing an issue in the integration test Signed-off-by: Kshitij Tandon <tandonks@amazon.com> * Using trace in place of warn in logger Signed-off-by: Kshitij Tandon <tandonks@amazon.com> --------- Signed-off-by: Kshitij Tandon <tandonks@amazon.com>
…earch-project#1268) * Allowing non-rollup and rollup indices to be searched together Signed-off-by: Kshitij Tandon <tandonks@amazon.com> * Fixing an issue in the integration test Signed-off-by: Kshitij Tandon <tandonks@amazon.com> * Using trace in place of warn in logger Signed-off-by: Kshitij Tandon <tandonks@amazon.com> --------- Signed-off-by: Kshitij Tandon <tandonks@amazon.com>
…earch-project#1268) * Allowing non-rollup and rollup indices to be searched together Signed-off-by: Kshitij Tandon <tandonks@amazon.com> * Fixing an issue in the integration test Signed-off-by: Kshitij Tandon <tandonks@amazon.com> * Using trace in place of warn in logger Signed-off-by: Kshitij Tandon <tandonks@amazon.com> --------- Signed-off-by: Kshitij Tandon <tandonks@amazon.com>
Description
Introduced a new setting ROLLUP_SEARCH_SOURCE_INDICES which when set to true allows users to search non-rollup and rollup indices together.
It's just a way to perform searches together of non-rollup and rollup indices. In order to use it, users need to update the cluster settings and set the above setting to true.
Related Issues
Resolves #1213
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.