various fixes for Elasticsearch 5 compatibility #1299
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
haarg
force-pushed
the
haarg/es5-compat
branch
2 times, most recently
from
842b04a to
739a174
Compare
To support newer Elasticsearch versions, we want a new Search::Elasticsearch version. It still supports older versions, as long as the correct module is installed and it is instructed to use it. We've already made the necessary changes to explicitly configure it to use Search::Elasticsearch::Client::2_0::Direct, so we are now free to upgrade.
Elasticsearch 5 wants these to be booleans.
Elasticsearch 5 doesn't support payloads on type: completion fields. We were storing data in the payload, but not using it at all. Remove it.
This isn't supported in the same way in newer ES, and we aren't fetching an excessive amount of data, so it doesn't really matter.
Use types from MooseX::Types::ElasticSearch instead of ElaticSearchX::Model::Document::Types for ES and Location types. It's just a reexport in ElaticSearchX::Model::Document::Types anyway.
ElasticSearchX::Model's set delete method uses search_type: scan, but that is not supported in Elasticsearch 5. Add a module to monkey patch ESXM to use sort: _doc instead.
Elasticsearch 5 will automatically convert string mapping types to be text or keyword types. We want to support both ES 2.4 and 5, so keep the declared mappings using the string type. When we check this against what is deployed, ignore the difference if a string is converted to text or keyword.
As far as I can tell, these are unused. These fields are using the string type, which will be automatically be converted to text or keyword types. With term_vector and fielddata, this automatic conversion can't be done. Removing them allow the same mapping to work on both Elasticsearch 2.4 and 5.
ES5 doesn't support "fields" to select qhat to return. Instead we can use _source.
Newer ES versions don't allow selecting fields. Instead, you need to use stored_fields, but it works rather differently. _source will do basically everything we need, and works on both old and new ES.
We don't need pod in both analyzed and non-analyzed form. We could remove the pod.analyzed field, but it would be more complicated to upgrade. Trying to index pod as non-analyzed with doc_values will be a problem for large pod content. Make both the main pod field and pod.analyzed analyzed, with doc_values off. We can consider dropping the sub-field in the future.
Elasticsearch 6 return hits.total as an object. Add a function that can cope with both forms, and use it in all of the places we retrieve the total.
We're nearly prepared to upgrade to Elasticsearch 5, and the code should work with the new version. Add the client prerequisite, so using ES5 is just a configuration change.
haarg
force-pushed
the
haarg/es5-compat
branch
from
739a174 to
b621a68
Compare
mickeyn
previously approved these changes
Oct 25, 2024
haarg
dismissed
mickeyn’s stale review
The merge-base changed after approval.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
_sourcerather thanfields. Requires a hack in ESXM.