This is a fork of the excellent library by @basemkhirat, who sadly seems to have abandoned it by now.
As we rely on this library quite heavily, we will attempt to keep it up to date and compatible with newer Laravel and Elasticsearch versions.
Goals of this fork:
- Support for newer versions of Elasticsearch, Laravel and PHP
- Limited backwards compatibility: PHP version requirement of
>= 7.3 - Strong typing and accurate documentation comments
- New features and convenience methods
If you're interested in contributing to this fork, please submit a PR or open an issue.
Laravel, Lumen and Native php elasticseach query builder to build complex queries using an elegant syntax
- Keeps you away from wasting your time by replacing array queries with a simple and elegant syntax you will love.
- Elasticsearch data model for types and indices inspired from laravel eloquent.
- Feeling free to create, drop, mapping and reindexing through easy artisan console commands.
- Lumen framework support.
- Native php and composer based applications support.
- Can be used as a laravel scout driver.
- Dealing with multiple elasticsearch connections at the same time.
- Awesome pagination based on LengthAwarePagination.
- Caching queries using a caching layer over query builder built on laravel cache.
- PHP >=
7.3
See Travis CI Builds. laravel/laravel>= 5.* orlaravel/lumen>= 5.* or any other application using composer
See Full Documentation.
This section describes the installation process for all supported application types.
Whether you're using Laravel, Lumen or another framework, start by installing the package using composer:
composer require matchory/elasticsearchIf you're using an older release of Laravel than 5.5, add the service provider and facade to your config/app.php:
'providers' => [
// ...
Matchory\Elasticsearch\ElasticsearchServiceProvider::class,
// ...
],
// ...
'aliases' => [
// ...
'ES' => Matchory\Elasticsearch\Facades\ES::class,
// ...
],Lastly, publish the service provider to your configuration directory:
php artisan vendor:publish --provider="Matchory\Elasticsearch\ElasticsearchServiceProvider"After installing the package from composer, add package service provider in bootstrap/app.php:
$app->register(Matchory\Elasticsearch\ElasticsearchServiceProvider::class);Copy the package config directory at vendor/matchory/elasticsearch/src/config/ to your project root folder alongside with your app/ directory:
cp -r ./vendor/matchory/elasticsearch/src/config ./configIf you haven't already, make Lumen work with facades by uncommenting this line in bootstrap/app.php:
$app->withFacades();If you don't want to enable facades in Lumen, you can access the query builder using app("es"):
app("es")->index("my_index")->type("my_type")->get();
# This is similar to:
ES::index("my_index")->type("my_type")->get();You can install package with any composer-based application. While we can't provide general instructions, the following example should give you an idea of how it works:
require "vendor/autoload.php";
use Matchory\Elasticsearch\Connection;
$connection = Connection::create([
'servers' => [
[
"host" => '127.0.0.1',
"port" => 9200,
'user' => '',
'pass' => '',
'scheme' => 'http',
],
],
// Custom handlers
// 'handler' => new MyCustomHandler(),
'index' => 'my_index',
'logging' => [
'enabled' => env('ELASTIC_LOGGING_ENABLED',false),
'level' => env('ELASTIC_LOGGING_LEVEL','all'),
'location' => env('ELASTIC_LOGGING_LOCATION',base_path('storage/logs/elasticsearch.log'))
],
]);
# access the query builder using created connection
$documents = $connection->search("hello")->get();After publishing the service provider, a configuration file has been created at config/es.php. Here, you can add one or more Elasticsearch connections, with
multiple servers each. Take a look at the following example:
# Here you can define the default connection name.
'default' => env('ELASTIC_CONNECTION', 'default'),
# Here you can define your connections.
'connections' => [
'default' => [
'servers' => [
[
"host" => env("ELASTIC_HOST", "127.0.0.1"),
"port" => env("ELASTIC_PORT", 9200),
'user' => env('ELASTIC_USER', ''),
'pass' => env('ELASTIC_PASS', ''),
'scheme' => env('ELASTIC_SCHEME', 'http'),
]
],
// Custom handlers
// 'handler' => new MyCustomHandler(),
'index' => env('ELASTIC_INDEX', 'my_index')
]
],
# Here you can define your indices.
'indices' => [
'my_index_1' => [
"aliases" => [
"my_index"
],
'settings' => [
"number_of_shards" => 1,
"number_of_replicas" => 0,
],
'mappings' => [
'posts' => [
'properties' => [
'title' => [
'type' => 'string'
]
]
]
]
]
]If you'd like to use Elasticsearch with Laravel Scout, you can find the scout specific settings in
config/scout.php.
With the artisan commands included with this package, you can create or update settings, mappings and aliases. Note that all commands use the default connection
by default. You can change this by passing the --connection <your_connection_name> option.
The following commands are available:
$ php artisan es:indices:list
+----------------------+--------+--------+----------+------------------------+-----+-----+------------+--------------+------------+----------------+
| configured (es.php) | health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size |
+----------------------+--------+--------+----------+------------------------+-----+-----+------------+--------------+------------+----------------+
| yes | green | open | my_index | 5URW60KJQNionAJgL6Q2TQ | 1 | 0 | 0 | 0 | 260b | 260b |
+----------------------+--------+--------+----------+------------------------+-----+-----+------------+--------------+------------+----------------+Note that creating operation skips the index if exists.
# Create all indices in config file.
php artisan es:indices:create
# Create only 'my_index' index in config file
php artisan es:indices:create my_index Note that updating operation updates indices setting, aliases and mapping and doesn't delete the indexed data.
# Update all indices in config file.
php artisan es:indices:update
# Update only 'my_index' index in config file
php artisan es:indices:update my_index Be careful when using this command, as you will lose your index data!
Running drop command with --force option will skip all confirmation messages.
# Drop all indices in config file.
php artisan es:indices:drop
# Drop specific index on sever. Not matter for index to be exist in config file or not.
php artisan es:indices:drop my_index First, why reindexing?
Changing index mapping doesn't reflect without data reindexing, otherwise your search results will not work on the right way.
To avoid down time, your application should work with index alias not index name.
The index alias is a constant name that application should work with to avoid change index names.
Assume that we want to change mapping for my_index, this is how to do that:
-
Add
aliasas examplemy_index_aliastomy_indexconfiguration and make sure your application is working with it."aliases" => [ "my_index_alias" ]
-
Update index with command:
php artisan es:indices:update my_index
-
Create a new index as example
my_new_indexwith your new mapping in configuration file.$ php artisan es:indices:create my_new_index
-
Reindex data from
my_indexintomy_new_indexwith command:php artisan es:indices:reindex my_index my_new_index # Control bulk size. Adjust it with your server. php artisan es:indices:reindex my_index my_new_index --bulk-size=2000 # Control query scroll value. php artisan es:indices:reindex my_index my_new_index --bulk-size=2000 --scroll=2m # Skip reindexing errors such as mapper parsing exceptions. php artisan es:indices:reindex my_index my_new_index --bulk-size=2000 --skip-errors # Hide all reindexing errors and show the progres bar only. php artisan es:indices:reindex my_index my_new_index --bulk-size=2000 --skip-errors --hide-errors
-
Remove
my_index_aliasalias frommy_indexand add it tomy_new_indexin configuration file and update with command:php artisan es:indices:update
First, follow Laravel Scout installation.
All you have to do is updating the following lines in config/scout.php:
# change the default driver to 'es'
'driver' => env('SCOUT_DRIVER', 'es'),
# link `es` driver with default elasticsearch connection in config/es.php
'es' => [
'connection' => env('ELASTIC_CONNECTION', 'default'),
],Have a look at Laravel Scout documentation, too!
Each index type has a corresponding "Model" which is used to interact with that type. Models allow you to query for data in your types or indices, as well as insert new documents into the type. The work pretty much like Eloquent models you might know from Laravel already.
namespace App;
use Matchory\Elasticsearch\Model;
class Post extends Model
{
protected $type = 'posts';
}The above example will use the default connection and default index in config/es.php. You can override both in the next example.
namespace App;
use Matchory\Elasticsearch\Model;
class Post extends Model
{
# [optional] Default: default elasticsearch driver
# To override default connection name of es.php file.
# Assumed that there is a connection with name 'my_connection'
protected $connection = 'my_connection';
# [optional] Default: default connection index
# To override default index name of es.php file.
protected $index = 'my_index';
protected $type = 'posts';
}Once you have created a model and its associated index type, you are ready to start retrieving data from your index. For example:
use App\Post;
$posts = App\Post::all();
foreach ($posts as $post) {
echo $post->title;
}The all method will return all the results in the model's type. Each elasticsearch model serves as a query builder, you may also add constraints to
queries, and then use the get() method to retrieve the results:
$posts = App\Post::where('status', 1)
->orderBy('created_at', 'desc')
->take(10)
->get();// Retrieve a model by document key...
$posts = App\Post::find("AVp_tCaAoV7YQD3Esfmp");To create a new document, simply create a new model instance, set attributes on the model, then call the save() method:
namespace App\Http\Controllers;
use App\Post;
use Illuminate\Http\Request;
use Illuminate\Http\Response;
use App\Http\Controllers\Controller;
class PostController extends Controller
{
/**
* Create a new post instance.
*
* @param Request $request
* @return Response
*/
public function store(Request $request): Response
{
// Validate the request...
$post = new Post;
$post->title = $request->title;
$post->save();
}
}The save() method may also be used to update models that already exist. To update a model, you should retrieve it, set any attributes you wish to update, and
then call the save method.
$post = App\Post::find(1);
$post->title = 'New Post Title';
$post->save();To delete a model, call the delete() method on a model instance:
$post = App\Post::find(1);
$post->delete();Scopes allow you to define common sets of constraints that you may easily re-use throughout your application. For example, you may need to frequently retrieve all posts that are considered "popular". To define a scope, simply prefix an Eloquent model method with scope.
Scopes should always return a Query instance.
namespace App;
use Matchory\Elasticsearch\Model;
use Matchory\Elasticsearch\Query;
class Post extends Model
{
/**
* Scope a query to only include popular posts.
*
* @param Query $query
* @param int $votes
* @return Query
*/
public function scopePopular(Query $query, int $votes): Query
{
return $query->where('votes', '>', $votes);
}
/**
* Scope a query to only include active posts.
*
* @param Query $query
* @return Query
*/
public function scopeActive(Query $query): Query
{
return $query->where('active', 1);
}
}Once the scope has been defined, you may call the scope methods when querying the model. However, you do not need to include the scope prefix when calling the method. You can even chain calls to various scopes, for example:
$posts = App\Post::popular(100)->active()->orderBy('created_at')->get();To define an accessor, create a getFooAttribute method on your model where Foo is the "studly" cased name of the column you wish to access. In this
example, we'll define an accessor for the title attribute. The accessor will automatically be called by model when attempting to retrieve the value of
the title attribute:
namespace App;
use Matchory\Elasticsearch\Model;
class post extends Model
{
/**
* Get the post title.
*
* @param string $value
* @return string
*/
public function getTitleAttribute(string $value): string
{
return ucfirst($value);
}
}As you can see, the original value of the column is passed to the accessor, allowing you to manipulate and return the value. To access the value of the
accessor, you may simply access the title attribute on a model instance:
$post = App\Post::find(1);
$title = $post->title;Occasionally, you may need to add array attributes that do not have a corresponding field in your index. To do so, simply define an accessor for the value:
public function getIsPublishedAttribute(): bool
{
return $this->attributes['status'] === 1;
}Once you have created the accessor, just add the value to the appends property on the model:
protected $appends = ['is_published'];Once the attribute has been added to the appends list, it will be included in model's array.
To define a mutator, define a setFooAttribute method on your model where Foo is the "studly" cased name of the column you wish to access. So, again, let's
define a mutator for the title attribute. This mutator will be automatically called when we attempt to set the value of the titleattribute on the model:
namespace App;
use Matchory\Elasticsearch\Model;
class post extends Model
{
/**
* Set the post title.
*
* @param string $value
* @return string
*/
public function setTitleAttribute(string $value): string
{
return strtolower($value);
}
}The mutator will receive the value that is being set on the attribute, allowing you to manipulate the value and set the manipulated value on the model's
internal $attributes property. So, for example, if we attempt to set the title attribute to Awesome post to read:
$post = App\Post::find(1);
$post->title = 'Awesome post to read';In this example, the setTitleAttribute function will be called with the value Awesome post to read. The mutator will then apply the strtolower function
to the name and set its resulting value in the internal $attributes array.
The $casts property on your model provides a convenient method of converting attributes to common data types. The $casts property should be an array
where the key is the name of the casted attribute, and the value is the type you wish to cast the column to. The supported cast types are: integer,
float, double, string, boolean, object and array.
For example, let's cast the is_published attribute, which is stored in our index as an integer (0 or 1) to a boolean value:
namespace App;
use Matchory\Elasticsearch\Model;
class Post extends Model
{
/**
* The attributes that should be cast to native types.
*
* @var array
*/
protected $casts = [
'is_published' => 'boolean',
];
}Now the is_published attribute will always be cast to a boolean when you access it, even if the underlying, stored value in the index as an integer:
$post = App\Post::find(1);
if ($post->is_published) {
//
}You can use the ES facade to access the query builder directly, from anywhere in your application.
ES::create('my_index');
# or
ES::index('my_index')->create();use Matchory\Elasticsearch\Facades\ES;
use Matchory\Elasticsearch\Index;
ES::index('my_index')->create(function(Index $index) {
$index->shards(5)->replicas(1)->mapping([
'my_type' => [
'properties' => [
'first_name' => [
'type' => 'string',
],
'age' => [
'type' => 'integer'
]
]
]
])
});
# or
ES::create('my_index', function(Index $index){
$index->shards(5)->replicas(1)->mapping([
'my_type' => [
'properties' => [
'first_name' => [
'type' => 'string',
],
'age' => [
'type' => 'integer'
]
]
]
])
});ES::drop("my_index");
# or
ES::index("my_index")->drop();To run a query, start by (optionally) selecting the connection and index.
$documents = ES::connection("default")
->index("my_index")
->type("my_type")
->get(); # return a collection of resultsYou can shorten the above query to:
$documents = ES::type("my_type")->get(); # return a collection of resultsExplicitly setting connection or index name in the query overrides configuration in config/es.php.
ES::type("my_type")->id(3)->first();
# or
ES::type("my_type")->_id(3)->first();ES::type("my_type")->orderBy("created_at", "desc")->get();
# Sorting with text search score
ES::type("my_type")->orderBy("_score")->get();ES::type("my_type")->take(10)->skip(5)->get();ES::type("my_type")->select("title", "content")->take(10)->skip(5)->get();ES::type("my_type")->where("status", "published")->get();
# or
ES::type("my_type")->where("status", "=", "published")->get();ES::type("my_type")->where("views", ">", 150)->get();ES::type("my_type")->where("views", ">=", 150)->get();ES::type("my_type")->where("views", "<", 150)->get();ES::type("my_type")->where("views", "<=", 150)->get();ES::type("my_type")->where("title", "like", "foo")->get();ES::type("my_type")->where("hobbies", "exists", true)->get();
# or
ES::type("my_type")->whereExists("hobbies", true)->get();ES::type("my_type")->whereIn("id", [100, 150])->get();ES::type("my_type")->whereBetween("id", 100, 150)->get();
# or
ES::type("my_type")->whereBetween("id", [100, 150])->get();ES::type("my_type")->whereNot("status", "published")->get();
# or
ES::type("my_type")->whereNot("status", "=", "published")->get();ES::type("my_type")->whereNot("views", ">", 150)->get();ES::type("my_type")->whereNot("views", ">=", 150)->get();ES::type("my_type")->whereNot("views", "<", 150)->get();ES::type("my_type")->whereNot("views", "<=", 150)->get();ES::type("my_type")->whereNot("title", "like", "foo")->get();ES::type("my_type")->whereNot("hobbies", "exists", true)->get();
# or
ES::type("my_type")->whereExists("hobbies", true)->get();ES::type("my_type")->whereNotIn("id", [100, 150])->get();ES::type("my_type")->whereNotBetween("id", 100, 150)->get();
# or
ES::type("my_type")->whereNotBetween("id", [100, 150])->get();ES::type("my_type")->distance("location", ["lat" => -33.8688197, "lon" => 151.20929550000005], "10km")->get();
# or
ES::type("my_type")->distance("location", "-33.8688197,151.20929550000005", "10km")->get();
# or
ES::type("my_type")->distance("location", [151.20929550000005, -33.8688197], "10km")->get(); ES::type("my_type")->body([
"query" => [
"bool" => [
"must" => [
[ "match" => [ "address" => "mill" ] ],
[ "match" => [ "address" => "lane" ] ]
]
]
]
])->get();
# Note that you can mix between query builder and array queries.
# The query builder will will be merged with the array query.
ES::type("my_type")->body([
"_source" => ["content"]
"query" => [
"bool" => [
"must" => [
[ "match" => [ "address" => "mill" ] ]
]
]
],
"sort" => [
"_score"
]
])->select("name")->orderBy("created_at", "desc")->take(10)->skip(5)->get();
# The result query will be
/*
Array
(
[index] => my_index
[type] => my_type
[body] => Array
(
[_source] => Array
(
[0] => content
[1] => name
)
[query] => Array
(
[bool] => Array
(
[must] => Array
(
[0] => Array
(
[match] => Array
(
[address] => mill
)
)
)
)
)
[sort] => Array
(
[0] => _score
[1] => Array
(
[created_at] => desc
)
)
)
[from] => 5
[size] => 10
[client] => Array
(
[ignore] => Array
(
)
)
)
*/ES::type("my_type")->search("hello")->get();
# search with Boost = 2
ES::type("my_type")->search("hello", 2)->get();
# search within specific fields with different weights
ES::type("my_type")->search("hello", function($search){
$search->boost(2)->fields(["title" => 2, "content" => 1])
})->get();$doc = ES::type("my_type")->highlight("title")->search("hello")->first();
# Multiple fields Highlighting is allowed.
$doc = ES::type("my_type")->highlight("title", "content")->search("hello")->first();
# Return all highlights as array using $doc->getHighlights() method.
$doc->getHighlights();
# Also you can return only highlights of specific field.
$doc->getHighlights("title");ES::type("my_type")->search("hello")->first();ES::type("my_type")->search("hello")->count();# These queries are suitable for large amount of data.
# A scrolled search allows you to do an initial search and to keep pulling batches of results
# from Elasticsearch until there are no more results left.
# It’s a bit like a cursor in a traditional database
$documents = ES::type("my_type")->search("hello")
->scroll("2m")
->take(1000)
->get();
# Response will contain a hashed code `scroll_id` will be used to get the next result by running
$documents = ES::type("my_type")->search("hello")
->scroll("2m")
->scrollID("DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAFMFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABSxZSUDhLU3ZySFJJYXFNRV9laktBMGZ3AAAAAAAAAU4WUlA4S1N2ckhSSWFxTUVfZWpLQTBmdwAAAAAAAAFPFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABTRZSUDhLU3ZySFJJYXFNRV9laktBMGZ3")
->get();
# And so on ...
# Note that you don't need to write the query parameters in every scroll. All you need the `scroll_id` and query scroll time.
# To clear `scroll_id`
ES::type("my_type")->scrollID("DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAFMFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABSxZSUDhLU3ZySFJJYXFNRV9laktBMGZ3AAAAAAAAAU4WUlA4S1N2ckhSSWFxTUVfZWpLQTBmdwAAAAAAAAFPFlJQOEtTdnJIUklhcU1FX2VqS0EwZncAAAAAAAABTRZSUDhLU3ZySFJJYXFNRV9laktBMGZ3")
->clear();$documents = ES::type("my_type")->search("hello")->paginate(5);
# Getting pagination links
$documents->links();
# Bootstrap 4 pagination
$documents->links("bootstrap-4");
# Simple bootstrap 4 pagination
$documents->links("simple-bootstrap-4");
# Simple pagination
$documents->links("simple-default");These are all pagination methods you may use:
$documents->count()
$documents->currentPage()
$documents->firstItem()
$documents->hasMorePages()
$documents->lastItem()
$documents->lastPage()
$documents->nextPageUrl()
$documents->perPage()
$documents->previousPageUrl()
$documents->total()
$documents->url($page)ES::type("my_type")->search("hello")->where("views", ">", 150)->query();ES::type("my_type")->search("hello")->where("views", ">", 150)->response();ES::type("my_type")->ignore(404, 500)->id(5)->first();Package comes with a built-in caching layer based on laravel cache.
ES::type("my_type")->search("hello")->remember(10)->get();
# Specify a custom cache key
ES::type("my_type")->search("hello")->remember(10, "last_documents")->get();
# Caching using other available driver
ES::type("my_type")->search("hello")->cacheDriver("redis")->remember(10, "last_documents")->get();
# Caching with cache key prefix
ES::type("my_type")->search("hello")->cacheDriver("redis")->cachePrefix("docs")->remember(10, "last_documents")->get();ES::raw()->search([
"index" => "my_index",
"type" => "my_type",
"body" => [
"query" => [
"bool" => [
"must" => [
[ "match" => [ "address" => "mill" ] ],
[ "match" => [ "address" => "lane" ] ]
]
]
]
]
]);ES::type("my_type")->id(3)->insert([
"title" => "Test document",
"content" => "Sample content"
]);
# A new document will be inserted with _id = 3.
# [id is optional] if not specified, a unique hash key will be generated.# Main query
ES::index("my_index")->type("my_type")->bulk(function ($bulk){
# Sub queries
$bulk->index("my_index_1")->type("my_type_1")->id(10)->insert(["title" => "Test document 1","content" => "Sample content 1"]);
$bulk->index("my_index_2")->id(11)->insert(["title" => "Test document 2","content" => "Sample content 2"]);
$bulk->id(12)->insert(["title" => "Test document 3", "content" => "Sample content 3"]);
});
# Notes from the above query:
# As index and type names are required for insertion, Index and type names are extendable. This means that:
# If index() is not specified in subquery:
# -- The builder will get index name from the main query.
# -- if index is not specified in main query, the builder will get index name from configuration file.
# And
# If type() is not specified in subquery:
# -- The builder will get type name from the main query.
# you can use old bulk code style using multidimensional array of [id => data] pairs
ES::type("my_type")->bulk([
10 => [
"title" => "Test document 1",
"content" => "Sample content 1"
],
11 => [
"title" => "Test document 2",
"content" => "Sample content 2"
]
]);
# The two given documents will be inserted with its associated idsES::type("my_type")->id(3)->update([
"title" => "Test document",
"content" => "sample content"
]);
# Document has _id = 3 will be updated.
# [id is required]# Bulk update
ES::type("my_type")->bulk(function ($bulk){
$bulk->id(10)->update(["title" => "Test document 1","content" => "Sample content 1"]);
$bulk->id(11)->update(["title" => "Test document 2","content" => "Sample content 2"]);
});ES::type("my_type")->id(3)->increment("views");
# Document has _id = 3 will be incremented by 1.
ES::type("my_type")->id(3)->increment("views", 3);
# Document has _id = 3 will be incremented by 3.
# [id is required]ES::type("my_type")->id(3)->decrement("views");
# Document has _id = 3 will be decremented by 1.
ES::type("my_type")->id(3)->decrement("views", 3);
# Document has _id = 3 will be decremented by 3.
# [id is required]# increment field by script
ES::type("my_type")->id(3)->script(
"ctx._source.$field += params.count",
["count" => 1]
);
# add php tag to tags array list
ES::type("my_type")->id(3)->script(
"ctx._source.tags.add(params.tag)",
["tag" => "php"]
);
# delete the doc if the tags field contain mongodb, otherwise it does nothing (noop)
ES::type("my_type")->id(3)->script(
"if (ctx._source.tags.contains(params.tag)) { ctx.op = 'delete' } else { ctx.op = 'none' }",
["tag" => "mongodb"]
);ES::type("my_type")->id(3)->delete();
# Document has _id = 3 will be deleted.
# [id is required]# Bulk delete
ES::type("my_type")->bulk(function ($bulk){
$bulk->id(10)->delete();
$bulk->id(11)->delete();
});See the release page.
Basem Khirat - basemkhirat@gmail.com - @basemkhirat
Moritz Friedrich - moritz@matchory.com
Thanks to everyone who has contributed to the original project and
everyone else who has contributed to this fork!
Please use Github for reporting bugs, and making comments or suggestions.
MIT
Have a happy searching..