In my current job, we had to parse a string from several thousand rows of data and extract a section of the field based on some criteria. We initially did this using full-text search in MySQL. It worked but the rate of accuracy was not what we expected, therefore we decided to look into another solution.
I had used Elasticsearch in a previous role which was very much search based and I knew that we could get better results if we were to swap out the full text swap with Elasticsearch. I needed to refresh my knowledge of Elasticsearch and decided to create a demo of a Laravel application which is what we use for our codebase and implement a search feature with both MySQL and Elasticsearch. Come with me as I take you through the implementation.
In order to make the search realistic, I decided to make use of Books data which is contained in a json file. We will be loading this data in MySQL which will then be indexed in Elasticsearch using a model observer.
If you want to see this in action, you can clone the repository https://github.com/stevepop/es-laravel and follow the instructions in the README.
If you want to follow along, the first thing to do is create a new Laravel application;
laravel new <your-project-name>
You can copy the data file at https://github.com/stevepop/es-laravel/blob/master/books.json to the 'storage/app/data' folder.
The next thing to do is creating and configuring our Database. For this example, I am using an SQLite database. You will need to ensure that you have Sqlite installed.
Create file for the database; touch database/database.sqlite
Copy the .env file and add the SQLite configuration to your .env file
cp .env.example .env
DB_CONNECTION=sqlite
Now that we have our database set up, let's create a migration file;
php artisan make:model Book -m
Update the up() method of the generated migration file in app/migrations
Schema::create('books', function (Blueprint $table) {
$table->id();
$table->string('title');
$table->integer('isbn')->nullable();
$table->integer('page_count')->default(0);
$table->dateTime('published_date')->nullable();
$table->text('short_description')->nullable();
$table->longText('long_description')->nullable();
$table->string('thumbnail_url')->nullable();
$table->string('status');
$table->json('authors');
$table->json('categories');
$table->timestamps(); });
Run the migration; php artisan migrate
Now that we have our database in place, we can now setup Elasticsearch. For the sake of this demo, I have included a docker-compose.yml file. We will be running a Docker instance of Elasticsearch. All the is needed is to execute the command;
docker-compose up
You need to ensure you have Docker installed before executing the above command. This will spin up an Elasticsearch container as well as Kibana which is a platform to enable us visualise and run queries directly on our Elasticsearch instance. Once the container is up and running, you should be able to visit http://localhost:5601/app/kibana#/home to view the Kibana dashboard.
We have Elasticsearch up and running but before we connect to it from our application, we need to install the Elasticsearch php client. We do this by requiring it with composer;
composer require elasticsearch/elasticsearch
Now that we can use Elasticsearch within our application, our first test that it works is to load our data in the database. This is done using a command class;
<?php
namespace App\Console\Commands;
use App\Book;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Storage;
class LoadBooksData extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'books:load';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Load book data from json file';
/**
* Create a new command instance.
*
* @return void
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
$json = json_decode(Storage::get('data/books.json'), true);
$this->info('Loading books from json. This might take a while...');
foreach ($json as $item) {
$bookData = [
'title' => $item['title'],
'isbn' => $item['isbn'] ?? null,
'page_count' => $item['pageCount'],
'published_date' => isset($item['publishedDate']) ? $item['publishedDate']['$date'] : null,
'thumbnail_url' => $item['thumbnailUrl'] ?? null,
'short_description' => $item['shortDescription'] ?? null,
'long_description' => $item['longDescription'] ?? null,
'status' => $item['status'],
'authors' => $item['authors'],
'categories' => $item['categories'],
];
Book::create( $bookData);
// PHPUnit-style feedback
$this->output->write('.');
}
$this->info("\n Loading Complete!");
}
}
The above command simply loads the data from the books.json loops through each data and creates a record in the database. Before we run this command however, we need to do a few more things. First, let us update out Book model. We are casting 'authors' and 'categories' as json ands 'published_date' as a Carbon date object.
<?php
namespace App;
use App\Search\Searchable;
use Illuminate\Database\Eloquent\Model;
class Book extends Model
{
use Searchable;
protected $guarded = ['created_at', 'updated_at'];
protected $casts = [
'authors' => 'json',
'categories' => 'json',
];
protected $dates = ['published_date'];
}
In order to ensure that our data is indexed in Elasticsearch when created, we have to create a model observer to hook into the model events. You can place this in the app\Search folder.
<?php
namespace App\Search;
use App\Book;
use Elasticsearch\Client;
class ElasticsearchObserver
{
/** @var \Elasticsearch\Client */
private $elasticsearch;
public function __construct(Client $elasticsearch)
{
$this->elasticsearch = $elasticsearch;
}
public function saved($model)
{
$this->elasticsearch->index([
'index' => $model->getSearchIndex(),
'type' => $model->getSearchType(),
'id' => $model->getKey(),
'body' => $model->toSearchArray(),
]);
}
public function deleted($model)
{
$this->elasticsearch->delete([
'index' => $model->getSearchIndex(),
'type' => $model->getSearchType(),
'id' => $model->getKey(),
]);
}
}
When a Book model is saved, the Observer's saved method is called and it executes the code that indexes that model in the database.
We need to create a trait that will be used within the Book model when the user makes a search. You can place this in the app\Search folder.
<?php
namespace App\Search;
trait Searchable
{
public static function bootSearchable()
{
//toggle the search feature flag on and off
if (config('services.search.enabled')) {
static::observe(ElasticsearchObserver::class);
}
}
public function getSearchIndex()
{
return $this->getTable();
}
public function getSearchType()
{
if (property_exists($this, 'useSearchType')) {
return $this->useSearchType;
}
return $this->getTable();
}
public function toSearchArray()
{
// By having a custom method that transforms the model
// to a searchable array allows us to customize the
// data that's going to be searchable per model.
return $this->toArray();
}
}
In order to make the search flexible so that we can use the same interface to search with Eloquent or Elasticsearch, we will create two repositories. The first is the repository to search within the database and return rows where the search term is in the title or in the description.
<?php
namespace App\Books;
use App\Book;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Contracts\Pagination\LengthAwarePaginator;
class EloquentRepository implements BooksRepository
{
public function search(string $query = ''): LengthAwarePaginator
{
return Book::query()
->where('short_description', 'like', "%{$query}%")
->orWhere('title', 'like', "%{$query}%")
->paginate();
}
}
<?php
namespace App\Books;
use App\Book;
use Elasticsearch\Client;
use Illuminate\Contracts\Pagination\LengthAwarePaginator;
use Illuminate\Support\Arr;
use App\Books\BooksRepository;
use Illuminate\Database\Eloquent\Collection;
class ElasticsearchRepository implements BooksRepository
{
/** @var \Elasticsearch\Client */
private $elasticsearch;
public function __construct(Client $elasticsearch)
{
$this->elasticsearch = $elasticsearch;
}
public function search(string $query = ''): Collection
{
$items = $this->searchOnElasticsearch($query);
return $this->buildCollection($items);
}
private function searchOnElasticsearch(string $query = ''): array
{
$model = new Book;
$items = $this->elasticsearch->search([
'index' => $model->getSearchIndex(),
'type' => $model->getSearchType(),
'body' => [
'query' => [
'multi_match' => [
'fields' => ['title^5', 'short_description', 'categories'],
'query' => $query,
],
],
],
]);
return $items;
}
private function buildCollection(array $items): Collection
{
$ids = Arr::pluck($items['hits']['hits'], '_id');
return Book::findMany($ids)
->sortBy(function ($book) use ($ids) {
return array_search($book->getKey(), $ids);
});
}
}
The search method above calls the searchOnElasticsearch method which contains a query to search in Elasticsearch and returns the results using the buildCollection private method.
Finally we create the interface for the above implementations;
namespace App\Books;
use Illuminate\Contracts\Pagination\LengthAwarePaginator;
use Illuminate\Database\Eloquent\Collection;
interface BooksRepository {
public function search (string $query = ''): Collection;
}
Finally, we need to bind the interface to the concrete implementations. This is done in a Service provider. For the sake of this exercise, we will register this interface in the AppServiceProvider's register method.
<?php
namespace App\Providers;
use App\Books\ElasticsearchRepository;
use Elasticsearch\Client;
use App\Books\BooksRepository;
use Elasticsearch\ClientBuilder;
use App\Books\EloquentRepository;
use Illuminate\Support\ServiceProvider;
class AppServiceProvider extends ServiceProvider
{
/**
* Register any application services.
*
* @return void
*/
public function register()
{
$this->app->bind(BooksRepository::class, function($app) {
// Use Eloquent if elasticsearch is switched off
if (! config('services.search.enabled')) {
return new EloquentRepository();
}
return new ElasticsearchRepository (
$app->make(Client::class)
);
});
$this->bindSearchClient();
}
/**
* Bootstrap any application services.
*
* @return void
*/
public function boot()
{
//
}
private function bindSearchClient()
{
$this->app->bind(Client::class, function ($app) {
return ClientBuilder::create()
->setHosts($app['config']->get('services.search.hosts'))
->build();
});
}
}
In order to make the search flexible, we add a configuration to enable/disable elasticsearch so that searches can still work even if elasticsearch is not available or if it is switched off for debugging purposes.
In the config/services.php add the following;
'search' => [
'enabled' => env('ELASTICSEARCH_ENABLED', false),
'hosts' => explode(',', env('ELASTICSEARCH_HOSTS')),
],
The hosts value returns all the elasticsearch hosts if there are many nodes running elasticsearch which is usually the case in production environments.
Ensure that you have the configuration values set in your .env file
ELASTICSEARCH_ENABLED=true
ELASTICSEARCH_HOSTS="localhost:9200"
To display the data, I have created a template using tailwindCSS;
@extends('layouts.master')
@section('content')
<div class="container mx-auto">
<div class="rounded p-4">
<div class="bg-white rounded p-6 mb-6 flex justify-between">
<div class="p-2">
<span class="font-bold text-lg">Books</span> <small>({{ $books->total() }})</small>
</div>
<div class="p-4 w-64">
<form action="{{ url('search') }}" method="get">
<div class="form-group">
<input type="text" name="q" class="border p-2 w-full" placeholder="Search..."
value="{{ request('q') }}" />
</div>
</form>
</div>
</div>
<div class="bg-white rounded p-6">
<table class="table-auto">
<thead>
<tr>
<th class="px-4 py-2">Title</th>
<th class="px-4 py-2">ISBN</th>
<th class="px-4 py-2">Categories</th>
<th class="px-4 py-2">Published Date</th>
</tr>
</thead>
<tbody>
@forelse ($books as $book)
<tr>
<td class="border px-4 py-2">{{ $book->title }}</td>
<td class="border px-4 py-2">{{ $book->isbn }}</td>
<td class="border px-4 py-2">
@foreach($book->categories as $category)
<span
class="inline-block bg-gray-200 rounded-full px-3 py-1 text-sm font-semibold text-gray-700 mr-2">
{{ $category }}
</span>
@endforeach
</td>
<td class="border px-4 py-2">
{{ $book->published_date ? $book->published_date->format('d/m/Y') : null }}
</td>
</tr>
@empty
<tr>
<td>No books found</td>
</tr>
@endforelse
</tbody>
</table>
<div class="bg-white p-4 mt-6">
{{ $books->links() }}
</div>
</div>
</div>
</div>
@stop
In order for us to view and search the data, we need to create the routes and controllers;
Route::get('/', 'BooksController@index');
Route::get('/search', 'BooksController@search');
<?php
namespace App\Http\Controllers;
use App\Book;
use Illuminate\Http\Request;
use App\Books\BooksRepository;
use Illuminate\Support\Collection;
use Illuminate\Pagination\Paginator;
use Illuminate\Pagination\LengthAwarePaginator;
class BooksController extends Controller
{
public function index()
{
$books = Book::paginate();
return view('books.index', compact('books'));
}
function search(BooksRepository $repository)
{
$results = $repository->search((string) request('q'));
$books = $this->paginate($results);
return view('books.index', [
'books' => $books
]);
}
/**
* The attributes that are mass assignable.
*
* @var array
*/
public function paginate($items, $perPage = 5, $page = null, $options = [])
{
$page = $page ?: (Paginator::resolveCurrentPage() ?: 1);
$items = $items instanceof Collection ? $items : Collection::make($items);
return new LengthAwarePaginator($items->forPage($page, $perPage), $items->count(), $perPage, $page, $options);
}
}
Now we are ready for the magic! To load the books data, execute the command;
php artisan books:load
This command will read the Books.json file, insert the records in the database and for each record inserted, the ElasticsearchObserver will index the document.
If you go to the homepage you should see the list of books displayed. To ensure that the search works, enter a string in the search input at the top of the page and press enter. To test searching with Eloquent, set the ELASTICSEARCH_ENABLED to false and search again. This should still work as expected. You may not notice the difference in speed because of the small set of data but the power of Elasticsearch really kicks in when you have massive amounts of data with millions of rows.
This has been a long post but I hope it has given you some understanding of how to integrate Elasticsearch into your Laravel application.