ElasticSearch and ServiceStack

In this post,  I will show you how to integrate Full text searching using ElasticSearch and ServiceStack.

Options for Full Text Search

Before I dig into the steps required for ElasticSearch, I will mention some other options for implementing full-text in your solution:

  • Sphinx  – used by craigslist
  • Solr – used by CNet, Netflix, digg.com
  • ElasticSearch – used by Foursquare, Github
    (source: StackOverflow, thanks Tommy for highlighting use cases)
  • given that I use PostgreSQL, I could also just use FTS in PostgreSQL 

In the .NET. world, the above solutions have got .NET connectors. For

  • SpinxConnector.NET  – very good! Unfortunately it costs EUR199 for a single dev license. If it were free, I probably would have used this one.
  • SolrNET – is an active GitHub project with quite a few contributors. If you are looking to implement Solr in .NET, this is probably the way to go.
  • NEST  for ElasticSearch – this is the official high level ElasticSearch client for .NET. I chose to go with this option & there are some code examples below on how I implemented it.

Installing ElasticSearch in Windows

Summary:
1st step – install Java
& set the JAVA_HOME variable
2nd step – download & install ElasticSearch
3rd step – install NEST
as a NugetPackage in VisualStudio

Details on above:

1st step – install Java

a. Go to Java.com.

b. Click on the Free Java Download button, and then tap or click Agree and Start Free Download & install

c. in order for ElasticSearch to run, we will need to set the JAVA_HOME variable. Otherwise you will get an error ‘no JAVA_HOME variable set’

For setting the JAVA_HOME in Windows,

  • click on Start Button & Configure Java
  • Navigate to the [JAVA] Tab  & click on [View…]
  • Copy the Path (as per below screenshot) without the \bin\javaw.exeScreen Shot 2016-01-19 at 7.50.12 PM
  • In Windows 7,
    – Right-click the My Computer icon on your desktop and select Properties.
    – Click the Advanced tab, then click the Environment Variables button.
    – Under System Variables, click New.
    – Enter the variable name as JAVA_HOME.
    – Enter the variable value as the installation path as per above
    – Apply changes

Screen Shot 2016-01-21 at 7.30.12 AM.png

2nd step – download & install ElasticSearch

a. Download ElasticSearch in zip format from the official ElasticSearch homepage & unzip into a specified directory. (For me this is , C:\Projects\ElasticSearch)

b. To run ElasticSearch in a console app,
– go to directory (cd C:\Projects\ElasticSearch\bin)
– & run ElasticSearch in console (elasticsearch.bat)

Since ElasticSearch is itself a REST webservice (which by default runs on port 9200), you can now open a browser and go to localhost:9200 & be presented with something similar as per below.

Screen Shot 2016-01-21 at 7.33.25 AM.png

In the long run, you would want to run this as a Windows service. However, whilst we are developing, I will run this as a console app.

3rd step – install NEST in Visual Studio

ElasticSearch is a RESTful web service which exposes great JSON output. To make interacting with this service easier, I will work with NEST. This is the official high-level .NET client which can map our requests and responses to strongly typed objects. It comes with a powerful DSL which maps one-to-one with ElasticSearch. I will highlight a few code samples in the ‘Show me the Code’ sections.

Adding NEST to our project could not be easier. We can simply install it using Nuget package manager.

PM > Install-Package NEST

 

Show me the Code – Indexing Data

The above code,

  • firstly shows how we can take a Request Dto and using ServiceStack’s auto-mapping feature convert the requestDto to our Listing object.  (This is similar to Jimmy Bogard‘s widely used AutoMapper)
  • secondly, we persist our Listing object to our own database (in this case I am using PostgreSQL). [Again, ServiceStack’s ORMLite functions out-of-the-box with almost any well-known backend RDBS database, including SQL, MySQL,  PostgreSQL, Firebird, Oracle)
  • and finally, we get to the indexing part:
    – firstly we get a connection to the Elastic webservice
    – & then call .Index to index the object.
    Now we let ElasticSearch webservice do the magic. Indexing is as easy as that!

 

Show me the Code – Searching Data

I have to be honest, the ElasticSearch search syntax was not the easiest for me to understand. I found it best working with Sense (see screenshot below). Sense is a chrome extension (you can get it on the Google App store for free, which allows you to construct Elastic search terms & query your ElasticSearch webservice in a more raw format). Since NEST maps to the ElasticSearch DSL one-to-one, it is easier first to figure out & play around with constructing queries in this format. Once you are happy with what you want to achive, you can then simply construct them in NEST.

Screen Shot 2016-01-22 at 7.33.15 AM.png

Constructing Search requests in NEST & querying ElasticSearch

In the above code, I have

  • my ElasticSearchClient. This is injected at run-time using ServiceStack’s IOC container. (see code & how it is done by going to this stackoverflow post. [Note, I had some initial issues with this. So the post shows me asking the question & Demis (lead developer) answering that question within a day! (this is typical of SS support)]
  • I then show all the request parameters, which are part of the ‘GetListings’ request Dto. You will note that for most of these, if they are null, I am using the wildcard operator (*) to match to anything.
  • Thirdly, I start constructing the Search request. For me, it was easiest for breaking it out into separate parts & then constructing the SearchRequest in the end. (I feel it gives the best oversight for me).
  • Finally, I call Elastic.Search(searchRequest). You will note that pageNumber, pageSize, I can enter in the searchRequest. Also #totalRecords are returned from the ElasticSearch response object. Beautiful, this makes my paging functionality very easy!

 

Summary

In this post I managed to implement quick, relevant searches for end-users coming to search my Classifieds site (see reasons of building a Classifieds site here & my income reports of the site here) using ElasticSearch & integrating with the ServiceStack web services framework. ElasticSearch is easily extendable. Thus for now it may be overkill, but I know it will meet my future needs & was far easier to set up and go than I had anticipated. Kudos to the ElasticSearch team & ServiceStack for providing awesome frameworks.

Finally, for those looking to get future updates of my story of building this Classifieds site & sharing my monetization strategy (yip, I am hoping to make some money off the Classifieds site); then sign up here for my mailing list (takes you to my mailchimp link. Promise I will not ever spam!)