ElasticSearch and ServiceStack

In this post,  I will show you how to integrate Full text searching using ElasticSearch and ServiceStack.

Options for Full Text Search

Before I dig into the steps required for ElasticSearch, I will mention some other options for implementing full-text in your solution:

  • Sphinx  – used by craigslist
  • Solr – used by CNet, Netflix, digg.com
  • ElasticSearch – used by Foursquare, Github
    (source: StackOverflow, thanks Tommy for highlighting use cases)
  • given that I use PostgreSQL, I could also just use FTS in PostgreSQL 

In the .NET. world, the above solutions have got .NET connectors. For

  • SpinxConnector.NET  – very good! Unfortunately it costs EUR199 for a single dev license. If it were free, I probably would have used this one.
  • SolrNET – is an active GitHub project with quite a few contributors. If you are looking to implement Solr in .NET, this is probably the way to go.
  • NEST  for ElasticSearch – this is the official high level ElasticSearch client for .NET. I chose to go with this option & there are some code examples below on how I implemented it.

Installing ElasticSearch in Windows

Summary:
1st step – install Java
& set the JAVA_HOME variable
2nd step – download & install ElasticSearch
3rd step – install NEST
as a NugetPackage in VisualStudio

Details on above:

1st step – install Java

a. Go to Java.com.

b. Click on the Free Java Download button, and then tap or click Agree and Start Free Download & install

c. in order for ElasticSearch to run, we will need to set the JAVA_HOME variable. Otherwise you will get an error ‘no JAVA_HOME variable set’

For setting the JAVA_HOME in Windows,

  • click on Start Button & Configure Java
  • Navigate to the [JAVA] Tab  & click on [View…]
  • Copy the Path (as per below screenshot) without the \bin\javaw.exeScreen Shot 2016-01-19 at 7.50.12 PM
  • In Windows 7,
    – Right-click the My Computer icon on your desktop and select Properties.
    – Click the Advanced tab, then click the Environment Variables button.
    – Under System Variables, click New.
    – Enter the variable name as JAVA_HOME.
    – Enter the variable value as the installation path as per above
    – Apply changes

Screen Shot 2016-01-21 at 7.30.12 AM.png

2nd step – download & install ElasticSearch

a. Download ElasticSearch in zip format from the official ElasticSearch homepage & unzip into a specified directory. (For me this is , C:\Projects\ElasticSearch)

b. To run ElasticSearch in a console app,
– go to directory (cd C:\Projects\ElasticSearch\bin)
– & run ElasticSearch in console (elasticsearch.bat)

Since ElasticSearch is itself a REST webservice (which by default runs on port 9200), you can now open a browser and go to localhost:9200 & be presented with something similar as per below.

Screen Shot 2016-01-21 at 7.33.25 AM.png

In the long run, you would want to run this as a Windows service. However, whilst we are developing, I will run this as a console app.

3rd step – install NEST in Visual Studio

ElasticSearch is a RESTful web service which exposes great JSON output. To make interacting with this service easier, I will work with NEST. This is the official high-level .NET client which can map our requests and responses to strongly typed objects. It comes with a powerful DSL which maps one-to-one with ElasticSearch. I will highlight a few code samples in the ‘Show me the Code’ sections.

Adding NEST to our project could not be easier. We can simply install it using Nuget package manager.

PM > Install-Package NEST

 

Show me the Code – Indexing Data

The above code,

  • firstly shows how we can take a Request Dto and using ServiceStack’s auto-mapping feature convert the requestDto to our Listing object.  (This is similar to Jimmy Bogard‘s widely used AutoMapper)
  • secondly, we persist our Listing object to our own database (in this case I am using PostgreSQL). [Again, ServiceStack’s ORMLite functions out-of-the-box with almost any well-known backend RDBS database, including SQL, MySQL,  PostgreSQL, Firebird, Oracle)
  • and finally, we get to the indexing part:
    – firstly we get a connection to the Elastic webservice
    – & then call .Index to index the object.
    Now we let ElasticSearch webservice do the magic. Indexing is as easy as that!

 

Show me the Code – Searching Data

I have to be honest, the ElasticSearch search syntax was not the easiest for me to understand. I found it best working with Sense (see screenshot below). Sense is a chrome extension (you can get it on the Google App store for free, which allows you to construct Elastic search terms & query your ElasticSearch webservice in a more raw format). Since NEST maps to the ElasticSearch DSL one-to-one, it is easier first to figure out & play around with constructing queries in this format. Once you are happy with what you want to achive, you can then simply construct them in NEST.

Screen Shot 2016-01-22 at 7.33.15 AM.png

Constructing Search requests in NEST & querying ElasticSearch

In the above code, I have

  • my ElasticSearchClient. This is injected at run-time using ServiceStack’s IOC container. (see code & how it is done by going to this stackoverflow post. [Note, I had some initial issues with this. So the post shows me asking the question & Demis (lead developer) answering that question within a day! (this is typical of SS support)]
  • I then show all the request parameters, which are part of the ‘GetListings’ request Dto. You will note that for most of these, if they are null, I am using the wildcard operator (*) to match to anything.
  • Thirdly, I start constructing the Search request. For me, it was easiest for breaking it out into separate parts & then constructing the SearchRequest in the end. (I feel it gives the best oversight for me).
  • Finally, I call Elastic.Search(searchRequest). You will note that pageNumber, pageSize, I can enter in the searchRequest. Also #totalRecords are returned from the ElasticSearch response object. Beautiful, this makes my paging functionality very easy!

 

Summary

In this post I managed to implement quick, relevant searches for end-users coming to search my Classifieds site (see reasons of building a Classifieds site here & my income reports of the site here) using ElasticSearch & integrating with the ServiceStack web services framework. ElasticSearch is easily extendable. Thus for now it may be overkill, but I know it will meet my future needs & was far easier to set up and go than I had anticipated. Kudos to the ElasticSearch team & ServiceStack for providing awesome frameworks.

Finally, for those looking to get future updates of my story of building this Classifieds site & sharing my monetization strategy (yip, I am hoping to make some money off the Classifieds site); then sign up here for my mailing list (takes you to my mailchimp link. Promise I will not ever spam!)

2 thoughts on “ElasticSearch and ServiceStack

    • Hi Deon, whilst AutoQuery is great & very quick to implement it would take quite a bit of effort to build in partial searches, relevant searches & boosting.

      Consider a user searches for a Ford Mustang 2011. Constructing this query in AutoQuery and more importantly returning relevant results would be quite difficul. eg consider Ford is in the Title in one Record, or mentioned multiple times in the description in another. These should get higher relevancy for search results.

      ElasicSearch & the others above make this much easier & that’s why you would go with such a solution .

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s