ElasticSearch and ServiceStack

In this post,  I will show you how to integrate Full text searching using ElasticSearch and ServiceStack.

Options for Full Text Search

Before I dig into the steps required for ElasticSearch, I will mention some other options for implementing full-text in your solution:

  • Sphinx  – used by craigslist
  • Solr – used by CNet, Netflix, digg.com
  • ElasticSearch – used by Foursquare, Github
    (source: StackOverflow, thanks Tommy for highlighting use cases)
  • given that I use PostgreSQL, I could also just use FTS in PostgreSQL 

In the .NET. world, the above solutions have got .NET connectors. For

  • SpinxConnector.NET  – very good! Unfortunately it costs EUR199 for a single dev license. If it were free, I probably would have used this one.
  • SolrNET – is an active GitHub project with quite a few contributors. If you are looking to implement Solr in .NET, this is probably the way to go.
  • NEST  for ElasticSearch – this is the official high level ElasticSearch client for .NET. I chose to go with this option & there are some code examples below on how I implemented it.

Installing ElasticSearch in Windows

Summary:
1st step – install Java
& set the JAVA_HOME variable
2nd step – download & install ElasticSearch
3rd step – install NEST
as a NugetPackage in VisualStudio

Details on above:

1st step – install Java

a. Go to Java.com.

b. Click on the Free Java Download button, and then tap or click Agree and Start Free Download & install

c. in order for ElasticSearch to run, we will need to set the JAVA_HOME variable. Otherwise you will get an error ‘no JAVA_HOME variable set’

For setting the JAVA_HOME in Windows,

  • click on Start Button & Configure Java
  • Navigate to the [JAVA] Tab  & click on [View…]
  • Copy the Path (as per below screenshot) without the \bin\javaw.exeScreen Shot 2016-01-19 at 7.50.12 PM
  • In Windows 7,
    – Right-click the My Computer icon on your desktop and select Properties.
    – Click the Advanced tab, then click the Environment Variables button.
    – Under System Variables, click New.
    – Enter the variable name as JAVA_HOME.
    – Enter the variable value as the installation path as per above
    – Apply changes

Screen Shot 2016-01-21 at 7.30.12 AM.png

2nd step – download & install ElasticSearch

a. Download ElasticSearch in zip format from the official ElasticSearch homepage & unzip into a specified directory. (For me this is , C:\Projects\ElasticSearch)

b. To run ElasticSearch in a console app,
– go to directory (cd C:\Projects\ElasticSearch\bin)
– & run ElasticSearch in console (elasticsearch.bat)

Since ElasticSearch is itself a REST webservice (which by default runs on port 9200), you can now open a browser and go to localhost:9200 & be presented with something similar as per below.

Screen Shot 2016-01-21 at 7.33.25 AM.png

In the long run, you would want to run this as a Windows service. However, whilst we are developing, I will run this as a console app.

3rd step – install NEST in Visual Studio

ElasticSearch is a RESTful web service which exposes great JSON output. To make interacting with this service easier, I will work with NEST. This is the official high-level .NET client which can map our requests and responses to strongly typed objects. It comes with a powerful DSL which maps one-to-one with ElasticSearch. I will highlight a few code samples in the ‘Show me the Code’ sections.

Adding NEST to our project could not be easier. We can simply install it using Nuget package manager.

PM > Install-Package NEST

 

Show me the Code – Indexing Data

The above code,

  • firstly shows how we can take a Request Dto and using ServiceStack’s auto-mapping feature convert the requestDto to our Listing object.  (This is similar to Jimmy Bogard‘s widely used AutoMapper)
  • secondly, we persist our Listing object to our own database (in this case I am using PostgreSQL). [Again, ServiceStack’s ORMLite functions out-of-the-box with almost any well-known backend RDBS database, including SQL, MySQL,  PostgreSQL, Firebird, Oracle)
  • and finally, we get to the indexing part:
    – firstly we get a connection to the Elastic webservice
    – & then call .Index to index the object.
    Now we let ElasticSearch webservice do the magic. Indexing is as easy as that!

 

Show me the Code – Searching Data

I have to be honest, the ElasticSearch search syntax was not the easiest for me to understand. I found it best working with Sense (see screenshot below). Sense is a chrome extension (you can get it on the Google App store for free, which allows you to construct Elastic search terms & query your ElasticSearch webservice in a more raw format). Since NEST maps to the ElasticSearch DSL one-to-one, it is easier first to figure out & play around with constructing queries in this format. Once you are happy with what you want to achive, you can then simply construct them in NEST.

Screen Shot 2016-01-22 at 7.33.15 AM.png

Constructing Search requests in NEST & querying ElasticSearch

In the above code, I have

  • my ElasticSearchClient. This is injected at run-time using ServiceStack’s IOC container. (see code & how it is done by going to this stackoverflow post. [Note, I had some initial issues with this. So the post shows me asking the question & Demis (lead developer) answering that question within a day! (this is typical of SS support)]
  • I then show all the request parameters, which are part of the ‘GetListings’ request Dto. You will note that for most of these, if they are null, I am using the wildcard operator (*) to match to anything.
  • Thirdly, I start constructing the Search request. For me, it was easiest for breaking it out into separate parts & then constructing the SearchRequest in the end. (I feel it gives the best oversight for me).
  • Finally, I call Elastic.Search(searchRequest). You will note that pageNumber, pageSize, I can enter in the searchRequest. Also #totalRecords are returned from the ElasticSearch response object. Beautiful, this makes my paging functionality very easy!

 

Summary

In this post I managed to implement quick, relevant searches for end-users coming to search my Classifieds site (see reasons of building a Classifieds site here & my income reports of the site here) using ElasticSearch & integrating with the ServiceStack web services framework. ElasticSearch is easily extendable. Thus for now it may be overkill, but I know it will meet my future needs & was far easier to set up and go than I had anticipated. Kudos to the ElasticSearch team & ServiceStack for providing awesome frameworks.

Finally, for those looking to get future updates of my story of building this Classifieds site & sharing my monetization strategy (yip, I am hoping to make some money off the Classifieds site); then sign up here for my mailing list (takes you to my mailchimp link. Promise I will not ever spam!)

Simulate Human Typing with jQuery in less than 5min

simulateTyping.gifMy Classifieds landing page looks a bit boring. I need to add a cool factor & provide a ‘something is happening here’ feeling for new visitors. So how about simulating someone typing a few popular searches, such as ‘looking for a car’ or ‘looking for an old laptop’.

I found just the right library for it & it can be implemented in less than 5min. It is a free jquery library called Typed.js which simulates typing. [The library is written by Matt Boldt (see his site here, Github here)]

Got it implemented in less than 5 minutes. Herewith the steps needed

  1. Reference the library & configure its usage 
  2. Reference the Html element.

Adding Facebook Authentication using ServiceStack

I am building a Classifieds site and will be sharing income reports and stats once launched  (click here to sign up to my mailing list)

In this post, I will be sharing how I added Facebook authentication to my website using ServiceStack. One users are logged in, they are able to post classified ads.

loginFaceBook

How to add Facebook authentication using ServiceStack

Facebook offers OAuth integration for developers to allow users to log in to external applications using their Facebook account. ServiceStack has a done the hard work for us already for Facebook auth integration (others include Twitter, Google) & all we have to do is plug it into our app. These are the steps:

Steps:

Registering with Facebook

  1. You will need to register with Facebook as a developer
    – go to https://developers.facebook.com & click on register
  2. Click on Add a new App & in my case I will add a new WebsitefaceBookAddANewApp.png
  3. Since my Site is not launched yet, I elect the [create new AppId] & choose a unique name for it.
  4. Take note of the AppId and AppSecret,which we will need for our configuration Files
    & (importantly) you will also need to add your Website URL by clicking on [Add Platform]. (Otherwise you receive an ugly error)
    faceBookAppIds.png

Configuring Facebook Authentication in our ServiceStack app

  1. We will need to add the AuthFeature plugin. This allows multiple auth providers (Twitters). We will add FacebookAuthProvider to this list.
  2. You can provide the AppId, AppSecret (as provided by Facebook) in code (as per below) or via Web.Config
  3. Redirect, Callback Urls need to be set. These are used by ServiceStack’s OAuth & include a callback URL as a query string to the Facebook.
  4. Set permissions. By default you will have access to the user’s public profile, list of friends & email. You can set additional permissions (note these have to be approved first by Facebook) using Permissions. [For a full list of Facebook permissions go to https://developers.facebook.com/docs/facebook-login/permissions]

In our website, we will create a login button, which is shown when the user is not authenticated. If authenticated, we will show the user’s default profile.

Login Button shown if not authenticated

Screen Shot 2016-01-14 at 6.57.55 AM

Profile Image & Logout shown if authenticated

Screen Shot 2016-01-14 at 6.57.18 AM

 

 

Finally, lets require [Authenticate]ion for specific user actions

In my Classifieds website, I only want to allow authenticated users to post Classified ads. This is easily done with ServiceStack. I can just decorate the necessary method or Dto with [Authenticate].

Digging into some data: behind the scenes of AuthFeature,

AuthFeature uses IUserAuthRepository to store user authentication details. We can use the standard OrmLiteAuthUserRepository class (as in the above code). This will create the tables as per below [I am using PostreSQL as the backend. Again this works seamlessly with ServiceStack’s ORMLite].

OrmAuthTables

ormAuthTables.png

Pagination using ServiceStack, BootStrap & jQuery

In this post, I’ll explain how I introduced paging into my Classifieds project. There are a number of different ways to accomplish this, including writing your own jQuery plugin.

However, where possible, I prefer to take some shortcuts as I am in need of a quick solution & there are some great / well tested plugins out there. So for my Bootstrap classifieds site, I’ll be using

  • ServiceStack – (this is my go-to webservice framework for this entire Classifieds project). This will return the resulting list of records. Importantly, it also includes total page count, current page number which we can make use of on the client side.
  • jQuery Pagination plugin (optimised for Bootstrap) – is a well-written jquery pluging which I will use for the client-side pagination [ link to GitHub project]

paging

API – Reqest and Responses for Paging queries

The first thing I’ll do is write the necessary request (namely, GetListings) and response (GetListingsResponse) objects.

Thus an example request would use the api route ~/listings?CategoryId=1&PageNr=1, which returns the first page of listings within categoryId =1. This happens to be a Category of ‘Houses for Sale’ in my case]

Messages – Request Parameters as properties
Warning, here comes a tangent (skip this passage if you are only interested in the paging solution).

ServiceStack promotes a message based approach. In my case, I know there will be additional future request filters/parameters I will want to add such as (e.g. CarMake). Using this message based design, these additional request parameters can simply be added as properties. This promotes a very extendable & clean code base without breaking any existing code.

Compare this to to the more typical approach of having to amend the signature of a method (a typical method with request parameters is given below). Adding new parameters can get annoying quickly in these cases, as it requires modification of the existing method signature! (for more on this topic within an MVC project, see Jimmy Bogard’s post)

hmmmmphh – much uglier.

jQuery Pagination Plugin

Installing the plugin:
You will need to download the jquery file from their homepage & then reference it in your page.

http://~/scripts/jquery.twbsPagination.min.js

Configuring the plugin:

From the plugin’s homepage we can see that we can configure the plugin for a number of things. For my purposes, I will be using the response of the GetListings request & make use of the TotalPageCount & CurrentPageNr to pass into the jquery options function.

$(‘#pagination-control’).twbsPagination({
totalPages: @Model.TotalPageCount,
visiblePages: 5,
startPage: @Model.CurrentPageNr,
href: ‘?’ + params + ‘&PageNr={{number}}’
});

 

Setting the Page url using href propery

I will use the href option to pass the page url (corresponding to my listings route). I want to pass all the page’s current request parameters (e.g. the CategoryId) as well as pass the newly selected PageNr to the route. Not being a Javascript guy (!!), I ended up with a bit of an ugly implemetion for this. [If anyone has some cleaner code or better suggestion, please leave them in the comments]

Uploading Images – ServiceStack and Dropzone

I am building a Classifieds site (see list of reasons here).

In this post, I will be showcasing how I used DropZone (an awesome open-source library for drag-n-drop functionality) and ServiceStack (as our back-end service for uploading, resizing & storing the files) in providing users with a drag and drop facility for uploading their images. Once images are dropped, the service handles the resizing of images & saving of images onto the server.

uploadingImages_3

Installing & configuring DropZone
Installing dropzone
– You can install it using Nuget. Install-Package dropzone.
– Alternatively, you can download the latest release packages here.

The most important file is the dropzone.js. You will need to reference this in your code if you want to make use of it as per below. I suggest also referencing dropzone.css to make use of Dropzone’s awesome styling.

Dropzone configuration options
There are multiple configuration options available & DropZone makes it really easy.
For the Classifieds site, I wanted
– a maximum file size of 3Mb
– 10 files maximum per post,
– limiting parallel uploads to 1. This  prevents memory stream being accessed by another image before the image is finished [if there is another way of handling this, please provide feedback in the comments]

Finally,
– on successfully uploading, I want to pass the saved file location’s url to a hidden input field (“Files”).  Thus when a user posts the form, I can link the image URLs to the classified ad.

Again, Dropzone makes this very easy by listening to events. In this case the ‘success’ event.


Using ServiceStack for uploading, resizing & storing images
Dropzone requires a server-side implementation to upload, resize & restore images. I will be using ServiceStack for this.

Firstly, you notice that the dropzone element includes an action parameter.  “/images/upload/” This is the Servicestack api route which will handle the upload.

Now, the server-side code for ServiceStack. We first need to create the Request and Response DTOs . For those familiar with WFC or WebAPI, note how much simpler this is (a full description on differences is given in this stackoverflow post [new window]). The response DTO includes a simple URL which is the image’s saved location. (you can access response properties from DropZone – see DropZone on success script above)

The upload, resizing and storing is all housed in the ImageService (this code has been adopted and marginally changed from http://imgur.servicestack.net/. Thanks Demis). Specifically,
– I wanted uploaded files to receive a unique name. Thus brought in the RandomString(int length) method when creating a unique file name. Thanks dtb @stackoverflow.
– my ResponseDTO includes a URL (giving the image’s filename), which can be used by DropZone

 

Coming up in future posts:

  • I will make a decision between deploying to Azure or AWS
  • I will launch the actual Classifieds Site and use Facebook Ads & share the results
  • More cool ServiceStack features

Get notified

New Project: Classifieds

“Noooo! Why another classifieds site? Aren’t they evil, ugly and well boring.”
This was more or less the answer my wife gave me, after I told her that I would be working on a classifieds site in my spare time [drama added].

Hmm. So why am I doing a Classifieds site?
Reason numero uno: Namibia (that is a country, for those of you who don’t know) does not have a Classifieds site
Well that’s not completely true. There are online Namibian classifieds sites out there, but they are ugly. I think Namibia deserves a pretty Classifieds site. Yay for that
Reason numero duo: it’s been on my to-do list for five years after friends & family have asked me to do it. Their reason. See reason number 1.
Additionally, it’s not so much about building a boring classifieds site. Rather I will be using some cool tools I want to experiment with in helping me build this site. I think you can make the most boring of projects/tasks interesting, by simply striving for awesomeness. ServiceStack, a web services framework, is helping me making this journey really enjoyable. [See all the [Tools] and Resources I am making use of]
Earning money online: I am hoping to earn some money from this venture. I will be sharing income reports once I launch the site (inspired by Mr Pat Flynn )

I aim to do all this by sharing my work online (thank to Derek Sivers for the inspiration).
– the source code is open & hosted on github.com [link to be added]
– I will share milestones using this blog
– share stats & income reports of the actual Classifieds Site [once I launch it]

Sign up to my mailing list to stay updated