Scaling ElasticSearch From 0 to 3 Billion :: Part 3 :: Way Way Too Many Types

In most languages types are your friends. You want types to describe the entity you're dealing with as clearly as possible. You make your base class Animal from which your Dog inherits from, then you make your SheepDog inherit from Dog and finally BorderCollie gets you something specific. (This is a terrible example, please don't quote me on this.)

Let's use a more realistic example. You have 50 different types of MyBaseEvent. When you index your types to ElasticSearch you think "I'll explicitly say these are type MyFirstEvent and MySecondEvent and so on." Don't.

Read More

Scaling ElasticSearch From 0 to 3 Billion :: Part 2 :: The 100% CPU Death Spiral of an ES Cluster

Watching your AWS cluster fluctuate nodes is about as bad as it gets. There's so little insight into why, you're stuck making a lot of uneducated guesses. You turn off your bulk indexing and hope that helps. You try and kill client connections and hope that helps. Neither did. It turns out it's taking it self down trying to recover. In a lot of information around ES they talk about keeping your node count and shard count as low as possible. What they don't really get too far into is why though.

Read More

Scaling ElasticSearch From 0 to 3 Billion :: Part 1

ElasticSearch is one of the most widely used full text search engines available to developers these days. It's ease of setup makes it an almost no brainier. But that ease can lull you in to a terrible false sense of security.

This series of posts will go into detail of all of the hard lessons learned as we at PlayFab built a cluster that went from 0 to 3 billion events.

Read More

Tuning EntityFramework :: Keeping the Context Alive to Save Overhead

In a lot of articles showing you how to get started with EntityFramework they have a good basis if you're using something like a desktop application where you can easily share the context object around your different classes. In the world of ASP.NET though things become a little more confusing.

Read More

Spicing Up The Required Field Notice

Whoever came up with the saying "There's only so many ways to skin a cat." clearly has never seen the innumerable form styles that the internet spits out every day. The problem is simple: we as developers/designers/people need to convey to a user, that we can't talk to, what they must put in for the site to give them the response they want. Why this simple thing is so hard to actually do probably stems from the fact that there's no skeuomorphism to fall back onto to train users with. There's no defined cue on a paper form to indicate "required" short of writing "required" next to the field name.

Read More

Paging Over Azure Table Results

After upgrading to Azure Storage 4 I noticed that a lot of the old methods for getting continuation tokens no longer work. I wasn't able to find a very clean work around so instead I cheated a little bit.

Read More

Inheriting Legacy :: Why We Can't Have Nice Things

Recently I've been tasked with moving a classic ASP site off of Server 2003 onto something a bit newer because the server is dying (not to mention being sunsetted soon).

While trying to figure out why the database refused to connect (jumping from SQL Server 2000 to SQL Server 2013 presents some interesting issues in a dead language) I found this awesome snippet of try.

Read More

How to Create a Clean Markdown Editor for ASP.NET MVC

For two different projects I found myself needing a nice full screen Markdown editor that works well in browsers. After not really finding a finished solution that I liked I ended up rolling my own with a combination of a few different systems.

I had a few requirements since I was also going to be a user of the editor, so here are the main features.

Main Features

  • Automatic/live preview
  • Ctrl + S to save
  • Code support via highlight.js
  • Unsaved changes notification.
  • Markdown help popup

markdown editor window preview

Read on to make your own!

Read More

Search On Azure :: Using Lucene.NET

Lucene.NET is a port of the popular Java search library Lucene ( And how very awesome it is.

Recently I was porting a SQL server database to SQL Azure, however the original database contained a full text catalog for the website's search functionality, something that SQL Azure does not support. You can get around this by using LIKE, however depending on how your index is built your performance could be terrible. LIKE '%{word}%' does NOT use the column's index on query, it does a table scan, which in SQL Azure is very costly (slow). LIKE '%{word}' and LIKE '{word}%' will use a column's index though so if your data is structured where it only needs to match the start or tail of the index's contents you will be in luck. Because of the drastic performance hit on doing a table scan for each search it was apparent we needed a new way to handle search, enter Lucene.

Read More