March 11, 2015

2 Easy Ways to Display Lots of Data Faster in a Rails App

Rendering views is almost always the slowest part of a rails app. The more data you need to display, the more rendering needs to take place, the slower the app loads. This post is going to show you how to use a couple tools to display lots of data in a rails app in a faster way.

In our app, bookacoach (A site that helps sports academies manage all their training online), I need to display a lot of data in 2 forms. 1. Data in a table form. 2. Data in a calendar form.

Speeding up pages with lots of data

A great way to speed up the loading of pages with lots of data is to let everything do what it’s good at. Databases are really good at storing, searching, and sorting. So lets’ have the database handle that work (through our ORM, ActiveRecord). Rails is really good getting data from the database to the views through a controller, so let it do that. jQuery in conjunction with AJAX is really good at loading data from a source and then changing the data the user sees, so we’ll use it for that.

For both our needs, displaying lots of data in a table form, and displaying lots of data in a calendar form, there happen to be some awesome pre-built tools I can implement.

1. Displaying data in table form

Quite aptly named, DataTables, is a jQuery plugin that is great for displaying, searching and sorting data in a table form. DataTables can load a JSON data source using AJAX, so it’s perfect for what I want to do. There’s even a great little wrapper for DataTables’ AJAX functionality in rails again aptly named ajax-datatables-rails, and that’s what I used in our app.

The ajax-datatables-rails gem is pretty straightforward to use. You can read the docs to see how you would set it up for a DataTable displaying data from a single model. It’s a great little gem because it uses the database to do all the searching and sorting, something the database is very good (and fast) at.

In our app, I needed to display a lot of different models (different kinds of trainings) all on the same DataTable and still be able to sort, search, and paginate the data. What I ended up doing was creating a TrainingSearchEntry model. Each TrainingSearchEntry object shares the same attributes, even though the attributes/methods in each of the different training models may be different. Each training has_one: training_search_entry that is created or updated each time a training is created or updated. This allowed me to have a standard interface for my DataTable to search and sort, even if the trainings backing the data are all vastly different. The end result is something like this:

A couple tricks learned along the way

Correctly sorting single-day events with multi-day events

In our app there basically two different categories trainings can fall into, time-wise. Either a single-day event (e.g. May 3rd 2PM - 3:30 PM) or multi-day event (e.g. May 3rd 2PM - 3:30 PM, May 10th 2PM - 3:30 PM, and May 15th 1PM - 2PM) You’ll notice one of the columns in my DataTable is “Date & Time”. So how do you sort in the database a bunch of different trainings with their own models based on their date and time when some of them have only one date and time and some have multiple? This is one of the reasons I decided have a dedicated TrainingSearchEntry model. When building the date_and_time value for any given TrainingSearchEntry object, I have code that creates the time in this format sortable-datestring|Human Readable Datestring. So for example, a lesson that happens on Sat, 04/11/15 12:30 PM - 1:00 PM would have a date_and_time of this "201504111230|Sat, 04/11/15 12:30 PM - 1:00 PM EST". A multi-day training might have one that looks like this "201504021045|Thu, 04/02/15 - Thu, 04/09/15". As you can see, the earliest date is put in a sortable format before the "|" character.

Then in my app/datatable/training_datatable.rb I have some handy little code that splits that string up so the sure doesn’t see the weird string used for sorting in the database, but instead only sees the human readable part.

def build_hidden_search_and_sort_string(string)
  if string
    array = string.split('|')
    "<span class='date-sort-span'>#{array[0]}</span>#{array[1]}"
  else
    ''
  end
end

The class date-sort-span is then hidden with CSS. This allows DataTables to sort by date correctly.

Filtering based on training type.

Creating filters also takes advantage of hiding the text on the left side of a "|" character. Each TrainingSearchEntry has a title attribute that is something similar to this is: some categorical description|Title of training. For example a camp’s title is setup like this "is: camp not day|Camp #{training.title}". I use the same build_hidden_search_and_sort_string method from above to hide the text to the left of the "|" character and show the text on the right to the user.

To make the filters work with DataTables is a simple as searching for the required text (e.g. “is: camp not day”) when the filter is clicked. That’s done with this JavaScript:

// Get the instance of the DataTable
var table = $("#scheduled-training-table").DataTable();

// On click of the filter, search for the phrase, then draw the results on the table
$( "#camp-filter" ).click(function() {
  table.search('is: camp not day').draw();
});

Loading indicator

By default, DataTable shows some small text when loading that is pretty easy to miss. I wanted to give the users a harder to miss indicator that something is going on while their data is loading. It also makes the whole process seem even faster because there’s no “Hey I clicked that column to sort, but nothing is happening.” It gives the user a visual cue that something is happening and tells them when it’s done.

Luckily, DataTable also gives you a really easy class to hook into when it’s loading. It unhides a div with the class .dataTables_processing anytime it’s loading. From that, it was easy to just add on the awesome CSS-only spinner I got here

2. Displaying data in calendar form.

FullCalendar is a great jQuery plugin for displaying events on a calendar. It’s very versatile and easy to use. You can get it wrapped up for rails in this gem. I also got some inspiration and direction from reading the source of the fullcalendar-rails-engine. I didn’t need all that it had to offer since I already had my own way of managing events and times. It did however get me started on using the AJAX functionality of FullCalendar.

FullCalendar handles JSON feeds easily http://fullcalendar.io/docs/event_data/events_json_feed/. So all I needed to do was give it some events in JSON to put on the calendar.

It’s as easy as giving the url that hits this controller method as the event source to FullCalendar:

def get_time_slots
  start_at = parse_time_param(params[:start])
  end_at   = parse_time_param(params[:end])

  collect_time_slots(start_at, end_at)

  formatted_time_slots = build_formatted_time_slots
  render json: formatted_time_slots.to_json

end

To get the time_slots (the model used in our app to represent anything dealing with time) in a certain range, I pushed the work down to the database using this scope:

scope :in_time_range, -> (start_at, end_at) {where('
      (start_time >= :start_at and end_time <= :end_at) or
      (start_time >= :start_at and end_time > :end_at and start_time <= :end_at) or
      (start_time <= :start_at and end_time >= :start_at and end_time <= :end_at) or
      (start_time <= :start_at and end_time > :end_at)',
      start_at: start_at, end_at: end_at)}

Then once I had the time_slots collected, I formatted them so that FullCalendar could understand them using a method similar to this:

def build_formatted_availability_time_slot(raw_time_slot)
  {
    id: raw_time_slot.id,
    start: raw_time_slot.start_time.in_time_zone(get_time_zone).iso8601,
    end: raw_time_slot.end_time.in_time_zone(get_time_zone).iso8601,
    url: edit_time_slot_path(raw_time_slot.id),
    title: 'Lesson Availability',
    textColor: 'white',
    color: '#31b0d5'
  }
end

Finally I just render that result as JSON and FullCalendar handles the rest of the work of actually showing the events on a calendar. This process again allows the user to see all of the events on their calendar as far back or as far forward as they want without ever loading more than a month at any one time. This keeps the pages very fast, as opposed to loading all the events at once.

Loading indicator

FullCalendar doesn’t having a loading indicator built in, but it’s really easy to add one. All you have to do is add a div right above your calendar like this:

<div id="calendar-loading" style="display: none;"></div>

and then add this to your FullCalendar JavaScript:

loading: function(bool) {
  if (bool)
    $('#calendar-loading').show();
  else
    $('#calendar-loading').hide();
}

Now you have a div that shows when the FullCalendar is loading, and hides when it is not. I just use that same CSS-Spinner as my loading indicator.

Conclusion

I needed to display thousands of rows of data in a table and thousands of events on a calendar. I tried using caching but the first (uncached) load of the page was still too expensive with that amount of data to display. Pushing all of the sorting and searching to the database and then using AJAX and jQuery to render the data helped speed up our pages incredibly.

So if you need to display data in table form, check out DataTables and the ajax-datatables-rails gem. If you need to display data in calendar form, check out FullCalendar and its AJAX functionality.