LayerVault Simple version control for designers. Try it free.

Rails in Realtime

Update: We’ve written a complementary post that goes into more depth. Read this first and then check out Rails in Realtime, Part 2.

LayerVault is built using the popular web framework, Ruby on Rails. The framework, at times known for its divisiveness, has allowed LayerVault to grow from a single box to a swarm of machines over the past year. Recently, there has been a wave of great JavaScript-based frameworks that make creating a “realtime” app a cinch. Because LayerVault is a perfect facsimile of a team’s filesystem, having the web interface update immediately is incredibly important. Every ⌘R is two key presses too many. Finder doesn’t require you to refresh every time a file changes, why should a web app? We’re happy with Rails, so transitioning to Meteor, Ember.js or Backbone.js whole hog isn’t the right move for us.

The phrase “realtime” is thrown around about as much as “local” and “disruptive” these days, so the phrase is often ambiguous. For the purposes of this post, a “realtime” app is the following: page refreshes are not required to see the most up-to-date state of information and new information is communicated in tens of milliseconds instead of hundreds or thousands.

This blog post will cover some of the patterns that we use to allow LayerVault customers to never worry about pressing ⌘R. This is all done by using vanilla *.html.erb templates and never rendering any parts of the page using JavaScript. Once a database record is changed, that change appears in <500ms in a customer’s browser.

Introduction

The way LayerVault handles realtime data hinges upon a few assumptions. They are as follows:

  • Templates should only be defined once. The last thing we want is an ERB template and a Mustache/Handlebars/Hogan template for the same piece of information.
  • JavaScript is never used to build a template. It’s the wrong tool for the job.
  • Realtime updates should be treated as volatile. That is, it’s not the end of the world if a page does not update in realtime.

Today, we use the following technologies to get this done:

  • Rails 3.2.8
  • Socket.IO v0.9
  • jQuery 1.8.1
  • Dalli gem
  • memcached
  • localStorage
  • timeago
  • delayed_job

Socket.IO

To communicate that new information is available, we went with the Socket.IO library and haven’t looked back. Socket.IO is a NodeJS library designed for realtime communication with graceful fallbacks. The client browser doesn’t have Websockets? Fall back to polling.

Recent versions of Socket.IO have a concept of rooms that we use liberally. The pub-sub model is nothing new, but Socket.IO is the first library that I’ve used that’s made it mostly painless.

Each client, whether it be a browser or a native application, subscribes to all of the rooms it cares about. If you are visiting the page for a certain folder, your in-browser Socket.IO client could subscribe to changes to folder/42, where 42 is the folder’s ID.

Alerting the concerned parties about changes then becomes a piece of cake. When a change is registered on the web app side, a Rails Observer sends off a volatile message to the appropriate room. It’s then up to the clients to decide whether or not to pull new information.

Scheduling messages

We use Rails Observers to monitor specific models for changes. Once a model is changed or is touched, we queue up a new realtime event:

class FileObserver < ActiveRecord::Observer
  observe :lv_file

  def after_save(record)
    record.delay.report_updated
  end
end

Our code pops off a new delayed_job job to connect to the realtime server and send the message. We offloaded this into a queued job because the Socket.IO machine is a separate service, and if it goes down it shouldn’t prevent writes from happening on the web application.

Russian Doll Caching, all the way down to the client

Russian Doll Caching has been a godsend for web developers and Rails 3.0 makes it easy to implement. (David Heinemeier Hansson has a great post about it for the unfamiliar.) This, coupled with the Dalli gem to communicate with memcached, makes intelligent caching easy.

Russian Doll Caching hinges on the idea that cache keys should be generated based upon the model’s id and its updated_at. The data that is cached is always generated HTML. The cache fragments are often nested, hence its namesake. Sometimes we’ll even mix in a view context, e.g. “Directory Page,” but many times a well-constructed partial won’t require such a context.

To add on this idea, we’ve found it also beneficial to make the clients use these same cache keys. For example, The LayerVault Activity Feed uses localStorage to store HTML snippets under the exact same cache keys as the backend uses. When a model changes, a message is published in the appropriate Socket.IO room with the new cache key. If that key is not available via localStorage, a new request is shot off.

Caching HTML often causes some nose-scrunching from hardened veterans. If one is not careful, you can still issue far too many database requests. Because LayerVault’s data structure is a tree (“users have folders that have files that have versions”), we can often short-circuit entire requests if the first cache key is a hit:

def index
  render if Rails.cache.exist?(['Directory Page', current_user])

  # The normal request starts here.
end

Phone home attributes

A single Directory Page or File page in LayerVault has many different moving parts. Even if a file gets updated deep in a folder hierarchy, that change is manifested at many different levels, e.g. “My Project changed a few seconds ago.” To keep us from worrying about updating multiple disparate parts of the page in unison, each section can have a “phone home” data-* attribute. A phone home attribute is the location where the latest information for that section can be found.

For example, this is the code that represents a subfolder count in a project:

<div class="SectionCount" data-url="/folders/count">
  <%= pluralize(@folders.size, 'folder') %>
</div>

We choose to represent our phone home attributes with a data-url attribute. Once Socket.IO tells us (because we are listening to the appropriate room) that something has changed, updating these folder counts is trivial:

$('.SectionCount').each(function () {
  var $section = $(this);

  $.get($section.attr('data-url'))
  .success(function (response) {
    $section.html(response);
  });
});

This pattern provides a consistent way to update only the relevant parts of the page the millisecond they change.

To cut down on flashes caused by replacing large parts of the page, we will sometimes intercept these requests and use a few of our Cosmos plugins to gracefully transition the data to its new state.

The Road Ahead

"LVJAX"

PJAX is a great tool for further speeding up a web application. It greatly cuts down on the information sent across the wire, without having to go full-on client-side MVC.

After playing around with PJAX for a few days, it falls short of exactly what we need it to do. The problem lies within an assumption carried over from ye olden days of browsers: pressing the back button returns the previous page in its previous state. This is fine if you are navigating a series of static pages that don’t change often. When you have a more complicated application with many parts updating in realtime, this solution is insufficient.

LayerVault needs the ability to conditionally expire previous browser states. Thus, a URL becomes less of a pre-defined resource (i.e. “a page that I was previously on”) and more a window into room. A change that a teammate makes to a file should be reflected when pressing the back button. Page refreshes should only be made when required.

In the near future, we will augment PJAX with this idea in mind.

Cache digests

Rails 4.0 will introduce the idea of cache digests. You can read up on it on its GitHub page. The current cache key generation gets developers 95% of the way, but they run into issues when updating templates or when dependencies change. The current method of generating a cache key is pretty dumb, albeit quite effective.

Cache digests represent a method of generating a cache key based on the digest of a template and its dependencies. It makes the task of key-based cache expiration even more brainless. That’s a great thing.

This slightly complicates the task of keeping a the localStorage cache keys matched up with the server-generated cache keys, but it will be a solvable problem.

Conclusion

Building a realtime web application does not require you to begin with a realtime framework. At its core, LayerVault is a simple CRUD app. Data persistence is more important than data availability. By structuring an application in an intelligent way, a traditional framework with some realtime mixins delivers an amazing user experience.

If you found this post interesting, you should following LayerVault on Twitter or check out some of our open-source projects.

We’re happy to answer any question on Hacker News.

Kudos to Geoff Stearns and Michael Hartl for catching typos.

  1. tech-aficionado reblogged this from layervault
  2. namitc reblogged this from layervault
  3. turing-machine reblogged this from layervault
  4. technotalk reblogged this from layervault
  5. technicalpicklejar reblogged this from layervault
  6. livercake reblogged this from jonpaullussier
  7. rekidetiar reblogged this from jonpaullussier
  8. jonpaullussier reblogged this from layervault
  9. ox86 reblogged this from layervault
  10. scottchiang reblogged this from layervault
  11. evandrix reblogged this from layervault
  12. carloscastro reblogged this from layervault
  13. kellysutton reblogged this from layervault and added:
    handle terabytes...changes reflected...here: Rails in...
  14. layervault posted this