Heartbleed Vulnerability

On April 7th, the “Heartbleed” vulnerability was disclosed and a corresponding patch was released. Onehub, with many others, was potentially vulnerable to the exploit; though we have no reason to believe any data was compromised. Within 24-hours all affected services were patched, and all of our SSL certificates were revoked as a precautionary measure. Onehub Enterprise is not vulnerable.

In early 2013 we began to offer Perfect Forward Secrecy to users running modern browsers which further mitigates any risk by preventing retroactive decryption of private traffic. We will continue to closely monitor the security and privacy of our customer’s data and will react swiftly against new threats that may emerge. Please contact us at support@onehub.com if you have any additional questions or concerns.

Firesheep: What It Is And Why You Should Be Scared

Recently, there has been a lot of discussion surrounding the Firefox add-on Firesheep. Firesheep is a tool that makes it easy for its user to assume (hijack) the identity of other users on a number of popular websites. I want to take a moment to explain not only how Firesheep works, but how when you are using the Onehub site, we protect you from the vulnerabilities that Firesheep exposes.

The Internet, and the protocol it is primarily built on, HTTP, are inherently stateless, meaning that by default no information is kept on either the client or server between requests. In other words, if you request the homepage from a website, then the product page, the website has no way of knowing that these two requests came from the same person, thereby making things like user logins and shopping carts impossible to track. In order to work around this limitation, and provide the seamless experience we have all come to expect, websites place a small piece of data on each user’s computer: a cookie. These cookies (specific to each website) are sent back to the website with every browser request, and they can use the information contained therein to recognize that all of your requests are coming from you, and you alone. In essence, once you are signed-in, you are your cookie. Cookies have a bad reputation amongst some users, however they are an indespensible piece of technology, and without them the modern Internet would not function.

An Example

Let us use a simple story. Charles would like to visit onehub.com to check the latest activity in his Workspaces.

Charles opens his web browser, Mozilla Firefox, and types onehub.com into the location bar. On his behalf, Firefox sends a request to the onehub.com servers for the homepage. In a few milliseconds the server replies with the HTML, CSS, JavaScript and images that make-up the onehub.com homepage. Firefox takes this data and renders onehub.com in Charles’ browser. So far so good. In order to see the latest activity, Charles will need to sign-in, and then visit the activity tab. So he clicks the “Sign In” button on the top-right of page and again, Firefox sends a request for a webpage to the onehub.com servers, but this time for the sign-in screen. This is where it gets interesting.

The sign-in screen is secured via SSL/TLS (indicated by the S in the HTTPS in Charles’ browser). This ensures that the onehub.com server really is who it claims to be, and encrypts all the data exchanged between Charles’ computer and the server. Confident that his email and password will not be exposed to the world, Charles fills out the form and presses enter. With the email and password in hand, the server does a few checks. It first looks in the database to see if a user with Charles’ email exists, if this is the case, it checks the password for that account against the password Charles typed in earlier. Provided both of these pieces of data match, the server replies back with the requested data, Charles’ home screen. In addition, the server provides a small piece of data in a cookie, a session identifier. This is a randomly generated string of characters, something like “a74822fdbce86b1541fec9c1f92d366f”. In addition to sending this identifier to Charles’ browser in a cookie, it is also stored by the onehub.com servers. In fact, if you are signed in to onehub.com right now you may look in your cookies for _onehub_session_id which will have your unique session identifier.

The session identifier functions as a stand-in for the user’s credentials, it saves Charles from having to provide this information on every single request.

Now that Charles is on the homepage, he clicks on the activity tab. Firefox again requests a specific webpage from the onehub.com servers, but this time it also provides the data in the cookie, the session identifier.

The server accepts this request, and looks up the session identifier in the database. The database provides two pieces of information to the server about this particular identifier. First, if this identifier was created recently enough (you may extend the length a session identifier is valid by choosing ‘Keep Me Signed In’), and second, that it belongs to Charles. The server can now reply with the data for Firefox to render Charles’ activity page.

This type of data exchange happens millions of times per-second across the Internet and it is this same technique that lets you do everything from sharing a photo on Flickr to keeping a shopping cart on Amazon.com.

So how does Firesheep work?

It is actually relatively simple. Most websites (onehub.com included) secure their sign-in forms with SSL/TLS. This means your email or username and password are encrypted when sent to the server. However, once a user is signed-in these sites do not encrypt the data exchanged between the browser and the server (onehub.com does). Firesheep watches all the traffic flying around on the network it is running on, like the coffeeshop which Charles is working from this morning. It listens specifically for the session identifiers for many popular websites including Facebook and Twitter. Firesheep then provides an easy way to change Charles’ session identifier to another one that Firesheep has found on the network. Charles is transparently able to assume (hijack) someone else’s identity.

How is Onehub secure?

We do two things differently than the majority of sites. First, we encrypt all traffic, all the time. This makes the session identifiers opaque to tools like Firesheep, provided the user visits the SSL/TLS secured webpage. Second, when providing the cookie to the browser, onehub.com marks the cookie as ‘Secure’. The implications of this second measure are subtle. Suppose Charles visits http://onehub.com (without the S). Without the cookie marked as ‘Secure’, Firefox would transmit the cookie to the onehub.com servers unencrypted. While onehub.com could redirect his browser to visit https://onehub.com (with the S), the damage is already done: Charles has broadcast his session identifer in an insecure fasion. Firesheep and other tools would be able to intercept it before the server can instruct the browser to try again via the secure connection.

Internet Security is a tough problem, it requires education of both end-users (Charles) and system administrators (Onehub) to ensure data is not leaked. The team here at Onehub works very hard to ensure your data is safe. We will continue to monitor new threats as they arrive, and adjust our systems and policies where appropriate.

Tasty Cookies with nginx

In a previous post I talked about the best way to handle putting an application into maintenance mode. In addition to sending the appropriate status code (503) for machines, nginx is configured to serve a helpful page to inform users of the maintenance window. Today, I would like to build on this configuration to make it possible for a few users to test the application while it is in maintenance mode for the rest of the world. There are a few different ways this could be handled, but HTTP cookies are a simple and flexible solution.

Recall these directives which instruct nginx to check for our maintenance page and set the status code to 503:

if (-f $document_root/system/maintenance.html) {
  return 503;
}

Let’s modify this with a variable to track state. Now, we can add an additional check for anything we would like to turn $maint off. $http_cookie holds all the cookies for the request:

set $maint off;

if (-f $document_root/system/maintenance.html) {
  set $maint on;
}

if ($http_cookie ~* "topsekrit" ) {
  set $maint off;
}

if ($maint = on) {
  return 503;
}

You can now set a cookie with a value of “topsekrit” and nginx will let you circumvent the maintenance page to test your new code before you release it to the world.

Rails Maintenance Pages Done Right

A maintenance page is a common feature in a rails deployment. This feature enables the developers to put their application into “maintenance mode”, returning 503 (Service Temporarily Unavailable) and a helpful page that describes what is happening and when the user can expect service to resume.

In an nginx configuration file this would be something like this:

if (-f $document_root/system/maintenance.html) {
  return 503;
}

error_page 404 /404.html;
error_page 500 502 504 /500.html;
error_page 503 @503;
location @503 {
  rewrite ^(.*)$ /system/maintenance.html break;
}

The page can then be activated or deactivated from a simple Capistrano task:

# Swap in the maintenance page
namespace :web do
  task :disable, :roles => :web do
    on_rollback { run "rm #{shared_path}/system/maintenance.html" }

    run "if [[ !(-f #{shared_path}/system/maintenance.html) ]] ; then ln -s #{shared_path}/system/maintenance.html.not_active #{shared_path}/system/maintenance.html ; else echo 'maintenance page already up'; fi"
  end

  task :enable, :roles => :web do
    run "rm #{shared_path}/system/maintenance.html"
  end
end

This pretty typical configuration is lacking two desirable features. First, what if you want your logo image on your maintenance page? Lets try this:

if (-f $document_root/system/maintenance.html) {
  return 503;
}

error_page 404 /404.html;
error_page 500 502 504 /500.html;
error_page 503 @503;
location @503 {
  # Serve static assets if found.
  if (-f $request_filename) {
    break;
  }

  rewrite ^(.*)$ /system/maintenance.html break;
}

This configuration will serve any static assets nginx can find on disk, but it will continue to return the 503 HTTP Status Code.

Finally, nginx, like most webservers will not let you POST to a static file. If a user loads your login page, then you switch on your maintenance page, the user will be greeted with the standard nginx 405 (Method Not Allowed) page. The solution is to capture 405 errors in your @503 location block, serving the maintenance page. In addition, you will have to enable @recursiveerrorpages@, since you are first, intentionally, throwing a 503 error, and then the user is throwing a 405 by posting to your static file:

recursive_error_pages on;

if (-f $document_root/system/maintenance.html) {
  return 503;
}

error_page 404 /404.html;
error_page 500 502 504 /500.html;
error_page 503 @503;
location @503 {

  error_page 405 = /system/maintenance.html;

  # Serve static assets if found.
  if (-f $request_filename) {
    break;
  }

  rewrite ^(.*)$ /system/maintenance.html break;
}

Now you can serve your stylish maintenance page no matter what the situation is.

Setting up a rails stack with nginx and clustered mongrels

Linux, MySQL, nginx, Mongrel , Rails, and friends.

Big Picture:

An ideal setup uses two different servers or virtual machines to split up the load and facilitate scaling. The first machine runs nginx serving static content and acting as a reverse proxy passing dynamic requests to a pool of mongrels running the rails application. The second machine is a dedicated MySQL database server. Even if you only have one physical box, two virtual machines (Xen based) is preferred, it is easier to scale and doesn’t require as much memory context switching. The hosting provider we picked to run our service, Engine Yard, uses a similar setup based on Gentoo and Xen.

Linux Setup:

All of the tools and applications used in this stack are actively developed and available in most package management systems. This document will assume you use Debian/GNU Linux, but will work equally well on Ubuntu. If choose to use Fedora, Gentoo or another distribution, you will need substitute commands for your package-management tool.

Installing and configuring Linux is outside the scope of this document, but you should strive for a minimalist system based on a recent release of the 2.6 kernel. Debian and Ubuntu both offer a number of “pre-made” configurations in their installers, just avoid installing anything more than the base system as it will just add more software that you will have to disable or remove later.

Conventions:

$ indicates a standard shell prompt.

# indicates the command needs to be run as the root user, either by switching to the root user account or assuming root privileges with a tool like sudo.

Tools:

While installing and managing a Linux server, you will often need the power of the root account. However, because the root user is so powerful, it dangerous to use all the time. A better solution is to use a standard user account and only assume the privileges of the root user when necessary. Sudo was designed to do exactly this.

# apt-get install sudo

Once the package is installed, the list of users able to assume root powers is controlled by /etc/sudoers. This file must be edited with the visudo command, which will open your default editor (or vi if your shell does not have an editor set).

# visudo

If you are unfamiliar with vi, press ‘i’ to put it in insert mode, then add a line to enable your user.

username ALL=(ALL) ALL

You can read more on the sudo package to better understand this configuration file. Once you have completed your edit, press escape (ESC) to exit editor mode. To save the file type ’:’ to enter command mode and then type ‘w’ and ‘q’ (meaning write and quit). The file should be properly saved and your user will now be able to assume root privileges.

Database Server:

MySQL is our database of choice, it is popular in the Rails community, proven to scale, and fast (especially in a read heavy environment).

To install, use the mysql-server package.

# apt-get install mysql-server

Your package manager should handle all the dependencies, including the mysql-client package, which we will use to improve our Ruby to MySQL performance. The installer may prompt you for a few questions, aim to accept the defaults and enter in information specific to your server.

Database Configuration:

The configuration file for MySQL is usually located in /etc/mysql as my.cnf. Most MySQL packages also include a number of other simple configurations to help MySQL perform better. Take a look at my-large.cnf and my-small.cnf in /usr/share/mysql and update your /etc/mysql/my.cnf to reflect your hardware or virtual machine’s configuration. MySQL tuning is a complex subject, but some basic configuration can go a long way to helping the application to perform well.

Before adding any databases, it is also important to instruct MySQL to use more international character encodings and collations. Storing the data in this way will make internationalization a simpler task in the future.

In /etc/mysql/my.cnf:

[client]
default-character-set = utf8[server]
default-character-set = utf8
default-collation = utf8_general_ci

These directives instruct MySQL to use UTF-8 for everything. If you have not already, restart your database server.

# /etc/init.d/mysql restart

You can now add the database to the server.

$ mysqladmin -u root create database_name

Consult the MySQL documentation to change the root user’s password and to add a user with privileges on the specific database. It may be simpler to use a MySQL administration GUI to achieve these tasks.

Application Server

The application server requires the most complex setup.

Ruby & Ruby Gems

To start, you will need Ruby and Ruby Gems.

# apt-get install ruby rubygems

ruby is a transitional package, currently it will install Ruby 1.8.6 (as of March 20, 2008), unless you know of a specific feature or bug to avoid, it is simplest to let the the package-management system handle which version you get. Ruby Gems is a package management system for system for third party ruby libraries, it functions much like CPAN for Perl or PEAR for PHP. We’re going to want a number of gems, but first lets update gems itself, from within itself—how meta.

# gem update —system

Ruby gems should now download and recompile itself. Once that has completed, you can install the gems we’ll need. The first one is a bit trickier than the others.

# gem install mysql

The mysql gem provides fast C bindings between Ruby and MySQL, if you aren’t already using these on your local machine you should give them a spin, the performance increase is well worth the minimal amount of effort required to install them. If you have your MySQL installed in a non-standard location, you may have to pass in mysql_config.

# gem install mysql –– ––with-mysql-config=/your/path/to/mysql_config

With mysql all setup and out of the way, its easy to install everything else you need.

# gem install rails mongrel mongrel_cluster

rails is optional, you may choose to freeze rails inside your application, which would override this gem. mongrel and mongrel_cluster are the gems that provide us with our application server. Before diving into mongrel, there are a few more gems you may want that are entirely optional.

# gem install tmail swiftiply

tmail is an e-mail library, and is the same library that the ActiveMailer component of Rails uses. Rails 2.0 now looks for this gem before loading its own bundled copy; by installing the gem you can get extra features and bug fixes that may have been released.

swiftiply is a gem that patches mongrel to use the event based network programming model, commonly called “evented mongrels”. You can find more on Ezra Zygmuntowicz’s Brainspl.at and my journal where I did some elementary benchmarking.

Mongrel

Mongrel will serve as our application server, containing an instance of the Ruby interpreter and our Rails application.

Mongrel is a fast HTTP library and server for Ruby that is intended for hosting Ruby web applications of any kind using plain HTTP rather than FastCGI or SCGI.

From inside your rails application you can start a mongrel server and interact with your application. By default, rails will start in development mode.

$ mongrel_rails start

A daemon with the application should now be running on port 3000. ctrl+c to stop the server, lets move on to mongrel_cluster

Mongrel Cluster

Mongrel Cluster will manage a pool of mongrels that nginx will reverse proxy requests to.

Mongrel_cluster is a GemPlugin that wrappers the mongrel HTTP server and simplifies the deployment of webapps using a cluster of mongrel servers. Mongrel_cluster will conveniently configure and control several mongrel servers, or groups of mongrel servers, which are then load balanced using a reverse proxy solution.

The wiki has more extensive documentation, right now we will only look at the basics.

# mongrel_rails cluster::configure -e production -p 8000 -N 3 -c /path/to/your/application -a 127.0.0.1 —user mongrel —group mongrel

This line creates the file application/path/config/mongrel_cluster.yml. Most of the switches are self explanatory, -a 127.0.0.1 configures the IP that the server binds to, by making it local you make it impossible to hit a mongrel directly without passing through the proxy. -p and -N configure the start port and the number of mongrels to start, so 8000, 8001, and 8002 in this case. —user and —group are the system user and group the servers will run under; because the mongrels run on an unprivileged port, they may run as an unprivileged user to improve security. -e is the environment our rails application will run in and -c is the path to our application (mongrel cluster will look for its configuration file at this path + config/mongrel_cluster.yml, you may use -C if you want to directly specify a YAML config file).

—-
user: mongrel
group: mongrel
cwd: /path/to/my/application
address: 127.0.0.1
port: "8000"
servers: 3
environment: production
pid_file: log/mongrel.pid

Evented Mongrel

To use evented mongrels, you just need to pass an environmental variable EVENT=1 while starting your server. With mongrel_cluster that will look like this:

# EVENT=1 mongrel_rails cluster::start

Don’t forget to pass it even if you are just restarting the servers! At the top of the mongrel.log you should see it notify you that it is using events.

Web Server

Currently our application server and web server are on the same virtual machine. In the future, we may choose to scale by moving the web server and static content to another virtual machine (or perhaps a CDN like Akamai) and have pure application servers. For now, we will keep things simple.

nginx

nginx is a very fast proxy that we will also use to serve our static content.

# apt-get install nginx

Configuration is very straight forward, everything is done in /etc/nginx/nginx.conf. The following configuration is based on “Ezra Zygmuntowicz’s Brainspl.at” nginx.conf.

# user and group to run as
user  www-data www-data;# number of nginx workers
worker_processes  6;# pid of nginx master process
pid /var/run/nginx.pid;# Number of worker connections. 1024 is a good default
events {
  worker_connections 1024;
}# start the http module where we config http access.
http {
  # pull in mime-types. You can break out your config
  # into as many include's as you want to make it cleaner
  include /etc/nginx/mime.types;  # set a default type for the rare situation that
  # nothing matches from the mimie-type include
  default_type  application/octet-stream;  # configure log format
  log_format main '$remote_addr - $remote_user [$time_local] '
                  '"$request" $status  $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';  # main access log
  access_log  /var/log/nginx_access.log  main;  # main error log
  error_log  /var/log/nginx_error.log debug;  # no sendfile on OSX
  sendfile on;  # These are good default values.
  tcp_nopush        on;
  tcp_nodelay       off;
  # output compression saves bandwidth
  gzip            on;
  gzip_http_version 1.0;
  gzip_comp_level 2;
  gzip_proxied any;
  gzip_types      text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;  # this is where you define your mongrel clusters.
  # you need one of these blocks for each cluster
  # and each one needs its own name to refer to it later.
  upstream mongrel {
        fair; # This changes from round-robin based to load based.
    server 127.0.0.1:8000;
    server 127.0.0.1:8001;
    server 127.0.0.1:8002;
  }  # the server directive is nginx's virtual host directive.
  server {
    # port to listen on. Can also be set to an IP:PORT
    listen 80;    # Set the max size for file uploads to 50Mb
    client_max_body_size 50M;    # sets the domain[s] that this vhost server requests for
    # server_name www.[yourdomain].com [yourdomain].com;    # doc root
    root /path/to/your/application/public;    # vhost specific access log
    access_log  /var/log/nginx.vhost.access.log  main;    # this rewrites all the requests to the maintenance.html
    # page if it exists in the doc root. This is for capistrano's
    # disable web task
    if (-f $document_root/system/maintenance.html) {
      rewrite  ^(.*)$  /system/maintenance.html last;
      break;
    }    location / {
      # needed to forward user's IP address to rails
      proxy_set_header  X-Real-IP  $remote_addr;      # needed for HTTPS
      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect false;
      proxy_max_temp_file_size 0;      # If the file exists as a static file serve it directly without
      # running all the other rewite tests on it
      if (-f $request_filename) {
        break;
      }      # check for index.html for directory index
      # if its there on the filesystem then rewite
      # the url to add /index.html to the end of it
      # and then break to send it to the next config rules.
      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }      # this is the meat of the rails page caching config
      # it adds .html to the end of the url and then checks
      # the filesystem for that file. If it exists, then we
      # rewite the url to have explicit .html on the end
      # and then send it on its way to the next config rule.
      # if there is no file on the fs then it sets all the
      # necessary headers and proxies to our upstream mongrels
      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }      if (!-f $request_filename) {
        proxy_pass http://mongrel;
        break;
      }
    }    error_page   500 502 503 504  /500.html;
    location = /500.html {
      root   /path/to/your/application/public;
    }
  }
}

Now that you have nginx configured, start it and you should have a solid rails stack up and running.

# /etc/init.d/nginx start

Request Cycle

Unlike more traditional setups, each individual request does not necessarily take the same path. Nginx monitors port 80 and accepts all incoming requests. Using the rewrite rules it looks for a file in the public directory with /url/path + .html. If it finds this static HTML page, it replies. It also searches for index.html to retrieve folder indexes. These static HTML files are generated by rails’ page caching mechanism, and provide enormous performance gains at a cost of flexibility. If nginx is unable to find an HTML page, it forwards the request to the cluster of mongrels. The mongrel that gets the request depends on which proxy algorithm you choose; the fair option spread the requests based on availability. Once received, the mongrel then runs the request through the rails application: the database is hit, ruby does some magic, and a page is completed and passed back to the mongrel which logs the request as complete and passes it back to nginx. If page caching is enabled, rails writes the page that was just generated to the public directory (lazy caching) before handing it off to the mongrel. Nginx takes this page and forwards it to the browser, acting as a proxy (gateway). The general process is repeated on each request, but the mongrel that is given the dynamic request will be different, and a cached page may be found, avoiding the need to engage the mongrels all together.