Encrypting your files with Rails – Part II

This is the second post in a two part series (part 1 is here) on adding encryption to attachment_fu for Rails applications.

What about making the file available for download? AVOIDING THE ISSUE OF SCALABILITY FOR A MOMENT (since sendfile is not the right way to serve files from Rails), we want to use a variant of sendfile to do the decryption and send the file. Here’s a modified version of send_file that uses an extra hash parameter (acme) to decrypt if provided:

module ActionController
   module Streaming

   def send_file_x(path, options = {}) #:doc:
      raise MissingFile, "Cannot read file #{path}" unless File.file?(path) and File.readable?(path)

      options[:length]   ||= File.size(path)
      options[:filename] ||= File.basename(path)
      send_file_headers! options

      @performed_render = false
      logger.warn("Sending file #{path}")
      if options[:stream]
        render :status => options[:status], :text => Proc.new { |response, output|
          logger.info "Streaming file #{path}" unless logger.nil?
          len = options[:buffer_size] || 4096
          if options[:acme]
            c = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
            c.decrypt
            c.key = key = options[:acme]
            c.iv = iv = Digest::SHA1.hexdigest("OneFishTwoFish")
          end
          File.open(path, 'rb') do |file|
            while buf = file.read(len)
              if options[:acme]
                output.write(c.update(buf))
              else
                output.write(buf)
              end
            end
          end
        }
      else
        logger.info "Sending file #{path}" unless logger.nil?
        File.open(path, 'rb') { |file| render :status => options[:status], :text => file.read }
      end
    end
  end
end

The code could be made more efficient by not performing the options[:acme] test each time a buffer is written. Our controller action that downloads a file would call it like so:

send_file_x(@file_item.stored_filename,
  :filename      =>  @file_item.filename,
  :type             =>  @file_item.content_type,
  :disposition  =>  'attachment',
  :stream    =>  'true',
  :buffer_size  =>  4096,
  :acme => @file_item.acme)

In a production environment, send_file consumes to many server resources – the rails application, and method used to service it (FastCGI, Mongrel, etc.) are tied up serving the file.

It’s more likely the case that the rails application will be behind a reverse proxy like nginx; in that case, a directive is sent to the server to provide the file (usually through an HTTP header). For nginx, serving a non-encrypted, static file would be done by sending a header with the location of a file:

if defined?(NGINX_FOR_DOWNLOAD) && NGINX_FOR_DOWNLOAD
  # code omitted – set up file name and path
  response.headers['X-Accel-Redirect'] = NGINX_PATH_FOR_ _DOWNLOAD + path
  response.headers['Content-Type'] = file_item.content_type
  render :nothing=>true;
else
  send_file_x(File.join(RAILS_ROOT, FILE_STORAGE_PATH, path_parts, file_item.filename),
      :type         => file_item.content_type,
      :disposition  => 'attachment',
      :stream    => 'true',
      :buffer_size  => 4096,
      :acme      => nil,
      :encoding     => 'utf8',
      :filename     => URI.encode(file_item.filename))
End

For more information on nginx and rails, learn more about NginxXSendfile.

To perform a similar feat of decrypting and sending a file for nginx, a new module would need to be written for nginx that takes an additional header variable ‘X-Accel-Redirect-Key’ and uses that as the key to send the file, decrypting as it goes.

Encrypting your files with Rails – Part I

This is the first post in a two part series on adding encryption to attachment_fu for Rails applications.

Let’s say that you’re building an application that will be used by a number of different people, and it involves storing information in files, and providing that information to the right person(s) at the right time.

There are choices about where to store the ‘files’ for users – in a database, in a file system (on disk), or in a specialized store (e.g. Amazon S3). For this example, we’re going to choose storing files ‘on disk’ in a file system – locally, across the network, NAS, SAN – wherever, as long as we can ‘see’ the information as a file.

One traditional way to enforce permissions on files is to have the file system itself enforce whether someone can or can’t have access to a file. In the old days, on *nix systems, that meant juggling user and group databases, and working within the ‘user-group-other’ paradigm. More modern attributed file systems make this easier, however that might not work with your web application, because you might not want to synchronize file system attribute information with your web identity information.

You might ultimately decide to control access yourself using some sort of User information DB (homegrown, LDAP, etc.), and explicitly control access by protecting the URLs which download specific files.

As a further measure, you might want to consider encrypting information on a file level. When a file is uploaded, you would generate an encryption key, encrypt the file on disk with that key, store the key separately, and use that key when a file is accessed (as close to the point that the file is downloaded as possible).

For prototype or low-volume applications, let’s look at what it would take to modify attachment_fu and a download controller action to accomplish this, and then extrapolate to what it might take in a higher-performance environment.

Attachment_fu is a nice plug-in to quickly enable file uploads. Plenty of examples are available on how to use it – we’re going to modify it to generate a an encryption password when a file is uploaded, then encrypt that file as it’s stored into the file system.
Here’s the original code for saveto_storage in attachmentfu/backends/filesystembackend.rb.

      #Saves the file to the file system
      def save_to_storage
        if save_attachment?
          # TODO: This overwrites the file if it exists, maybe have an allow_overwrite option?
          FileUtils.mkdir_p(File.dirname(full_filename))
          File.cp(temp_path, full_filename)
          File.chmod(attachment_options[:chmod] || 0644, full_filename)
        end
        @old_filename = nil
        true
      end

Borrowing some example code from OpenSSL for doing aes encryption (you could use a different method if you like), we’ve modified save_to_storage to accept a key acme, which is used to encrypt the file as it is copied from temporary storage.

      #Saves the file to the file system
      def save_to_storage(acme=nil)
        if save_attachment?
          # TODO: This overwrites the file if it exists, maybe have an allow_overwrite option?
          FileUtils.mkdir_p(File.dirname(full_filename))
          if acme.nil?
            File.cp(temp_path, full_filename)
          else
            c = OpenSSL::Cipher::Cipher.new("aes-256-cbc")
            c.encrypt
            #
            c.key = key = acme
            c.iv = iv = Digest::SHA1.hexdigest("OneFishTwoFish")
            output = File.open(full_filename,'wb')
            File.open(temp_path, 'rb') do |file|
              while buf = file.read(4096)
                output.write(c.update(buf))
              end
              file.close
            end
            output.write(c.final)
            output.close
          end
          File.chmod(attachment_options[:chmod] || 0644, full_filename)
        end
        @old_filename = nil
        true
      end

Where do we get ‘acme’? filesystembackend is called from attachmentfu.rb – afterprocessattachment needs to generate a key, pass it to savetostorage, and also make the key available in the model attachmentfu is mixed-in with.

 # Cleans up after processing.  Thumbnails are created, the attachment is stored to the backend, and the temp_paths are cleared.
    def after_process_attachment
      if @saved_attachment
        if respond_to?(:process_attachment_with_processing) && thumbnailable? && !attachment_options[:thumbnails].blank? && parent_id.nil?
          temp_file = temp_path || create_temp_file
          attachment_options[:thumbnails].each { |suffix, size| create_or_update_thumbnail(temp_file, suffix, *size) }
        end

        myacme = nil
        if attachment_options[:encrypted_storage]
          myacme = Digest::SHA1.hexdigest(Clock.now.to_i.to_s+rand.to_s)
        end

        save_to_storage(myacme)

        @temp_paths.clear
        @saved_attachment = nil
        write_attribute :acme, myacme
        callback :after_attachment_saved
      end
    end

Note that we’re using Clock.now.toi.tos+rand.to_s as the key… for a real-world example, you would likely additionally seed this with a phrase only known to you.

Your model code should have an attribute named acme, of type string. When your model object is saved, acme will also be updated.

UPDATE: Read Part II