Speeding up bundle install with in-memory file system

On some of the servers I work with, due to cheap hard drives in software RAID configuration, I’ve found that bundle install can be extremely slow (take half an hour to complete). This obviously became unacceptable during deploys.

I thought that it might have something to do with how bundler writes a lot of small files during the installation of the gems. So I decided to try putting the deploy bundle directory (where all the gems are being installed) onto the in-memory filesystem. On Ubuntu this is /dev/shm.

It works flawlessly. The install time improved from half an hour down to a few seconds. After the bundle install is complete however, we do not want to leave the installed gems in the memory, as during restart they would be purged. So we just copy the directory back to the disk. Strangely enough, copying the whole directory from /dev/shm does not trash the disk so much and it only takes up to a minute for a few hundred MB of gems.

It’s cool to be able to find and utilize such a useful and simple part of Linux to solve and work around a slow hardware problem, while for everything else the server does, it’s still perfectly usable and more than capable of performing it.

Here’s my Capistrano 3 lib I use in my deploys that integrates this speedup:

namespace :bundler_speedup do
  task :symlink_to_shm do
    on roles(:all) do
      bundle_shm_path = fetch(:bundle_shm_path)
    
      # Make sure bundle dir exists
      execute "if [ ! -d #{shared_path}/bundle ]; then mkdir #{shared_path}/bundle; fi" 

      # todo: what if #{shared_path}/bundle is a symlink - meaning an interrupted install from previous time?

      cmds = []
      # Copy the bundle dir to /dev/shm/
      cmds << "cp -r #{shared_path}/bundle #{bundle_shm_path}"
      # Remove the shared bundle dir and symlink the shm dir instead
      cmds << "mv #{shared_path}/bundle #{shared_path}/bundle.old"
      cmds << "ln -s #{bundle_shm_path} #{shared_path}/bundle"
      # We're ready to do a fast in-memory bundle install now...
      execute cmds.join(' && ')
      
      info "shared/bundle was copied to /dev/shm for in-memory bundle install"
    end
  end

  task :remove_from_shm do
    on roles(:all) do
      bundle_shm_path = fetch(:bundle_shm_path)
      cmds = []
      # Copy the shm bundle to shared
      cmds << "cp -r #{bundle_shm_path} #{shared_path}/bundle.new"
      # Remove the symlink and move in the dir on disk
      cmds << "rm #{shared_path}/bundle"
      cmds << "mv #{shared_path}/bundle.new #{shared_path}/bundle"
      # Remove the in memory bundle
      cmds << "rm -rf #{bundle_shm_path}"
      cmds << "rm -rf #{shared_path}/bundle.old"
      # Bundle is persisted and in place
      execute cmds.join(' && ')
      
      info "shared/bundle was restored from bundle install within /dev/shm"
    end
  end
  
  before 'bundler:install', 'bundler_speedup:symlink_to_shm'
  after 'bundler:install', 'bundler_speedup:remove_from_shm'
end

namespace :load do
  task :defaults do
    set :bundle_shm_path, -> { "/dev/shm/#{fetch(:application).gsub(' ', '_').downcase}_bundle" }
  end
end

In a Rails project, place it in lib/capistrano/tasks/bundler_speedup.rake. Capistrano should auto-load this for you.

This code is released under the MIT license.

Principal – Integrated Business Management

Integrated business management web app for a sales company with CRM features. The functionality this Rails app provides is extensive:

  • Management of distributor accounts across multiple countries.
  • Capture, edit and build lead portfolio.
  • Logging of phone calls, appointments and notes.
  • Management of orders.
  • Invoicing.
  • Integrated credit card payments (Stripe, MyGate).
  • Management of stock and its reordering.
  • Payouts and commissions.
  • Client support pages.
  • Product download, activation and update facilities.

For me, the most important part of this project is the extensive test suite that was developed from day one, which allows for all the integrated functionality to coexist in one, easy to manage and extend codebase; and rapid development and deployment of new features.

UI Tour

Leads overview.
Leads overview. Leads can be added, edited, commented on, moved between different statutes, filtered and searched. (Note: these screenshots do not contain any real client information. This data is randomly generated for the purpose of this presentation.)
Logging and appointment.
Logging an appointment.
Logging a phone call with a lead. (Note: these screenshots do not contain any real client information. This data is randomly generated for the purpose of this presentation.)
Logging a call for a lead.
Captured order.
Captured order.
Excerpt from a stock management screen.
Excerpt from a stock management screen.
On-site credit card payment.
On-site credit card payment.

Rails stats

Lines of code, test coverage & other project stats.
Lines of code, test coverage & other project stats. Almost 14 000 lines of code – this is one of the biggest projects I’ve created, developed and worked with in production. Despite the relatively large amount of code, I am extremely happy with how Rails handles a decent sized project like this, all in one integrated, business specific and complete codebase.

EQAFE – Ecommerce

This is one of the first projects where I started using Rails in production. EQAFE developed—over the years—from a simple checkout page to sell a single book to an ecommerce platform featuring thousands of products, affiliate program, gifts and more.

Deliverables:

  • UI design and implementation.
  • Backend design and implementation (Rails).
  • Automated test suite.
  • ~4 years running in production (DevOps).

Tour

Front page provides an index of the latest products.
Front page provides an index of the latest products.
Product details.
Product details.
A lot of the functionality is neatly tucked away in the always accessible dropdown menus.
A lot of the functionality is neatly tucked away in the always accessible dropdown menus.

Rails stats

eqafe-code-stats
It’s interesting to observe how each project can have quite a different allocation of the amount of code. Here for example, test vise, most of the code is located in controller tests – this tells us a lot about the app itself – a lot of the control flow is located in the controllers, most of the interesting stuff happens there, like for example users are redirected to different locations depending on their starting point and actions.

DIP Lite: E-learning Platform

DIP Lite is a free e-learning platform. Its fully automated in-browser experience connects the users with a personally assigned “buddy”: an experienced person that overviews and assist the user throughout the course. The user walks the course on her own pace as well as engage in daily exercises, complete with email notifications and communication with the course buddy provided by the course platform.

For this project I delivered the following:

  • Initial design of the flow and functionality of the platform based on the goals set by the client.
  • Mockups of the UI and user flows.
  • Graphic design.
  • Logo creation.
  • Implementation within Rails.
  • Automated test suite.
  • Further feature development and updating/upkeep of the codebase.
  • Running this project in production for 3 years (DevOps).

Tour

Front page.
Front page.
After the lightning fast registration users immediately start with Lesson 1.
After the lightning fast registration users immediately start with Lesson 1.
Reviewing user's feedback.
Reviewing user’s feedback.
Admin dashboard.
Admin dashboard.

Rails stats

Code stats.
Code stats.

Equal Money System Website

The goal for this project was to create a presentation which immediately engages the user and allows them to interact with the website and information in a meaningful fashion. Therefore, instead of a static presentation, we came up with easy to use and immediately available voting in relation to the goals of the project that is being presented.

Deliverables:

  • UIX design and its implementation within HTML & CSS.
  • Backend implementation in Rails, MySQL, including a fully integrated admin interface to manage the website content, votes and translations.
  • Custom lightweight Wiki.
  • Automated test suite.
  • Utilizing web APIs of other products of the client.
  • Running, maintaining & upgrading the app in production since 2012 (Unicon, Nginx, Ubuntu & Upstart).
  • Associated PhpBB forum.

Tour

Front page with immediately available point of interaction: voting on the projects goals.
Front page with immediately available point of interaction: voting on the projects goals.
Viewing a "goal" with voting results and visitor's feedback.
Viewing a “goal” with voting results and visitor’s feedback.
Votes are managed and responded to by moderators from the build in & easy to use interface.
Votes are managed and responded to by moderators from the build in & easy to use interface.
The Wiki editor is custom build, features live side-by-side preview and formatting help.
The Wiki editor is custom build, features live side-by-side preview and formatting help.

Rails stats

ems-stats

DIP Pro – Online Course Delivery Platform

E-learning project similar to DIP Lite but for more serious and advanced students, providing much more features including credit card payments, commission payouts, file (course material) delivery, multiple different courses divided into years and months and API for integration with other projects.

Visit desteniiprocess.com

Deliverables:

  • Initial design of the flow and functionality of the platform based on the goals set by the client.
  • Mockups of the UI and user flows.
  • Graphic design.
  • Logo creation.
  • Implementation within Rails.
  • Automated test suite.
  • Further feature development and updating/upkeep of the codebase.
  • Running this project in production for ~4 years (DevOps).

Tour

Front page explains what is it all about.
Front page explains what is it all about.
Courses overview.
Courses overview.
The dashboard students sees after confirming her email address.
The dashboard a new student sees after confirming the email address.
If a particular screen is still empty, it contains information about what it does.
If a particular screen is still empty, it contains information about what it does.
Integrated online payments.
Integrated online payments.

Rails stats

dip pro stats