Speeding up bundle install with in-memory file system

On some of the servers I work with, due to cheap hard drives in software RAID configuration, I’ve found that bundle install can be extremely slow (take half an hour to complete). This obviously became unacceptable during deploys.

I thought that it might have something to do with how bundler writes a lot of small files during the installation of the gems. So I decided to try putting the deploy bundle directory (where all the gems are being installed) onto the in-memory filesystem. On Ubuntu this is /dev/shm.

It works flawlessly. The install time improved from half an hour down to a few seconds. After the bundle install is complete however, we do not want to leave the installed gems in the memory, as during restart they would be purged. So we just copy the directory back to the disk. Strangely enough, copying the whole directory from /dev/shm does not trash the disk so much and it only takes up to a minute for a few hundred MB of gems.

It’s cool to be able to find and utilize such a useful and simple part of Linux to solve and work around a slow hardware problem, while for everything else the server does, it’s still perfectly usable and more than capable of performing it.

Here’s my Capistrano 3 lib I use in my deploys that integrates this speedup:

namespace :bundler_speedup do
  task :symlink_to_shm do
    on roles(:all) do
      bundle_shm_path = fetch(:bundle_shm_path)
    
      # Make sure bundle dir exists
      execute "if [ ! -d #{shared_path}/bundle ]; then mkdir #{shared_path}/bundle; fi" 

      # todo: what if #{shared_path}/bundle is a symlink - meaning an interrupted install from previous time?

      cmds = []
      # Copy the bundle dir to /dev/shm/
      cmds << "cp -r #{shared_path}/bundle #{bundle_shm_path}"
      # Remove the shared bundle dir and symlink the shm dir instead
      cmds << "mv #{shared_path}/bundle #{shared_path}/bundle.old"
      cmds << "ln -s #{bundle_shm_path} #{shared_path}/bundle"
      # We're ready to do a fast in-memory bundle install now...
      execute cmds.join(' && ')
      
      info "shared/bundle was copied to /dev/shm for in-memory bundle install"
    end
  end

  task :remove_from_shm do
    on roles(:all) do
      bundle_shm_path = fetch(:bundle_shm_path)
      cmds = []
      # Copy the shm bundle to shared
      cmds << "cp -r #{bundle_shm_path} #{shared_path}/bundle.new"
      # Remove the symlink and move in the dir on disk
      cmds << "rm #{shared_path}/bundle"
      cmds << "mv #{shared_path}/bundle.new #{shared_path}/bundle"
      # Remove the in memory bundle
      cmds << "rm -rf #{bundle_shm_path}"
      cmds << "rm -rf #{shared_path}/bundle.old"
      # Bundle is persisted and in place
      execute cmds.join(' && ')
      
      info "shared/bundle was restored from bundle install within /dev/shm"
    end
  end
  
  before 'bundler:install', 'bundler_speedup:symlink_to_shm'
  after 'bundler:install', 'bundler_speedup:remove_from_shm'
end

namespace :load do
  task :defaults do
    set :bundle_shm_path, -> { "/dev/shm/#{fetch(:application).gsub(' ', '_').downcase}_bundle" }
  end
end

In a Rails project, place it in lib/capistrano/tasks/bundler_speedup.rake. Capistrano should auto-load this for you.

This code is released under the MIT license.

Effort – Personal To-do and Project manager

I have open sourced a Rails app that I’ve been personally using for years. The code is available on Github under the MIT license. From the README:

I’ve modeled this app for my own personal use, note keeping and personal project management loosely after Basecamp. The single most important point for me is to have To-do lists that work in a particular way – that’s why I’ve build this for myself.

I am open-sourcing it to see if somebody finds it useful and can maybe build on it. Let’s see what happens.

This is a standard Rails 4 app, build the “Rails way”. Test coverage is minimal, just enough for the purposes of this app at this stage.

effiort todo lists
To-do lists – the most important and the most used part of this project.

Script to update PhpBB 3.0.x to 3.1.x

Recently I had to upgrade a dozen PhpBB boards to the latest version. Previously I would do this by hand, which would take days. This time though, while reading through the update notes, I noticed that it is possible to update the database through the console (I assume this was only introduced in 3.1). That was the necessary prerequisity to be able to automate everything. I ended up with the following script:

#!/usr/bin/env bash
echo "Upgrading PhpBB instal in '$1' to 3.1 with files from '$2'..."

cd $1

echo "Deleting old files..."
shopt -s extglob
rm -r !(config.php|images|files|store)
shopt -u extglob

echo "Copying in 3.1 files..."
rsync -a --exclude='.git' $2/ $1/

echo "Migrating the database..."
php ./bin/phpbbcli.php db:migrate --safe-mode

echo "Removing the install dir..."
rm -r install

echo "Done."

The usage is as follows (assuming you named the script phpbb-update-to-3.1):

phpbb-update-to-3.1 /www/forum-to-update /tmp/phpbb-3.1-files-for-upgrade

The second parameter is the path to where you have prepared your 3.1 files, according to this guide. You should put your custom styles in this folder too.

If you have custom folders or files within your current install, make sure you add those to the rm -r !(config.php|images|files|store) line in the update script. This rm deletes everything except the files within the curly braces.

This saved me a lot of time and prevented mistakes. Enjoy!

Bash: Underscore a String

I find it weird that I did not find a ready made script for this immediately through a google search. So this is what I arrived at after some digging:

#!/usr/bin/env bash
read input
str="$(echo $input | tr '[:upper:]' '[:lower:]')"
printf ${str// /_}

Use it with stdin:

echo "Underscore Me" | underscore
underscore_me

I use this in TextMate when I need to convert normal english string/text into a parameter/key.

MailCatcher – My favorite development SMTP server

Big part of development of webapps is to be able to effectively send and receive email from these apps. Using and external SMTP server is slow: you have to wait for the email to be send and then receive it through a real email client. Having a local SMTP server on the dev machine is a much better solution. I use MailCatcher. It’s open source, simple and today I fixed the only thing that was bothering me on it:

Dockerized Percona MySQL Server with automated replication, tools & tests

Docker & Percona Server I developed this container to solve specific needs and alleviate pain points present when dealing with deployment and administration of MySQL on servers that I manage.

I decided to look into Docker during a migration from MySQL 5.5 to 5.6 on one of the production servers. The server hosts multiple applications and services and is running in a hot spare configuration (another server mirrors this server, acts as the MySQL slave, etc.). Thus I wanted a migration strategy where I can have the 5.6 installation ready and running on the server, so that I can test it with production data and when ready just turn the switch to replace the old 5.5. Docker turned out to be the cleanest solution.

Since the MySQL server is such a critical part of the infrastructure I decided to develop the container utilizing test driven development. This allowed me to quickly add new features, like performance optimization and replication over a ssh tunnel (to support servers in different public clouds). Having this functionality in a standalone, tested and isolated unit is amazing. Before, all this complexity would be managed by Chef provisioning, which is much harder to test and experiment with on the production server. Having this functionality contained in a Docker container allows me to just use Chef for orchestration and deployment of the containers themselves, witch requires much simpler logic, compared to provisioning a full MySQL server install, configuration, replication and upgrading.

This project is released under the MIT license.