One of the major nerd goals for Airbnb in 2011 was upgrading to Rails 3. Our production instances made the final switch in the week leading up to Thanksgiving, but it didn’t happen all at once.
Breaking the Upgrade into 3 Steps
We added the required pieces throughout the past year, and, looking back, breaking the upgrade into three major steps was easier to manage than trying to cram it into a single deploy. The first two steps are independent from actually upgrading to the Rails 3 gem, and any Rails 2.3 app out there should do them even if there’s no intention of upgrading to 3.0.
Step 1: Add Bundler
This was our first step into Rails 3 land, and we did it at the end of 2010. Our app was running Rails 2.3.8 at the time, and the production instances were a mess. Each instance had its own set of gems and its own versions of those gems. Naturally, none of the instances matched.
One of our engineers picked the latest version of each gem initialized by config.gem in production to create our first Gemfile. He installed Bundler using their Rails 2.3 guide and added any missing gems when trying to start the app. He used the recommended ~> notation for gem versions at first, but that upgraded our Paperclip gem to 2.3.3. That version had performance problems in our environment, so the engineer specified exact versions for our gems instead, the versions we had fully tested.
He then ran each of our Rake tasks and added any gems those tasks needed to run. That was the last step before our Gemfile was ready to deploy in production.
Adding Bundler at the end of 2010 was a one-engineer task that took about a week working on-and-off. If Bundler had been added the same time as the rest of the Rails 3 upgrade, almost a full year later, it would have been a much bigger ordeal.
After installing Bundler, Airbnb ran without any thoughts of Rails 3 for a while. Our production environment was far cleaner, each production instance was running the same set of gems, and our deploy process was simpler.
Step 2: Install the rails_xss plugin
Templates in Rails 3 auto-escape all strings, which is a major change from 2.3 and breaks a lot of assumptions. Auto-escaping HTML is a good thing, but many of our templates assumed Ruby strings would be interpreted as HTML.
Skip ahead from our Bundler install about 6 months to Spring 2011. To bring the Rails 3 behavior into our app, Tobi Knaup, one of our engineers, chose to install the rails_xss plugin. The plugin escapes strings by default like Rails 3. He knew this was likely to break a ton, if not all, of our templates, so he setup a Rails XSS Hacksession to get every engineer’s help in the installation.
Tobi listed every controller in our app at the time and divided them among the engineers. The engineers went through the templates used by each controller and made sure string output in templates was marked as html_safe or rendered with raw where appropriate.
Most of the XSS work was done in a single day with all of the engineers’ help, but there was cleanup on pages that weren’t used very often that continued for several weeks after installing the plugin.
With rails_xss installed, Airbnb was more secure and closer to Rails 3.
Step 3 (the big one): Add the Rails 3 Gem
This is the well-known upgrade step that includes updating the ‘rails’ gem to 3.x. Thankfully our Gemfile was already mature, and our templates expected most strings to be HTML-escaped when we got to this step around Thanksgiving 2011.
We knew the real upgrade required major code changes, so four engineers started the upgrade on a Saturday afternoon when there were minimal changes being committed. We asked other engineers to hold off on changes over the weekend to prevent any merge nightmares.
We kept the Rails upgrade branch separate with its own Gemset in case we needed a local working version of master before the upgrade was done. The most basic version of our workflow looked like this:
- Create a new rails3 branch
- Clone the rails3 branch into a separate directory from the usual working directory (instead of simply switching branches)
- Create a new gemset for the rails3 branch’s directory
- Install the rails_upgrade plugin
- Follow the steps from the rails_upgrade README, skipping the step that generates a Gemfile since we were already using Bundler
- Try running
rails consoleandrails server - Fix any raised errors
- Return to step 6 if there were errors in step 7
- Done! Sort of…
Things that caused headaches during the upgrade:
- The auto-generated routes file from the rails_upgrade plugin was wrong. The routes for our internal API (the API that powers our iPhone app) were busted. The problems weren’t isolated to those routes, but those were the most notable. While we did use the auto-generated file as a starting point, we spent a lot of time tracing problems back to the broken routes file.
- Timezones that worked in Rails 2.3 raised errors in 3.0. We didn’t change the timezones we had stored, but various ones started raising errors. Benjamin Oakes’s post about Rails Timezones explains the problem and how to start fixing it.
- The lib directory is no longer auto-loaded. This was not called out prominently anywhere, but a quick search turns up several ways to address it. We added our lib directory to the autoload paths in application.rb
config.autoload_paths += %W( #{Rails.root}/lib )
The app was running on Rails 3 by the end of the weekend, and specs were passing, but there were plenty of pages that hadn’t been tested.
Monday morning all of the engineers took ownership of features they had recently worked on and thoroughly tested them similar to how we approached the Rails XSS install. By the middle of the week we merged the rails3 branch into master and deployed it to our production instances.
What made the upgrade successful
If everything goes smoothly, the Rails 3 upgrade should be transparent to users. That can make the upgrade a hard sell to anyone outside engineering considering the effort that goes into it. Installing Bundler was a small project done by a single engineer, but installing the XSS plugin and doing the actual Rails 3 upgrade took considerable hours that had to be planned.
We coordinated with our product team for each of the big steps. They were planned between major product updates to make sure the codebase wasn’t changing while we were upgrading.
- Published in Code
- January 04, 2012 by Ross Allen
- 15 Comments
15 Comments
I agree with @producersteve about the 1.9.2 upgrade woes. I’m running 2.3.x also on my main app and have completed the bundler and 1.9.2 upgrades. The final piece of jumping to Rails 3 is really a tough sell for me to the business owner.But yes congrats with the upgrade!
Hello nerds,I’m one of the maintainers of the rails_upgrade gem. I know that the routes code is a bit ghetto at the moment, and would really like to fix that up. Do you reckon you would be able to find some time to send some patches in to fix this up? I’m sure that an application the size of AirBnB’s contains a lot of great examples!<3 Radar
@producersteve – that is a good point, forgot to mention we are still on Ruby 1.8.7We’ll most likely upgrade to 1.9.2 as another step down the road. It has definitely helped to take the upgrade piecemeal like this (it can be overwhelming to do all at once) and 1.8.7 seems to have worked fine with Rails 3.
@producersteve Step 3 could be expanded into another post, we had a lot more than just those few paragraphs to mention.For templating we used the https://github.com/rails/jquery-ujs plugin maintained by the Rails team to support remote forms and remote links, but things still occasionally broke. We added the ‘prototype_legacy_helper’ gem to support tools pages that we wanted to ensure still worked.We were using searchlogic as well and replaced any queries using it with the ActiveRecord equivalent, but luckily there were relatively few considering the size of our codebase. With 70K calls like you have, rewriting all of them is not reasonable. Since we scrapped the gem we didn’t look into alternatives.
Was 3.1 just a bridge too far due to the asset pipeline? Seems sorta strange to not upgrade fully to the latest release, but at the same time totally understandable given the extra amount of work it would probably entail.
@ryanbigg I can grab our final Rails 2.3 routefile and try generating it again with rails_upgrade and then diff against our current version to see what the problems were.@mossity We decided to go for the latest 3.0.x since the jump from 2.3 -> 3.0 was enough to tackle at once. We have started talking about 3.1 (especially for the asset pipeline), but there’s no plan for it yet.
When you do upgrade to ruby 1.9, one thing to keep in mind that is not obvious at all is that 1.9 changes the default Date format to international (dd/mm/yy). See http://stackoverflow.com/questions/5372464/ruby-1-87-vs-1-92-date-parse/5373165#5373165 and use the delocalize gem to avoid a lot of headaches. Congrats on the smooth upgrade.
We’re in the middle of the same thing at Storenvy right now. Unfortunately, we’ve also had to upgrade a bunch of other tools at the same time which has turned it into a pretty big project. (Rspec -> Rspec 2, restful_auth -> Devise, prototype -> jQuery, mostly, plus a few others). Honestly, upgrading to Rspec 2 has been a bigger hurdle than just upgrading to Rails 3. So. Much. Syntax!
Does the "502 Bad Gateway" at http://airbnb.com right now have anything to do with the upgrade?
@ryanflorence Sorry about that Ryan! That was unrelated to the Rails 3.0 upgrade, we were having a problem with our Redis setup which was choking nginx.Should be fixed now.
I’m happy my blog post about Rails Timezones helped you out. Congrats on moving from Rails 2.3 to 3.0!
Any changes in performance?
Here’s a few more that we at barrelrun.com encountered, might be useful for you and other people (note that we upgraded all the way to 3.1 so some of these might apply to 3.1 not 3.0):delete config/initializers/new_rails_defaults.rb. It is no longer necessary (per its own comments) and in fact causes a weird exception if you don’t remove it:../../config/initializers/new_rails_defaults.rb:14: undefined method `generate_best_match=’ for ActionDispatch::Routing:Module (NoMethodError)You will find that your delete actions will not work anymore, for this to work, you need to install rails_ujs gemIF you use will_paginate, add require ‘will_paginate/array’ to application.rb fileIF you use Devise, add require ‘devise/orm/active_record’ to ./config/initializers/devise.rb fileFinally, if you do upgrade to Rails 3.1, make sure you fix the form_for and fields_for depreciation warnings and use the new <%= form_for …%> notation
@mvilrokx We are using the ‘rails_ujs’ gem to continue supporting "remote" links and tags. We added support for the :popup attribute too, which opens the link in a new window. The effect is the same as target="_blank", but a bunch of our links expected the :popup attribute to work.We also updated most uses of tags that use concat to use the updated syntax, <%= %>, like you mentioned. form_for, content_for, etc.@primdahl There wasn’t a noticeable change in performance, faster or slower.
I’m surprised you didn’t mention any issues with:- string encodings, or difficulty in upgrading Ruby from 1.8.7 to 1.9.2 (I’ve experienced difficulty with this on some systems, namely gentoo boxes)- gems (or plugins) you rely on being incompatible with Rails 3- gems (or plugins) you rely on requiring an upgrade and completely changing their own dependencies or APIsThen again I guess that if you had those issues, they were collapsed into that mythical #6-7-6-7-6-7… loopThe project I work on relies heavily on the searchlogic gem, which is not compatible with Rails 3. People have recommended alternatives but none of them have compatible APIs. I’m of the mindset that if we comb through 70k LOC to change all our AR calls, I would prefer to change the calls to vanilla AR.At any rate interesting post. I’m glad to hear that you were successful upgrading Rails — it certainly gives me hope.
Comments are closed.