skip navigation

Here you will find ideas and code straight from the Software Development Team at SportsEngine. Our focus is on building great software products for the world of youth and amateur sports. We are fortunate to be able to combine our love of sports with our passion for writing code.

The SportsEngine application originated in 2006 as a single Ruby on Rails 1.2 application. Today the SportsEngine Platform is composed of more than 20 applications built on Rails and Node.js, forming a service oriented architecture that is poised to scale for the future.

About Us
Home

Deploying when Removing Columns with Rails with Zero Downtime

07/11/2011, 8:00am CDT
By Luke Ludwig

By overriding the ActiveRecord::Base.columns method we are able to avoid the ActiveRecord::StatementInvalid exception thrown when removing a column from a table.

 At TST Media we deploy updates to our NGIN application frequently without downtime during peak traffic periods. These updates range from minor bug fixes to multiple feature rollouts with migrations against the database. A key component that allows us to accomplish this is a rolling deploy capistrano task, where slices are taken out of the load balancer list and restarted one at a time, which enables us to maintain current traffic without interruptions.

When migrations need to be ran we do a rolling deploy with migrations, in which the migrations are ran first and then servers are restarted one at a time. Difficulties arise when migrations are involved that modify or remove existing columns. Adding a new column to a table is not an issue as the Rails processes running against the old code base will not know that the new column exists. Removing a column from a table will cause problems. To understand why, lets take a look at the ActiveRecord::Base columns method:

      # Returns an array of column objects for the table associated with this class.
      def columns
        unless defined?(@columns) && @columns
          @columns = connection.columns(table_name, "#{name} Columns")
          @columns.each { |column| column.primary = column.name == primary_key }
        end
        @columns
      end

This method asks the database what columns belong to the table and memoizes the list of columns for each ActiveRecord class the first time it is called in the @columns variable. This will last for the length of the running process since this is done at the class scope. If a column were to be removed from the database while a rails process is running, the rails process will still think that column exists since the columns method is memoized. Any custom application code that uses this column would still expect that column to exist as well. Therefore we don't remove columns during the rolling deploy, but instead remove them afterwards.

When removing a column from a table we make sure this is done in a migration by itself. Then we comment out the actual line that removes the column within the migration so that on the rolling deploy the column does not get removed. Once the rolling deploy is complete, one might think we can now uncomment the migration that removes the column and rerun it. However this will cause an ActiveRecord::StatementInvalid exception to be thrown, even though the application code no longer refers to the column directly. ActiveRecord still expects the column to be there due to how the ActiveRecord::Base.columns method works.

ActiveRecord references the columns method for a variety of reasons. For example, when creating an instance of a model and saving it to the database, ActiveRecord will try to set the column which no longer exists to null. Here is an example stack trace:

ActiveRecord::StatementInvalid (Mysql2::Error: Unknown column 'testing' in 'field list': INSERT INTO `objects` (`title`, `created_on`, `testing`) VALUES ('Test', '2011-07-05 08:45:32', NULL)):
  mysql2 (0.2.7) lib/active_record/connection_adapters/mysql2_adapter.rb:314:in `execute'
  activerecord (3.0.5) lib/active_record/connection_adapters/abstract/database_statements.rb:282:in `insert_sql'
  mysql2 (0.2.7) lib/active_record/connection_adapters/mysql2_adapter.rb:325:in `insert_sql'
  activerecord (3.0.5) lib/active_record/connection_adapters/abstract/database_statements.rb:44:in `insert'
  activerecord (3.0.5) lib/active_record/connection_adapters/abstract/query_cache.rb:16:in `insert'
  arel (2.0.9) lib/arel/select_manager.rb:217:in `insert'
  activerecord (3.0.5) lib/active_record/relation.rb:14:in `__send__'
  activerecord (3.0.5) lib/active_record/relation.rb:14:in `insert'
  activerecord (3.0.5) lib/active_record/persistence.rb:270:in `create'
  activerecord (3.0.5) lib/active_record/timestamp.rb:47:in `create'
  activerecord (3.0.5) lib/active_record/callbacks.rb:281:in `create'
  activesupport (3.0.5) lib/active_support/callbacks.rb:428:in `_run_create_callbacks'
  activerecord (3.0.5) lib/active_record/callbacks.rb:281:in `create'
  activerecord (3.0.5) lib/active_record/persistence.rb:246:in `create_or_update'
  activerecord (3.0.5) lib/active_record/callbacks.rb:277:in `create_or_update'
  activesupport (3.0.5) lib/active_support/callbacks.rb:453:in `_run_save_callbacks'
  activerecord (3.0.5) lib/active_record/callbacks.rb:277:in `create_or_update'
  activerecord (3.0.5) lib/active_record/persistence.rb:39:in `save'
  activerecord (3.0.5) lib/active_record/validations.rb:43:in `save'
  activerecord (3.0.5) lib/active_record/attribute_methods/dirty.rb:21:in `save'
  activerecord (3.0.5) lib/active_record/transactions.rb:240:in `save'

This may seem like a rare special case scenario, but in a large code base such as NGIN's this scenario arises frequently enough that we needed a solution. We found an elegant and simple pattern to avoid this problem. We override the ActiveRecord::Base.columns method on the class whose column is being removed. Then we call super to retrieve the columns and strip out the columns marked for removal. Now the class can behave as if the column does not exist. This prevents the above ActiveRecord::StatementInvalid exception from happening once the columns are removed.

    class << self
      RemovedColumns = {'column_to_remove' => true}
      def columns
        cols = super
        cols.reject { |col| RemovedColumns.has_key? col.name }
      end
    end

To recap, anytime we remove a column from a table we override the columns method as shown above. Upon deploy we carefully make sure the migration that actually removes the column is not ran. Once the deploy has completed, we then run this migration to remove the column. While there are a few manual steps involved in ensuring the migration is ran at the correct time, this process has successfully enabled us to deploy without downtime while running migrations that remove columns.

Sport Ngin Pulse Subscribeto our newsletter

Tag(s): Home  Ruby  High Availability