Test-Driven Metaprogramming, by Example

If you've been following along in my blog, you'll know that the study group I attend is currently working our way through the Rebuilding Rails book, together.  In chapter 7, we tackle the meaty subject of ActiveRecord.  This one area where the metaprogramming facilities of Ruby really shine.

I wanted to tackle this work using Test-Driven Development.  There were some interesting challenges when it came to some required metaprogramming.  I learned a lot through the process.

The Story So Far...

Guided along by Noah's narrative, we have developed a core class called SQLiteModel — an ActiveRecord pattern implementation.  Application programmers would subclass for their concrete models and use the base functionality to find, create, update, and delete.  Each instance of this class corresponds directly to a row in the database.

Trimmed down (and mostly folded), here's the outline of the class, so far:

class SQLiteModel
  def self.connect(path_to_db)
    @db = SQLite3::Database.new path_to_db
    @dialect = SQLiteDialect.new table, schema
  end
  def self.table...
  def self.schema...
  def self.create(values)...
  def self.find_by_id(id)...
  def self.count...
  def [](attribute)...
  def []=(attribute, new_value)...
# def save
#   update_sql = @dialect.sql_for_update @values
#   @db.execute update_sql
# end

end
Listing 1 — sqlite_model.rb @ 3ae4ce6e7c.  Just as I tried to implement an instance method that wanted to access a metaclass variable.

As you can see, we can connect to a database, determine the name of the table that backs this specific SQLiteModel type, obtain schema information, insert new records, find records by id, find out how records there are and get/set values of individual rows.

(The experienced observer will note that @db and @dialect are not instance variables (i.e. variables specific to each instance of SQLiteModel).  You can tell because they are declared within a singleton method.  Therefore they are class variables.)

Things were going great!  However, when it came time to provide individual instances of SQLiteModel (i.e. objects that represent individual rows in the underlying table) with the ability to save changes, we hit a wall...

Accessing Singleton Class Instance Variables in Direct Instance Methods

# def save
#   update_sql = @dialect.sql_for_update @values
#   @db.execute update_sql
# end

Listing 2 — In sqlite_model.rb, the save() method.  This implementation won't work because @dialect and @db are instance variables on SQLiteModel's singleton class, not variables on individual instances. 

The problem here is that I want to access two variables defined at the class level: "@dialect" and "@db".

"A wall?!" my pair might say, "but all you have to do is provide an attribute reader on the singleton class and then call that reader."

"huh?," I would exclaim with a puzzled look on my face.  To which my partner would turn to the editor and write this:

class SQLiteModel
  def self.connect(path_to_db)
    @db = SQLite3::Database.new path_to_db
    @dialect = SQLiteDialect.new table, schema
  end
  
  def self.dialect
    @dialect
  end
  
  def self.db
    @db
  end

  ...
  
  def save
    update_sql = self.class.dialect.sql_for_update @values
    self.class.db.execute update_sql
  end
end
Listing 3 — In sqlite_model.rb, a straight-forward way for instances to access class singleton variables.

"ooooohhh!" I'd say, slapping my forehead.  But then we'd sit there and stair at those last two executable lines, trying not to think about how the "self.class." phrase would liter the codebase.  On the one-hand, rather literal ("my class' dialect attribute") but also perhaps a bit verbose (especially when repeated everywhere?).

The Essential Challenge

In effect we have discovered a new¹ desired feature from the language.  Namely, in the same way that we can declare instance attributes (via attr_reader, attr_writer, or attr_accessor) that we would like to be able to declare a class attributes that is inherited by subclasses and accessible by instances.  Ruby is so pliable, we can actually add this feature to the language (and the Rails core team do so.  Here's a description of the feature: http://guides.rubyonrails.org/active_support_core_extensions.html#class-attributes )

So as to give credit where it's due, I had no idea of how that would be implemented.  Had I not had a working example in Rails (namely, https://github.com/rails/rails/blob/3-2-stable/activesupport/lib/active_support/core_ext/class/attribute.rb), at this point, I'd would have some spike work to do, personally.  But I'm lazy and so I read their code. :)

The Code

What follows is a test-driven approach to implementing class_attribute.  I originally did this in the following series of commits: https://github.com/jtigger/ruby-on-rulers/compare/3ae4ce6e7cc770dcf30fd645e340e81d560f28fc...7297d58f0a8d1a90a51d09653f654a47349110d7

Step 1: Setup and Teardown

In the first step, I had to discover how to clean-up class definitions so that the results of the unit tests were independent of each other.  Lee Jarvis posted a helpful technique to do this on Stackoverflow.  Here's how we do it in our unit tests...

class ClassExtTest < Test::Unit::TestCase

  def setup

    TestClasses.module_eval "class Foo; end"

    @foo = TestClasses::Foo.new

  end

  

  def teardown

    TestClasses.send(:remove_const, "Bar")

    TestClasses.send(:remove_const, "Foo")

  end

...

end

Listing 4 — A class definition can be "deleted" by calling remove_const on the containing module.

Step 2: Add Attribute Reader

And the next natural step is to begin using the class macro.  We create a private namespace (we'll call it TestClasses).

module TestClasses
end

class ClassExtTest < Test::Unit::TestCase
  ...  
  def test_class_attribute__creates_variable_reader_on_base_class
    TestClasses::Foo.class_eval "class_attribute :ree"
    assert_nil TestClasses::Foo.ree
  end
  ...
end
Listing 5 — the first test case.

which sets us up with the simplest thing that could possibly work:

class Class
  def class_attribute(attr)
    define_singleton_method(attr) { nil }
  end
end

Listing 6 — Monkey patching Class.  At first, we're just getting the attribute reader on the class.

We run it and the tests pass.  So far, so good. (git commit of steps 1 & 2)

Step 3: Verify the Class Attribute is Inherited 

class ClassExtTest < Test::Unit::TestCase

  def setup

    TestClasses.module_eval "class Foo; end"

    @foo = TestClasses::Foo.new

    TestClasses.module_eval "class Bar < Foo; end"

    @bar = TestClasses::Bar.new

  end

  ...

  def test_class_attribute__creates_variable_reader_on_subclass

    TestClasses::Foo.class_eval "class_attribute :ree"

    assert_nil TestClasses::Bar.ree

  end

  ...

end

Listing 7 — the second test case.

This one comes for free since public and protected methods defined on base classes are inherited by their subclasses.

Step 4: Add Attribute Writer

Building on what we've constructed thus far, we can both drive the ability to assign values to the class attribute and verify that it works for subclasses.

class ClassExtTest < Test::Unit::TestCase
  ...
  def test_class_attribute__GIVEN_subclass_has_no_value_defined_THEN_assignments_to_variable_applies_to_both_base_class_and_subclass
    TestClasses::Foo.class_eval "class_attribute :ree"
  
    TestClasses::Foo.ree = 42
  
    assert_equal 42, TestClasses::Foo.ree, "assignment failed for the base class"
    assert_equal 42, TestClasses::Bar.ree, "assignment worked for the base class, but not the subclass"
  end
  ...
end

Listing 8 — the third test case

However, as I got started down the road of implementing the assignment, I realized that I wanted to have the ability to remove methods (regardless of visibility):

class Class
  def class_attribute(attr)
    define_singleton_method(attr) { nil }
    
    # the writer simply redefines the reader
    define_singleton_method("#{attr}=") do |new_value|
      singleton_class.class_eval do
        # TODO: if "attr" is already defined, remove it...
        define_method(attr) { new_value }
      end
    end
  end
end

Listing 9 — I want to remove any existing definition of the writer method before redefining it.

Detour: Cleanly Removing Existing Methods

Out of the box, Ruby's Module class has a "undef_method", but if you call it for a method that does not exist, it raises an exception.  So, we'd like to check first that the method, in fact, exists; and if so, remove it.  But Module::method_defined?() only matches on public and protected methods, so to be thorough, we'd want to also call Module::private_method_defined?().  Not only is this getting a little complicated, but this is very likely something we'll be doing again and again.

So, I detoured: commented out the test in Listing 8, and set out to test-drive a method that would do just this...

class ModuleExtTest < Test::Unit::TestCase
  def setup
    TestClasses.module_eval "class Foo; end"
    @foo = TestClasses::Foo.new
  end
  
  def teardown
    @foo = nil
    TestClasses.send(:remove_const, "Foo")
  end
  
  def test_GIVEN_a_method_is_defined_on_a_module_THEN_remove_possible_method_undefines_it
    TestClasses::Foo.class_eval do
      public; def public_bar; nil; end
      protected; def protected_bar; nil; end
      private; def _private_bar; nil; end
    end
    
    assert TestClasses::Foo.method_defined?(:public_bar)
    assert TestClasses::Foo.method_defined?(:protected_bar)
    assert TestClasses::Foo.private_method_defined?(:_private_bar)
    
    TestClasses::Foo.class_eval do
      remove_possible_method :public_bar
      remove_possible_method :protected_bar
      remove_possible_method :_private_bar
    end
    
    assert !TestClasses::Foo.method_defined?(:public_bar), "expected method 'public_bar' to be removed, but it's still there."
    assert !TestClasses::Foo.method_defined?(:protected_bar), "expected method 'protected_bar' to be removed, but it's still there."
    assert !TestClasses::Foo.private_method_defined?(:_private_bar), "expected method '_private_bar' to be removed, but it's still there."
  end
  
  def test_GIVEN_a_method_is_NOT_defined_on_a_module_THEN_remove_possible_method_quietly_ignores
    TestClasses::Foo.class_eval do
      def ree
      end
    end
    foo = TestClasses::Foo.new
    assert !foo.respond_to?(:bar)
    
    TestClasses::Foo.class_eval do
      remove_possible_method :bar
    end
    
    assert !foo.respond_to?(:bar), "expected method 'bar' to be removed, but it's still there."
  end
end
Listing 10 — Specification of the "remove_possible_method" method.

Which, over the two iterations (we define and drive to one test at a time), yielded this monkey patch on Module:

class Module
  def remove_possible_method(name)
    if method_defined?(name) || private_method_defined?(name)
      undef_method(name)
    end
  end
end
Listing 11 — Adding remove_possible_method to Module.

With that new capability under our belts, we can now satisfy our attribute writer test (detour is captured in this commit: https://github.com/jtigger/ruby-on-rulers/commit/e3d24887128de2f1568011ce136e907807c44e9d):

class Class
  def class_attribute(attr)
    # define the reader
    define_singleton_method(attr) { nil }
    
    # the writer simply redefines the reader
    define_singleton_method("#{attr}=") do |new_value|
      
      singleton_class.class_eval do
        remove_possible_method(attr)
        define_method(attr) { new_value }
      end
    end
  end
end
Listing 12 — the first useful implementation of class_attribute.

Step 5: Ensure Writer Affects the Target Class Only

With all my current tests passing green, now is a good time to make sure we cover other highly important cases.  The most obvious one being to ensure that subclasses can override the value set in their super classes:

class ClassExtTest < Test::Unit::TestCase
  ...
  def test_class_attribute__GIVEN_subclass_HAS_a_value_defined_THEN_assignments_to_variable_applies_to_only_subclass
    TestClasses::Foo.class_eval "class_attribute :ree"
    
    TestClasses::Foo.ree = 42
    TestClasses::Bar.ree = 157
    
    assert_equal 42, TestClasses::Foo.ree, "assignment failed for the base class"
    assert_equal 157, TestClasses::Bar.ree, "assignment worked for the base class, but not the subclass"
  end
  ...
end
Listing 13 — An important use-case: assigning a value to the class attribute in the subclass.

As it turns out, this is how the implementation works, but it's lovely to make this fact plain and verifiable in a test.

Meanwhile... Back at the Ranch

And now we have just added a new feature to Ruby: the ability to define an attribute on a class that is inherited by subclasses and visible to direct instances.  Returning back to our original goal, now we can simply declare "db" and "schema" as class attributes...

class SQLiteModel
  class_attribute :db
  class_attribute :dialect
  
  def self.connect(path_to_db)
    self.db = SQLite3::Database.new path_to_db
    self.dialect = SQLiteDialect.new table, schema
  end
  ...
  def save
    update_sql = dialect.sql_for_update @values
    db.execute update_sql
  end
  ...
end
Listing 14 — SQLiteModel @ dcd61e9b8a5.  Class attributes simplify the syntax in this simplest case.  Not illustrated are the capabilities of subclasses to set their own value for these attributes.

And the syntax for using them is simplified.  That alone is almost reason to invest in this feature.  But there's more.  Even with the minimal implementation we did, here, subclasses of SQLiteModel can simply set their own value, overriding that for themselves.  This means Models default to sharing a DB connection, but can have different DB connections, if desired; same is true for dialect.

The story isn't over...

This is the starting point, not the conclusion.  In fact, there are a number of cases that we'll likely want to cover:

• what about singleton classes?  does this code work properly if the class being decorated is one of those?
• what about direct instances?  would it be useful for individual instances to also override the value for themselves (while allowing all the rest of the instances to continue to use the class-level value)?
• since this is close to core language modification, perhaps we should be providing a means of determining if an instance has such an attribute (known as instance predicate methods).

More from the giants who came before me:

https://github.com/rails/rails/commits/master/activesupport/lib/active_support/core_ext/class/attribute.rb

and

https://github.com/rails/rails/commits/master/activesupport/test/core_ext/class/attribute_test.rb 

Cheers!

---

¹ Actually, this is a feature that Rails implements called "class_attribute".  This implementation has been all but lifted from their code (link at the bottom of the article. ↩