Surprise 10min Benchmark: eval, class_eval, instance_eval, define_method, bind

Mathew Abonyi — Thu, 11 Jan 2007 03:42:37 -0600

What is the fastest way to dynamically define a method? define_method? class_eval? instance_eval? With strings or procs? Maybe we can pull out some arcane magic and use instance_method and bind (depending on what we want to do).

We all need to use some kind of eval at some point and, while it will not usually effect performance when used sparingly, sometimes it just has to make a serious appearance. When I saw how slow RSpec is in comparison to the venerable Test::Unit, I discovered the way it handles its contexts: each one is individually instance_eval’d into an empty ContextEvalModule. It isn’t the only reason for being slower, but it is one and one which can really mount up on big projects.

So, moving on from RSpec, Test::Unit and my baseline, MicroTest (see previous post), I became perversely interested in benchmarking all the ways of doing it and found some surprises.

Results (Ruby 1.8.4)

                            user     system      total        real
eval/str              112.540000   0.060000 112.600000 (112.615607)
instance_eval/str      99.510000   0.040000  99.550000 ( 99.564661)
class_eval/str        112.750000   0.060000 112.810000 (112.824374)
instance_eval/proc     56.750000   0.040000  56.790000 ( 56.802483)
class_eval/proc        56.440000   0.020000  56.460000 ( 56.464818)
define_method/class   112.420000   0.060000 112.480000 (112.489430)
define_method/send    111.540000   0.050000 111.590000 (111.610502)
def/unbind/bind        73.750000   0.060000  73.810000 ( 73.875800)
def                    52.670000   0.030000  52.700000 ( 52.705886)
method/bind            13.520000   0.010000  13.530000 ( 13.523194)

Analysis

I put it roughly in the order which I originally expected, with the slowest at the top.

The first thing that stunned me was the speed of define_method. It’s as slow as a snail. It’s practically eval, despite using a proc. I’ve been using it in preference before, thinking it was faster than any of the evals (proc or string). Apparently not.

The second revelation was how proc-based evals are about 50% faster than a def/unbind/bind cycle, which only has def as the evaluating part. That any eval is close to def is simply surprising. The instance_eval variety is the one used by RSpec, so I guess they picked the best one already.

The third shock was how slow def is! Granted, this is a lot of iterations, but it should be quicker than that, surely?

So it seems pretty plain that method unbinding as a tuning technique is rubbish, despite not evaluating anything, whereas a proc-based instance_eval or simple def are the clear winners.

But wait… what about Ruby 1.8.5?

Seeing the decrepitude of define_method made me wonder about the latest release of Ruby. I downloaded it, compiled it, got it working and ran the same bench:

Results (Ruby 1.8.5)

                            user     system      total        real
instance_eval/str     109.620000   0.050000 109.670000 (109.685393)
class_eval/str        113.840000   0.060000 113.900000 (113.910749)
eval/str              114.520000   0.060000 114.580000 (114.596238)
define_method/class    70.220000   0.040000  70.260000 ( 70.264506)
define_method/send     68.030000   0.040000  68.070000 ( 68.066688)
def/unbind/bind        73.690000   0.030000  73.720000 ( 73.741549)
instance_eval/proc     59.050000   0.030000  59.080000 ( 59.084445)
class_eval/proc        59.530000   0.030000  59.560000 ( 59.566885)
def                    52.870000   0.030000  52.900000 ( 52.895184)
method/bind            12.470000   0.010000  12.480000 ( 12.484069)

Anaylsis

I guess they fixed the problem and more besides. define_method is now marginally faster than def/unbind/bind (which has not budged an inch). I think I’ll continue to take the performance hit on define_method with 1.8.4, knowing I’ll want to upgrade later in the future and not change techniques.

The old recommendations hold true

instance_eval with a proc for class, instance or singleton evaluation
instance_eval for changing proc bindings
define_method for converting procs into methods
def for everything else
eval when you really know you need it

Note: These findings are only for def meth; true; end and may vary in complex cases.

Benchmark Code

require 'benchmark'

class A; def meth1; true; end; end

# defining using eval/class/instance
str = "def meth2; true; end"
a_binding = A.send(:binding)

# proc for class/instance
proc1 = Proc.new { def meth2; true; end }

# proc for define_method
$proc2 = Proc.new { true }

# unbound method for bind
um = A.instance_method(:meth1)

# number of iterations
n = 12000 * 600

Benchmark.bm(22) do |x|

  x.report('instance_eval/str') do
    for i in 1..n; A.instance_eval(str); end
  end

  x.report('class_eval/str') do
    for i in 1..n; A.class_eval(str); end
  end

  x.report('eval/str') do
    for i in 1..n; eval(str, a_binding); end
  end

  x.report('define_method/class') do
    for i in 1..n; class A; define_method(:meth2, &$proc2); end; end
  end

  x.report('define_method/send') do
    for i in 1..n; A.send(:define_method, :meth2, &$proc2); end
  end

  x.report('def/unbind/bind') do
    for i in 1..n
      class A; def meth2; true; end; end
      A.instance_method(:meth2).bind(A.new)
    end
  end

  x.report('instance_eval/proc') do
    for i in 1..n; A.instance_eval(&proc1); end
  end

  x.report('class_eval/proc') do
    for i in 1..n; A.class_eval(&proc1); end
  end

  x.report('def') do
    for i in 1..n; class A; def meth2; true; end; end; end
  end

  x.report('method/bind') do
    for i in 1..n; um.bind(A.new); end
  end

end

Wood for the Trees: Surprise 10min Benchmark: eval, class_eval, instance_eval, define_method, bind