We process a lot of HTML documents using Nokogiri. On large HTML files we sometimes get the (SystemStackError) “stack level too deep” exception. This happens because Nokogiri recursively processes the dom, and on large documents it exceeds the maximum stack size of 1024 KB on our system.

In Ruby versions before Dec. 2012, the Ruby stack size was limited by the C stack size which can be changed using the ‘ulimit -s’ command. In Ruby 2.0 the stack size is now limited via its own environment variable RUBY_THREAD_VM_STACK_SIZE.

We address the problem by setting this env variable to a higher value when launching our ruby processes.

Here is some code to try this yourself:

First launch a Ruby console

irb

Then inspect your current max stack size

irb(main):052:0> ENV['RUBY_THREAD_VM_STACK_SIZE']
=> "1048576"

You can also look at all the default settings for the VM:

irb(main):001:0> RubyVM::DEFAULT_PARAMS
=> {:thread_vm_stack_size=>1048576,
    :thread_machine_stack_size=>1048576,
    :fiber_vm_stack_size=>131072,
    :fiber_machine_stack_size=>524288}

Now run the program below and modify the upper limit from 7600 to 7700. You will see it work with 7600 and exit with SystemStackError for 7700:

upper_limit = 7600 # 7600 works, 7700 doesn't

def so(arg, count)
  r = if count < upper_limit
    ('A') + so(arg, count + 1)
  else
    arg
  end
  r
end

so('a', 0)

In order to make it work with 7700, quit your irb and re-launch it with modified max stack size:

export RUBY_THREAD_VM_STACK_SIZE=2000000
irb

Link: [ruby-cvs:45648] ko1:r38478 (trunk): * vm.c: support variable VM/Machi via permalink.gmane.org