Enrico Stahn

The company I’m currently working for has one of those TextDrive life time accounts. We moved one of our projects from a dedicated server to this account which results in some errors based on the different ferret versions. The pragmatic way was to rebuild the index (<Modelname>.rebuild_index) which I did but without success because I straight run into the following error:

NoMemoryError (failed to allocate memory):
    (eval):3:in `each_hash'
    (eval):3:in `all_hashes'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:396:in `select'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all'
    /vendor/rails/activerecord/lib/active_record/base.rb:424:in `find_by_sql'
    /vendor/rails/activerecord/lib/active_record/base.rb:994:in `find_every'
    /vendor/rails/activerecord/lib/active_record/base.rb:415:in `find'
    /vendor/acts_as_ferret-0.4.0/lib/local_index.rb:219:in `reindex_model'
    /vendor/acts_as_ferret-0.4.0/lib/local_index.rb:217:in `reindex_model'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:59:in `transaction'
    /vendor/rails/activerecord/lib/active_record/transactions.rb:95:in `transaction'
    /vendor/acts_as_ferret-0.4.0/lib/local_index.rb:216:in `reindex_model'
    /vendor/acts_as_ferret-0.4.0/lib/local_index.rb:47:in `rebuild_index'
    /vendor/acts_as_ferret-0.4.0/lib/local_index.rb:46:in `rebuild_index'
    /vendor/acts_as_ferret-0.4.0/lib/local_index.rb:23:in `ensure_index_exists'
    /vendor/acts_as_ferret-0.4.0/lib/local_index.rb:14:in `ferret_index'
    /app/models/place.rb:59:in `find_storage_by_contents'
    /app/models/place.rb:54:in `full_text_search'
    /app/controllers/places_controller.rb:38:in `search'
    ...

Well, I was a bit surprised as it worked fine on the dedicated server and we’d the same version of act_as_ferret on both machines.

Let’s test

# ruby script/console production
>> Place.find(:first).nil?
=> false
>> Place.find(:all, :limit => 10).nil?
=> false
>> Place.find(:all, :limit => 100).nil?
=> false
>> Place.find(:all, :limit => 1000).nil?
(eval):3:in `each_hash': failed to allocate memory (NoMemoryError)
        from (eval):3:in `all_hashes'
        from ./script/../config/../config/../vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:396:in `select'
        from ./script/../config/../config/../vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all'
        from ./script/../config/../config/../vendor/rails/activerecord/lib/active_record/base.rb:424:in `find_by_sql'
        from ./script/../config/../config/../vendor/rails/activerecord/lib/active_record/base.rb:994:in `find_every'
        from ./script/../config/../config/../vendor/rails/activerecord/lib/active_record/base.rb:415:in `find'
        from (irb):8:in `irb_binding'
        from /usr/local/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'
        from /usr/local/lib/ruby/1.8/irb/workspace.rb:52

Fetching 1000 rows causes the error which is the default reindex batch size of acts_as_ferret.

so far so good

Let’s change the batch size as the documentation describes, it shouldn’t be a big deal. Well, shouldn’t be, but the current documentation seems to be build for the trunk and not for the stable version (0.4.0), so we have to change one line of the acts_as_ferret sources.

Open local_index.rb and change line 210 so that it fit your needs. I got good results with 100.

    def reindex_model(index, model = aaf_configuration[:class_name].constantize)
      # index in batches of 1000 to limit memory consumption (fixes #24)
      # TODO make configurable through options
      batch_size = 100
      ...
    end