ruby-dependency-mess.md

Dependency mess in Ruby gems

I decided to follow up on my email about requires vs. autoload in Ruby. I thought I'd do more research to figure out how to best manage requiring code in a Ruby gem. I decided to take a look at a few popular Ruby projects to get inspired.

The results are... not inspiring.

Projects

Bundler

Code: https://github.com/bundler/bundler/tree/060e2f953c8f08dd7bed9711863e6341eebe5fe2

Ruby's dependency management system.

• Uses autoload.
• Weird circular-dependency: lib/bundler/ui.rb autoloads lib/bundler/ui/rg_proxy.rb but the latter require the former.
• Unnecessary requires:
  • lib/bundler/gem_tasks.rb requires bundler/gem_helper even though autoload for GemHelper is already in lib/bundler.rb.
  • Both lib/bundler.rb and lib/bundler/rubygems_ext.rb (and a bunch of others) require pathname.
• Requires bundler in spec_helper.rb but also sometimes requires particular files in some specs.

Capybara

Code: https://github.com/teamcapybara/capybara/tree/03092d5c7f3673d68548ee6e36ea1978449ecac5

Popular framework for feature tests.

• Used to use autoload but removed for thread safety in https://github.com/teamcapybara/capybara/commit/c3e75f8988f640d3587bffbc48ead011ef2665d3
• Many files required in lib/capybara.rb but then also many required in other files. Unclear how it's decided which requires go where from the first glance.
• Requires capybara in spec_helper.rb but also sometimes requires particular files in some specs.

Jekyll

Code: https://github.com/jekyll/jekyll/tree/76a0fc38887e684f65237dd35861a54dc4ab97e2

Popular static website generator.

• Uses autoload for most of its code, but there are some exceptions where require is used (why?).
• Namespacing doesn't match file paths (e.g. Jekyll::CollectionReader is in lib/jekyll/readers/collection_reader.rb).
• Some unnecessary requires but not too bad:
  • set required in lib/jekyll.rb and lib/jekyll/cleaner.rb.
• Requires jekyll in test/helper.rb but sometimes requires individual files too in particular tests (why?).

Pry

Code: https://github.com/pry/pry/tree/cebad88656f5c092ce01c30c6c0a279cef68ca8f

Popular REPL, replacement for IRB.

• Does not use autoload.
• Sometimes has a file to load all classes in a module (e.g. lib/pry/helpers.rb), sometimes not (lib/pry/command.rb defines Pry::Command that is both a class and a namespace, but constants namespaced with Pry::Command:: are not loaded in that file).
• Inconsistencies between modules and paths, e.g. files in lib/pry/commands have classes namespaced with Pry::Command:: (singular).
• Some third-party gems required in lib/pry.rb, some in other files.
• Unnecessary requires:
  • coderay required in lib/pry.rb and one more place.
• Requires pry/testable in spec/helper.rb which in turn requires pry. Does not require individual files in tests.

Rack

Code: https://github.com/rack/rack/tree/ab008307cbb805585449145966989d5274fbe1e4

Webserver interface that most Ruby web frameworks use.

• Uses autoload. Oddly, some non-top-level autoloads are defined in lib/rack.rb but some in separate files. However, file paths match namespaces in all cases.
• Rack has no third-party dependencies so we don't see any requires for other gems.
• No stdlib requires in lib/rack.rb. It seems each file tries to be explicit about its stdlib dependencies.
• Unnecessary requires:
  • Even though autoloads for File and Utils are defined in lib/rack.rb, lib/rack/file.rband lib/rack/utils.rb are explicitly required in other files.
• Requires whatever it wants in tests.

Rubocop

Code: https://github.com/bbatsov/rubocop/tree/6fb5a84976567549302881de4923e21819040756

The most popular linter for Ruby.

• Does not use autoload.
• Uses a lot of require_relative, not sure why.
• Almost completely flat require structure, lib/rubocop.rb has over 500 lines of requires.
• Some third-party gems required in lib/rubocop.rb, some in other files.
• Unnecessary requires:
  • lib/rubocop.rb requires set but many cops also require it.
• Requires rubocop and a bunch of things in spec_helper.rb but then requires more things in individual spec files.

Conclusions

I can't help but feel that Ruby's require/autoload system is broken if even the most popular and mature projects can't get it right. The biggest problem seems to be global scope where everything is dumped when it's required. This makes enforcing correct requires per file almost impossible or at least impractical (you could run each spec file in a separate rspec process, but who does that?).

Therefore, the simplest approach seems to be to use as few requires/autoloads in as few places as possible and for any interaction with the code (specs, external use) simply require the whole gem.

Autoload can be beneficial because it frees you from figuring the right order of loading files which can be cumbersome if you don't do explicit requires per file (you can just list everything alphabetically), but has its limitations too. Sometimes, you don't want to load things lazily, you need to be sure they are always loaded.