A ruby script to generate a complete R10k ready Puppetfile from a librarian puppet style file.

lp2r10k.rb

#!/usr/bin/env ruby

require 'librarian/puppet'

lockfile = Librarian::Puppet::Lockfile.new(
  Librarian::Puppet::Environment.new, 'Puppetfile.lock'
)

puppet_modules = {}

lockfile.load(File.read(lockfile.path)).manifests.each do |mod|
  mod_type = mod.source.class.name.split('::').last.downcase.to_sym
  puppet_modules[mod_type] = {} unless puppet_modules.key?(mod_type)
  puppet_modules[mod_type][mod.name.to_sym] = {}
  case mod_type
  when :forge
    puppet_modules[mod_type][mod.name.to_sym][:version] = mod.version.to_s
  when :git
    puppet_modules[mod_type][mod.name.to_sym][:git] = mod.source.uri
    puppet_modules[mod_type][mod.name.to_sym][:ref] = mod.source.ref
  end
end

puppetfile = []
File.readlines('Puppetfile').each do |line|
  break if line =~ /#{Regexp.escape("# Puppetfile.lock dependencies")}/
  puppetfile << line
end
puppetfile.pop if puppetfile.last == "\n"

puppetfile << "\n# Puppetfile.lock dependencies"

puppet_modules.each do |mod_type, sources|
  sources = sources.sort_by { |k, _v| k }
  puppetfile << "\n# #{mod_type} modules"
  sources.each do |mod, data|
    case mod_type
    when :forge
      next if puppetfile.grep(/#{Regexp.escape(mod)}/).any?
      puppetfile << "mod '#{mod}', '#{data[:version]}'"
    when :git
      next if puppetfile.grep(/#{Regexp.escape(data.to_h[:git])}/).any?
      puppetfile << "mod '#{mod}',"
      puppetfile << "    :git => '#{data[:git]}',"
      puppetfile << "    :ref => '#{data[:ref]}'"
    end
  end
end

puts puppetfile

For a long time there's been debate and unhappy faces when it comes to using librarian-puppet vs r10k vs g10k.

The main reason for librarian-puppet is it's awesome (but slow) dependency resolution. This script uses the librarian-puppet gem to parse and effectively reverse engineer your Puppetfile.lock to a complete Puppetfile with versions for forge module dependencies.

For example, a Puppetfile like this:

#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-stdlib'
mod 'puppetlabs-git'
mod 'puppetlabs-haproxy'
mod 'puppetlabs-java'
mod 'theforeman-foreman'

Gives the following output:

#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-stdlib'
mod 'puppetlabs-git'
mod 'puppetlabs-haproxy'
mod 'puppetlabs-java'
mod 'theforeman-foreman'

# Puppetfile.lock dependencies

# forge modules
mod 'puppet-archive', '2.3.0'
mod 'puppet-extlib', '2.0.1'
mod 'puppetlabs-concat', '2.2.1'
mod 'puppetlabs-apache', '2.3.1'
mod 'puppetlabs-apt', '2.4.0'
mod 'puppetlabs-postgresql', '5.3.0'

You have to run a librarian-Puppet install first to generate the .lock file. To replace your Puppetfile with the output of this do something like:

ruby lp2r10k.rb > Puppetfile.tmp ; mv Puppetfile.tmp Puppetfile

Performance improvements with the above complete Puppetfile and various tools:

With regular Puppetfile, no .lock file and librarian-puppet:
librarian-puppet install --clean  15.78s user 9.11s system 21% cpu 1:53.87 total

With complete Puppetfile and librarian-puppet:
librarian-puppet install --clean  16.14s user 9.59s system 38% cpu 1:06.24 total

With complete Puppetfile and r10k:
r10k puppetfile install --puppetfile Puppetfile --moduledir modules  1.50s user 1.58s system 24% cpu 12.628 total

With complete Puppetfile and g10k:
g10k -usemove -puppetfile -moduledir modules  0.37s user 1.57s system 44% cpu 4.399 total

Impressive!! From 2 minutes down to 5 seconds?? that's an amazing improvement

That is on a MacBook Pro with a Core i7. Of course. Mileage may vary when it comes to EC2 instances which generally have less powerful CPU’s and are prone to context switching due to a lower number of cores. Taking advantage of g10k’s concurrency (downloads, gz extracts) should still see an impressive improvement.