Code for generating digital averages of faces extracted from a macos photo library, or any collection of images with face metadata in an XMP file.

.Generating averages of faces.md

What?

Inspired by the work of Jason Salavon I wanted to extract all the photos of my face and generate the average of them.

This is the ruby code for doing that.

Pre-requisites

osxphotos

Install osxphotos and use it to extract the images of the face you want to average. In my case, that's me, but probably you'll want to use your own name.

I chose to place mine in ~/Pictures/Averages and to divide them up into folders by year. The ruby code that follows assumes that you have a root folder with sub-folders of jpgs you want to extract faces from by reading XMP metadata files. This is the command I used:

osxphotos export --person "Murray Steele" --sidecar xmp --convert-to-jpeg --jpeg-quality 1.0 --jpeg-ext jpg --directory "{created.year}" --skip-edited --download-missing --verbose --only-photos ~/Pictures/Averages

You can experiment with osxphotos options to get different results, but the ruby code might not work if you try exif or json metadata, or a different, more nested, destination folder structure.

The macos photos library is a SQLite DB and I could have written some ruby code to execute the relevant SQL to extract the metadata myself. That was my original plan TBH, but the osxphotos project has done all the heavily lifting already, and I'd probably end up using it as source for tranlisterating bits of the python to ruby for my purposes. This seemed boring so I decided not to.

imagemagick

Install imagemagick because the minimagick gem I use is a wrapper to the command line and doesn't install it for you.

Install

Having run the osxphotos step above, copy the ruby files below into the place you extracted the images to. Then run bundle install.

Process

Run the ruby script, passing in the folder of images you want to process. In this example we generate averages of all the photos of my face from 2019:

bundle exec ruby average.rb 2019

This will leave 2019-averaged.jpg in the same folder as the ruby script. It'll also leave a bunch of semi processed images in the 2019 folder. These can all be deleted as the code isn't smart enough to skip bits it's already done. The whole folder can be deleted if you don't want to run it again.

You can comment out various parts of the process to avoid running things twice if you encounter problems.

E.g. if you've extracted the faces, but normalizing them is broken, you can comment out the call to extract_faces and run the script from there. Or, if normalizing is fine, but orienting is broken, comment out the calls to both extract_faces and normalize_faces.

If I put more effort into this it would work via command line options or be smart enough to not repeat itself. I might also have made the filenames less verbose (foo.jpg.face.jpg.normalized.jpg.oriented.jpg is ... a choice).

Output

Marvel at your digitally average face. It might look something like these images averaging my face from 2024, 2019 and 2016:

• 2024 (143 images):
• 2019 (321 images):
• 2016 (44 images):

In my opinion the output looks better when based on more photos as it becomes more dreamy and blurry. Contrast the 2019 version which was calculated from 321 images, with the 2024 version based on 143 images, or the 2016 version based on only 44 images. An actual specific face is much clearer in the 2016 version, whereas it becomes increasinly abstract in 2024 and 2019 as the image count used increases.

Future work?

In a future version it might be interesting to extract some more data about the face position and do more than simply scale the faces to the same size to align them. Perhaps rotating the image to align the eyes, and scale the image more accurately based on the detected eye positions.

average.rb

require 'nokogiri'
require 'mini_magick'

folder_name = ARGV[0]

def to_pixel(percentage, full) = percentage * full

def to_crop(face_rect, image_height, image_width, pad = 1.6)
  face_height = to_pixel(face_rect[:height], image_height) * pad
  face_width = to_pixel(face_rect[:width], image_width) * pad
  face_center_x = to_pixel(face_rect[:center_x], image_width)
  face_center_y = to_pixel(face_rect[:center_y], image_height)
  face_top_left_x = face_center_x - (face_width / 2.0)
  face_top_left_y = face_center_y - (face_height / 2.0)
  "#{face_width.floor}x#{face_height.floor}+#{face_top_left_x.floor}+#{face_top_left_y.floor}"
end

def get_rect_from_xmp(image_name)
  xmp_name = "#{image_name}.xmp"
  xmp_doc = Nokogiri::XML(File.read(xmp_name))
  area = xmp_doc.xpath(".//mwg-rs:RegionList//rdf:li[.//mwg-rs:Name[text() = 'Murray Steele']]//mwg-rs:Area", :"mwg-rs" => 'http://www.metadataworkinggroup.com/schemas/regions/', :rdf => 'http://www.w3.org/1999/02/22-rdf-syntax-ns#')
  {
    height: area.xpath('.//stArea:h/text()', stArea: "http://ns.adobe.com/xmp/sType/Area#").first.text.to_f,
    width: area.xpath('.//stArea:w/text()', stArea: "http://ns.adobe.com/xmp/sType/Area#").first.text.to_f,
    center_x: area.xpath('.//stArea:x/text()', stArea: "http://ns.adobe.com/xmp/sType/Area#").first.text.to_f,
    center_y: area.xpath('.//stArea:y/text()', stArea: "http://ns.adobe.com/xmp/sType/Area#").first.text.to_f
  }
rescue
  {}
end

def extract_faces(folder)
  Dir["#{folder}/*.jpg"].each do |image_name|
    next if image_name.include? 'face'
    puts "Extracting face for: #{image_name}"
    image = MiniMagick::Image.open(image_name)

    center_rect = get_rect_from_xmp(image_name)
    if center_rect[:height] == center_rect[:width]
      puts "Face rect looks weird, skipping #{center_rect}"
      next
    else
      image.crop to_crop(center_rect, image.height, image.width)

      image.write "#{image_name}.face.jpg"
    end
  end
end


def upper_size_of_faces(folder)
  sizes = Dir["#{folder}/*.face.jpg"].map do |image_name|
    image = MiniMagick::Image.open(image_name)
    [image.width, image.height]
  end
  return {} if sizes.size.zero?
  {
    max_width: sizes.max_by(&:first)&.first,
    max_height: sizes.max_by(&:last)&.last,
    mean_width: sizes.sum(&:first) / sizes.size.to_f.ceil,
    mean_height: sizes.sum(&:last) / sizes.size.to_f.ceil,
    median_width: sizes.sort_by(&:first)[sizes.size / 2].first.ceil,
    median_height: sizes.sort_by(&:last)[sizes.size / 2].last.ceil
  }
end

def normalize_faces(folder, max_width:, max_height:)
  resize_command = "#{max_width}x#{max_width}!"
  Dir["#{folder}/*.face.jpg"].each do |image_name|
    image = MiniMagick::Image.open(image_name)
    puts "#{image_name} - resizing from #{image.width}x#{image.height} to #{resize_command}"
    image.resize resize_command
    image.write "#{image_name}.normalized.jpg"
  end
end

extract_faces folder_name
sizes = upper_size_of_faces(folder_name)

exit! if sizes.empty?

normalize_faces(folder_name, max_width: sizes[:mean_width], max_height: sizes[:mean_width])

normalized_image = MiniMagick::Image.open(Dir["#{folder_name}/*.normalized.jpg"].first)
normalized_size = [normalized_image.width, normalized_image.height]

def orientation(folder)
  Dir["#{folder}/*.normalized.jpg"].each do |image_name|
    image = MiniMagick::Image.open(image_name)
    puts "#{image_name} - orientation: #{image.data['orientation']} exif:orientation: #{(image.data['properties'] || {})['exif:Orientation']}"
  end
end

def average_faces(folder, normalized_size)
  count = 1
  first_file, *rest_files = *Dir["#{folder}/*.oriented.jpg"]
  pixels = MiniMagick::Image.open(first_file).get_pixels
  rest_files.each do |image_name|
    puts "#{image_name} - processing pixels"
    count += 1
    image = MiniMagick::Image.open(image_name)
    pixels_to_add = image.get_pixels
    pixels_to_add.each.with_index do |row, row_idx|
      row.each.with_index do |pixel, pixel_idx|
        pixel.each.with_index do |colour_value, colour_idx|
          pixels[row_idx][pixel_idx][colour_idx] += colour_value
        end
      end
    end
  end
  pixels.map! do |row|
    row.map! do |pixel|
      pixel.map! { |colour| (colour / count.to_f).floor }
    end
  end
  #
  #
  # all_pixels = Dir["#{folder}/*.oriented.jpg"].map do |image_name|
  #   image = MiniMagick::Image.open(image_name)
  #   puts "#{image_name} - pulling pixels"
  #   image.get_pixels
  # end
  # puts "Zipping"
  # head, *rest = *all_pixels
  # next_zip = head.zip(*rest)
  # next_next_zip = next_zip.map { |x| h, *r = *x; h.zip(*r) }
  # next_next_next_zip = next_next_zip.map { |x| x.map { |y| h, *r = *y; h.zip(*r) }}
  # puts "Averaging"
  # averages = next_next_next_zip.map { |row| row.map { |cell| cell.map { |colour| (colour.sum / colour.size.to_f).floor }}}
  image = MiniMagick::Image.get_image_from_pixels(pixels, normalized_size, 'rgb', 8, 'jpg')
  image.write("#{folder}-averaged.jpg")
end

def fix_orientation(folder)
  Dir["#{folder}/*.normalized.jpg"].each do |image_name|
    image = MiniMagick::Image.open(image_name)
    puts "#{image_name} - orientation: #{image.data['orientation']}"
    image.combine_options do
      case image.data['orientation']
      when 'TopLeft'
        # 1: Do nothing
      when 'TopRight'
        # 2: Flip horizontally
        image.flop
      when 'BottomRight'
        # 3: Rotate 180 degrees
        image.rotate 180
      when 'BottomLeft'
        # 4: Flip vertically
        image.flip
      when 'LeftTop'
        # 5: Rotate 90 degrees and flip horizontally (transpose)
        image.rotate 90
        image.flop
      when 'RightTop'
        # 6: Rotate 90 degrees
        image.rotate 90
      when 'RightBottom'
        # 7: Rotate 90 degrees and flip vertically (transverse)
        image.rotate 90
        image.flip
      when 'LeftBottom'
        # 8: Rotate 270 degrees
        image.rotate 270
      else
        # do nothing - it's probably the same as TopLeft
      end
      image.orient 'top-left'
    end
    image.write("#{image_name}.oriented.jpg")
  end
end

fix_orientation(folder_name)

average_faces(folder_name, normalized_size)

Gemfile

source 'https://rubygems.org'

gem 'mini_magick'
gem 'nokogiri'

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    mini_magick (5.0.1)
    nokogiri (1.17.2-aarch64-linux)
      racc (~> 1.4)
    nokogiri (1.17.2-arm-linux)
      racc (~> 1.4)
    nokogiri (1.17.2-arm64-darwin)
      racc (~> 1.4)
    nokogiri (1.17.2-x86-linux)
      racc (~> 1.4)
    nokogiri (1.17.2-x86_64-darwin)
      racc (~> 1.4)
    nokogiri (1.17.2-x86_64-linux)
      racc (~> 1.4)
    racc (1.8.1)

PLATFORMS
  aarch64-linux
  arm-linux
  arm64-darwin
  x86-linux
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  mini_magick
  nokogiri

BUNDLED WITH
   2.5.16