Delete all versions of all files in s3 versioned bucket using AWS CLI and jq.

delete_all_object_versions.sh

#!/bin/bash

bucket=$1

set -e

echo "Removing all versions from $bucket"

versions=`aws s3api list-object-versions --bucket $bucket |jq '.Versions'`
markers=`aws s3api list-object-versions --bucket $bucket |jq '.DeleteMarkers'`
let count=`echo $versions |jq 'length'`-1

if [ $count -gt -1 ]; then
        echo "removing files"
        for i in $(seq 0 $count); do
                key=`echo $versions | jq .[$i].Key |sed -e 's/\"//g'`
                versionId=`echo $versions | jq .[$i].VersionId |sed -e 's/\"//g'`
                cmd="aws s3api delete-object --bucket $bucket --key $key --version-id $versionId"
                echo $cmd
                $cmd
        done
fi

let count=`echo $markers |jq 'length'`-1

if [ $count -gt -1 ]; then
        echo "removing delete markers"

        for i in $(seq 0 $count); do
                key=`echo $markers | jq .[$i].Key |sed -e 's/\"//g'`
                versionId=`echo $markers | jq .[$i].VersionId |sed -e 's/\"//g'`
                cmd="aws s3api delete-object --bucket $bucket --key $key --version-id $versionId"
                echo $cmd
                $cmd
        done
fi

First time s3 user here. AWS sneakily by default enables object versions and does not make it obvious via UI, so you won't notice it until your bucket size reached ~10tb and billing account exploded.

I'm homeless now, but on the bright side, here's a faster version of the script: it deletes 999 items per loop until the bucket is empty.

(inspired by https://gist.github.com/nashjain/6119aecd5e8919d0818773a118d05ed6)

#!/bin/bash
# deletes ALL old versions of uploaded files from s3 bucket

# MacOS / BSD date:
# deleteBefore=$(date -v-7d +%F)
# Linux date:
deleteBefore="$(date -u -d '7 day ago' +%F) 

bucket=$1
fileName='aws_delete.json'
rm $fileName
echo "Removing versions older than ${deleteBefore} from $bucket..."
export AWS_PROFILE={YOUR_S3_PROFILE_NAME}

# repeat the following command until there are no more versions to delete
while true; do
# s3api delete-objects can handle upto 1000 records
  versionsToDelete=$(aws s3api list-object-versions --max-items 999 --bucket "$bucket" --query "Versions[?(LastModified<'$deleteBefore')].{Key: Key, VersionId: VersionId}")
  if [ -z "$versionsToDelete" ]; then
    break
  fi
  echo "Deleting versions: $versionsToDelete"
  cat <<EOF >$fileName
{"Objects":$versionsToDelete, "Quiet":true}
EOF
  aws s3api delete-objects --bucket "$bucket" --delete file://$fileName
done

echo "Delete successsful"

# We leave the aws_delete.json file, in case you want to later see what happened.