Delete all versions of all files in s3 versioned bucket using AWS CLI and jq.

delete_all_object_versions.sh

#!/bin/bash

bucket=$1

set -e

echo "Removing all versions from $bucket"

versions=`aws s3api list-object-versions --bucket $bucket |jq '.Versions'`
markers=`aws s3api list-object-versions --bucket $bucket |jq '.DeleteMarkers'`
let count=`echo $versions |jq 'length'`-1

if [ $count -gt -1 ]; then
        echo "removing files"
        for i in $(seq 0 $count); do
                key=`echo $versions | jq .[$i].Key |sed -e 's/\"//g'`
                versionId=`echo $versions | jq .[$i].VersionId |sed -e 's/\"//g'`
                cmd="aws s3api delete-object --bucket $bucket --key $key --version-id $versionId"
                echo $cmd
                $cmd
        done
fi

let count=`echo $markers |jq 'length'`-1

if [ $count -gt -1 ]; then
        echo "removing delete markers"

        for i in $(seq 0 $count); do
                key=`echo $markers | jq .[$i].Key |sed -e 's/\"//g'`
                versionId=`echo $markers | jq .[$i].VersionId |sed -e 's/\"//g'`
                cmd="aws s3api delete-object --bucket $bucket --key $key --version-id $versionId"
                echo $cmd
                $cmd
        done
fi

Came up with that version, using headless commands & specifying region & profile :
https://gist.github.com/andy-b-84/9b9df3dc9ca8f7d50cd910b23cea5e0e

This gist was very useful.

This error occurs when the aws command's default output format is not json:

parse error: Invalid numeric literal at line 2, column 0

This has a very simple fix:

Wherever aws command output is passed to jq, let the script specify --output=json.

For instance:

versions=`aws s3api list-object-versions --bucket $bucket |jq '.Versions'`

becomes

versions=`aws --output=json s3api list-object-versions --bucket $bucket |jq '.Versions'`

@kayomarz I think that might be a setting on your side - I don't need --output=json.

@kayomarz I think that might be a setting on your side - I don't need --output=json.

@l0b0 Yes, my aws CLI is configured with output = table (aws CLI output is no longer json) and this script results in parse error: Invalid numeric literal at line 2, column 0.

Using --output=json mentioned above can be used to fix the error.

you can use jq -r flag to remove quotation chars " from query results instead of sed btw

for some bizarre reason, this line does not work for me:
version_id="$(jq --raw-output ".[${index}].VersionId" <<< "$1")"

First time s3 user here. AWS sneakily by default enables object versions and does not make it obvious via UI, so you won't notice it until your bucket size reached ~10tb and billing account exploded.

I'm homeless now, but on the bright side, here's a faster version of the script: it deletes 999 items per loop until the bucket is empty.

(inspired by https://gist.github.com/nashjain/6119aecd5e8919d0818773a118d05ed6)

#!/bin/bash
# deletes ALL old versions of uploaded files from s3 bucket

# MacOS / BSD date:
# deleteBefore=$(date -v-7d +%F)
# Linux date:
deleteBefore="$(date -u -d '7 day ago' +%F) 

bucket=$1
fileName='aws_delete.json'
rm $fileName
echo "Removing versions older than ${deleteBefore} from $bucket..."
export AWS_PROFILE={YOUR_S3_PROFILE_NAME}

# repeat the following command until there are no more versions to delete
while true; do
# s3api delete-objects can handle upto 1000 records
  versionsToDelete=$(aws s3api list-object-versions --max-items 999 --bucket "$bucket" --query "Versions[?(LastModified<'$deleteBefore')].{Key: Key, VersionId: VersionId}")
  if [ -z "$versionsToDelete" ]; then
    break
  fi
  echo "Deleting versions: $versionsToDelete"
  cat <<EOF >$fileName
{"Objects":$versionsToDelete, "Quiet":true}
EOF
  aws s3api delete-objects --bucket "$bucket" --delete file://$fileName
done

echo "Delete successsful"

# We leave the aws_delete.json file, in case you want to later see what happened.