Last active
July 25, 2022 21:09
-
-
Save nashjain/6119aecd5e8919d0818773a118d05ed6 to your computer and use it in GitHub Desktop.
Let's assume you backup a file every hour on AWS S3 and you want to clean up versions that are older than a week. However for the older versions, you want to leave one version per day just in case if you need them. Following script does that for you.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
deleteBefore=`date --date="1 week ago" +%F` | |
bucket=$1 | |
fileToDelete=$2 | |
fileName='aws_delete.json' | |
rm $fileName | |
echo "Removing all versions of $fileToDelete from $bucket" | |
versionsToDelete=`aws s3api list-object-versions --bucket "$bucket" --prefix "$fileToDelete" --query "Versions[?(LastModified<'$deleteBefore' && (contains(LastModified, 'T0') || contains(LastModified, 'T1') || contains(LastModified, 'T20') || contains(LastModified, 'T21') || contains(LastModified, 'T22')))].{Key: Key, VersionId: VersionId}"` | |
cat << EOF > $fileName | |
{"Objects":$versionsToDelete, "Quiet":true} | |
EOF | |
aws s3api delete-objects --bucket "$bucket" --delete file://$fileName | |
# s3api delete-objects can handle upto 1000 records | |
echo "Delete successsful" | |
# We leave the aws_delete.json file, in case you want to later see what happened. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For deleting all objects with the above command, use this snippet to loop through all object list 1000 objects at a time:
#Get old version Objects
echo "Old version objects under this prefix:"
cat "all-objects-$bucket.json" | jq '.[] | select(.IsLatest | not)' | jq -s '.' > "old-objects-$bucket.json"
no_of_obj=$(cat old-objects-$bucket.json | jq 'length')
i=0
while [ $i -lt $no_of_obj ]
do
next=$((i+999))
oldversions=$(cat "old-objects-$bucket.json" | jq '.[] | {Key,VersionId}' | jq -s '.' | jq .[$i:$next])
cat << EOF > deleted-files-start-index-$i.json
{"Objects":$oldversions, "Quiet":true}
EOF
echo "Deleting records from $i - $next"
aws s3api delete-objects --bucket "$bucket" --delete file://deleted-files-start-index-$i.json
let i=i+1000
done