AWS - Upload files to S3 via curl

s3_upload.sh

#!/bin/bash -x
#Date: 21/7/2017
#Author: Mohan
#Purpose: To upload files to AWS S3 via Curl
#Uploads file at the top level folder by default


#S3 parameters
S3KEY="XXXXXXXXXXX"
S3SECRET="XXXXXXXXXXXXXXXX"
S3BUCKET="storage-backup"
S3STORAGETYPE="STANDARD" #REDUCED_REDUNDANCY or STANDARD etc.
AWSREGION="s3-eu-west-1"

putS3(){

  file_path=$1
  aws_path=$2
  bucket="${S3BUCKET}"
  date=$(date -R)
  acl="x-amz-acl:private"
  content_type="application/x-compressed-tar"
  storage_type="x-amz-storage-class:${S3STORAGETYPE}"
  string="PUT\n\n$content_type\n$date\n$acl\n$storage_type\n/$bucket$aws_path${file_path##/*/}"
  signature=$(echo -en "${string}" | openssl sha1 -hmac "${S3SECRET}" -binary | base64)
  curl -s --retry 3 --retry-delay 10 -X PUT -T "$file_path" \
       -H "Host: $bucket.${AWSREGION}.amazonaws.com" \
       -H "Date: $date" \
       -H "Content-Type: $content_type" \
       -H "$storage_type" \
       -H "$acl" \
       -H "Authorization: AWS ${S3KEY}:$signature" \
       "https://$bucket.${AWSREGION}.amazonaws.com$aws_path${file_path##/*/}"
}

usage(){
  echo "Usage: $0 <absolutepath_to_file> <s3_folder_path>"
  echo "$0 '/tmp/storage_backup/storage-backup_07192017_112945.tar.gz' '/'"
}

#validate positional parameters are present
if [ $# -ne 2 ]; then
  usage
  echo "Exiting .."
  exit 2
fi

putS3 $1 $2

Hi
In ubuntu I had to change "function usage" for " usage()" for solve error -->s3_upload.sh: 15: s3_upload.sh: function: not found.

Now I have this error-->
"The request signature we calculated does not match the signature you provided. Check your key and signing method."

I was test credentials in aws cli and works fine.

Any idea for this issue ?

Kind Regards

iam encounter same error，have you found the solution?

tag

amazonaws.com$aws_path${file_path##/*/}
should be changed to
amazonaws.com$aws_path/${file_path##/*/}

If you have a curl version >= 7.75.0 you can add files to s3 like this

#!/bin/sh

FILE=$1

S3_ACCESS_KEY="<ACCESS_KEY>"
S3_SECRET_KEY="<SECRET_KEY>"
S3_BUCKET="<BUCKET_NAME>"
S3_REGION="<REGION>"

curl --progress-bar -X PUT \
    --user "${S3_ACCESS_KEY}":"${S3_SECRET_KEY}" \
    --aws-sigv4 "aws:amz:${S3_REGION}:s3" \
    --upload-file ${FILE} \
    https://${S3_BUCKET}.s3.${S3_REGION}.amazonaws.com

is working this script?

Hi,

I come across this question.

In ubuntu I had to change "function usage" for " usage()" for solve error

@jamontesg
is it an issue with Bash vs sh 'POSIX' syntax ? Were you using Bash ?

see Bash documentation functions or https://linuxize.com/post/bash-functions/ for example.

HTH

For me I had to change the host
-H "Host: $bucket.s3.${AWSREGION}.amazonaws.com" \

I have a local S3 bucket, if that makes any difference, but I also got the signature error, which I fixed by changing
string="PUT\n\n$content_type\n$date\n$acl\n$storage_type\n/$bucket$aws_path${file_path##/*/}"

to

string="PUT\n\n$content_type\n$date\n$acl\n$storage_type\n/$bucket/$aws_path${file_path##/*/}"