Introduction:
We can list, get and put the objects in S3 Bucket if you are using a single account on AWS by applying simple policies as you are the main account owner and the bucket too. But there are some limitations arises when you have multiple accounts and you are uploading objects from one account user/role to another account bucket. Here in this article, I am addressing multiple issues and their resolution in this regard.
LISTING / PUTTING AN OBJECT TO ANOTHER ACCOUNT BUCKET.
GETTING AN OBJECT FROM YOUR OWN BUCKET HAVING DIFFERENT OWNERS (REPHRASING: OBJECT PUT BY ANOTHER ACCOUNT OWNER TO YOUR
ACCOUNT BUCKET AND YOU ARE NOT ABLE TO GET THAT OBJECT )
Where to Use?
Currently, we are uploading database dump from Production to Staging they are currently on the same account, therefore, it works without any issue. But once we have separate Stage or multiple accounts we will be facing these issues.
Instructions:
How to replicate the issue?
You must have at least two accounts. I am considering account name as below
Sandbox ( AccountID: 1235158225945 )
Development ( AccountID: 1627109621622 )
Sandbox and Development account has dedicated IAM roles/users
Sandbox account IAM role: DbAnonymizerRestoreSandbox
Development account IAM role: DbAnonymizerRestore
An instance in both accounts with different IAM roles/users
Sandbox account Instance name with IAM role as DbAnonymizerRestoreSandbox : sandboxinstance
Development account Instance name with IAM role as DbAnonymizerRestore : developmentinstance
Bucket in Sandbox Account.
It means your destination bucket is in Sandbox account and you want to access it from another account in our case you are
accessing bucket from developmentinstance.
Name it like: db-dump.sandbox.testexample.com
Here we have the instance, bucket and IAM roles. We are missing the policies yet which will be added on requirements below.
So to summarise above with our current infra and requirements "We have a bucket in a single account and IAM roles in separate account: These IAMroles want to access bucket ( separate account ) contents with full access".
aws s3 ls s3://db-dump.sandbox.testexample.com/
It will list down all the content present in the bucket once you have the policy allowed for the IAM role DbAnonymizerRestoreSandbox to access this bucket. Here is the policy ( which is responsible for Put, Get, List ) the bucket. For now, ignore the last allowing policy s3:PutObjectAcl I will explain it later in the next section.
Listing bucket Content
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::db-dump.sandbox.testexample.com",
"arn:aws:s3:::db-dump.sandbox.testexample.com/*"
]
}
]
}
After adding the policy in SandboxAccount and run below commands from sandboxinstance we have an output below
arn:aws:iam::1235158225945:policy/s3-dump-allow
############# listing the objects inside the bucket ##############
centos@ip-172-30-1-148 ~ $aws s3 ls s3://db-dump.sandbox.testexample.com/
2019-11-04 09:34:43 5 file
2018-06-22 11:09:17 9954041709 testexample.anonymized.sql
2018-06-25 10:12:33 6304512898 testexample.full.sql
########### Copy objects from bucket to instance ##################
centos@ip-172-30-1-148 ~ $aws s3 cp
s3://db-dump.sandbox.testexample.com/file.
download: s3://db-dump.sandbox.testexample.com/file2 to ./file
########## Create file and Copy to bucket ########################
centos@ip-172-30-1-148 ~ $touch newfile
centos@ip-172-30-1-148 ~ $ aws s3 cp newfile
s3://db-dump.sandbox.testexample.com/
upload: ./newfile1 to s3://db-dump.sandbox.testexample.com/newfile
######### Again check if the file uploaded in the bucket #########
centos@ip-172-30-1-148 ~ $aws s3 ls s3://db-dump.sandbox.testexample.com/
2019-11-04 11:47:45 0 newfile
2019-11-04 09:34:43 5 file
2018-06-22 11:09:17 9954041709 testexample.anonymized.sql
2018-06-25 10:12:33 6304512898 testexample.full.sql
At this stage, we can list, put and get the object from the same account.
Major Issue:
Now you need to SSH the instance which resides in account Development. Once you successfully login to the instance you need to list, put and get the objects from the Bucket s3://db-dump.sandbox.testexample.com/
Listing / Putting an object from one account.
From the above output, it clearly shows that even I am not able to list the objects. To fix this make sure you need to add IAM role (DbAnonymizerRestore) as a principle inside s3://db-dump.sandbox.testexample.com/ bucket policy. Screenshot attached for reference
Listing - Copying - Getting Objects
Listing object from another account
########### Listing the object from Development Instance IAMrole #########
[root@db-anonymizer centos]# aws s3 ls s3://db-dump.sandbox.testexample.com/
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::1627109621622:role/DbAnonymizerRestore"
]
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::db-dump.sandbox.testexample.com",
"arn:aws:s3:::db-dump.sandbox.testexample.com/*"
]
}
]
}
Screenshot attached for reference.
Now you can easily list and put the objects inside the bucket from another account to s3://db-dump.sandbox.testexample.com/ See below for reference. Make sure you have already the policy to access the bucket mentioned in 1st section.
Allowing Another Account IAM role
############# listing the objects from anotheraccount instance
##############
[root@db-anonymizer centos]# aws s3 ls
s3://db-dump.sandbox.testexample.com/
PRE db/
2019-11-04 09:34:43 5 file
2019-11-04 11:47:45 0 newfile
2018-06-22 11:09:17 9954041709 testexample.anonymized.sql
2018-06-25 10:12:33 6304512898 testexample.full.sql
############# copying the objects to bucket from anotheraccount instance
##############
[root@db-anonymizer centos]# aws s3 cp fileforsandbox
s3://db-dump.sandbox.testexample.com/
upload: ./assum2 to s3://db-dump.sandbox.testexample.com/fileforsandbox
All Good till now.
Summarizing above you can easily copy and listing the objects to another bucket.
Getting an object which is put by another account to your account bucket.
Go back to Sandbox Account Instance and check if the copied object is present or not.
############# listing the objects from anotheraccount instance
##############
centos@ip-172-30-1-148 ~ $ aws s3 ls s3://db-dump.sandbox.testexample.com/
PRE db/
2019-11-04 12:13:30 0 assum
2019-11-04 09:34:43 5 file
2019-11-04 11:47:45 0 newfile
2018-06-22 11:09:17 9954041709 testexample.anonymized.sql
2018-06-25 10:12:33 6304512898 testexample.full.sql
############# copying the objects from anotheraccount instance
##############
centos@ip-172-30-1-148 ~ $ aws s3 cp
s3://db-dump.sandbox.testexample.com/assum.
fatal error: An error occurred (403) when calling the HeadObject
operation: Forbidden
You are getting an error with forbidden. To debug this we need to check the owner by using the below command.
Allowing Another Account IAM role
Listing - Getting object from another account
############# listing the objects with owner detail and you found the
file with different owner ##############
centos@ip-172-30-1-148 ~ $ aws s3api list-objects --bucket
db-dump.sandbox.testexample.com
Sandbox account uploaded files with this owner name: "DisplayName": "aws.sandbox", and the files which are uploaded by Development shows "DisplayName": "aws.outsource.importer".
Solution
So to resolve this forbidden issue and getting an object from another owner you need to copy the files in a different way using:
############# listing the objects with owner detail and you found the
file with different owner ##############
[root@db-anonymizer centos]# aws s3 cp --acl bucket-owner-read
filetocopywithfullaccess s3://db-dump.sandbox.testexample.com/
upload: ./filetocopywithfullaccess to
s3://db-dump.sandbox.testexample.com/filetocopywithfullaccess
I have uploaded the object with ACL. If you are not adding s3:PutObjectAcl you will get an error like: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Object Details
Putting Object with ACL
############# listing the objects from anotheraccount instance
##############
centos@ip-172-30-1-148 ~ $ aws s3 ls s3://db-dump.sandbox.testexample.com/
PRE db/
2019-11-04 12:13:30 0 assum
2019-11-04 09:34:43 5 file
2019-11-04 12:16:00 0 file3fromsandbox
2019-11-04 12:34:24 0 filetocopywithfullaccess
2019-11-04 11:47:45 0 newfile
2018-06-22 11:09:17 9954041709 testexample.anonymized.sql
2018-06-25 10:12:33 6304512898 testexample.full.sql
############# Getting objects which is uploaded by anotheraccount
instance ##############
centos@ip-172-30-1-148 ~ $ aws s3 cp
s3://db-dump.sandbox.testexample.com/filetocopywithfullaccess.
download: s3://db-dump.sandbox.testexample.com/filetocopywithfullaccess to
./filetocopywithfullaccess
Successful
Related articles
AWS provides the solution in this article https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-owner-access/
Tested and Deployment:
I have tested it with an importer-outsource and sandbox account and it works like a charm. So we need to follow while deploying multiple environments.
Putting Object with ACL