Motivation
If you have a strict process when running AWS applications or creating resources with restrictive AWS IAM permission, you must have gone through the same pain as me:
- Run your AWS command / resource creation
- Fail due to missing permission
- Update the permission manually (or even more painful: the permissions are defined as code, you need to wait for a PR review)
- Run again
- Fail due to another missing permission
- Update again
- Loop…
In the past, I spent hours grinding with these permissions and saw my number of commits become top 10 in my organisation on GitHub. My manager at the time told me a good developer should make their development experience better.
Now I have a chance to help mitigate this issue and hopefully it helps others too.
Prerequisite
Ideally, you have a sandbox AWS account where you can have god mode IAM permission, and a production account where you need to have highly restrictive IAM permission.
If you only have one account, you need to be able to create a god mode IAM role to do this process.
How to do it
Step 0
Create a CloudTrail in your sandbox account, most importantly, enable CloudWatch Logs
Step 1
In your sandbox account, create a god mode IAM role with full access to the services you are interacting with, e.g. S3 full access, IAM full access, with trust policy that allows your own sandbox account role to assume it.
Example trust policy
// Trust policy of the god mode role: cloudtrail-test
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:sts::12345678:assumed-role/your_sandbox_role/your_user_email"
},
"Action": "sts:AssumeRole"
}
]
}Here, I create a role named cloudtrail-test , and later I will use my own role appadmin to assume it.
Step 2
Add the following profile to your ~/.aws/config, so that your own sandbox role can assume this god mode role
// The sandbox profile you should already configured
[sandbox]
...// The god mode role you just created
[profile sandbox-cloudtrail]
role_arn = arn:aws:iam::12345678:role/cloudtrail-test
source_profile = sandbox
The reason why we need another god mode role rather than your own appadmin role is, we don’t want your own operations to happen in the CloudTrail and add irrelevant permissions to the query results.
Run the following to validate you assume the role successfully:
AWS_PROFILE=sandbox-cloudtrail aws sts get-caller-identity
// It should print the following
{
"UserId": "BLABLABLABLA:botocore-session-12345678",
"Account": "12345678",
"Arn": "arn:aws:sts::12345678:assumed-role/cloudtrail-test/botocore-session-12345678"
}Step 3
Run whatever AWS CLI / Terraform / AWS CDK using this profile.
For example: AWS_PROFILE=sandbox-cloudtrail aws s3 ls
Step 4
Wait for a few minutes (may be 5), go to CloudWatch -> Logs Insights -> choose your CloudWatch log group created in step 0, run the following query
fields @timestamp, @message
| sort @timestamp desc
| filter @message like 'cloudtrail-test'
| filter @message not like 'Alex.Luo' # This is for filter out the operations where I assume this roleIn each log record, you will see a lot of stuff, but find the following properties:
- eventName
- eventSource
- eventTime
- requestParameters.xxx
- userIdentity.sessionContext.sessionIssuer.arn
They are corresponding to:
- Action
- Service
- Time
- Resource
- Role for executing the action
Step 5
Find all the relevant actions and resources from the log, put them in your restrictive service IAM role/policy, and test it by running your command again.
A more realistic example
In my recent project, I needed to create an AWS IAM role (for Snowflake storage integration) with Terraform, and I used this method to refine the role that Terraform uses on CD pipeline.
After using the god mode role to run Terraform in dev environment, I queried what actions were used by running:
fields @timestamp, @message
| sort @timestamp desc
| filter @message like 'cloudtrail-test'
| filter @message not like 'GetCallerIdentity' # Filter out all other possible operations during the time
| filter @message like 'u-snowflake-access' # The resource you are interested inThe following actions appear in the log
AttachRolePolicyGetPolicyVersionGetPolicyCreatePolicyListEntitiesForPolicyListPolicyVersions
I added them to the role my CD pipeline uses, and ran Terraform in staging environment via GitHub Actions. The job succeeded, I ran the following query to retrieve used actions:
fields @timestamp, @message
| sort @timestamp desc
| filter @message like 'github-actions-role'
| filter @message like 's-snowflake-access'The following actions appear in the log
AttachRolePolicyGetPolicyVersionGetPolicyCreatePolicy
You can see the first 4 actions in the first query were used to create the staging resource.
Close enough! For other operation (update, delete etc), you might need to fine tune the permission again, but the initial creation process is easier now.
Advantages over other similar services
AWS does provide products like IAM simulator to help with this issue. Compared to those products, this approach is more helpful when you run CDK / CloudFormation, because:
- IAM simulator can only simulate one service at a time, this approach captures all services at the same time
- Sometimes you don’t even know what actions your CDK is executing until you hit a failure. In IAM Simulator, you need to specify the actions upfront.
Take it to another level
Ideas for making this even easier:
- IAM Access Analyzer
- Automate the process by using Glue (claw the CloudTrail log from S3) → Athena (grab the relevant actions and resources) → S3 event bridge → Lambda (Submit a PR to the production IAM definition repo)
Reference
Using AWS Athena and CloudTrail to shape permissions for a Machine Account
