In this post we'll go over how we found a limitation in the current AWS CloudTrail logging features that limit detection capabilities of possible abuse against AWS DynamoDB, in the event of the user's AWS IAM keys being compromised.
The objective of this article is to make users aware of this limitation and discuss alternatives for increasing the detection of attacks that might abuse it. This information was shared initially with AWS and only after going through the disclosure process with their security team did we decide to write it.
Our methodology for doing these simulations is quite straightforward. We start by selecting one service, of the many AWS has to offer, and we perform all possible actions that an adversary could do to benefit themselves.
We look at what logging is generated (most of the time in AWS CloudTrail) by these services and we start developing our capabilities (workflows, visualisations, alerting, remediation) based on that.
Recently, in a detection engineering hackaton, we focused on AWS DynamoDB.
In the detection mentioned above we were looking at the provider of AWS DynamoDB logs (event.provider), the action that results in listing tables (event.action), done through the awscli (user_agent.name) tool and by a particular user (user.name).
The detection can vary depending on specific needs. From excluding IPs that are allowed to make requests via awscli to particular awscli versions. Detections are usually tweaked according to client environments and as long as a log exists, we can develop custom workflows for alerting and remediation.
Continuing with our attack, as we now have the details (table name) we need, we'd like to export the records of the 3cs-detect-table. Still within dynamodb sub-command of awscli we can run:
$ aws dynamodb scan --table-name 3cs-detect-table --query "Items[*].[id.N,name.S]" --output text
3 frank
2 john
4 cees
1 tiago
As an attacker, you'd have to guess what to query in the table from which you'd like to export content, but as you'll shortly see, brute-forcing it would not be a problem.
It's also relatively safe to say that a table will hold some sort of "id" with "N"=numerical content.
Based on that assumption we set out to build a detection that would find the invocation of the scanning/reading of a table when ran through awscli. To our surprise, there was no record in AWS CloudTrail of us scanning/reading the table through awscli.
While the scanning operation would be normal when used in an application (and those requests would come from the SDK, not the CLI), our goal was to detect when this functionality of AWS DynamoDB was invoked through awscli.
Unsure of our results, we reached out to AWS.
The objective of this article is to make users aware of this limitation and discuss alternatives for increasing the detection of attacks that might abuse it. This information was shared initially with AWS and only after going through the disclosure process with their security team did we decide to write it.
Introduction & Methodology
A big part of the work done for our MDR platform comes from adversary simulation. Even though a lot of effort is put into developing our detection capabilities against a known killchain, we're constantly looking for alternative ways of attacking AWS workloads, regardless if the techniques are seen being used in the wild or not.Our methodology for doing these simulations is quite straightforward. We start by selecting one service, of the many AWS has to offer, and we perform all possible actions that an adversary could do to benefit themselves.
We look at what logging is generated (most of the time in AWS CloudTrail) by these services and we start developing our capabilities (workflows, visualisations, alerting, remediation) based on that.
Recently, in a detection engineering hackaton, we focused on AWS DynamoDB.
What was found
The simulation was simple, even though scary to think of: An adversary had gained AWS IAM user credentials that had, at a bare minimum, permissions to read AWS DynamoDB tables. With these credentials the adversary's goal was to pull all information from a table (without going through AWS Glue).
Let's assume that the attacker does not know which tables exist in the account for which credentials were stolen. Through awscli, dynamodb allows us to list-tables:
$ aws dynamodb list-tables
{
"TableNames": [
"3cs-detect-table"
]
}
"TableNames": [
"3cs-detect-table"
]
}
This event will generate a log that we can quickly identify as ListTables.
In the detection mentioned above we were looking at the provider of AWS DynamoDB logs (event.provider), the action that results in listing tables (event.action), done through the awscli (user_agent.name) tool and by a particular user (user.name).
The detection can vary depending on specific needs. From excluding IPs that are allowed to make requests via awscli to particular awscli versions. Detections are usually tweaked according to client environments and as long as a log exists, we can develop custom workflows for alerting and remediation.
Continuing with our attack, as we now have the details (table name) we need, we'd like to export the records of the 3cs-detect-table. Still within dynamodb sub-command of awscli we can run:
$ aws dynamodb scan --table-name 3cs-detect-table --query "Items[*].[id.N,name.S]" --output text
3 frank
2 john
4 cees
1 tiago
Now, we did make an assumption when running the above command: the schema of the table.
As an attacker, you'd have to guess what to query in the table from which you'd like to export content, but as you'll shortly see, brute-forcing it would not be a problem.
It's also relatively safe to say that a table will hold some sort of "id" with "N"=numerical content.
Based on that assumption we set out to build a detection that would find the invocation of the scanning/reading of a table when ran through awscli. To our surprise, there was no record in AWS CloudTrail of us scanning/reading the table through awscli.
While the scanning operation would be normal when used in an application (and those requests would come from the SDK, not the CLI), our goal was to detect when this functionality of AWS DynamoDB was invoked through awscli.
Unsure of our results, we reached out to AWS.
AWS Response
AWS has a track record of promptly addressing all remarks or concerns about security of the their cloud, and we got a chance of seeing that for ourselves.
Below is a timeline of the interactions:
- Report/Questions are sent to the AWS Security Team (27/03)
- Acknowledgement of the report and follow up date scheduled (27/03)
- Extension of the follow up date due to current health situation (03/04)
- Closure of the case and further details about the finding (10/04)
According to AWS Security this issue had been reported before and the AWS DynamoDB team is "actively working on adding logging for these types of activities" as they are currently "gathering data on the feature".
While it is not possible to have the logging of this activity for all regions, AWS did information that
users who want the functionality enabled can do so through a Support Request for workloads in US East (Ohio) and US West (Oregon) regions.
Even though AWS didn't confirm, we're pretty sure that this feature will be rolled out to other regions soon and eventually be part of standard AWS CloudTrail functionalities.
Final Remarks
It's funny how creativity plays such a big role in the development of attack vectors. In this case it all started with a "what would AWS CloudTrail register if we tried to export an entire DynamoDB table straight to a file using stolen creds?"
As important as it is to maintain commitment to the development of detection techniques against a known standard - 3CS MDR covers 80% of the AWS MITRE matrix - detection engineering is never complete and requires constant tuning, development and forward thinking.
After getting confirmation from AWS that it would not be possible to detect these activities with the current version of AWS DynamoDB (at least in regions outside of the ones mentioned above) we decided to increase the risk score of the MDR rule that detects the usage of list-tables (as we already cover exfiltration attempts based on AWS Glue).
Join us
If you like this kind topic consider registering on our Community Slack where we talk threat hunting, NIDS and traffic analyses, general AWS security & more!
Want to take it a step further and already have some AWS threat hunting experience under your belt? If our vision resonates with you in any way, reach out to us at jobs@ so we can talk.