Intro: Bypassing CloudTrail
With a major focus in cloud security architecture, we’ve released several attack vectors and security tools around AWS. Pacu brought the first AWS exploitation framework, CloudGoat a vulnerable-by-design cloud environment, and today we make another release – this time bypassing CloudTrail logging and the many defensive tools which rely on it.
As reported by Wired this morning, Rhino Security Labs identified how AWS honeytokens can be bypassed using AWS services unsupported by Amazon’s own CloudTrail logging service.
In this article we’ll do a deeper technical walkthrough of the CloudTrail bypass techniques, how they can be used against honeytokens (and similar AWS alerting systems), and release proof-of-concept tools to demonstrate.
Just looking for the tools?
New CloudTrail-bypass Pacu Module here. Standalone PoC script here.
Background on HoneyTokens
Honeytokens are a “tripwire”-like defense similar to honeypots. Where honeypots are highly-monitored dummy systems to act as an appealing target to attackers and pentesters, honeytokens are credentials, documents, or other ‘tokens’ which are designed to attract unwitting hackers. When found – and used – these booby-trapped resources trigger a secret alarm to the company’s security team, notifying them of a breach.
AWS Honeytokens
These ideas have expanded to AWS, where two kinds of honeytoken services have developed: those hosted by a 3rd party company (dummy keys created in the vendor’s account), and those where honeytoken users are hosted in the users AWS account directly (DIY style).
In either case, AWS honeytokens use the CloudTrail service to detect when a certain set of AWS keys are used. The keys are created with all permissions denied, so whenever someone uses those keys, an access denied error is logged to CloudTrail and triggers an alert.
The keys have been stripped of all permissions so regardless of the API request, the call is denied and the attacker caught (or that’s the idea anyway).
IAM Enumeration in the Dark
Upon looking into CloudTrail some more, we discovered a list of AWS services that CloudTrail does not support logging for (available here) . This means any API calls to these services – successful or not – do not generate any logs.
After testing several unsupported services, we found many failed (denied access) API calls still returned the current user’s Amazon Resource Name (ARN). While this isn’t necessarily ‘sensitive’ information, as an attacker it provides an early means of enumeration – again, without generating any logs to alert from.
The following screenshot shows one of those error messages, in this case from a DescribeFleets API call made to AWS AppStream.
Combining this lack of logging and verbose error message providing the ARN, we now have basic account enumeration for a set of compromised AWS keys that can’t be detected.
ARN Formats and Detecting of Honeytokens
An ARN can be in different formats (and disclose different types of data), depending on whether its an IAM user or IAM role. For users, the ARN will contain AWS account ID, IAM user path (if applicable), and the IAM user name. For roles, AWS account ID, IAM role path (if applicable), IAM role name, and the session name will be available.
Examples of both can be seen below (111111111111 is the example account ID).
IAM User:
arn:aws:iam::111111111111:user/the-path/TheUserName
Assumed IAM Role:
arn:aws:iam::111111111111:role/the-path/TheRoleName/TheSessionName
These previous discoveries lead to testing of known AWS honeytoken services, which we confirmed disclosed identifying information in the ARN string.
By leveraging the ‘access denied’ error message (and ARN) in AppStream and other services, we could parse the IAM user path or IAM user name – leaking the keys as a known canary token.
This allowed us to determine if a set of compromised keys were being used as honeytokens without triggering their alerts. An attacker could use this for information discovery and remain undetected, even if they come across a set of rigged AWS keys.
Example Case: SpaceCrab
One affected example is the popular AWS honeytoken project, SpaceCrab. The affected default configuration uses the IAM user path “/SpaceCrab/” for all users it created, which would allow an attacker to detect SpaceCrab was in use, before ever triggering an alert.
The following screenshot shows an example set of keys that were created with the IAM user path set to “/SpaceCrab/”, which is then disclosed in an access denied error message.
It should be noted that while the default settings are affected, this can be (mostly) resolved by changing the IAM user path in the SpaceCrab config. While the underlying unlogged enumeration will continue to exist until CloudTrail supports all AWS services, a convincing user path can be enough to trick an attacker into additional (logged!) API calls.
PoC Script: awshoney-check
Although the vendors responded to this issue quickly, there are still keys out there running under the old configurations, which means that we can still identify honeytokens undetected.
This PoC script will alert you if the provided AWS keys are honeytokens: Github link.
General help and usage information that is printed out with the –help argument:
usage: awshoney_check.py [-h] [-p PROFILE]
This script takes a set of AWS API keys and tries to determine whether they are honeytoken keys. AWS honeytokens are strategically placed in various locations, such as a private Git repo, on the file system of a certain server, and other similar places. Their main purpose is to detect when an attacker has breached their environment. An attacker would likely use AWS keys that they find, but aws honeytokens have no permissions and immediately alert the account owner when they are used. This is a "defense in depth" protection to detect malicious users with access to certain areas of an environment. This script abuses the AWS API by trying to run a command against an AWS service that is not supported by CloudTrail. Because the targeted service is not supported by CloudTrail, there will never be an alert that the keys were compromised and used. This makes it possible to detect whether or not a set of AWS keys are boobytrapped without triggering the alerts behind them.
optional arguments:
-h, --help show this help message and exit
-p PROFILE, --profile PROFILE
The AWS CLI profile to use. You will be prompted if
this argument is omitted
Currently this script supports detection for two separate canary/honey token services, https://canarytokens.org/ by Thinkst Canary and SpaceCrab by Atlassian. All tokens generated from https://canarytokens.org/ are vulnerable to this detection and only default configurations of SpaceCrab are affected.
This script has also been integrated into the AWS exploit framework Pacu as a module: iam__detect_honeytokens.
Disclosure
We reached out to both affected vendors (CanaryTokens and SpaceCrab) and received quick responses to resolve the issues as much as possible. Would like to thank both parties for their promptness and professionalism.
Given the primary issues lies with CloudTrail not logging certain services (and users not able to disable those APIs), we also reached out to AWS Security with a detailed explanation and screenshots.
Amazon responded that ARN’s are not sensitive, CloudTrail services are working as intended, and that it’s not a security concern.
Conclusion
While CloudTrail has been touted as the singular logging source for all of AWS, unsupported services provide opportunities for attackers and penetration testers to hide malicious calls – such as enumerating account information. These can be used to identify honeytokens, but likely much more as well.
