CloudGoat is Rhino Security Labs’s tool for deploying “vulnerable by design” AWS infrastructure. This blog post will walk through the new detection_evasion scenario, where you will attempt to move through an AWS environment, capturing flags at various points, all without being detected by automated systems.

Previously, CloudGoat scenarios have focused exclusively on exploiting misconfigured resources. While these techniques are necessary, focusing on them exclusively can paint a simplistic picture of what enterprise AWS environments look like, and what your priorities should be as an attacker. In most enterprise AWS environments, attackers need to take into consideration what the blue team might be doing as well. 

Every exploit has a probability of success, a potential payoff (privilege escalation, access to secrets, etc.), and a probability of being detected. These factors are rarely certain, but should be taken into consideration during engagements. This scenario is our attempt at creating educational content around the latter factor. 

SPOILER ALERT: The rest of this post is a walkthrough of the detection_evasion scenario. If you want to work through the scenario on your own first, click here to find CloudGoat on Github. This scenario is categorized as “Hard”, so if you have never completed a CloudGoat scenario before, consider starting with an “Easy” scenario. 

There are a wide variety of services anent logging in AWS, including Access logs, CloudTrail logs, and CloudWatch Events. At the time of writing, this scenario uses CloudTrail logs to trigger alerts. There can be a significant lag-time between the generation of an event in CloudTrail, its delivery to a CloudWatch log group, and the triggering of an alert. This is important to keep in mind both as an attacker and as a defender.

There are two alerts set up; one for detecting honeytokens, and one for detecting the usage of access keys from untrusted IP addresses. Both of these alerts are triggered based on the following metric filters on a CloudWatch log group.

Below is the filter pattern for detecting honeytokens (phase 1). The associated metric value will be incremented by 1 every time credentials belonging to one of the ARN’s below is used, which will trigger an alert. The bypass for this alert will be discussed in the “Honeytokens In AWS” section.

The filter pattern for detecting the usage of EC2 instance profile credentials from untrusted IP addresses (phase 2) is a little more interesting. This pattern will trigger an alarm every time there is an event in CloudTrail that uses both credentials from a CloudGoat EC2 instance and has a “sourceIPAddress” other than that of the corresponding instance. Bypass techniques for both EC2 instances will be described in the “Manipulating AWS CloudTrail Logs” section.

If you’re not familiar with the concept of honeytokens, they are essentially credentials created by blue teams with the intention of functioning as a sort of “early warning” system that something has gone awry. For example, you might place honeytokens in the commit history of an internally hosted GitHub repo to detect that a malicious actor has begun digging for secrets. 

There are multiple ways to implement honeytokens. You can use a service that does it for you, like https://canarytokens.org, free open-source solutions such as https://github.com/spacesiren/spacesiren, or ‘roll your own’ honeytoken services. These solutions essentially exist on the following spectrum.

The reason pre-built solutions are easier to detect is that they use predictable strings in the ARN of the principal associated with the honeytoken. Honeytokens from CanaryTokens all have the same AWS Account ID, and SpaceSiren user names are all uuid4 strings from the default python library.  

If you want to learn more about the process of detecting honeytokens, this post explains it in more detail. For the present purposes, it should be sufficient to say that you can use Pacu’s iam__detect_honeytokens module to automatically detect credentials created by CanaryTokens and SpaceSiren. If you’re aware of additional honeytoken services, please feel free to reach out to us @RhinoSecurity and we’ll add them to Pacu.

In this scenario, you simply need to run “pacu run iam__detect_honeytokens” for each pair of credentials and avoid the honeytokens for the remainder of the scenario.

As mentioned earlier, there are two paths for the second phase of this scenario. Both paths require the user to acquire credentials from the Instance Metadata Service (IMDS), and then use them in such a way as to create CloudTrail logs where the “sourceIPAddress” is that of the EC2 instance from which they were acquired.

The first – “easy” – path uses an EC2 instance that is connected to the public internet. This path is relatively trivial to complete because you can just install the AWS CLI, and use the credentials of that instance from that instance. If you move the credentials elsewhere (such as your laptop) and try to use them, an alert will be triggered.

The second – “hard” – path does not have access to the public internet. Instead, it exists inside of a VPC which only has access to a few AWS services through VPC Endpoints. This means the technique used on the first path will not work. The solution to this is to use VPC Endpoints to spoof the IP address of the “hard” instance when calling AWS API endpoints. Hunters.ai shares this technique in detail on their blog. We have created a public repo with terraform code that will deploy all the resources necessary to complete this bypass.

This walkthrough describes the structure of the CloudGoat detection_evasion scenario from a high level, covering AWS honeytokens and IP spoofing in CloudTrail as techniques necessary to complete the scenario. 

Hopefully, this CloudGoat scenario provides some insight into a different side of AWS penetration testing. If you want to contribute to CloudGoat, feel free to open issues and pull requests on GitHub, and follow us on Twitter @RhinoSecurity for updates.