What is different about cloud log management versus on-premises? The answer may seem simple, but several elements such as CloudTrail vs CloudWatch come into play.
In this article, we will cover some of the most important differences, and then dig deeper into a specific example of AWS CloudTrail vs CloudWatch.
What are cloud logs?
Let’s say you run an Apache web server (remember to check if it is updated so as not to be vulnerable to the new CVE-2022-42889) in your cloud provider, AWS for example. When it fails a web request, this is logged to /var/log/httpd/error_log on the machine. This log entry would happen even if Apache was deployed in a brick-and-mortar environment.
Therefore, we are dealing with something that is not a cloud log.
On the other hand, cloud logs come directly from your cloud services. What’s only found in the cloud are what we assume to be true cloud logs.
To define cloud logs, here is this example that shows an Identity and Access Management (IAM) user, called ‘Brett,’ using the AWS CLI to call the Amazon EC2 StartInstances action. It uses the ec2-start-instances command for the instance name ‘i-ebeaf9e2’ because its previous state was ‘stopped.’
{"Records": [{
"eventVersion": "1.0",
"userIdentity": {
"type": "IAMUser",
"principalId": "EX_PRINCIPAL_ID",
"arn": "arn:aws:iam::123456789012:user/Brett",
"accessKeyId": "EXAMPLE_KEY_ID",
"accountId": "123456789012",
"userName": "Brett"
},
"eventTime": "2014-03-06T21:22:54Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "StartInstances",
"awsRegion": "us-east-2",
"sourceIPAddress": "<External-IP>",
"userAgent": "ec2-api-tools 1.6.12.2",
"instanceId": "i-ebeaf9e2",
"currentState": {
"code": 0,
"name": "pending"
},
"previousState": {
"code": 80,
"name": "stopped"
}
}]}}
}]}Code language: JavaScript (javascript)The current state is ‘pending’ as a result of the action.
How should you manage your logs?
At a high level, you have to decide how you are going to implement these three processes:
- Creating logs
- Transmitting logs
- Storing logs (including rotating and deleting logs)
Best practices
AWS log management best practices is a big topic. For example, if you follow the Payment Card Industry (PCI) standard, you have to store logs for at least one year. We don’t have room to cover best practices here, but the folks over at Loggly have a nice list, and AWS maintains a list specifically for security events as well.
Some best practices are:
- If you send logs remotely, use TCP and encryption.
- For AWS CloudTrail, if you want logs longer than 90 days, you have to create a Trail.
- In a microservices environment, it is not feasible or practical to SSH into containers to see what is happening, so logs become our eyes. Collect all the data you can because the more data you capture, the more visibility you have.
If you want to learn more about best practices, dig deeper with 26 AWS Security Best Practices.
New challenges inside cloud log management
The cloud brings new challenges with it:
- Multi-cloud: It means dealing with different logging specific functionality, and types of logs.
- Cloud logs can be massive: Even a medium-sized cloud environment may generate millions of log events per day. Strategies to control costs related to sheer logging volume are one consideration here. Another solution is to add runtime detection to the cloud detection and response, and try to avoid storing a large lake of data and speed up the alerting process. The Falco plugin may help with that.
- Security concerns: Logs are a very valuable target for attackers, either to steal critical information or to find ways to hide their tracks by deleting them.
With all this in mind, what solutions can we apply to cover the different threats we may face?
Public cloud logging services compared
The major public cloud providers have a very important audit log capability: to log all the changes made in their cloud. Think of this as a log for every change on every device in your entire datacenter. Without this, you would have no paper trail and no record of changes made.
There are logs for virtual machines, security, databases, and many other services.
All the major cloud providers offer a logging agent for their virtual machines. For example, AWS offers the CloudWatch agent which can be injected into an EC2 instance using several techniques, including Userdata at provisioning time.
A sample of the logs available in Google Cloud, Microsoft Azure, and AWS is shown in the following table:
| Google Cloud | Microsoft Azure | Amazon Web Services | |
| Audit logging Service Name | Cloud Audit Logs | Azure Activity Logs | AWS CloudTrail |
| Other Logs |
|
|
|
To write something useful about all of these would make this blog too long, so instead, let’s take a look at two major logs in one of the clouds. In this case, we will compare and contrast AWS CloudTrail vs CloudWatch.
AWS CloudTrail vs CloudWatch
What is AWS CloudTrail?
By utilizing AWS CloudTrail, you can facilitate operations and risks auditing, as well as govern and ensure compliance within your AWS account.
Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS software development kit (SDK) and application programming interfaces (API).
CloudTrail is enabled by default on your AWS account. You can easily view recent events in the CloudTrail console by going to Event history.
For an ongoing record of activity and events in your AWS account, create a trail. Because they will be deleted from the default S3 bucket after 90 days, it is important to be quick in detecting and responding to a vulnerability or abuse of our accounts and to ensure that the logs used to detect the source of the problem do not disappear.
What is Amazon CloudWatch?
Amazon CloudWatch is a performance monitoring service, but it can do a few more tricks than that. CloudWatch also includes the capability to detect anomalous behavior, visualize logs, and set alarms. However, keep in mind that very little of this extended functionality is available out of the box. You may need to write code in AWS Lambda and configure triggers in Amazon EventBridge to get this working.
Source: https://aws.amazon.com/cloudwatch/
By default, AWS CloudWatch exists to monitor the performance of your AWS resources and applications. AWS CloudWatch graphs metrics over time, and is not out-of-box a way to get security alerts. If you want alerts, you have to create your own, based on metrics. And if you want to create an alert for a discrete event, you have to generate a non-zero metric and key off of that.
What kinds of threats can be identified in advance in the cloud logs?
We will choose which type of cloud log management service we need based on the type of event that needs to be detected. For this example, let’s configure CloudTrail to find a simple event: making an S3 bucket public.
(You would be forgiven if you thought that public s3 bucket misconfiguration was no longer relevant, but in fact, this cloud mistake is still very relevant. For example, there is a GitHub project created just to track widely known public s3 bucket incidents, as well as a website with a search engine to find exposed cloud storages.)
Let’s continue with our example. First, we make the bucket public, which thankfully includes a dire warning and a flow that makes you click twice to avoid doing this by mistake:
Within a few minutes, this event shows up in CloudTrail:
"requestParameters": {
"publicAccessBlock": "",
"bucketName": "brett-public-test-bucket",
"PublicAccessBlockConfiguration": {
"xmlns": "http://s3.amazonaws.com/doc/2006-03-01/",
"RestrictPublicBuckets": false,
"BlockPublicPolicy": false,
"BlockPublicAcls": false,
"IgnorePublicAcls": false
},
Code language: JavaScript (javascript)Here, we see “BlockPublicPolicy” and the other parameters set to “false.” Together, these settings make this bucket public on the Internet.
In this scenario, CloudWatch could be useful to detect massive bucket usage, but not so useful for detecting a bucket changing from private to public.
Finding the event in CloudWatch vs. CloudTrail
We found a few ways to find this event in CloudWatch, but there is no easy button.
There are some complex ways to integrate CloudTrail with CloudWatch, including henry and s3-public-alerts. However, these projects have not been updated in some time, and we could not successfully get them to work.
We followed this set of steps from AWS exactly, and we were able to see our bucket alert, but only in AWS Config. Also, the blog post is a little bit out of date and AWS services have been updated since then, so maybe that’s why we had some challenges.
If the scenario we have is completely different, our main cloud detection and response tool could be CloudWatch.
Let’s say, for example, that we have been affected by an attacker that has taken control of our EC2 machines and has started cryptomining. We would detect the attack based on the percentage of CPU used, which would have increased dramatically. For this, CloudTrail is not much help, but CloudWatch gives us valuable information.
Other services within AWS also use them, such as GuardDuty.
Therefore, depending on the threat we are facing, we must be clear about which service is the most suitable for detecting it.
Useful tools
During the writing of this article, we experimented with several useful tools focused around what we initially thought would be a very simple event: finding out “who, what, when, and where” somebody changed an S3 bucket from private to public.
| Description | Link to the Project or Article |
| How to Use AWS Config to Monitor for and Respond to Amazon S3 Buckets Allowing Public Access | https://aws.amazon.com/blogs/security/how-to-use-aws-config-to-monitor-for-and-respond-to-amazon-s3-buckets-allowing-public-access/ |
| List of S3 Hacks | https://github.com/nagwww/s3-leaks |
| Leaky Bucket Finder | https://github.com/willh/henry |
| Alerts about AWS S3 Buckets or Objects made Public | https://github.com/cleancloud/s3-public-alerts |
Our results were mixed, but the point stands that cloud logs are not something to take lightly.
You really have to understand their functionality and how the different parts work together. Finally, you have to understand their limitations.
Other examples to be detected by this cloud log service could be:
- Exploitation of vulnerable lambdas
- Cloud Lateral movement by the attackers
- Changes in DNS configuration
- Networking misconfiguration with Security Groups
CloudTrail vs CloudWatch final words
When we compare AWS CloudTrail vs CloudWatch, we find that although these are sometimes both described as logging functions, they are very different. And while they can be made to work together, this often requires a lot of additional configuration.
Cloud log management can be quite different than in the traditional datacenter. For example, there is no single audit log for the entire datacenter like there is in the cloud. There are several surprising limitations.
The visibility you have within the cloud depends entirely on how you manage logs, and should be a priority. Each cloud provider offers many types of logs, but at the top of the list is the audit log. This is also a key place to look for security issues. Another key piece is the strategy you take when storing them, since a closer option to realtime helps in the early cloud detection and response of malicious events.
Logging in the cloud needs is as important as everything else you do in the cloud. If done incorrectly, it can lead to confusion. But if done well, your cloud will benefit greatly from logging best practices.
If you want to know how to configure and manage these services, Sysdig can help you improve your Cloud Security Posture Management (CSPM). Dig deeper with the following resources:
- Improving AWS security services with Sysdig Secure
- How to Secure Amazon EC2 with Sysdig
- How to Secure AWS Route 53 with Sysdig
- Hunting AWS RDS security events with Sysdig
- Hunting malware with Amazon GuardDuty and Sysdig
Register for our Free 30-day trial and see for yourself!