AlertOps and CloudTrail

 AlertOps’ alert management system can be integrated with CloudTrail to receive and respond to critical (predefined status mappings) alarms/alerts through email, SMS, push notification or phone alerts. AlertOps would ensure that the alert would reach the appropriate team by using proper workflows, escalation policies and schedules. Based on your ruleset, incidents can be automatically opened and closed, depending on whether CloudTrail reports a problem or a recovery.

The above scenario and scope for integration is due to the fact that AlertOps has a very flexible and simple API/Webhook configuration feature that can be leveraged with CloudTrail's logging and notification capability.

AlertOps has to be subscribed to a particular SNS Topic in order to pull messages from the topic and manage them accordingly. This SNS topic would be attached to a CloudWatch/CloudTrail event/log metric alarm, that would execute this particular topic, when an event/log alarm occurs. 

Setup a CloudTrail ‘Trail’

1. Ensure CloudTrail is enabled on your AWS account when you create the account. When activity occurs in any AWS service that supports CloudTrail, that activity is recorded in a CloudTrail event along with other AWS service events in Even History. You can view, delete and download them.
2. Go to the CloudTrail console, go to the ‘Trails’ section and click ‘Create Trail’
3. Give a proper trail name. For storage location you can either create a new S3 bucket or select an already configured bucket.
4. By default, your log files are encrypted with SSE-S3 encryption. You can choose to enable or disable this. You can leave the other settings as default.
5. You can create an SNS topic here itself and configure it later. Let us leave this for now.
6. In the next step, enable CloudWatch Logs. We will be sending events to CloudWatch and then set up alarms on metrics, to send out notifications to AlertOps.
7. Under Log Group, select new, or existing and link it to a CloudWatch log group.
8. You can specify an IAM role, that CloudTrail will assume in order to send events to your CloudWatch log group.
9. Review and finish creating the trail. If everything goes well, you will have created a Trail or a configuration where in, you can send events to CloudWatch for metric alarms.

AlertOps Inbound Integration

We can define an inbound integration in AlertOps to receive the event information from the SNS Topic through an alarm from CloudWatch.

AlertOps would ensure based on these notifications received, that it would always reach out and assign to the correct person/team by utilizing its escalation policies, schedules, and workflow features. AlertOps provides Inbound Integrations to integrate with numerous monitoring, chat and ITSM tools. You can select the CloudTrain Integration Template.

At a high level, the flow looks like the diagram shown below – CloudWatch configured with a log group (which you created in the previous step), receives events from CloudTrail. CloudWatch configured with an alarm for this particular log group trail (for a metric), triggers an alarm that is connected to a SNS topic. This SNS Topic would push the message to the HTTPS endpoint provided by AlertOps. AlertOps must be subscribed to the SNS Topic in order to receive notifications. (We will create an example configuration to illustrate this in the upcoming section)

Configure Inbound Integration in AlertOps 

1. Under 'Configuration' select 'Integrations'. From the Inbound Integration section, select 'API' from the dropdown and then click the 'Add API' button.
2. Select AWS CloudTrail from the list of available integration options
3. Once you select the integration, you can then specify basic settings like the integration name, escalation policy, names of the recipients/groups for which the alerts must be assigned to.
4. Once you click save, the API Integration will be created, and you will be given a unique URL which acts as the access point and needs to be configured at the source (in this case the SNS Topic), to send notifications. You can find the integration you just created, and you can give advanced settings and define various configurations for the alerts to be received and processed. For example, you can define when to open and close alerts based on the response obtained from the API call, filters etc.
5. You can similarly define URL mappings as you want, owing to the flexibility provided by AlertOps’ OpenAPI/Plug-and-Play integrations. You can provide other filters and match with regex expressions as well. You can also test the generated URL.
6. CloudTrail logs trigger CloudWatch Alarms that would send notifications to AlertOps.

Configure Integration in CloudTrail/CloudWatch/AWS SNS 

Now that we have setup AlertOps with the AWS CloudTrail API Inbound Integration, along with a unique API URL; we can now define a Webhook connection in Amazon SNS to access this API and send out event notifications to AlertOps.

Let’s recall what we have done so far – we have created a “trail” so that it sends events to CloudWatch log group; we have also set up an inbound integration in AlertOps and have a unique URL to which these events must send notifications to.

However, CloudWatch or CloudTrail doesn’t send event information directly to an external client, instead it sends the notification to an attached SNS Topic, which in turn would connect to an external HTTPS endpoint and push the notification (refer to the previous section for a high-level flow diagram). Let us configure an SNS topic and attach it to the CloudWatch log group metric rule.

Create an SNS Topic and a Subscription:

1. Go to Services – select Amazon SNS.
2. In the left tab – select Topics – Create Topic
3. Select Standard – Give a name to the topic. You can configure other options as you need to.
4. Once you create the topic, in the left tab – select Subscriptions – Create Subscription
5. In the Topic ARN option, select the name of the topic you just created.
6. For protocol – select HTTPS, and in the endpoint – paste the API URL which you obtained when you created the inbound integration. You can configure other options as you need to.
7. Once you create the subscription, go to Topics, select the topic you created – you must have a screen as below,
8. In the “Subscriptions” section, you will have a status that says, “Pending Confirmation”. This means that AlertOps hasn’t yet subscribed to this topic to receive notifications. (The below screenshot shows “Confirmed”)
9. To subscribe to the topic, go to your AlertOps dashboard, under integrations go to ‘InboundLog’.
10. You should have an entry in the log, from AWS, however there wont be an alert created. Select the message ID detail. In the body section, you should have a field that says, “Subscribe URL” and a link as a value. Copy and paste the link in a new tab and you must get a confirmation template. (You can save it if you want to). 
11. Once you have subscribed to the topic, navigate to your AWS console. Now if you open the Subscriptions under Amazon SNS, you should see a status that says “Confirmed”. You can edit the topic and subscription configurations as and how you want it.

Configure CloudWatch Log Group for sending out CloudTrail events:

In this configuration (for an example scenario), we will create a CloudWatch alarm that will be triggered when there are more than 3 failed sign-in attempts to the AWS console.

1. Go to CloudWatch console, in the left navigation pane, select Logs, and select the log group which you created while creating the “trial”. 
2. Select the log group, and under ‘Actions’ select ‘Create Metric Filter’
3. In ‘Create Filter Pattern’, enter the following: 

{ ($.eventName = ConsoleLogin) && ($.errorMessage = "Failed authentication") } 

4. You can test the pattern if you want to. For different filter pattern techniques, refer to the link in the Reference section. Click ‘Next’

5. Create the filter name as “ConsoleSignInFailures”

6.Enable ‘Create New’ metric namespace and give the namespace as ‘CloudTrailMetrics’, Metric name as ‘ConsoleSigninFailureCount’ and Metric Value as ‘1’. Leave others default and click ‘Next’. Review and create the metric filter.

7. Once you create the metric filter, select it, and click “Create Alarm”. Give the settings as per the below screenshot.

8. In the next page, you can configure the SNS topic which you created in the previous section so that a notification will be sent to the AlertOps’ unique API URL when an alarm state is raised. Review and create the alarm! 

9. Refer guide for AWS Cloudwatch integration - https://help.alertops.com/en/articles/1715406-aws-cloudwatch-sns  , to see how to configure/setup Cloudwatch to send out notifications.

That’s it! You have created a “trail” that will send out logs to CloudWatch log group, which will trigger an alarm for a particular metric. Try logging in 3 or more times (failing which), this alarm notification will be sent and can be viewed in the ‘Inbound Log’ section in AlertOps Dashboard.

Alert Triggering Information

AlertOps will automatically create an incident when a new alert is received from AWS with an Message^NewStateValue status of “ALARM”.

If an alert with status “ALARM” matches an existing Open Alert, AlertOps will recognize the new alert as a duplicate and ignore the alert. The alert will be recorded in the Inbound Messages table as “Mapped Appended”.

AlertOps will automatically close the same incident when an alert with an Message^NewStateValue status “OK” is received.

References

AlertOps Integration Guides

General Restful API Guide

AWS CloudWatch Docs

AWS CloudTrail

AWS Metric Filter Syntax