AlertOps and AWS Health/PHD

AlertOps’ alert management system can be integrated with PHD to receive and respond to critical (predefined status mappings) alarms/alerts through email, SMS, push notification or phone alerts. AlertOps would ensure that the alert would reach the appropriate team by using proper workflows, escalation policies and schedules. Based on your ruleset, incidents can be automatically opened and closed, depending on whether PHD reports a problem or a recovery.

The above scenario and scope for integration is due to the fact that AlertOps has a very flexible and simple API/Webhook configuration feature that can be leveraged with PHD’s monitoring and action capabilities.

Setup PHD

AWS Health provides a console, called the Personal Health Dashboard, to all customers. You do not need to write code or perform any actions to set up the dashboard.

PHD starts learning activity immediately, once you have your account setup.

The Personal Health Dashboard has a bell icon in the console navigation bar. Choose the bell icon to see whether your account is affected by recent events. You can then choose an event to navigate to the Personal Health Dashboard for more information.  

AlertOps - Inbound Integration

We can define an inbound integration in AlertOps to receive the event information from the SNS Topic through a target from Amazon CloudWatch. 

AlertOps would ensure based on these notifications received, that it would always reach out and assign to the correct person/team by utilizing its escalation policies, schedules, and workflow features. AlertOps provides Inbound Integrations to integrate with numerous monitoring, chat and ITSM tools. You can configure a custom inbound integration for AWS PHD Events.

At a high level, the flow looks like the diagram shown below – CloudWatch configured with an event rule, sends out PHD event notifications to the SNS Topic it is attached to. This SNS Topic would push the message to the HTTPS endpoint provided by AlertOps. AlertOps must be subscribed to the SNS Topic in order to receive notifications.

Configure Inbound Integration in AlertOps

1. Under 'Configuration' select 'Integrations'. From the Inbound Integration section, select 'API' from the dropdown and then click the 'Add API' button.
2. Select AWS Personal Health Dashboard from the list of available integration options.
3. Once you select the integration, you can then specify basic settings like the integration name, escalation policy, names of the recipients/groups for which the alerts must be assigned to.
4. Once you click save, the API Integration will be created, and you will be given a unique URL which acts as the access point and needs to be configured at the source (in this case SNS Topic), to send alerts. You can find the integration you just created, and you can give advanced settings and define various configurations for the alerts to be received and processed. For example, you can define when to open and close alerts based on the payload obtained from the API call, filters etc.
5. AlertOps automatically creates an alert when the status variable (detail^state) contains 'stopped' or 'terminated'. The incident will also be closed automatically when the status 'running' is received from AWS.
6. You can similarly define URL mappings as you want, owing to the flexibility provided by AlertOps’ OpenAPI/Plug-and-Play integrations. You can provide other filters and match with regex expressions as well. You can also test the generated URL. 
7. The above example configuration trigger is provided for an event related to EC2 instance states. Using the payload received from AWS, you can map any event in the URL mapping option as you want.

Configuration of AWS to send alerts to AlertOps

To create an SNS Topic and a subscription:

1. Go to Services – select Amazon SNS. 
2. In the left tab – select Topics – Create Topic 
3. Select Standard – Give a name to the topic. You can configure other options as you need to. 
4. Once you create the topic, in the left tab – select Subscriptions – Create Subscription 
5. In the Topic ARN option, select the name of the topic you just created.
6. For protocol – select HTTPS, and in the endpoint – paste the API URL which you obtained when you created the inbound integration. You can configure other options as you need to. 
7. Once you create the subscription, go to Topics, select the topic you created – you must have a screen as below, 
8. In the “Subscriptions” section, you will have a status that says, “Pending Confirmation”. This means that AlertOps hasn’t yet subscribed to this topic to receive notifications. (The below screenshot shows “Confirmed”)
9. To subscribe to the topic, go to your AlertOps dashboard, under integrations go to ‘Inbound Log’. 
10. You should have an entry in the log, from AWS, however there wont be an alert created. Select the message ID detail. In the body section, you should have a field that says, “Subscribe URL” and a link as a value. Copy and paste the link in a new tab and you must get a confirmation template. (You can save it if you want to).
11. Once you have subscribed to the topic, navigate to your AWS console. Now if you open the Subscriptions under Amazon SNS, you should see a status that says “Confirmed”. You can edit the topic and subscription configurations as and how you want it.

Configuring CloudWatch Event Rule for PHD Events

 (this example is for an EC2 instance state)

1. Go to your CloudWatch console, under events in the left navigation pane, select Get Started. 
2. Configure the ‘Event Source’ by selecting ‘Service name’ as ‘EC2’ and ‘Event Type’ as ‘EC2 Instance State Change Notification’. You can configure the event pattern as and how you require. Alternatively you can go to the PHD Dashboard and create a rule from there. 
3. On the right side under ‘Targets’ select ‘SNS Topic’ and the Topic as the topic which you created in the previous step. You can configure what input you want to send. Click “Configure Details” and give a name to the event rule. Save
4. Once you create the rule, you should see its status as “Enabled” (with a green dot). You can edit the rule as and when you want to
5. You can simulate an event by stopping and terminating an EC2 instance. As a result, you should the sample correlating alert in AlertOps under Integrations – Inbound Log or the Alerts section.

Thats it! You have now integrated Amazon Personal Health Dashboard with AlertOps!

The following alert triggering information is given for EC2 instance states, this event payload can be changed and mapped for any service of whose health, AWS PHD reports.

Alert Triggering Information

AlertOps will automatically create an incident when a new alert is received from AWS PHD, when the detail^state field contains “stopped” or “terminated”. 

If an alert with status “stopped” or “terminated” matches an existing Open Alert, AlertOps will recognize the new alert as a duplicate and ignore the alert. The alert will be recorded in the Inbound Messages table as “Mapped Appended.”

AlertOps will automatically close the same incident when an alert with detail^state contains “running”

References

AlertOps Integration Guides

General Restful API Guide

AWS Personal Health Dashboard

AWS CloudWatch Docs