AlertOps and Wavefront

AlertOps’ alert management system can be integrated with Wavefront to receive and respond to all (predefined status mappings) alerts through email, SMS, push notification, or phone alerts. AlertOps would ensure that the alert/job status would reach the appropriate team by using proper workflows, escalation policies, and schedules. Based on your ruleset, incidents can be automatically opened and closed, depending on what kind of event/trigger Wavefront reports.

The above scenario and scope for integration are due to the fact that AlertOps has a very flexible and simple API/Webhook configuration feature that can be leveraged with Wavefront's alert and notification capabilities.

AlertOps - Inbound Integration

We can define rulesets in AlertOps so that Wavefront can send out alerts to the AlertOps platform. AlertOps would ensure based on these notifications received, that it would always reach out and assign to the correct person/team by utilizing its escalation policies, schedules, and workflow features.

AlertOps provides Inbound Integrations to integrate with numerous monitoring, chat, and ITSM tools. You can configure an inbound integration for Wavefront.

At a high level, this is how the flow looks like, you define an API integration in the AlertOps platform by defining settings like Integration Name, Escalation rules, and recipient users/groups. Once an integration is defined, a unique API URL is generated. This acts as a webhook or the gateway through which notifications from Wavefront reach AlertOps and thus an incident/alert is created correspondingly. The API can be defined with various settings like URL mappings, filters, escalations, etc. as required. Wavefront has to be configured with a webhook alert target and they must be attached to alert triggers.

Configure Inbound Integration in AlertOps

1. From Configuration, select Integrations from the submenu. 
2. Scroll down to the 'Inbound Integrations' section, select 'API' from the dropdown menu, then click the 'Add API Integration' button.
3. Select 'VMware Tanzu' from the available integration options available.
4. Once you selected the integration, you can then specify basic settings like the integration name, escalation policy, and names of the recipients/groups to which the alerts must be assigned to.
5. Once you click save, the API Integration will be created, and you will be given a unique URL that acts as the access point and needs to be configured at the source (in this case Wavefront), to send alerts. You can find the integration you just created, and you can give advanced settings and define various configurations for the alerts to be received and processed. For example, you can define when to open and close alerts based on the payload obtained from the API call, filters, etc.
6. Make a note of the API URL, which will be used in Wavefront, so it calls a HTTP POST request to this URL with the body in JSON format containing the alert-specific information. AlertOps automatically creates an alert when the status variable (severity) contains 'SEVERE'. The incident will also be closed automatically when the status 'INFO' is received from Wavefront.
7. You can similarly define URL mappings as you want, owing to the flexibility provided by AlertOps’ OpenAPI/Plug-and-Play integrations. You can provide other filters and match them with regex expressions as well. You can also test the generated URL with the sample data provided.

Configure Integration in VMware Tanzu

1. In your VMware Tanzu Observability portal, click 'Browse' in the top navigation tabs and then select 'Alert Targets'
2. Under 'Types' in the left menu, select 'Webhooks' and then click ' Create Alert Target'
3. Give a name, and description, for triggers - select the following (as per the screenshot below)
4. Select type as 'Webhook' and under Default Recipients, paste the AlertOps Inbound Integration API URL.
5. Select Content Type as 'application/json'
6. Under Body Template, select 'Generic Webhook' - remove the existing template and paste the following,

{
  "alertId": "{{{alertId}}}",
  "alertTags": [
    {{#trimTrailingComma}}
      {{#alertTags}}
        "{{#jsonEscape}}{{{.}}}{{/jsonEscape}}",
      {{/alertTags}}
    {{/trimTrailingComma}}
  ],
  "notificationId": "{{{notificationId}}}",
  "imageLinks": "{{{imageLinks}}}",
  "reason": "{{{reason}}}",
  "name": "{{#jsonEscape}}{{{name}}}{{/jsonEscape}}",
  "severity": "{{{severity}}}",
  "severitySmoke": {{severitySmoke}},
  "severityInfo": {{severityInfo}},
  "severityWarn": {{severityWarn}},
  "severitySevere": {{severitySevere}},
  "condition": "{{#jsonEscape}}{{{condition}}}{{/jsonEscape}}",
  "url": "{{{url}}}",
  "createdTime": "{{{createdTime}}}",
  "startedTime": "{{{startedTime}}}",
  "sinceTime": "{{{sinceTime}}}",
  "endedTime": "{{{endedTime}}}",
  "snoozedUntilTime": "{{{snoozedUntilTime}}}",
  "subject": "{{#jsonEscape}}{{{subject}}}{{/jsonEscape}}",
  "hostsFailingMessage": "{{#jsonEscape}}{{{hostsFailingMessage}}}{{/jsonEscape}}",
  "updateMessage": "{{#jsonEscape}}{{{updateMessage}}}{{/jsonEscape}}",
  "additionalInformation": "{{#jsonEscape}}{{{additionalInformation}}}{{/jsonEscape}}"
}

 7. Click Save.

8. Now create another alert target similar to the previous one for INFO alerts. The previous one can be for SEVERE alerts. 

9. You will need to attach the alert target to the alerts that you create in VMware Tanzu. Make sure you attach the alert target for the SEVERE and INFO thresholds for your alerts. You can create alert targets for the other statuses as well and attach them accordingly. (Make sure corresponding mapping is done in AlertOps as well)

Thats it! You have configured a webhook alert target and attached it to alerts in VMware Tanzu Observability. Any alert will be sent to AlertOps for incident management.

Message logs, alert specific information can be viewed in the “Inbound Log” section in AlertOps Dashboard. Alerts can be viewed in the ‘Alerts’ tab as well.

Alert Triggering Information

AlertOps will automatically create an incident when a new alert is received from Wavefront when the severity contains "SEVERE".

If an alert with the status "SEVERE" matches an existing Open Alert, AlertOps will recognize the new alert as a duplicate and ignore the alert.

The alert will be recorded in the Inbound Messages table as “Mapped Appended.”

AlertOps will automatically close the same incident when an alert with severity contains "INFO".

References

AlertOps Integration Guides

General Restful API Guide

Wavefront Docs