Lightstep

A platform that combines tracing data and metrics to give full observability into a system.

1. AlertOps and Lightstep

AlertOps’ alert/incident management system can be integrated with Lightstep to receive and respond to critical (predefined status mappings) alerts through email, SMS, push notification or phone alerts. AlertOps would ensure that the alert would reach the appropriate team by using proper workflows, escalation policies and schedules. Based on your ruleset, incidents can be automatically opened and closed, depending on whether Lightstep reports a problem or a recovery. 

The above scenario and scope for integration is since AlertOps has a very flexible and simple API/Webhook configuration feature that can be leveraged with Lightstep’s monitoring/tracing and alerting capabilities.

Alerts in Lightstep: You can create alerts by defining thresholds that mark SLAs or metrics that you want to be notified about. Once you create alerts, you can define notification hooks or integrate with other services like Slack etc. to notify for incident management. Alerts can also be opened via the public APIs.

1. AlertOps - Inbound Integrations

We can define some rulesets in AlertOps so that Lightstep can send out notifications to the AlertOps platform. AlertOps would ensure based on these notifications received, that it would always reach out and assign to the correct person/team by utilizing its escalation policies, schedules, and workflow features. 

AlertOps provides Inbound Integrations to integrate with numerous monitoring, chat and ITSM tools. You can configure an inbound integration for Lightstep incidents. 

At a high level this is how the flow looks like, you define an API integration in the AlertOps platform by defining settings like Integration Name, Escalation rules, recipient users/groups. Once an integration is defined, a unique API URL is generated. This acts as webhook or the gateway through which notifications from Lightstep reach AlertOps and thus an incident/alert is created correspondingly. The API can be defined with various settings like URL mappings, filters, escalations etc. as required.

To configure an Inbound Integration in AlertOps to receive alerts from Lightstep,

• In the menu on left pane, select Integrations > Inbound Integrations > Add API Integration
• There are numerous integration options available in AlertOps, select Lightstep
• Once you select the integration, you can then specify basic settings like the integration name, escalation policy, names of the recipients/groups for which the alerts must be assigned to
• Once you click save, you will be given a unique URL which acts as the access point and needs to be configured at the source (in this case Lightstep), to send alerts. You can find the integration you just created, and you can give advanced settings and define various configurations for the alerts to be received and processed. For example, you can define when to open and close alerts based on the payload obtained from the API call, filters etc.

• Make a note of the API URL, which will be used in Lightstep, so it calls a HTTP POST request to this URL. AlertOps automatically creates an alert when the status variable (status) contains ‘Critical’ or ‘Warning’. The incident will also be closed automatically when the status ‘Resolved’ is received.

• You can similarly define URL mappings as you want, owing to the flexibility provided by AlertOps’ OpenAPI integrations. You can provide other filters and match with regex expressions as well. You can also test the generated URL.

  1. Configuration of Lightstep for AlertOps Integration

Lightstep Alerts allow you to set robust and configurable alerting policies that enable you to get notified about changes or issues affecting your application/infrastructure’s metric/tracing data. We must define a Notification Destination in Lightstep to send out alerts to AlertOps.

To setup a Notification Destination to send out alerts to AlertOps,

• In the left navigation pane select ‘Alerts’ and then select ‘Notification destinations. In the right corner select ‘Create a Destination’ and then select ‘Webhook’
• Give a name, URL, and headers if any. Click Create
• That’s it. Your notification destination is created. You can test if it works.

To define an alert for a sample metric,

In the left navigation pane of your Lightstep UI, select ‘Alerts’, in the top right corner, select ‘Create an Alert’. You can create alerts for metrics or streams. Select ‘Create Alerts for Metrics’

1. Give a name for the alert, description, and in the ‘Query’ section, the dropdown should list the metrics coming in from different environments you might have configured.
2. Select whichever query you want to monitor and alert. You can add tags, formulae, aggregates etc.
3. Under Alert Configuration, you can specify the threshold to send the notification (the ‘Critical and Warning’ thresholds). For this example, we define the query as ‘aws.ec2.cpuutilization’ and the thresholds as 2 and 1.5, then send a notification webhook with the alert payload to the notification destination you defined. You can also view the threshold analysis chart. Click ‘Save’.

You have created a Notification Destination and an Alert Rule that would send out notifications to AlertOps when the alert condition is triggered.

Alerts in AlertOps can be seen in the ‘Inbound Log’ or the ‘Alerts’ section.

Alert Triggering Information:

AlertOps will automatically create an incident when a new alert is received from Lightstep when the status field contains “Critical” or “Warning”. 

If an alert with status “Critical” or “Warning” matches an existing Open Alert, AlertOps will recognize the new alert as a duplicate and ignore the alert. The alert will be recorded in the Inbound Messages table as “Mapped Appended.” 

AlertOps will automatically close the same incident when an alert with status contains “Resolved”.

References

AlertOps Integration Guides

General Restful API Integration

Lightstep Docs

Lightstep Alerts