AlertOps and Sematext

AlertOps’ alert management system can be integrated with Sematext to receive and respond to critical (predefined status mappings) alerts through email, SMS, push notification or phone alerts. AlertOps would ensure that the alert would reach the appropriate team by using proper workflows, escalation policies and schedules. Based on your ruleset, incidents can be automatically opened and closed, depending on whether Sematext reports a problem or a recovery. 

The above scenario and scope for integration is because AlertOps has a very flexible and simple API/Webhook configuration feature that can be leveraged with Sematext’s monitoring and alerting capabilities.

Sematext has three types of alerts: Threshold, Anomaly and Heartbeat, and can operate on three sources of data - metrics, logs, experience

AlertOps - Inbound Integrations

We can define rulesets in AlertOps so that Sematext can send out notifications to the AlertOps platform. AlertOps would ensure based on these notifications received, that it would always reach out and assign to the correct person/team by utilizing its escalation policies, schedules, and workflow features. 

AlertOps provides Inbound Integrations to integrate with numerous monitoring, chat and ITSM tools. You can configure an inbound integration for Sematext incidents. 

At a high level this is how the flow looks like, you define an API integration in the AlertOps platform by defining settings like Integration Name, Escalation rules, recipient users/groups. Once an integration is defined, a unique API URL is generated. This acts as webhook or the gateway through which notifications from Sematext reach AlertOps and thus an incident/alert is created correspondingly. The API can be defined with various settings like URL mappings, filters, escalations etc. as required.

Configure Inbound Integration in AlertOps

1. Navigate to Configuration àIntegrations àAdd API Integration à API Integration Detail page.
2. Select Sematext.
3. Once you select the integration, you can then specify basic settings like the integration name, escalation policy, names of the recipients/groups for which the alerts must be assigned to.
4. Once you click save, you will be given a unique URL which acts as the access point and needs to be configured at the source (in this case Sematext), to send alerts. You can find the integration you just created, and you can give advanced settings and define various configurations for the alerts to be received and processed. For example, you can define when to open and close alerts based on the payload obtained from the API call, filters etc
5. Make a note of the API URL, which will be used in Sematext, so it calls a HTTP POST request to this URL with the body in JSON format containing the alert specific information. AlertOps automatically creates an alert when the status variable (alertConditionNormality) is ‘false’. The incident will also be closed automatically when the status ‘true’ is received
6. In Advanced Settings you can define URL mappings as you want in the Rules for Opening and Closing Alerts. You can provide other filters and match with regex expressions as well. You can also test the generated URL with the sample data provided.

Configure Integration in Sematext 

Sematext Alerts allow you to set robust and configurable alerting policies that enable you to get notified about changes or issues affecting your application/infrastructure’s metrics or logs.

To setup a Notification Hook (Webhook) to send out alerts to AlertOps,

• In the left navigation pane select ‘Alerts’ and then select ‘Notification Hooks’. In the top right corner select ‘New Notification Hook’.
• Select ‘Custom’. Give a hook name and paste the URL which you configured in the previous section in the URL field. Send data as ‘JSON’ and select HTTP method as ‘POST’.
• Add the following parameters exactly - this is the JSON payload.

• Define any headers if required. You can see a preview of the URL and the JSON Payload. You can also send a test notification to the HTTPS Endpoint you just configured. Click ‘Save Notification Hook’
• We have to add create an alert rule to invoke this notification hook.

To define a sample metric for an example server instance and setup alerting for that metric,

• In the left navigation pane select ‘Monitoring’. Find the app you created in the previous section and select the three dots under the ‘Actions’ column. Select ‘Alert Rules’
• In the top right corner, select ‘New Alert Rule’
• Define the Alert Metric for which you want to define the threshold and trigger condition. You can see a bunch of metrics like os.cpu.idle, os.load, os.memory etc. You can select metrics, filter, transform and group them as and how you want.
• Under ‘Alert Condition’ define the trigger condition (for example the condition could be ‘The Chosen Metric (os.cpuload) > 0.5 over 5 minutes)
• You can define the ‘Alert Rule Name’, ‘Description’ and ‘Runbook Plan’ under Metadata.
• Under ‘Notifications’, make sure ‘Enable Notifications’ and ‘Alert me when the value goes back to non-alert level’ is enabled. Under ‘Send to’ select the Notification Hook which you created. You can also schedule when you want to send alerts. Click ‘Save’.
• You can view the details/edit the alert by clicking the alert name. You can also find rule previews, heatmaps and histogram of the alerts.

You have created a Notification Hook and an Alert Rule that would send out notifications to AlertOps when the alert condition is triggered. You can define any kind of monitoring setup and define any kind of rules as you wish. Just make sure the payload for the notification hook is defined exactly as specified above.

Alerts in AlertOps can be seen in the ‘Inbound Log’ or the ‘Alerts’ section

Alert Triggering Information

AlertOps will automatically create an incident when a new alert is received from Sematext, when the alertConditionNormality field contains “false”.

If an alert with status “false” matches an existing Open Alert, AlertOps will recognize the new alert as a duplicate and ignore the alert. The alert will be recorded in the Inbound Messages table as “Mapped Appended.”

AlertOps will automatically close the same incident when an alert with alertConditionNormality contains “true”.

References

AlertOps Integration Guides

General Restful API Guides

Sematext Integration Guide