Close the loop from alert detection to automated fix

Prerequisites

• User role with alerts.view permission for viewing alerts and incidents
• User role with alerts.manage permission for creating/editing runbooks and approving/cancelling executions
• Alert rules must be configured and active for alerts to fire and trigger remediation
• Hosts must be online and reporting metrics for alert evaluation to work
• Scripts or workflows referenced by runbook actions must already exist in the system

Understanding runbooks

A runbook defines the automated response to a specific alert condition. When an alert fires and an investigation identifies a probable cause, the remediation engine searches for a matching runbook and creates an execution.

Automatic vs manual runbooks

System runbook library

25 system runbooks are seeded at installation covering common operational scenarios. System runbooks are global (no organization scope) and do not require approval by default. They serve as a starting library that you can supplement with organization-specific runbooks.

Event-aware runbook matching

Runbooks can match on event-driven alerts in addition to metric-based alerts. Two additional trigger fields are available for event sources:

When an event alert fires, the remediation engine matches against both the event-specific trigger fields and the standard severity trigger. This allows runbooks to respond to AD security events, vulnerability discoveries, SNMP device failures, and other non-metric alert sources.

Auto-approve by investigation confidence

Runbooks can set an auto-approve confidence threshold (0.0 to 1.0). When the causal investigation's confidence score meets or exceeds this threshold, the execution is auto-approved without manual intervention. This is separate from the health-based auto-approve.

Creating runbook definitions

1. Navigate to Remediation > Runbooks.
2. Click Create Runbook.
3. Configure trigger conditions -- the engine matches ALL non-null conditions:
   Trigger Metric TypeMatch on the alert's metric type (e.g., CPU usage percentage).

   Trigger Cause PatternPattern matched against the investigation's probable cause text.

   Trigger SeverityMinimum severity threshold: info, low, medium, warning, high, or critical.

   Trigger Heuristic RuleMatch on the investigation's heuristic rule name.
4. Configure the action:
   Action TypeOne of: Script, Workflow, Service Restart, or Command.

   Action ConfigurationAction-specific parameters (script, workflow, service name, command text, etc.).
5. Configure safety constraints:
   Requires ApprovalWhether manual approval is needed before execution (default: Yes).

   Max Executions Per DayMaximum times this runbook can execute in 24 hours (default: 5).

   Cooldown (minutes)Minimum time between executions of this runbook (default: 30).

   Auto-Approve ThresholdOptional host health percentage below which approval is skipped.
6. Configure validation (post-execution health check):
   Validation MetricMetric to check after the action completes.

   Validation OperatorComparison operator (less than, greater than, equals, etc.).

   Validation ThresholdTarget value the metric must satisfy.

   Stabilization WindowMinutes to wait before validating (default: 5).
7. Optionally configure a rollback action (same structure as the primary action). Rollback executes automatically if validation fails.
8. Click Save.

Linking runbooks to alert rules

Runbooks are matched to alerts automatically based on trigger conditions -- you do not manually link a runbook to a specific alert rule. The matching occurs at runtime during the alert evaluation cycle:

1. An alert fires for a host based on an alert rule's threshold and duration.
2. The causal engine runs an investigation, producing a probable cause, confidence score, and heuristic rule.
3. The remediation engine queries all active runbooks and matches trigger conditions against the alert's metric type, investigation cause pattern, severity, and heuristic rule.
4. If a match is found and safety constraints (daily limit, cooldown) pass, a remediation execution is created.
5. If no match is found, the engine may create a suggested runbook for the operator to review and configure.

Remediation execution

Triggered execution (automatic)

When the remediation engine matches a runbook to an alert, the execution proceeds through the lifecycle automatically. If the runbook requires approval, it pauses at "Pending Approval" until an operator approves.

Manual approval workflow

1. Navigate to Remediation > Executions.
2. Find executions with status "Pending Approval".
3. Review the runbook, alert, and investigation details.
4. Click Approve to allow execution, or Cancel to abort.
5. Monitor execution progress through the state transitions on the execution detail page.

Remediation lifecycle

Each remediation execution moves through a defined state machine:

Remediation history and reporting

1. Navigate to Remediation > Executions to browse all executions with filters for status, host, and runbook.
2. Click an execution to see full details: action job results, validation results, rollback results, error messages, and timestamps for each state transition.
3. Navigate to Remediation > Dashboard for aggregate statistics.

Causal analysis and root cause chain

When an alert fires, the causal analysis engine runs an automated investigation to identify the probable root cause. This investigation drives both the remediation matching and operator decision-making.

Investigation output

Incident timeline

For a unified view of the full causal chain from alert to resolution:

1. Navigate to an alert incident and click the Timeline tab.
2. The timeline shows alerts, investigations, and remediations in chronological order.
3. This provides the complete chain: what triggered, what was investigated, what was attempted, and the final outcome.

Permissions reference

Navigation reference

Troubleshooting