Sheet REL-07 · Endpoint Management Manual

Alerts & Monitoring

Alert rules, the alert queue, incidents, suppression, notification channels and routing rules.

Audience: Operations and NOC teams Focus: Alert rules and triage

Scope

How to create alert rules, work alerts and incidents, and route notifications to the right people.

Step-by-step guides for alert rule creation, incident management, notifications, and templates.

Related documents:

Creating Metric Alert Rules

severity (critical, high, medium, low, info) 2. Optional notification settings:
notification_channels: array of channel IDs for alert delivery
re_notification_interval_minutes: resend if still active
escalation_threshold: occurrence count to trigger escalation
escalation_to: the people to escalate to, as a list of portal_users.id values. Every id must be an active Portal person whose tenant is linked to this account, and — for an org-scoped rule — who can reach that channel IDs, and the ids it accepts are Portal’s, not the retired local users table’s.

Optional trigger actions:

System rules: Default rules are created per org during onboarding (CPU 85%/95%, Memory 90%/97%, Disk 80%/95%, Swap 80%, Process Count 500). These cannot be modified or deleted.

Creating Event Alert Rules

severity, optional severity_filter (drop events below this level)
Optional mute_until (datetime to silence until) 2. Event sources are validated against the event_source_registry table.

Managing Alert Snooze

To temporarily suppress alerts for an endpoint:

Snoozes are always endpoint-owned. Optional metric or event rule IDs may narrow the snooze to one rule, but the endpoint, rule, creator, and IDP actor must all resolve inside the same account and compatible organization scope. A snooze in one customer account or sibling organization cannot suppress another account’s or organization’s alert.

a. Working the Alert Queue

and the total that matches. Everything is done on the server, across all matching alerts, not just the page you have:

exist.

search – text contained in the alert title, the message or the device name (for example a host name such as UXAUDIT-SRV-07, or CVE-2026-1234).

unresolved_by_severity (open alerts per severity). A tab’s count always equals the list’s total for that tab.

can see in the queue, including alerts raised by Relay’s built-in rules (new critical CVE, CISA KEV match, AD events). Who can act is decided by the alert’s organization: if a role of yours reaches the organization, you can open and act on its alerts. An account-level alert (one that belongs to no organization) needs an account-wide role.

Alert not found: <ids> and nothing changes; remove those ids and send again.

active runbooks of the alert’s organization (and Relay’s system runbooks) whose triggers match the alert: event source and type, metric type, severity, investigation cause or heuristic.

Working with Incidents

View incidents:

Manage incidents:

Correlation behavior:

Per-endpoint: alerts on the same endpoint within 5 minutes group into one incident
Cross-endpoint (opt-in): enable cross_endpoint_correlation_enabled on the organization. When 3+ endpoints trigger the same rule within 5 minutes, a single cross-endpoint incident is created.
Storm: if 20+ alerts fire per minute for the same rule, they aggregate into a [STORM] incident

Auto-resolution: incidents auto-resolve when all member alerts resolve.

Understanding Dependency-Aware Suppression

When configured:

If an upstream endpoint (per service map dependencies) has an active critical/high alert, downstream non-critical alerts are suppressed
This prevents secondary alert noise during cascading failures
Critical alerts are NEVER suppressed (always fire)
To configure: create EndpointGroupDependency records in the service map

Managing Notification Channels

To configure notification channels, go to Alerting > Channels (the relevant workflow).

Reference channels in alert rules via notification_channels array

Type	`config`
`email`	`{"to_addresses": ["ops@example.com", …]}`: recipients only, 1 to 50 addresses. In the console, type each address and press Enter.

email sender (§6a). A request that still sends smtp_host, smtp_password, from_address or recipients is refused with a message that says so.

Channels are organization-scoped. Routing rules can only use active channels from the same account, and an org-scoped routing rule can only route to channels in that same organization.

as ••••••••. When you edit a channel, leave a secret blank (or send it back as URL, enter its header values again: they are sent to the URL, so Relay does not carry them over to a new one.

a. Email Sending (your server or Cadres’ default)

Go to Settings > Account > Email Sending (the relevant workflow). You need email_settings.view to see it and email_settings.manage to change it.

The Current sender card says where your account’s email comes from:

Card shows	Meaning
Cadres’ default sender	Mail goes from Cadres’ address. Nothing to set up.
Email sending is not configured	This deployment has no default sender and you have no server. Email channels cannot deliver until you add one.

To send from your own domain:

the username and password, and the from address and name.

Select Save. The server must be reachable on the internet: Relay refuses a host that resolves to a private or local address.
Select Send test (leave the address blank to send to yourself). If it fails, the message shows the server’s own answer, for example SMTPAuthenticationError: 535 5.7.8 …. Fix the setting and test again.
Make sure your domain’s SPF record authorizes this server or your mail provider, and that DKIM signing is on at the provider. Otherwise receivers may reject the mail or file it as spam.

Cadres’ default sender always sends from Cadres’ own address, never from your domain, so there is nothing to add to your DNS for it; adding it to your SPF record would not make it send as your domain.

The password is write-only. Relay never shows it. Leave the field blank to keep it, or tick Remove the stored password. If you change the host, you must enter the password again.

To stop using your server, either turn off Send through this server (keeps the settings) or select Revert to Cadres’ default (deletes them).

Problem	What to do
Test says “Email sending is not configured”	Add your own server, or ask Cadres to configure the default sender for this deployment.
Test says “resolves to … which is not a public address”	Use the server’s public host name.
`SMTPAuthenticationError`	Check the username and password with your mail provider.
Mail arrives in spam	Check SPF and DKIM for your from address’s domain.

Pausing All Automation (Kill Switch)

In an emergency, pause all automated actions for an organization:

Notification Routing Rules

To configure routing rules, go to Alerting > Routing Rules (the relevant workflow).

Route alerts to different channels based on conditions. Rules are evaluated in priority order; first match wins.

Creating a Routing Rule

name: Rule name (unique per account)
priority: Higher number = evaluated first (descending order)
severity: ["critical", "high"] — match alerts with these severities (OR logic)
alert_type: ["metric", "event"] — match metric-based or event-based alerts
source_type: ["snmp", "agent", "active_directory"] — match by source
time_window: {"start": "18:00", "end": "06:00", "timezone": "America/New_York"} — match during specific hours (supports overnight windows)
suppress: Set true to silence matching alerts (no notification sent)

All conditions use AND logic (all must match). Within each condition, values use OR logic (any can match). Empty conditions = match everything.

those condition IDs and channel IDs must be compatible with that same organization. Cross-account, inactive, or sibling-org-only channel references are rejected.

Permission required: alerts.manage

Testing a Rule

Before activating, test your rules against a simulated alert:

Returns which rule matched, whether it suppressed, and the target channel IDs
If fallback: true, no rules matched — default alert behavior applies

Managing Rule Priority

Control evaluation order:

Higher priority values are evaluated first
First matching rule stops evaluation
Reorder updates are scope-enforced: only rules in your effective organization scope (plus account-wide rules) are updated; out-of-scope IDs are ignored

How Routing Integrates with Existing Alerts

When an alert fires, routing rules are evaluated before the alert rule’s own channels
If a routing rule matches with suppress: true, no notification is sent
If no routing rules match, the alert rule’s own channels are used (fallback behavior)
Routing works for both metric alerts (core/alert_engine.py) and event alerts (core/event_alert_adapter.py)

Example: After-Hours Escalation

Route critical alerts to PagerDuty during off-hours, email during business hours:

The after-hours rule evaluates first (higher priority); during business hours it won’t match, so the business-hours rule catches it

Platform Runtime Health (Background Loops)

Where loop health surfaces

Interpreting statuses

starting — no attempt recorded yet since the tables were (re)created; expected briefly after a deploy for long-interval loops.

Cycle claims and lease takeovers

Loops that dispatch durable work (agent jobs, outbound deliveries) claim their cadence slot in loop_cycle_ledger before running it. Two log lines are normal and not faults:

cadence slot already claimed by another holder — skipping — a second process computed the same slot. Expected briefly around a failover or a deploy overlap. If it repeats every cycle for the same loop, two processes are both holding the role lease; check select * from loop_leases and the unit status of both.

To see what a loop actually did, query the ledger:

Rows are kept for 7 days (loop_cycle_ledger_retention).

Restarting a scheduler or worker role

sudo systemctl restart cadres-relay-dev-scheduler (or cadres-relay-dev-worker) sends SIGTERM, and the role drains rather than stopping instantly:

Loops that are between cycles exit immediately — an idle role stops in about 0.2 seconds.
A loop that is mid-cycle gets up to 10 seconds () to finish, close its ledger row and write its heartbeat. Its log line is <role> runtime draining (10s)....
Anything still running past that window is cancelled, and the log names it: drain deadline (10s) reached with N task(s) still running: [...] — cancelling; their cycles will not be finalized. A cancelled cycle leaves its ledger row claimed, which is what stops any other process re-running that slot.
The lease row is deleted on the way out, so the successor takes over in under a second instead of waiting out the 300-second TTL. Seeing the successor wait the full TTL means the previous process did not shut down cleanly — it was killed, or it crashed.

That slot’s work was skipped, not repeated — the next cadence slot picks it up. Retention reaps the rows after 7 days.

Alert Volume, Flapping, and Retention

Alert retention

What	Kept for	Loop
Auto-resolved alerts nobody acknowledged, not linked to an incident	30 days	`alert_retention`
All other resolved alerts, and resolved incidents with no remaining alerts	180 days	`alert_incident_retention`
Active alerts, and any alert you acknowledged	Never deleted on age	—

Acknowledging an alert takes it out of the short horizon: a human looked at it, so it keeps the full 180-day history.