Audit & Webhooks

Prerequisites

• audit_logs.view for viewing the audit trail
• audit_logs.export for CSV export of audit logs
• session_recordings.view for viewing terminal/desktop sessions and recordings
• monitoring.view for viewing webhook subscriptions and deliveries
• monitoring.manage for creating/modifying webhooks and triggering tests
• Notification endpoints require only authentication (no specific permission)

Understanding Audit Logs

Every mutation in SPOG is recorded as an audit log entry. The system uses a hash chain for tamper detection — each log entry's SHA256 hash includes the previous entry's hash, creating a blockchain-style integrity chain.

What Gets Logged

All API mutations across every router: host create/delete, script executions, credential changes, policy updates, user modifications, agent registrations, workflow actions, and more. Each entry captures:

Actor Types

Viewing Audit Logs

Search and Filter

Navigate to Audit Logs from the sidebar. Use the following filters to narrow results:

Results include the user email and resource name for readability.

Recent Activity

The Recent Activity view shows the last 10 entries (up to 50). Useful for dashboard widgets.

Audit Detail

Click on any audit log entry to view the full detail, including before and after snapshots of the changed resource. Entries are tenant-isolated — you can only view entries within your own account.

Audit Statistics & Export

Statistics

Navigate to Audit Logs > Statistics to view a summary for a configurable time window (default: 7 days):

CSV Export

Click Export on the audit log page to download a CSV file. Requires audit_logs.export permission. Supports the same filters as the list view.

Integrity Verification

Navigate to Audit Logs > Integrity to verify the hash chain. The verification walks the entire chain and reports any broken links. A broken link indicates potential tampering or data corruption.

Session Recordings

All terminal and desktop sessions are automatically recorded for compliance and forensic review.

Terminal Sessions

1. Navigate to Terminal > Sessions to view active terminal sessions (filtered by host access).
2. Each session shows: session ID, host, hostname, user email, start time, and shell type.

Terminal Recordings

1. Navigate to Terminal > Recordings. Optionally filter by host or user.
2. Each recording shows: duration, bytes in/out, user email, and start/end times.
3. Click a recording to view the full playback with events in asciicast v2 format.
4. Click Download .cast to export a file compatible with asciinema play.

Desktop Recordings

1. Navigate to Desktop > Recordings to view desktop session recordings.
2. Each recording shows: frame count, total bytes, duration, and file path.
3. Click Playback to stream the recording in real time via the browser.

Configuring Webhooks

Webhook subscriptions push event notifications to external HTTPS endpoints when things happen in SPOG.

Create a Subscription

1. Navigate to Webhooks from the sidebar.
2. Review the available event types on the Event Types tab.
3. Click Create Webhook and fill in the fields below.

Test a Webhook

1. Open the webhook detail page and click Test.
2. A test ping event is sent to the endpoint synchronously.
3. The result (success/failure, status code, and response body) is displayed immediately.
4. Test deliveries are not recorded in the delivery history.

Manage Subscriptions

• Click Edit on a webhook to update its name, URL, secret, headers, event types, or active status.
• Click Delete to remove the subscription and all its delivery history.

Webhook Event Types

Webhook Delivery & Retry Logic

Payload Format

Headers included with every delivery:

• Content-Type: application/json
• X-Webhook-Event: {event_type}
• X-Webhook-Signature: sha256={hmac_hex} (only if secret is configured)

Delivery State Machine

Retry Schedule

After 5 failed attempts, the delivery is permanently marked as failed. Successful deliveries that are not claimed within 30 days are cleaned up by the scheduler.

Monitoring Deliveries

• Navigate to the webhook detail page > Deliveries tab to view delivery history. Filter by status and event type.
• Each delivery shows: attempt count, response status code, error message, and timestamps.
• Click Retry on a failed delivery to manually re-attempt it (resets the attempt count).

Notification System

In-app notifications provide real-time alerts to users via the web UI. They are persisted in the database and pushed over WebSocket for instant delivery.

Viewing Notifications

1. Click the notification bell icon in the top bar to view your notifications.
2. Filter by read/unread status and category (job, deployment, alert, security, system).
3. Broadcast notifications (not targeted to a specific user) also appear in your feed.

Managing Read Status

• Click a notification to mark it as read.
• Click Mark All Read to clear all unread notifications.
• The unread count badge on the bell icon updates in real time.

Real-Time Delivery

1. Notifications are pushed to the browser in real time via WebSocket.
2. New notifications appear immediately without page refresh.
3. Dashboard updates are also pushed in real time when relevant data changes.
4. A keepalive ping runs every 30 seconds to maintain the connection.

Notification Channels (External Delivery)

Notification channels deliver alert and remediation notifications to external systems. They are separate from the in-app notification system described above and from the webhook event subscriptions described earlier. Alert rules and event alert rules reference notification channels by ID to route notifications externally when alerts fire.

Channel types

Creating a channel

1. Navigate to Notification Channels and click Create Channel.
2. Enter a name, select the channel type, select the organization, and fill in the type-specific configuration fields from the table above.
3. Channel names must be unique within an organization.
4. URL fields must use HTTP or HTTPS schemes.

Testing a channel

1. Open the channel detail page and click Test.
2. A test notification is sent synchronously and the result (success/failure) is displayed immediately.

Linking channels to alert rules

Both metric alert rules and event alert rules can reference notification channels. When an alert fires, the system delivers notifications to all linked channels in addition to the in-app notification. Alert rules also support:

Delivery format

Slack messages use color-coded attachments (red for critical, orange for high, yellow for medium, green for low, blue for info). Teams messages use MessageCard format with matching theme colors. Webhook payloads include subject, message, severity, and source fields.

Permissions Reference

Troubleshooting