Sheet POR-07 — Identity Manual

Audit Logs

Identity-visible change history, operator accountability, and the audit trail teams rely on during review and investigation.

Audience: Security, compliance, and tenant administrators Focus: Audit visibility and operator review

Identity systems earn trust when access and administrative change remain explainable later. This page keeps the audit-focused operator guidance from the underlying manual while excluding the private implementation paths used to collect or expose that data.

Scope

This guide covers the in-product audit-log review workflow for operators and administrators.

Portal now ships a dedicated audit-log browser at Monitoring -> Audit Logs, backed by . The same page also exposes an Export CSV action backed by for the current filtered result set.

Access

Audit-log review requires portal.audit.view.

Current roles with that permission:

super_user
admin

Review Workflow

Triage open anomaly alerts first

When you open Monitoring -> Audit Logs, review the Open anomaly alerts panel before scrolling into raw rows. The panel surfaces threshold-based detections from Portal’s audit-stream anomaly engine (audit-logs G-01 / IU-002-S04). Each row shows the severity, rule summary,

Review matching rows filters the audit-log table to the same actor / action / IP that produced the alert so you can land directly on the underlying evidence.
Acknowledge closes the alert workflow row. Only acknowledge after you have recorded triage and remediation in the incident record: acknowledgement writes an audit.anomaly_alert.acknowledged row on the immutable ledger naming the operator, the alert id, the rule, and any active support-access provenance. failure separately (audit retention is unaffected). attempted (e.g. the alert was opened during a recovery path).

You do not have to go looking for the panel. When an alert first opens, everyone in the tenant with portal.audit.view gets a notification in the header bell that links straight here — red for a critical rule, amber fires once per alert, so a burst that keeps refreshing an open alert will not flood the bell.

The detector is tenant-scoped. System / cross-tenant audit rows

The ad hoc SQL queries under remain as backup investigation paths — they are not the primary detection mechanism any more.

Working through the audit table

Navigate to Monitoring -> Audit Logs.
Confirm the current scope banner: - ordinary and operator-home sessions are restricted to the active tenant - support personnel must switch into an approved support-access context before reviewing a customer tenant
Select Apply filters to rerun the server-side search.
Select Export CSV when you need a reusable bulk-review artifact for the current filter set.
Select Verify signed export when you need to prove a previously issued manifest still matches the CSV body you were handed. The manifest tenant must match your active tenant or approved support-access tenant.
Review the table columns for timestamp, tenant, actor, action, outcome, resource, integrity, IP, request id, and the details summary.
Inspect the chain integrity banner at the top of the page before relying on any row as clean evidence (see “Audit Integrity Status” below). A green “verified clean” banner only certifies rows through the listed checked_through boundary; newer rows are pending the next scan.
Select Review on any row to inspect the full event payload, actor attribution, provenance, resource metadata, network context, and the per-row integrity disclosure.
Use Previous and Next to move through server-driven pages of results.

Audit Integrity Status

IU-002-S07 (closes audit register finding R-0032) surfaces the per-tenant hash-chain posture directly in the review UI. Before treating audit rows as clean compliance evidence, confirm the banner and the per-row integrity badge agree.

Allowed states:

Banner / badge	Meaning	Operator action
`Last clean check is stale` (yellow)	The last clean check is older than	Select Verify now; use the offline checker only for incident response
`pending next check` (yellow)	Row is hash-chained but newer than the last clean run	Re-check the page after the next scheduled scan
`not yet checked` (yellow)	No integrity check has been recorded for this chain	Select Verify now
`Audit chain integrity check found violations` (red)	Failed run for the chain	Do not use the tenant audit trail as clean evidence; treat as a security incident
`verification failed` / `chain failed elsewhere` (red)	Per-row failure detail	Treat as a security incident; use the offline checker for forensic certainty
`Latest integrity check failed to complete` (red)	Run finished with `error` status	Select Verify now once; if it repeats, investigate worker logs and use the offline checker

Behind the scenes the scheduled AuditIntegrityWorker writes the per-chain summary; the API serves a derived state per row. Verify now calls runs the same canonical walker synchronously for the active tenant. The offline CLI in backend/scripts/audit_integrity_check.py remains the independent incident-response entry point — see Tamper Detection.

Operational notes:

callers are restricted to the active tenant returned by the backend
operator tenant context does not provide all-customer audit search from this page
support personnel reviewing customer audit rows must request approval, switch into that customer tenant’s support-access context, then review or export the tenant audit trail
the browser always reloads from the backend; the page is not reconstructing a local audit cache
actor and resource filters accept exact numeric IDs rather than preloading a capped user list, so large tenants and support-access rows are still filterable from the browser
audit-detail email values are intentionally tokenized to sha256:<16-hex>@domain in the browser and CSV export; Portal preserves the domain for correlation but does not expose raw email local-parts there
ip_address and created_at are part of the stored audit row and should be used together when reconstructing a timeline
the canonical action catalog lives in backend/core/audit_registry/; the operator UI shows the raw action string and the registry is the source-of-truth lookup
every audit-log page read appends an audit.log.read row; every successful CSV export appends an audit.log.exported row in the affected tenant’s audit trail. These watcher-of-watchers rows let reviewers prove who inspected or exported audit evidence.
customer tenants can verify support-access review activity by filtering their audit trail for audit.log.read or audit.log.exported; operator-home sessions cannot use this page as an all-customer audit search.
unknown action names are accepted and warning-logged in W1 (lenient mode); PORTAL_STRICT_AUDIT_REGISTRY=true in W2+ rejects them at write time
JML template changes, imported or scheduled trigger creation, terminal trigger outcomes, manual run kickoff, callback receipt, callback-driven step completion, terminal run outcomes, and step retries now appear as governance.jml.* events, while the resulting access changes still appear under governance.access_grant.*
forwarded headers only affect ip_address when the request came through a configured trusted proxy boundary
when multiple trusted reverse proxies sit in front of Portal, controls how many trusted hops Portal walks from the right before it records the client IP

Practical Filters

Use action=user.updated or a similar exact action value when you need one event type.
For workload-identity investigations, start with action=nhi.token.denied, action=nhi.agent_action.denied, or action=nhi.identity.disabled, then and the requested or granted scope/resource fields.
For a tenant-scoped incident or support case, switch to that tenant’s authorized context first; the audit endpoint does not accept a different tenant id from the active context.
Use a date range for incident response instead of paging the whole table.
Filter action=mfa.totp.replay_rejected to see TOTP replay rejections. A cluster of these from a single user (or a single source IP across users) is a strong indicator of an active MFA replay campaign — escalate per your incident-response runbook. Replay rejection itself is transparent to the end-user (they see a generic “Invalid MFA code”), so this is the only for the underlying RFC 6238 §5.2 mechanism (W2-T12, finding 1A-SEC-02).
For suspected OIDC authorization-code replay, use CSV/export or a privileged backend query to filter action=oauth2.authorization_code.exchanged, outcome=failure, and details.trigger=presented_consumed_authorization_code. Treat details.descendants_revoked > 0 as evidence that Portal revoked the recover raw authorization codes or refresh tokens from logs; Portal intentionally does not log those values.
For brute-force or integration-abuse triage, filter action=auth.account.locked to surface threshold-crossing lockouts and action=auth.rate_limit.exceeded to surface SlowAPI 429 denials details.route_template, details.rate_limit_key, and details.limit. Use a tight 5–15 minute time window for a single campaign view. Correlate with the matching event_type=account_lockout and event_type=rate_limit_exceeded records in the application log stream; Portal intentionally does not record raw bearer tokens, cookies, authorization headers, or instantiated path parameters in audit details, so do not look for them.

Workload Identity Triage

Filter on the decision family first: nhi.identity.disabled, or nhi.trust_provider.verification.failed. the workload API denial body and the structured application log event.
Read the decision envelope in details: - token exchange rows carry requested and granted scopes, audience, resource, TTL, policy ids, and trust-provider ids

agent-action rows carry delegation id, agent and MCP identities, acting user and session, requested or granted action/tool/resource, and the safe diagnostics bundle
lifecycle rows carry previous state, target state, trust-provider id, last-used evidence, and review-item linkage

Treat details.reason_code as the canonical denial classifier. Free-text message values are explanatory copy, not the routing key.
Use the structured application log with the same event family (workload_token_exchange.*, agent_action.*, need latency or burst visibility across many repeated denials.

Live Browser Proof

The audit-log search and CSV export path has a repo-supported live browser proof. Activate it by setting PORTAL_RUN_AUDIT_EXPORT_SMOKE=1 in docs/ux/.creds alongside the acceptance credentials, then run:

audit-logs-export.csv
audit-logs-signed-*.csv
authenticated accessibility and Lighthouse summaries for the relevant workflow

The mocked CI spec (audit-logs-ci.spec.ts) covers the frontend render contract and runs as part of npm run test:playwright:ci-smoke. The two proofs are complementary: the mock guards browser UI regressions, and the live proof guards authenticated retrieval and CSV export behavior.

Recovery Guidance

If the result set is empty, first confirm you are using the correct tenant and date window.
If the actor is no longer present in the directory, actor_email may be null even though the audit row still exists.
If the event exists in the API but not the current page, widen the filters or move through the paginated result set.

Retention

Audit retention runs in two phases:

Visibility (soft window). The cleanup worker counts rows older than the effective retention window and appends audit.log_retention.evaluated; the review and export APIs hide expired rows from normal operator evidence views. The ledger row itself is untouched.
Physical destruction (hard-delete window). Rows older than (default 730 days; set 0 to disable destruction entirely) are physically deleted via the guarded audit_logs_retention_purge() SQL function — the only delete path past the append-only trigger. The effective window is never shorter than the visibility window (including per-tenant overrides). Each run appends audit.log_retention.purged with the deleted count and per-chain hash-boundary attestation.

For compliance evidence, treat audit.log_retention.evaluated as proof that the retention evaluator ran and audit.log_retention.purged as the record of physical deletion. Run the tamper checker before exporting evidence when audit integrity is material to the control assertion.

Privacy and erasure posture

Portal does not support per-row or per-subject erasure inside the retained audit ledger. Deleting or rewriting those fields would break the append-only contract and the per-tenant integrity chain.
GDPR/CCPA requests therefore require a truthful response: Portal can disable the user, sever tenant access, and stop future processing; historical audit evidence remains retained until the configured hard-delete window elapses (or indefinitely when hard-delete is disabled), after which rows are physically destroyed in whole-rows-past-cutoff granularity.
Customer-specific retention shortening is supported only through Tenant.contract_terms.audit_retention_days_override within the documented 30..3650 day range. It changes operator/API visibility windows and extends (never shortens) that tenant’s physical-destruction floor; it does not selectively rewrite the ledger.

Archival posture

audit_retention_mode=append_only_visibility is the shipped contract.
audit_retention_archive_table remains a legacy configuration key only; Portal does not currently ship physical archival or detached cold-storage export for immutable audit rows.

Operator Actions

Operator tenant context is valid for operator-tenant audit review only. It does not restore customer-tenant audit visibility through or . To review a customer tenant, request support access, obtain approval, switch into the support-access context for that tenant, and then review or export. The resulting watcher-of-watchers row is written into the customer tenant’s audit trail with the support-access grant id.

Tamper Detection

W2-T02 made audit_logs append-only at the DB layer (closes 7A-SEC-1, 7A-SEC-2, 8A-XSEC-17, 8B-XSEC-06). Portal now ships two paths for hash-chain integrity:

The scheduled AuditIntegrityWorker (IU-002-S07 / R-0032) runs the canonical chain walker on a configurable cadence (, default 3600s), persists a per-chain summary into audit_integrity_check_runs, and emits one audit.integrity_check.completed audit row per chain so worker activity is itself attributable. Operators consume the result through the Audit Integrity Status banner and the per-row badge in the review surface.
The authenticated Verify now action runs that walker for the active tenant and records audit.integrity_check.requested before verification.
The offline checker backend/scripts/audit_integrity_check.py shares the same canonical walker via core.audit_integrity and remains the manual / incident-response entry point — independent of the running Portal process. A failed worker check should block using the tenant audit trail as clean evidence; rerun the offline checker before clearing the status.

Architecture write-up: and

Hash chain versions

Each audit_logs row carries a hash_version column (Integer):

Version	Migration	Fields covered by `row_hash`

The IU-002-S01 migration rechains every surviving hashed row in each per-tenant chain to hash_version=3 and preserves retention-truncated chain starts. All new rows are written with hash_version=3. On a v2 — the offline checker reports any such row as legacy_hash_version (a remediation failure indicating that the IU-002-S01 migration did not complete), not as a healthy “OK” result. Pre-immutability rows (row_hash IS NULL) remain unchanged: no hash commitment exists to upgrade, and they are still skipped as chain resets.

Security impact of v3: an attacker who rewrites any covered evidence user_agent, etc. — on a hashed row will be detected as row_hash_mismatch by the offline checker. The remaining honest gap is pre-immutability rows that predate W2-T02; those carry no hash and cannot be retroactively chained.

When to run

credential, or the audit_logs triggers themselves were touched.

Pre-evidence collection — before generating a SOC2, ISO27001, or FedRAMP AU-9 evidence package referencing audit-log integrity.
On alarm — wire to whatever external integrity monitor your deployment uses (host-based file monitor on the trigger DDL, role-grant diff alerting, etc.).
Routinely — mature deployments run the checker hourly via cron and page on non-zero exit.

How to run

so each tenant’s chain is walked independently. It needs on audit_logs only — it does not need any write privilege.

Expected output

Healthy: OK: <N> rows scanned, <M> with hash chain, <K> pre-immutability skipped, all hashes match. Exit code 0. The pre-immutability count is the number of rows written before W2-T02 deployed; those rows have NULL row_hash and are treated as a per-tenant chain reset.
Tampered: FAILED: <N> integrity violations across <M> rows followed by up to 50 lines of the form Exit code 1.
Retention-truncated boundaries (informational, exit code 0): NOTE: <N> chain(s) start at a retention-truncated boundary. followed by Emitted when the oldest surviving row of a tenant chain references a predecessor destroyed by the retention hard-delete. Cross-check each printed hash against the details.chain_boundaries of the matching audit.log_retention.purged row — a boundary with no matching purge attestation is evidence of tampering and must be triaged as a deletion.

Triage path

Finding	Likely cause	First step
`unsupported_hash_version` on any row	Future hash algorithm rolled out without a checker update, or manual override	Verify the deployed `audit_integrity_check.py` matches the active migration; do not accept the row as healthy until both sides agree on the canonical contract
Many `row_hash_mismatch` results in one tenant	Backdated INSERTs with the trigger disabled, OR the canonicalisation contract changed	Compare the trigger DDL against the migration in version control before assuming tampering
`prev_hash_mismatch` on every row in a tenant after a known-good point	The chain was rebuilt by an attacker who replayed forward — full DBA compromise; outside the W2-T02 threat model	Escalate. Wave 6 P1 follow-up “signed audit-log export for SOC2 evidence” addresses this scenario via off-host signed log shipping

What it cannot catch

A full-DBA attacker who disables the trigger, replays every subsequent row’s hash chain with new hashes, and re-enables the trigger atomically. The W2-T02 implementation is honest about this gap; closing it requires off-host log shipping with detached signatures (Wave 6 P1).

Audit Signing Key Rotation

audit_signing_keys (F-09) - see

Confirm is set. Deployments using inline audit_signing_keyring_not_writable; rotate those by editing the variable.
Confirm an operator session with portal.system.admin and active operator mode.
Call with an empty body.
Never remove a retired key from the keyring file while signed artifacts that reference its kid remain inside retention - they become permanently unverifiable. The retired audit_signing_keys provenance rows should be kept for the same reason, though they hold no key material.

Bootstrap on a fresh install - required before evidence packets or signed exports can be issued - with python -m scripts.bootstrap_audit_signing_key. The script is idempotent. For the deploy-time placement of this step see Until bootstrap completes, and audit_signing_key_unavailable.

Verifying A Signed Audit Export

Download .
Recompute sha256 over the CSV bytes. support-access tenant. manifest.payload.content_hash as proof that the CSV is byte-identical to what Portal issued for that tenant.

Failed tenant mismatches are recorded as failed audit.export.verified attempts in the affected tenant trail once the manifest signature proves the artifact tenant. Operator-home sessions cannot verify customer artifacts unless they first switch into an approved support-access context for that tenant.

Scheduled Signed Evidence

Recurring compliance handoff no longer depends on someone remembering to pull a manual export. Tenant admins with portal.audit.view can pin the current Audit Logs filters into a scheduled signed evidence run; the AuditScheduledExportWorker materialises one signed artifact per closed cadence period and stores both the CSV bytes and the signed manifest in audit_export_artifacts. See for the design.

Create a schedule

Open the relevant workflow from a tenant membership session (support access is read-only here — see below).
Apply the audit-log filters you want the schedule to repeat (e.g. action=user.updated plus a free-text keyword). Date range and pagination are ignored: the worker computes the closed period itself.
In the Scheduled signed evidence section, set the schedule and Time of day (UTC) (HH:MM).
Click Create schedule. The schedule appears in the Active schedules table with its next run time. An audit.export_schedule.created row is written to the audit trail with the saved filter shape.

Review generated artifacts

The Generated evidence table lists the most recent artifacts (newest first). Each row shows generated time (UTC), period bounds, row count, and status.

completed artifacts expose a Download action that streams a CSV and manual signed export. Both files share the same signing key id as the manual the relevant workflow route, so existing verification flows work unchanged.
failed artifacts surface error_code so the operator can decide whether to narrow the schedule filters, bootstrap a missing signing key, or wait for the next cadence run. The worker continues processing other schedules even when one fails.

Pause or remove a schedule

Pause (Disabled) — the worker skips the schedule until it is re-enabled. Use this when an investigation produces noisy filters that would otherwise overflow the export cap.
Delete — the schedule is soft-deleted (status="deleted", deleted_at=now) and removed from the active list. Already-generated artifacts remain downloadable because the foreign key is `ON DELETE SET retention for true purging.

Pause/delete actions emit audit.export_schedule.disabled / audit.export_schedule.deleted audit rows.

Recovery: `audit_signing_key_unavailable`

If no usable audit signing keyring is loadable when the worker fires (unset , a missing/unparseable/too-permissive file, or no active key in it), the schedule’s next artifact is recorded as status=failed with error_code=audit_signing_key_unavailable. Recovery is the same operator bootstrap as the manual signed export:

Run python -m scripts.bootstrap_audit_signing_key per
Wait for the next cadence tick (or temporarily set to a shorter interval during the recovery window).
Confirm the next artifact reports status=completed and that audit.export.scheduled was written.

Support-access read-only mode

Support operators acting under an approved support-access grant remain able to review schedules and download stored artifacts read-only, but the support_access_schedule_management_denied if those routes are called. Persistent customer evidence schedules must be owned by a tenant membership session so the audit trail attributes the recurring decision to a tenant administrator rather than a transient support operator.

Trust Controls Dashboard

(operator session) and (Nexus service-key) return the same payload, built by core.operator_trust_controls.build_trust_controls_summary:

rls — RLS coverage report.
support_access.active_grants / break_glass_active — support and break-glass posture in the last 24 hours.
secret_exports.events_last_24h — signed/secret-bearing export volume in the last 24 hours.
nexus_service_key_activity — Nexus operator service-key audit activity (R-0214 / IU-013-S03).

Reviewing Nexus service-key activity

nexus_service_key_activity covers the trailing 24-hour window. Review:

events_last_24h — total Nexus service-key audit rows. Compare against recent baselines for the deployment.
by_action — which mutating Nexus operations were called and how often, ordered by descending count.
by_actor_ref — split by Nexus key (nexus-service[kid=…]). A spike confined to one kid narrows the investigation to that credential.
by_source_ip — distinct caller IPs for the period (omits rows where Portal could not resolve a client IP — fix the trusted-proxy configuration if you expect IPs but see none).
alerts — current alert codes derived from the persisted rows.

Alert response

The Cadres Nexus operator UI surfaces these alerts; Portal exposes the data only and Cadres Nexus owns the production workflow. When alerts appear, regardless of how they arrive (operator-UI banner or log aggregator picking up the inline event_type="nexus_service_key_anomaly" warning):

Alert	Operator action
`nexus_service_key_new_source_ip`	Check the alert’s `ip_address`; if it does not match a known Nexus host, treat the key as compromised, rotate it, and review the audited actions performed from that source.
`nexus_service_key_off_hours_use`	Investigate why a Nexus key was used outside business hours. If unintended, rotate the key and tighten Nexus-side scheduling; the off-hours window itself is tunable via `NEXUS_OPERATOR_KEY_BUSINESS_*` settings.

After rotating a key, confirm new activity attributes to the new kid and that nexus_service_key_activity.by_actor_ref no longer shows the retired entry.

Audit Log Streaming

Operator-grade real-time push of audit log rows to a tenant’s SIEM or log-collection endpoint. The architecture and API contract are documented at

Verify near-real-time delivery

Generate any auditable action (create a test user, rotate a service-client secret, etc.) and confirm the new row appears in the receiver within ~30s.
The Outbox tab (filter by event_type=audit.log.recorded) shows the pending → dispatched transition.
The receiver-side timestamp matches the Portal created_at for Splunk HEC (the time field is populated from the audit row, not the dispatch time).

Rotate the delivery secret

reveal modal — update the receiver’s verification config immediately.

Splunk HEC: Click Rotate token. Paste the new HEC token in the modal. The token is never echoed back; Splunk-side rotation must happen separately.

Recover dead-lettered audit stream events

If a delivery fails repeatedly (e.g. receiver outage), the corresponding audit.log.recorded outbox row eventually transitions to dead_letter.

Open Webhooks → DLQ (or the Outbox tab for outbox-level dead letters) and filter by event_type=audit.log.recorded.
Use Inspect payload to confirm the failure mode. The payload mirrors the persisted audit row and never contains the receiver’s secret/token.
Once the receiver is healthy, click Replay. The row re-enters the dispatch queue and delivery is reattempted.

Deactivate a stream

DELETE is a soft-deactivate: the subscription row and its delivery history remain so compliance auditors can prove which rows were dispatched and when. Re-activate via PATCH is_active=true (or the Reactivate button) when the receiver is back.

Limitations

tenant-owned targets.

Native Azure Event Hub SAS delivery is not shipped. Use http_json or or Function App proxy) until a native azure_event_hub target type ships.
http_json HMAC secrets and Splunk HEC tokens are returned only once on create/rotate. Lost secrets require rotation to recover.