Automation

Workflows, jobs, and script execution

Create reusable scripts, dispatch ad-hoc jobs to agents, build multi-step workflows with approval gates and PAM credential integration, and schedule recurring automation.

Technical Manual
Status: Available

Prerequisites

  • Role with jobs.view / jobs.manage for job operations
  • Role with scripts.view / scripts.manage for script operations
  • Role with workflows.view / workflows.manage for workflow operations
  • For PAM-integrated workflows: pam_vaults.view / pam_vaults.manage and configured identities
  • Target hosts must be online with active agents

Creating scripts

  1. Navigate to Scripts > Create Script
  2. Enter a descriptive name
  3. Select the language: PowerShell, Bash, or Python
  4. Write or paste the script content
  5. Set the timeout (maximum execution time in seconds, default: 300)
  6. Define input parameters (name, type, required flag, default value)
  7. Optionally set a Run-as Identity for privileged execution via PAM
  8. Toggle Requires Elevation if the script needs sudo/admin
  9. Click Save

Script libraries

Create reusable function libraries at Scripts > Libraries. Link a library to a script and its content is automatically prepended to the script before execution. Use libraries for shared utility functions across multiple scripts.

Executing a script

  1. On the script detail page, click Execute
  2. Select target host(s)
  3. Fill in any required parameters
  4. Click Run -- the system creates one job per host

Viewing results

Navigate to Scripts > [Script] > Executions. Each execution shows the host name, exit code, stdout/stderr output, duration, and success/failure status.

Running ad-hoc jobs

Jobs are the fundamental execution unit. Every agent operation -- scripts, commands, service control, patching -- is backed by a Job record.

Job types

TypeDescriptionPriority
scriptExecute PowerShell, Bash, or Python scriptsNormal
commandExecute a single commandNormal
service_controlStart, stop, restart, enable, or disable servicesCritical
process_controlKill or force-kill processesCritical
network_diagnosticPing, traceroute, nslookup, dig, netstat, port checkHigh
software_installInstall software packagesNormal
user_controlCreate, delete, enable, disable, reset password for local usersCritical
group_controlCreate, delete, add/remove members for local groupsCritical
Priority determines polling order Agents pick up jobs ordered by priority (critical > high > normal > low) then by scheduled time. Service, process, user, and group control jobs are always critical priority.

Job lifecycle

Status flow

Queued Running Completed Failed Timeout Cancelled

Job actions

ActionWhen AvailableEffect
CancelQueued, Pending, RunningStops execution. Agent may still complete current work.
RetryFailed, TimeoutRe-queues the job with an incremented retry count
DeleteCompleted, Failed, CancelledRemoves the job record

Timeouts

  • Running jobs: timed out if elapsed time exceeds 2x the payload timeout
  • Stale queued jobs: auto-failed after 30 minutes past the scheduled time with "Agent did not pick up job" message

Creating workflow definitions

Workflows chain multiple steps into repeatable automation sequences with branching, approval gates, and credential management.

  1. Navigate to Workflows > Create Workflow
  2. Enter a name and description
  3. Add steps using the visual editor
  4. Configure each step's success and failure routing
  5. Click Save

Available step types

Step TypeDescriptionWorkflow State
script / commandExecute a script or command on target hosts. Creates one Job per host.Running (until all jobs complete)
approvalRequire human approval to continue. Configurable approval types.Paused (until approved/rejected)
conditionalBranch based on previous step results or variable values.No pause -- advances immediately
delayWait for a specified duration before proceeding.Paused (until delay expires)
pam_checkoutCheck out privileged credentials from a PAM vault.Running (or Paused if approval needed)
pam_checkinReturn previously checked-out credentials.Running

Step configuration

Each step has the following properties:

Step IDUnique identifier within the workflow (e.g., "health_check")
TypeStep type (see table above)
NameDescriptive label shown in the UI
On SuccessNext step on success (or "end" to complete the workflow)
On FailureNext step on failure (or "fail" to abort the workflow)

Script step example

{
  "id": "health_check",
  "type": "script",
  "name": "Run health check",
  "script_id": 42,
  "parameters": {"threshold": "90"},
  "on_success": "deploy_step",
  "on_failure": "notify_failure"
}

Approval step example

{
  "id": "approve_deploy",
  "type": "approval",
  "title": "Approve production deployment",
  "approval_type": "one_of_many",
  "approvers": ["role:admin", "user:5"],
  "timeout_minutes": 60,
  "on_success": "deploy_step",
  "on_failure": "cancelled"
}

Variable substitution

Use {{path.to.value}} syntax in script parameters, command strings, approval titles, and checkout reasons to reference data from previous steps.

Step result variables

Auto-created when a script or command step completes successfully:

SyntaxDescriptionExample
{{step_ID.stdout}}Standard output from step{{step_health_check.stdout}}
{{step_ID.exit_code}}Exit code (integer){{step_scan.exit_code}}
{{step_ID.json.KEY}}Parsed JSON key (if stdout was JSON){{step_scan.json.ip_address}}

PAM credential variables

Auto-created on PAM checkout. Variables are stored under both the explicit variable_name from step config and the identity's name.

SyntaxDescription
{{db_admin.username}}Username of the checked-out identity
{{db_admin.credential_type}}Credential type (password, ssh_key, etc.)
{{db_admin.domain}}Domain (if identity has one)
{{db_admin.password}}BLOCKED -- returns ***CREDENTIAL_BLOCKED***
Password security The {{X.password}} variable is blocked for security. Credentials flow via the run_as mechanism (transport-encrypted), never as plaintext in job payloads. Use run_as: "{{db_admin}}" on script/command steps to execute as the checked-out identity.

Conditional expressions

Conditionals can reference any variable:

{
  "id": "check_result",
  "type": "conditional",
  "config": {
    "condition": "{{step_scan.exit_code}} == 0",
    "on_true": "continue",
    "on_false": "rollback"
  }
}

Legacy syntax

The variables. prefix also works for backward compatibility: {{variables.step_scan.stdout}}, {{variables.db_admin.username}}.

Approval workflows

Approval steps pause the workflow and wait for human authorization. Pending approvals appear in Workflows > Approvals.

Approval types

TypeBehavior
singularOne designated person decides
one_of_manyAny one of the listed approvers can approve
two_of_manyTwo approvers must agree
allEvery listed approver must approve
majorityMore than half of listed approvers must approve

Each approval request has an optional timeout (in minutes). If nobody responds before the deadline, the approval auto-rejects and routes to the failure path.

Running workflow instances

  1. Navigate to the workflow definition
  2. Click Start
  3. Select target hosts
  4. The system creates a workflow instance and begins processing steps

Instance statuses

queuedCreated but not yet started processing
runningActively executing a step
pausedWaiting for approval, delay, or PAM approval
completedAll steps finished successfully
failedA step failed and routed to the abort path
cancelledManually cancelled by a user

On the instance detail page, monitor the current step, view completed step results, and respond to pending approval requests.

PAM integration in workflows

Use pam_checkout and pam_checkin steps to integrate privileged credential management directly into your workflows.

Checkout step

  • Creates a privileged session for the specified identity
  • If the identity requires approval, the workflow pauses until approved
  • Credential data is stored in the workflow instance for use by subsequent steps
  • Use run_as: "{{identity_name}}" on script/command steps to execute with the checked-out credential

Checkin step

  • Ends the privileged session(s) created by the checkout step
  • If the identity has "Rotate After Use" enabled, triggers automatic password rotation
Credential security Credential values never appear in plaintext in job payloads. The run_as mechanism passes credentials via transport encryption directly to the agent.

Recurring jobs

Schedule jobs to repeat automatically using cron expressions.

  1. Create a job (script, command, etc.) as usual
  2. Toggle Recurring
  3. Enter a Cron Expression (standard 5-field syntax, e.g., 0 2 * * * for daily at 2 AM)
  4. Optionally set Max Recurrences (total execution limit)
  5. Optionally set an End Date
  6. Click Save

The scheduler creates child jobs at each cron tick. View execution history via the parent job's children. The template job tracks the next run time and recurrence count, and stops when limits are reached.

Permissions reference

PermissionGrants
jobs.viewView jobs and job details
jobs.manageCreate, cancel, retry, delete jobs
scripts.viewView scripts, libraries, and execution history
scripts.manageCreate/update/delete scripts, execute scripts, manage libraries
workflows.viewView workflow definitions, instances, and approvals
workflows.manageCreate/update/delete workflows, start instances, respond to approvals

Workflow execution lifecycle

Define Workflow Start Instance Execute Step Evaluate Result (on_success/fail) Next Step Complete / Fail PAUSE POINTS Approval Gate Delay Timer PAM Approval Each pause resumes automatically when condition is met

Troubleshooting

SymptomCauseFix
Job stuck in "queued"Agent offline or not pollingCheck agent heartbeat. Stale jobs auto-fail after 30 minutes.
Job timed outExecution exceeded timeoutIncrease the timeout in the script settings. Server uses 2x the agent timeout.
Script returns wrong exit codeScript error handling issueCheck stderr in job result. Ensure script uses exit with correct code.
Workflow stuck in "paused"Waiting for approval or delayCheck approval requests. For delay steps: the scheduler resumes automatically when the delay expires.
Recurring job not creating childrenScheduler issueCheck the next run time on the template job. Verify cron expression syntax.
"Agent did not pick up job"Agent offline, auth failure, or host decommissionedCheck agent status. Verify org secret. Check server logs for auth failures.
Approval timed outNobody responded before deadlineSet a longer timeout or notify approvers. Auto-rejected approvals route to the failure path.