Everything Is a Workflow

In the last post I said managed credentials pay off when you combine them with automation. This post is about the automation: the workflow engine most of the platform runs on.

Why it came first

I designed the workflow engine before almost anything else. The reasoning was simple. With a flexible way to define workflows and deliver jobs, you can extend what the agent does without recompiling it. The agent stays small and production hosts get fewer updates. New capability ships as a workflow instead of a new agent build rolled out to ten thousand servers.

A lot followed from that decision. Patching is a workflow. So is remediation, and so is a privileged maintenance task across a fleet. The features share one engine, and that shapes everything else.

How it works

A workflow is a graph of steps. Each step has a type, some configuration, and an on_success and on_failure. The engine walks the graph, sends out work, waits for results, fills in variables, handles failures and decides what runs next. None of it is clever. It’s a lot of plumbing that has to be correct.

Two step types send work to the agent on a host:

• Script runs a saved script on a host, a list of hosts or a host group, with parameters and, optionally, a PAM credential to run as.
• Command does the same for a single inline command, for when you don’t want to keep a script around.

The rest run inside the platform and never touch an agent:

• PAM Checkout takes a credential from the vault and puts it in a workflow variable for later steps. It can require a human approval first.
• PAM Checkin hands the credential back and rotates it for break-glass identities.
• Approval pauses until someone approves. It can need one approver, one of many, two of many, a majority or everyone, and it rejects automatically after a timeout you set.
• Conditional branches on an expression over workflow variables, with separate on_true and on_false paths.
• Parallel runs a group of steps at the same time and waits for all of them.
• Delay waits for a set time. Useful as a settling period after a change.
• Webhook calls an external HTTP endpoint with templated headers and body.
• Notification sends a message through the platform’s notification system.

Variables pass through the whole graph. One script’s output can become the next script’s parameters. A PAM checkout fills a credential variable, and a later script step names that variable to run as that account. Steps can also read safe fields about the target host, such as {{hosts.<id>.hostname}}, without you passing them in.

SOPs that scale

A workflow is an SOP that can run on one server or a hundred, or on a host group that changes size on its own. The engine fans out the jobs, tracks each one and rolls the results up to the parent step. If three hosts succeed and one fails, you see which one and why.

For risky changes you can run a canary. The workflow goes to a set percentage of hosts first. If enough succeed, it continues to the rest; if not, it stops and tells you why. It’s the same workflow definition, with no separate canary tool.

Before running anything against production you can simulate it. The engine walks every step and shows what would happen: which jobs would go out, which way each condition would go and where an approval would block. No host is touched. Plenty of tools advertise a dry run; I wanted one that works.

Runbooks decide when

Workflows say what to do. Runbooks say when.

A runbook links something happening in the platform to a workflow. Disk space crosses a threshold, a detection rule fires, an alert reaches a certain severity, or an investigation points at a likely cause. The runbook says: when this happens, run that.

Simple cases stay simple. Low disk on a Linux host runs a cleanup. A stuck Windows service gets restarted. Harder cases just get longer workflows. If an application cluster stops responding, the workflow might restart the SQL database, drain the load balancer, recycle the IIS app pools, run a synthetic check, and page on-call only if that check still fails.

Automated remediation needs guard rails, so runbooks have them. Rate limits stop a flapping alert from triggering the same fix fifty times. There are cooldowns, and approvals per runbook, with auto-approval rules based on how confident the investigation is or how healthy the service is. After the action runs, a stabilization window watches a chosen metric. If it doesn’t recover, the runbook escalates or rolls back. Automation you can’t stop is worse than none.

Three ways to edit

The editor has three views of the same workflow, and you can switch between them whenever you like.

The wizard lists the steps in order. You pick a step type, fill in its form and it validates as you go. If you’re not sure what a step type does, start here.

The visual view is a flow chart showing branches, parallel groups and success and failure paths. Click a node to edit it. It helps with anything that isn’t a straight line, and with explaining a workflow to someone who didn’t build it.

The JSON view shows the raw definition. It’s read-only on purpose, because the other two views are what you edit. It’s there to see exactly what will be saved, to copy a definition into source control, or to paste one into a ticket.

Versions and audit

Every workflow is versioned, and a published version can’t be changed. You can keep editing the draft, but the version that ran last night stays exactly as it ran, and the history tab shows what executed. A running workflow keeps a copy of its definition from when it started, so editing it mid-run changes nothing that’s already in progress.

Every step is recorded: each approval, PAM checkout, job and result. When someone asks what a workflow did, on which hosts, with whose credentials and who approved it, you can answer with one query instead of four log sources.

Sharing workflows

We’re all running the same operations on the same operating systems, and we all keep writing the same disk cleanup script and the same patch validation workflow.

The marketplace is meant to stop that. You can publish a workflow, script or runbook, tag and version it, and other Cadres customers can browse, install and rate it. It only works if people use it, which is a chicken-and-egg problem I’m aware of.

Each item has a trust level (unverified, community, verified or official) so you know what you’re installing. Channels (stable, candidate or pinned) control whether you get the latest release or a fixed version. Installs come from a snapshot taken at publish time, so a publisher can’t change a workflow after you’ve installed it.

If you’ve solved an awkward edge case nobody else has, publish it. If you’re facing something new, check whether someone already solved it.

Anything in the platform that needs more than one job (several steps or hosts, an approval, a credential, a branch, a wait) runs through this engine. Most of what “human by exception” means in practice is in here.

If you’re working through the same manual SOP on ten servers at a time, start a trial.