From a wall of AI ideas to one buildable architecture

The team's ask · paraphrasedWhen a Zabbix alert fires, someone checks CPU, memory, disk, services and logs separately, often repeating the same checks. We wanted that gathered automatically, with a summary and suggested action attached to the alert.

Where the intelligence isAlarm triage + evidence collection

Sub-task for the System One typed decisionAlarm → {known-benign, needs-collection, escalate} class and which collection playbook to run; after collection, cause class {cpu, memory, disk, service, network, other}

Stays with the reasoning LLMThe incident summary and suggested action written into Opsgenie

Gain from the typed layerVery high — this is Monitoring Agent and Disk Space generalised to every Zabbix alarm, with feasibility scored 10 by the team. It is the first worker to build in the group

Same worker asMonitoring Agent · Disk Space

What the follow-up session saidAlert Triage is the entry point of Flow B; the same alarm stream also feeds the Governance SOC triage (SOC Analyst) on the security side — one triage_decision table, two consumers.

The team's ask · paraphrasedWe keep applying the same fixes by hand for full disks, stopped services and memory-hungry processes. We wanted the right playbook suggested with human approval first, and run automatically for low-risk cases later.

Where the intelligence isPlaybook selection + risk-gated execution

Sub-task for the System One typed decisionAlarm + collected state → {playbook id, none} with confidence, plus a risk class {low, medium, high}; the confidence × risk pair decides auto-run vs approval

Stays with the reasoning LLMNone on the decision path; a one-line explanation for the approver

Gain from the typed layerVery high — choosing one of a fixed set of playbooks is the canonical typed decision, and the team already frames it as approval-first. This is the action arm of Alert Triage (Alert Triage Assistant): triage decides what is wrong, remediation decides which playbook

Same worker asAlert Triage Assistant · Disk Space

The team's ask · paraphrasedTracking down gateway errors, timeouts and backend failures on our load balancers means reading many logs and configs together. We wanted help pinpointing the source, including running connectivity and health checks when needed.

Where the intelligence isInvestigator profile: proxy layer

Sub-task for the System One typed decisionError class {backend down, timeout, DNS, connection limit, TLS, config} from the log pattern → picks the checks to run

Stays with the reasoning LLMCorrelating logs, health and config into a cause; choosing follow-up tests

Gain from the typed layerMedium — the reasoning is LLM; the error class saves most of it. Third profile of the same Incident Investigator worker (DB DB Incident Investigator, platform Incident Analyzer, proxy Load Balancer Troubleshooting)

Same worker asIncident Analyzer · DB Incident Investigator

The team's ask · paraphrasedSecrets or insecure settings can end up in code by mistake, and sorting through security scanner results by hand takes effort. We wanted AI to explain findings, suggest fixes and flag risks during code review.

Where the intelligence isSecurity-finding triage

Sub-task for the System One typed decisionPer scanner finding: {true positive, false positive, accepted risk, needs review} + severity — the same shape as SOC alarm triage

Stays with the reasoning LLMFix suggestion text for confirmed findings

Gain from the typed layerHigh — scanners produce far more findings than anyone reads; triage is the bottleneck, and it is a typed decision. Same "MR review" family as IaC Reviewer and Code Review Agent

Same worker asIaC Reviewer · Code Review Agent

The team's ask · paraphrasedWe judge deployment risk mostly by hand and don't use our deployment history systematically. We wanted clear risk signals, like a pending migration or recent failures, without the system making the call.

Where the intelligence isTyped risk indicators over the change set

Sub-task for the System One typed decisionPer deployment: each risk indicator as a typed flag with confidence (migration present, critical-service touched, recent failures, large diff, first-time author on service) and an overall risk class

Stays with the reasoning LLMOptional one-line explanation of why an indicator fired

Gain from the typed layerVery high — the entry itself says "the system does not decide, it shows indicators"; that is a typed-decision product by definition. Same score the Engineering SDLC agents use as agent_task.risk_score

What the follow-up session saidworkshop now names "Internal LLM" as the model — consistent with the NOC report script (NOC Network Health Report) already sending to "our LLM". Confirm in the session which model and who operates it.

The team's ask · paraphrasedDuring an incident our logs, metrics, cluster events and deploy history sit in separate places, so finding the root cause drags on. We wanted them correlated by time, with likely causes summarized for us.

Where the intelligence isTime-correlated root-cause reasoning

Sub-task for the System One typed decisionPer signal in the window: "anomaly onset?" and "related to a change?" flags — the LLM reasons only over signals that fired

Stays with the reasoning LLMCorrelation narrative, root-cause candidates

Gain from the typed layerMedium-high — the platform twin of DB Incident Investigator; one Incident Investigator worker with a DB profile and a platform profile

Same worker asDB Incident Investigator

The team's ask · paraphrasedDigging through CI/CD logs for pipeline failures is slow, depends on who's looking, and repeats for the same errors. We wanted AI to point to the likely cause and fix, using past failures.

Where the intelligence isCI log triage + fix suggestion

Sub-task for the System One typed decisionPer failed job: {regression, flaky, infra, env, data, dependency} class + fingerprint (dedupe "same failure again") + priority

Stays with the reasoning LLMExplanation of the probable cause and the fix steps, only for new fingerprints

Gain from the typed layerVery high — the same failures recur; once fingerprinted, most jobs never reach the LLM. This is the same worker as Engineering Automated Test Results: one CI-triage worker, two consumers

Same worker asAutomated Test Results

What the follow-up session saidEngineering said their CI log analysis is already live (Automated Test Results). Establish what runs and on which model; this DevOps entry and Automated Test Results are the same CI-triage worker.

The team's ask · paraphrasedWe mostly analyze Redis slowdowns, memory pressure, slow commands, connection spikes and replication issues by hand. We wanted its stats checked regularly for anomalies and capacity problems, with a likely cause explained during incidents.

Where the intelligence isHealth checks + incident explanation

Sub-task for the System One typed decisionPer instance per window: {healthy, watch, warn} + cause class {memory, slow command, connections, replication} — the Redis profile of DB Health Check

Stays with the reasoning LLMIncident explanation

Gain from the typed layerHigh — same shape as Kafka and DB health; one health-check worker with engine profiles

Same worker asDB Health Check

The team's ask · paraphrasedLogs and app data can fill our disks fast, and we usually only investigate after the alert. We wanted disk growth forecast ahead of time, and the cause found automatically when an alert fires.

Where the intelligence isTrend forecast + cause classification + safe cleanup

Sub-task for the System One typed decisionGrowth cause class {logs, journal, containers, app data, deleted-open, unknown} from the collected evidence; "safe to clean automatically?" decision

Stays with the reasoning LLMNone

Gain from the typed layerHigh — this is Disk Space and the disk part of Capacity Prediction done properly: statistics for the forecast, a typed cause class, a fixed cleanup playbook set. Same worker as Auto-Remediation for the action

Same worker asDisk Space · Capacity Prediction · Auto-Remediation

The team's ask · paraphrasedWhen a database alarm fires or looks likely, we wanted the root cause found for us: query plans, blocking, waits, recent deployments and similar past incidents, with a suggested fix.

Where the intelligence isMulti-step root-cause analysis (AWR/ASH, top SQL, plans, blocking, deploy correlation)

Sub-task for the System One typed decisionEntry gate: "is the alarm real?", incident type, "which check first?" routing — plus, inside the worker, the "is this normal?" verdict on each of the ~10 routine checks

Stays with the reasoning LLMRoot-cause reasoning, hypothesis chaining, deploy correlation, resolution text — the intelligence here cannot be cut

Gain from the typed layerMedium — the reasoning core stays LLM, but most tool calls in the loop are normal/abnormal checks that the typed classifier answers in ~100 ms for free; the LLM then reasons only over flagged checks

Same worker asIncident Analyzer

Revised viewRated ★ in the first pass (gate only). Under typed-decision economics the shape changes: the agent runs ~10 checks per incident, and each "is CPU / wait event / plan change abnormal?" is a typed decision. The typed classifier inside the worker cuts LLM tokens per incident by an estimated 70–80% without touching the reasoning step. Raised to ★★.

The team's ask · paraphrasedRoutine database health checks take longer as we add more databases. We wanted each one's metrics and logs reviewed automatically, with recommendations before an alarm ever fires.

Where the intelligence isMetric/log triage

Sub-task for the System One typed decisionPer check: {healthy, watch, warn} + category; one schema across five engines

Stays with the reasoning LLMPrecaution recommendation text before the alarm fires

Gain from the typed layerVery high — hundreds of checks per day, each in milliseconds

What the follow-up session saidConfirmed: routine checks across OEM, Zabbix and Grafana, suggestions before an alarm forms; when it produces an alarm it both reports and triggers the Incident Investigator. Both agents recommend only — no action on production.

The team's ask · paraphrasedDiagnosing a failing Kubernetes pod means running many commands across several sources. We wanted a read-only assistant that gathers the cluster state, connects the dots and suggests a likely cause and next steps.

Where the intelligence isMulti-source root-cause reasoning (K8s)

Sub-task for the System One typed decisionSymptom class {CrashLoop, OOM, Pending, ImagePull, network, node} from events → picks the runbook and the checks to run first

Stays with the reasoning LLMCorrelation, root-cause statement, troubleshooting steps

Gain from the typed layerMedium — the reasoning is LLM, exactly like DB Incident Investigator: same worker template, different connectors

Same worker asDevOps Copilot · Observability Copilot

What the follow-up session saidFolded into the DevOps Copilot programme (DevOps Copilot) as the Kubernetes specialist agent; the existing K8s/Loki/Prometheus tool-calling is reused, not rebuilt.

The team's ask · paraphrasedServers that should match slowly drift apart through manual changes to packages, kernel settings, services and configs. We wanted servers in the same group compared regularly, with the differences that matter explained.

Where the intelligence isDeterministic diff + significance classification

Sub-task for the System One typed decisionPer detected difference: {cosmetic, worth noting, risky, critical} and the affected area

Stays with the reasoning LLMExplanation of why a risky drift matters, only for risky/critical

Gain from the typed layerHigh — the diff is free; the value is in not reading 400 harmless differences. A typed significance verdict per difference is exactly the right shape

The team's ask · paraphrasedBad resource limits, risky settings and broken configurations in infrastructure-as-code changes can slip through manual review. We wanted AI to review those changes in the merge request and leave comments.

Where the intelligence isMR review (IaC)

Sub-task for the System One typed decisionPer changed resource: policy-violation class {none, limits, security, availability, style} + severity; decides whether the LLM comments at all

Stays with the reasoning LLMThe review comment text with the suggested change

Gain from the typed layerMedium — policy checks are largely rule-based (OPA/Checkov-style); the classifier ranks what the LLM should explain. Belongs to the same "MR review" family as Code Review Agent and DevSecOps Reviewer

Same worker asDevSecOps Reviewer · Code Review Agent

The team's ask · paraphrasedDiagnosing Kafka broker, partition, replication and disk issues means piecing together lots of metrics and CLI output. We wanted an assistant that spots replication problems, consumer lag and capacity risks, and helps plan restarts.

Where the intelligence isHealth checks + plan

Sub-task for the System One typed decisionPer broker/topic/partition per window: {healthy, watch, warn} + cause class — the Kafka profile of DB Health Check

Stays with the reasoning LLMRolling-restart plan text; explanation of a warn

Gain from the typed layerHigh — most listed tasks are typed checks over metrics; only the rolling-restart plan and explanations are prose

Same worker asDB Health Check

The team's ask · paraphrasedWe wanted an agent that watches our logs and metrics around the clock, spots anomalies, suggests a fix, pages the on-call engineer when needed, and warns us before bottlenecks hit.

Where the intelligence isAnomaly classification + recommendation

Sub-task for the System One typed decisionFor every log/metric window: anomalous? → {normal, warn, critical} + category; at 24/7 volume

Stays with the reasoning LLMResolution text, narrative for the on-call engineer

Gain from the typed layerVery high — calling an LLM on a continuous stream is uneconomical; the typed classifier filters the stream in under 100 ms

Same worker asAlert Triage Assistant · NOC Network Health Report

The team's ask · paraphrasedWe often set Kubernetes CPU and memory limits by guesswork, which wastes resources or leads to crashes and throttling. We wanted per-deployment recommendations based on how much each service actually uses.

Where the intelligence isPercentile statistics + rules

Sub-task for the System One typed decisionWorkload shape class {steady, bursty, batch} to pick the right sizing rule

Stays with the reasoning LLMNone

Gain from the typed layerLow — P95/P99 arithmetic and VPA-style recommenders do this without AI; the classifier only chooses the rule set

The team's ask · paraphrasedWe wanted to catch over-privileged database users quickly by regularly checking what rights each account holds and how often it actually uses them, then reporting the excess.

Where the intelligence isRules + contextual classification

Sub-task for the System One typed decision"Is this privilege excessive for this role?" score, with usage-frequency context — run continuously on every principal

Stays with the reasoning LLMNot needed

Gain from the typed layerMedium — closes the grey zone the rule engine leaves; the same bandwidth argument as Cleanup applies

The team's ask · paraphrasedWhen a low-disk alert fires, someone has to check the server, decide whether to clean up or expand, and get approval. We wanted that handled step by step, with on-call kept informed.

Where the intelligence isDecision: {cleanup, extend, escalate, wait}

Sub-task for the System One typed decisionTyped decision + confidence from server state; the approval threshold is tied to the confidence score

Stays with the reasoning LLMRationale text for the on-call engineer (short; a template also works)

Gain from the typed layerHigh — decision at alarm time, no LLM latency

Same worker asAlert Triage Assistant

The team's ask · paraphrasedPutting together what goes into each production release by hand takes time. We wanted release notes built from commits, merge requests and tickets, sorted into fixes, features, infrastructure and configuration.

Where the intelligence isChange categorisation + prose

Sub-task for the System One typed decisionPer commit/MR: category {bug fix, feature, infra, database, config} and "user-visible?" flag

Stays with the reasoning LLMOne paragraph of release-note prose per category

Gain from the typed layerHigh — categorisation is the whole job; the prose is a short single call. The same categorised change list feeds Deployment Risk Analyzer

The team's ask · paraphrasedPlanning OS patching around service dependencies, reboots and the order cluster nodes get updated is done by hand. We wanted a maintenance plan built from each server's role, then a controlled rolling update.

Where the intelligence isDependency-aware plan + controlled execution

Sub-task for the System One typed decisionPer server: role class, "reboot required?", "safe to patch now?" from health and cluster state; patch-order gating per batch

Stays with the reasoning LLMPlan narrative and the rationale for the proposed order; none needed for execution

Gain from the typed layerMedium — the plan is largely derivable from inventory and dependency data; the typed gates are what make rolling execution safe. Execution stays behind approval

The team's ask · paraphrasedWe wanted to track database resources like disk, memory, CPU and connections, and get a forecast of when each one will hit its limit.

Where the intelligence isTime-series forecasting

Sub-task for the System One typed decisionPer object daily: "breaches in {7, 30, 90, never} days?" as a classification over recent-trend features — coverage over precision

Stays with the reasoning LLMNarrative report

Gain from the typed layerLow-medium — the forecast itself stays with classical models (Prophet/ARIMA). The typed classifier adds breadth: a coarse horizon class on every tablespace/partition/host, where per-object model fitting would never be done

Revised viewRated — in the first pass ("not a typed-classifier task"). Correct for the forecast, but too quick: the typed classifier at that price makes it viable to classify every object daily into a breach-horizon bucket. Worse per-object accuracy than ARIMA, far wider coverage. Raised to ★.

The team's ask · paraphrasedWriting up an incident afterward, with the timeline, cause, impact and follow-ups, is slow manual work. We wanted a draft postmortem pulled together from our monitoring, logs and tickets that the team then checks.

Where the intelligence isDocument generation from the incident record

Sub-task for the System One typed decisionTimeline event class {detection, escalation, mitigation, change, recovery} to structure the draft

Stays with the reasoning LLMThe postmortem draft itself

Gain from the typed layerLow-medium — mostly generation; the value depends on the incident memory (DB Incident Investigator, Incident Analyzer) being populated first

The team's ask · paraphrasedWe wanted legal information requests handled automatically, with the final report checked for gaps and the missing data pinpointed so we can complete it from our log archives.

Where the intelligence isReport sufficiency analysis

Sub-task for the System One typed decisionPer-field {missing, present, unclear} check; class for which source completes it

Stays with the reasoning LLMIdentifying data to fill the gaps, report text

Gain from the typed layerMedium — the checking step gets cheap

Same worker asLegal Requests (IP/port)

What the follow-up session saidThe legal-request matching tool already exists on the NetSec side (Legal Requests (IP/port): QRadar + Splunk APIs, matches by time and source IP; 25k rows in ~1.5 h). This worker sits on top of it — it checks sufficiency and fills gaps, it does not redo the matching.

The team's ask · paraphrasedInvoices go through several approval steps. We wanted to upload an invoice image, have its details read automatically, and route it from approver to approver, with notifications by chat and email.

Where the intelligence isDocument extraction + approval routing

Sub-task for the System One typed decisionTyped extraction of the invoice fields (supplier, amount, VAT, date, cost centre) and "which approval chain?" routing

Stays with the reasoning LLMVision model for the image → text step when invoices arrive as photos; not needed for e-invoice XML

Gain from the typed layerMedium — the workflow is deterministic; the extraction is the AI step and it is the same primitive Finance needs for Smart PO Matching, so it is built once

The team's ask · paraphrasedDevelopers keep coming to us for deployment status, pod restarts, failed pipelines and log queries. We wanted an assistant they can ask in plain language, read-only first, then acting with approval.

Where the intelligence isRouter + specialist agents (GitLab, Kubernetes, observability, host & alarm, database, cloud) — phase 1 read-only Q&A by topic, phase 2 approval-gated actions by action type

Sub-task for the System One typed decisionIntent → tool routing (which system, which query template) and the read/write guardrail on every proposed action

Stays with the reasoning LLMQuestion understanding, query composition, answer synthesis

Gain from the typed layerMedium — the agent loop is LLM; the classifier keeps it on the safe, read-only path. Merge with Observability Copilot and K8s Troubleshooting (Kubernetes Troubleshooting): one copilot, several skills

Same worker asKubernetes Troubleshooting · Observability Copilot

What the follow-up session saidDecided in the room: no single big DevOps chatbot. A router plus specialist agents by topic (Flow D), phased by topic — phase 1 answers questions only, phase 2 takes actions, grouped by action type. DevOps already has tool-calling into Kubernetes, Loki and Prometheus (built in-house); that becomes the first connector layer of the core. DevOps Copilot, Kubernetes Troubleshooting and Observability Copilot are one programme and are not scored separately.

The team's ask · paraphrasedWe wanted unused objects in our databases, like stale indexes, old partitions and orphaned users, found regularly and reported with cleanup suggestions, so resources are used well and the warehouse stays tidy.

Where the intelligence isAssessment, not a scheduled worker: static scan of the application code base × the DB catalog, producing evidence per object

Sub-task for the System One typed decisionPer candidate object: reference class {referenced in live code, referenced only in dead/legacy code, unreferenced, unknown (dynamic SQL)} — a typed verdict that turns the catalog scan into evidence the DBA can act on

Stays with the reasoning LLMThe assessment skill itself (Claude Code): reads the repositories, resolves ORM / dynamic-SQL references the scanner cannot, explains why an object is or is not used, and writes the cleanup plan with rollback steps. It never drops anything.

Gain from the typed layerMedium — the value is confidence, not volume. The catalog rules already find candidates; what the team asked for is proof that nothing in production references an object before it is touched, starting with unused indexes

What the follow-up session saidConfirmed triggers: a monitoring alarm (OEM/Zabbix/Grafana), a Health Check finding, or a DBA asking directly. Output: a report with root cause and recommendation, and the DBA can continue by prompting. "Worked out CPU rose because query X went from 10–100 ms to 500 ms" is the target shape of an answer.

Revised viewReframed after the the session notes. The real concern is "do not break production": which SQL objects are unused, and how to clean them in a way that provably cannot affect production. So this is an on-demand assessment skill (code scan + catalog snapshot → evidence list → cleanup plan), not a weekly agent proposing drops. The new entry text (data-warehouse tidy-up and development-area suggestions) rides on the same scan.

The team's ask · paraphrasedOur cloud provider only emails us when spending passes a limit and never stops anything. We wanted to see quickly where a spike comes from and how to shut services down if needed.

Where the intelligence isSpike detection + attribution

Sub-task for the System One typed decisionspike source → {service, account, region, category}; "recommend emergency shutdown?" decision

Stays with the reasoning LLMExplanation of the shutdown method

Gain from the typed layerMedium — for spike detection, classical statistics (baseline + z-score) beat both the typed classifier and an LLM

What the follow-up session saidCloud cost and CDN transfer (CDN Data Transfer) were discussed together; the cost-attribution worker stays, the CDN item leaves the AI list.

The team's ask · paraphrasedInvestigating a problem means jumping between dashboards, log tools and different query languages. We wanted to ask about our infrastructure in plain language and have the queries written and the results summarized.

Where the intelligence isNatural language → query (PromQL/LogQL)

Sub-task for the System One typed decisionIntent → data source routing and query-template selection

Stays with the reasoning LLMQuery composition and result summary

Gain from the typed layerLow — text-to-query is LLM work; merge into the DevOps Copilot as its observability skill

Same worker asDevOps Copilot · Kubernetes Troubleshooting

What the follow-up session saidFolded into the DevOps Copilot programme (DevOps Copilot) as the observability specialist agent ("merge with the related projects" — workshop note).

The team's ask · paraphrasedOur disaster recovery site has to be ready to take over at any moment, so we wanted databases, file shares, servers, Kubernetes, services and third-party access checked regularly.

Where the intelligence isChecklist evaluation

Sub-task for the System One typed decision{ready, at risk, not ready} component score from heterogeneous check outputs

Stays with the reasoning LLMReadiness report text

Gain from the typed layerMedium — the checks are deterministic, the classification is the typed classifier

The team's ask · paraphrasedMuch of our CDN bill comes from the media we send users. We wanted to ship assets like team logos at lower resolution and let AI upscale them on the client side.

Where the intelligence isImage super-resolution model on the client

Sub-task for the System One typed decision— (not a decision task)

Stays with the reasoning LLMNone

Gain from the typed layerNone — this is a client-side vision model, not a typed decision; and before any ML, modern formats (AVIF/WebP), responsive sizes and caching usually capture most of the saving

What the follow-up session saidDecided: not an AI project. Serve x/2x/3x image variants by device resolution (mobile and desktop currently fetch the same images) — do it now as a plain engineering task. On-device AI upscaling was discussed and set aside as experimental.