What it actually does: four pillars, in depth
Each pillar rests on the same Tool · Agent · Repo pattern and operates on a specific slice of the 11-layer stack. Here's what each one does, the layers it touches, and a worked before/after.
Configure the data center & the nodes
Bring up bare or cloud GPU nodes and make them correct: install and verify drivers and kernel modules, configure the network interfaces and fabric (EFA / InfiniBand / RoCE / TCP), set hugepages, validate NIC counts, and confirm the node is healthy before any workload runs. It catches the classic traps automatically because it has already learned them.
- Instance assumed to have 16 NICs; workload launched against all 16.
- RDMA quietly enabled on a secondary subnet.
- Collectives hang, then fall back to TCP, reported only as “slow.”
- Enumerates the node: finds 8 NICs, not 16, and pins the workload accordingly.
- Refuses to enable RDMA on the secondary subnet until multi-rail routing validates.
- Confirms the fabric path end-to-end before the workload starts. No silent fallback.
- L10Knowledge base
- L9Autonomous loop
- L8Benchmark harness
- L7OS / infra / driver
- L6Network fabric
- L5CI / build environment
- L4Transport plugin
- L3Communication library
- L2Framework
- L1Launch config
- L0Compute kernel
Configure & operate Kubernetes
Stand up the orchestration layer end-to-end: device plugins, node selectors and affinity, tolerations, the collective-communication operator, and the workload manifests. Everything is ordered correctly (cluster exists → drivers installed → fabric validated → benchmark) so a step never runs before its prerequisite. Works on managed Kubernetes (e.g. EKS) or any conformant cluster.
- Workload pod scheduled before the device plugin registered GPUs.
- Pods land on nodes whose fabric was never validated.
- Manifests applied in author order, not dependency order.
- Sequences the bring-up: device plugin ready → drivers verified → fabric validated → workload.
- Node selectors and tolerations generated from the actual, verified node inventory.
- A prerequisite that hasn't passed blocks the step that depends on it.
- L10Knowledge base
- L9Autonomous loop
- L8Benchmark harness
- L7OS / infra / driver
- L6Network fabric
- L5CI / build environment
- L4Transport plugin
- L3Communication library
- L2Framework
- L1Launch config
- L0Compute kernel
Launch agents that train & optimize models
The co-design engine. Portent spawns sub-agents that generate candidate implementations, evaluate them through a cascaded gate (parse → compile → correctness → benchmark), and evolve the winners: an optimization harness that improves a real workload with attributable, single-variable experiments. It combines evolutionary search over a design space, multi-LLM consensus, and a bandit that routes work to whatever is paying off.
- A hand-tuned kernel change bundles three edits; nobody knows which one helped.
- A “faster” variant is accepted without a correctness check.
- One model's guess is trusted with no cross-check.
- One layer per change, so every speedup is attributable to exactly one variable.
- The cascaded gate rejects any candidate that fails parse, compile, or correctness before it's ever benchmarked.
- Multiple models cross-check; a synthesizer resolves disagreements.
- L10Knowledge base
- L9Autonomous loop
- L8Benchmark harness
- L7OS / infra / driver
- L6Network fabric
- L5CI / build environment
- L4Transport plugin
- L3Communication library
- L2Framework
- L1Launch config
- L0Compute kernel
Learn from everything, forever
Every run, success or failure, is mined into the knowledge base. A closed loop mines → categorizes (on four axes: layer × substrate × framework × subject) → scrubs any sensitive data → verifies against a zero-false-positive gate → (human-gated) promotes the best lessons to a canonical corpus. A coverage ratchet guarantees every canonical lesson stays retrievable, actionable, and attributed, and the bar never lowers.
- A hard-won fix lives in one engineer's head, then leaves with them.
- “Tribal knowledge” gets applied on a substrate where it doesn't hold.
- Account IDs and internal hostnames leak into shared notes.
- The fix is mined, attributed to its event, and recalled automatically next time.
- Every lesson is tagged with where it applies, so it won't leak across substrates.
- The scrubber strips IDs, IPs, hostnames, and keys before anything is stored.
- L10Knowledge base
- L9Autonomous loop
- L8Benchmark harness
- L7OS / infra / driver
- L6Network fabric
- L5CI / build environment
- L4Transport plugin
- L3Communication library
- L2Framework
- L1Launch config
- L0Compute kernel