Imp: running one CI job in one microVM
Format: 20-minute technical walkthrough.
Purpose: explain how Imp turns a Kubernetes request into an isolated Firecracker microVM, how credentials reach the guest, and where live proof still stops.
The numbered sections are a useful order for a walkthrough. The linked pages contain the full source-backed flows.
1. The question — 1 minute
How do we run an untrusted CI job with a normal Kubernetes control plane, but a stronger execution boundary than a container?
Imp keeps Kubernetes as the system that decides. It moves the actual job into a Firecracker microVM.
Visual: split-control architecture diagram.
2. Why a microVM — 2 minutes
| Option | Strength | Cost |
|---|---|---|
| Container runner | Fast and familiar | Shares the host kernel |
| Full VM | Strong isolation | Heavy and slower to create |
| Firecracker microVM | Separate guest kernel with a small VM footprint | Requires a careful host/guest lifecycle |
MicroVMs are not always better. They are useful when a stronger job boundary justifies the extra lifecycle work.
3. The simple mental model — 2 minutes
flowchart LR
k8s[Kubernetes decides] --> imp[Imp coordinates]
imp --> vm[Firecracker executes]
vm --> job[One CI job]
job --> cleanup[Cleanup and replacement]There are four parts. Kubernetes holds desired state. Imp coordinates. The microVM executes one job. Cleanup makes the capacity reusable.
4. Split control plane — 2 minutes
flowchart LR
api[Kubernetes API] --> operator[Imp operator]
operator --> vm[ImpVM desired state]
vm --> agent[Kubernetes agent Pod]
agent --> firecracker[Firecracker Linux process]
firecracker --> guest[Guest Linux system]The operator does not perform node-local work. The agent does not become the cluster scheduler. The split keeps each responsibility small.
An ImpVM is not a Kubernetes Pod. Kubernetes schedules the agent Pod. The agent starts Firecracker as a Linux process on the selected node. Firecracker then boots a separate guest Linux kernel for the job.
Evidence: internal/controller/impvmrunnerpool_controller.go and internal/agent/.
Visual: container architecture diagram.
5. From OCI layers to a VM disk — 2 minutes
flowchart LR
base[Base OCI image] --> rootfs[Ext4 root filesystem]
layer[Runner layer] --> rootfs
guestagent[Injected guest agent] --> rootfs
rootfs --> fc[Firecracker VM]The job does not run directly from a container image. Imp assembles a VM filesystem from those layers, then boots it under Firecracker.
Trade-off: layer reuse reduces setup work; cache identity must include every input that changes the filesystem.
6. Credential path — 3 minutes
sequenceDiagram
participant O as Operator
participant G as GitHub
participant S as VM-owned Secret
participant A as Node agent
participant V as Guest VM
O->>G: App JWT
G-->>O: Installation token
O->>G: JIT runner setup
O->>S: Store one-time setup
A->>V: VSOCK Exec environment
V-->>A: Runner exits
A->>S: Delete setupThe long-lived App credential stays with the operator. The VM receives only a one-time runner setup through its execution environment.
Evidence boundary: the lifecycle implementation is source-backed. A live runner-pool deployment and smoke test remain separate proof work.
This step prepares runner registration. It does not assign a GitHub job. Runner configuration explains the App and JIT path. Job scheduling explains capacity, node placement, and GitHub's later runner-selection decision.
7. The failure question — 2 minutes
What if the agent loses contact after sending the setup, but before it knows whether the runner started?
That is an ambiguous handoff. Replaying could create two runners or reuse a one-time setup. The implemented lifecycle already prevents an unconfigured VM after JIT setup fails. A lost guest acknowledgement still needs live failure testing before it can be claimed as a proven recovery path.
Visual: runner lifecycle implementation.
8. Who owns cleanup? — 2 minutes
| Event | Owner | Reason |
|---|---|---|
| Firecracker process exits for an ephemeral VM | Agent VM lifecycle path marks the VM Succeeded | The pool controller later removes terminal runner VMs |
| Guest or node disappears | Source implementation removes terminal VMs; live failure recovery remains to be demonstrated | A missing acknowledgement must not strand a VM |
| Runtime image changes | GitOps replaces OnDelete pods deliberately | Image declaration alone does not replace old pods |
Finalizer removal makes the Kubernetes object deletable. It does not prove that the runtime process is gone. Those are separate checks.
9. What is true today — 2 minutes
Source-backed today: App JWT exchange, installation tokens, JIT Secret ownership, VSOCK delivery, and deletion after runner exit.
Not yet claimed: the full unraid-lab runner-pool deployment, job assignment, teardown, and replenishment.
Source capability and live proof are separate. This page does not treat one as evidence of the other.
10. The recording to add later — 1 minute
After the smoke test passes, add an asciinema recording to Demonstration Evidence that shows:
- a harmless job queued for the runner group;
- a fresh VM booting and receiving its one-time setup;
- the job running;
- cleanup and pool replacement.
The recording is evidence, not the explanation. It confirms the path described in the diagrams.
11. Closing — 1 minute
Imp keeps Kubernetes for desired state and uses a Firecracker microVM for the job boundary. The difficult part is not booting a VM. It is proving the credential and cleanup paths stay safe when the happy path breaks.
Questions the design should answer:
- Where would you accept containers instead of microVMs?
- Which lifecycle state needs another recovery rule?
- What evidence would you require before trusting the runner pool with a production job?