Under the Hood

Imp turns Kubernetes objects into Firecracker microVMs. Its control plane decides what should exist. Its node agent does the host work that must happen on the selected machine.

This page separates current source behaviour, the implemented runner lifecycle, and live proof. That boundary matters: implemented source code is not the same thing as a working runner pool in a cluster.

Three views of the system

View	What it answers	Status
Current implementation	What the current Imp source is designed to do	Source-backed
Runner lifecycle implementation	How runner setup, ownership, cleanup, and replacement work in source	Source-backed
Demonstration evidence	What has been proven in unraid-lab	Waiting for smoke test

Current implementation

1. One desired state, two execution roles

flowchart LR
  user[User or controller] --> api[Kubernetes API]
  api --> operator[Imp operator]
  operator --> vm[ImpVM desired state]
  vm --> agent[Node agent]
  agent --> fc[Firecracker microVM]
  agent --> status[ImpVM status]
  status --> api

Plain explanation: the operator owns the cluster-wide decision. The node agent owns work that can only happen on the node: disks, networking, Firecracker, and guest communication.

Evidence in the repository:

• API contracts: api/v1alpha1/
• Pool reconciliation: internal/controller/impvmrunnerpool_controller.go
• Node-local runtime work: internal/agent/
• Firecracker/VSOCK path: internal/agent/vsock/

2. A runner VM is built from layers, then started

flowchart LR
  base[Base OCI image] --> rootfs[Ext4 root filesystem]
  runner[Runner OCI layer] --> rootfs
  agentbin[Injected guest agent] --> rootfs
  rootfs --> disk[VM disk]
  disk --> fc[Firecracker]
  fc --> guest[Guest agent over VSOCK]

Plain explanation: Imp does not run a container as the job. It builds a VM filesystem from container image layers, starts a microVM, and talks to a small guest agent over VSOCK, a host-to-guest channel.

When the guest agent is enabled, Imp writes a small /.imp/init wrapper into the VM disk. That wrapper starts the guest agent in the background, then runs the image's normal /sbin/init. Runner launch happens later through VSOCK.

Evidence in the repository:

• Root filesystem assembly: internal/agent/rootfs/
• Guest init injection: internal/agent/rootfs/init.go
• Runner launch boundary: internal/agent/runnerlaunch/launcher.go

3. An ImpVM is not a Kubernetes Pod

The Imp agent is a Kubernetes DaemonSet Pod. Kubernetes schedules that Pod onto a Linux node. The agent container mounts the host's /dev/kvm, the Firecracker binary, and the guest kernel.

An ImpVM is different. It is not another Pod created for the job. Once an ImpVM is assigned to a node, the agent starts Firecracker as a Linux process on that node. Firecracker uses KVM through /dev/kvm to boot a separate guest kernel and the ext4 root filesystem Imp built from OCI layers.

flowchart TD
  k8s[Kubernetes schedules one agent Pod per enabled node] --> agent[Linux Imp agent Pod]
  agent --> kvm[Host /dev/kvm]
  agent --> kernel[Guest kernel image]
  agent --> rootfs[Ext4 root filesystem built from OCI layers]
  kvm --> fc[Firecracker Linux process]
  kernel --> fc
  rootfs --> fc
  fc --> guest[Guest Linux kernel and processes]

Plain explanation: a Pod runs container processes that share the node's Linux kernel. An ImpVM is a guest Linux system with its own kernel. Kubernetes knows the agent Pod; Imp records the VM as an ImpVM custom resource and starts its Firecracker process on the selected node.

	Agent Pod	ImpVM
Kubernetes object	DaemonSet Pod	ImpVM custom resource
Who selects the node	Kubernetes scheduler	Imp's VM scheduler
What starts the work	Kubelet starts the agent container	Agent starts a Firecracker process
Kernel seen by workload	Node's shared Linux kernel	A separate guest Linux kernel
Image use	Container image becomes the process filesystem	OCI layers become an ext4 VM disk
Main runtime record	Pod status	ImpVM.status.runtimePID and phase

Evidence in the repository:

• Linux-only Firecracker driver: internal/agent/firecracker_driver.go:1-4
• Agent DaemonSet and host mounts: charts/imp/templates/agent/daemonset.yaml:1-104
• Root filesystem, kernel, KVM, and Firecracker launch: internal/agent/firecracker_driver.go:153-323, 473-530
• VM runtime PID: api/v1alpha1/impvm_types.go:159-175

4. GitHub App to one runner

sequenceDiagram
  participant O as Pool controller
  participant G as GitHub API
  participant S as VM-owned Secret
  participant A as Node agent
  participant V as Guest VM

  O->>G: App JWT
  G-->>O: Installation token
  O->>G: JIT runner setup
  O->>S: Store one-time setup
  A->>V: VSOCK Exec environment
  V-->>A: Runner exits
  A->>S: Delete setup

Plain explanation: the GitHub App key stays with the operator. The one-time setup reaches the guest through the VSOCK execution environment. It is not baked into the VM disk or placed in command arguments.

Evidence in the repository:

• App JWT and installation-token exchange: internal/runner/githubapp.go
• JIT configuration contract: internal/runner/jitconfig_contract_test.go
• VM-owned JIT Secret: internal/controller/impvmrunnerpool_controller.go
• VSOCK handoff and post-exit deletion: internal/agent/runnerlaunch/launcher.go

For the full chronological paths, read how a runner is configured and how a queued job reaches a runner.

Security invariants

Must happen	Must not happen
Keep App credentials in the operator-side credentials Secret	Put App keys or JIT data in ImpVM.spec.env
Send the one-time setup through guest Exec environment	Bake JIT data into the root filesystem or its cache key
Keep the JIT Secret owned by exactly one VM	Print tokens, private keys, or JIT payloads in logs
Delete the JIT Secret only after runner exit or bounded cleanup	Treat request acceptance as proof that the runner completed

Runner lifecycle implementation

The runner lifecycle is implemented in the current source. The pool controller creates an ephemeral VM, creates or safely recovers its VM-owned JIT Secret, and removes the new VM if setup cannot be minted or persisted. The agent hands the one-time setup to the guest and removes that Secret after the runner exits. Terminal VMs are deleted and the next reconciliation restores capacity within the pool limits.

flowchart TD
  reconcile[Pool controller reconciles capacity] --> vm[Create ephemeral VM]
  vm --> mint[Mint or recover VM-owned JIT Secret]
  mint --> handoff[Agent delivers one-time setup]
  handoff --> exit[Runner exits]
  exit --> cleanup[Agent deletes setup; controller deletes terminal VM]
  cleanup --> replace[Pool restores capacity if needed]

  mint -. mint or persistence failure .-> remove[Delete the new VM]

Plain explanation: one VM owns one setup Secret. If setup cannot be made safely, the controller removes that VM instead of letting it boot without a runner. When the runner finishes, the temporary setup and terminal VM are removed, then the pool can create replacement capacity.

The implementation also validates the named credential fields before use, keeps JIT data tied to the VM identity, and avoids creating a second Secret when a reconcile repeats after an interrupted write.

This is source-backed implementation, not a claim that the unraid-lab runner pool has passed an end-to-end smoke test.

Implementation evidence:

• internal/controller/impvmrunnerpool_controller.go:284-391
• internal/runner/auth.go
• internal/agent/runnerlaunch/launcher.go:73-145

Read the system in order

• Architecture for the core operator/agent split.
• Runner pool for capacity and job execution.
• Runner configuration for the GitHub App and JIT setup flow.
• Job scheduling for capacity, node placement, and GitHub job assignment.
• Lifecycle for VM lifecycle concepts.
• Demonstration evidence for the live-proof boundary.
• 20-minute technical presentation for a speaker-ready walkthrough.