IMP

Appearance

Troubleshooting ​

Triage Flow ​

  1. Identify affected scope: single VM, node, namespace, or cluster-wide.
  2. Inspect ImpVM conditions and recent events.
  3. Check operator logs around matching reconcile timestamps.
  4. Check agent logs on the affected node.
  5. Verify runtime and network dependencies.

Common Failure Classes ​

  • RuntimeStartFailed: VM process failed to initialize.
  • NetworkSetupFailed: IP allocation or wiring failed.
  • AgentTimeout: controller could not confirm node execution path.
  • status patch conflict loops: frequent 409 Conflict under contention.
  • InvalidSpec / missing classRef: VM stuck in Starting; operator emits InvalidSpec condition — create an ImpVMClass and set spec.classRef.
  • RuntimeStartFailed / FC_BIN not found: Firecracker binary not at expected host path; verify node extension install and agent FC_BIN env wiring.
  • One-shot VM shows panic after success: normal — init exits after command completes; check serial log for IMP_SUCCESS marker.

Quick Commands ​

  • kubectl describe impvm <name> -n <ns>
  • kubectl logs -n imp-system deploy/imp-controller-manager -c manager --since=30m
  • kubectl logs -n imp-system ds/imp-agent --since=30m
  • kubectl get events -A --sort-by=.lastTimestamp | tail -n 100

Escalation Signals ​

  • rising retry loops with no condition convergence
  • multi-node agent degradation
  • broad network allocation failures
  • sustained reconciliation backlog growth

Common Scenarios ​

VM stuck in Starting with InvalidSpec ​

bash
kubectl describe impvm <name> -n <ns> | grep -A5 Conditions

Cause: spec.classRef missing or references a non-existent ImpVMClass. Fix: Create the class and set classRef.

RuntimeStartFailed — Firecracker binary not found ​

bash
kubectl logs -n imp-system ds/imp-agent | grep FC_BIN

Cause: FC_BIN host path not set or extension not installed on node. Fix: Install talos-ext-firecracker on node, verify agent env.

One-shot VM shows kernel panic after completion ​

bash
kubectl logs <agent-pod> -n imp-system | grep -E "IMP_SUCCESS|serial"

Cause: After the guest command exits, init terminates and the kernel emits a panic line. Check the serial log for the IMP_SUCCESS marker — if present, the workload succeeded.