Firecracker v1.12.0

Added

• #5048: Added support for PVH boot mode. This is used when an x86 kernel provides the appropriate ELF Note to indicate that PVH boot mode is supported. Linux kernels newer than 5.0 compiled with CONFIG_PVH=y set this ELF Note, as do FreeBSD kernels.
• #5065 Added support for Intel AMX (Advanced Matrix Extensions). To be able to take and restore a snapshot of Intel AMX state, Xsave is used instead of kvm_xsave, so users need to regenerate snapshots.
• #4731: Added support for modifying the host TAP device name during snapshot restore.
• #5146: Added Intel Sapphire Rapids as a supported and tested platform for Firecracker.
• #5148: Added ARM Graviton4 as a supported and tested platform for Firecracker.

Changed

• #5118: Cleared WAITPKG CPUID bit in CPUID normalization. The feature enables a guest to put a physical processor into an idle state, which is undesirable in a FaaS environment since that is what the host wants to decide.
• #5142: Clarified what CPU models are supported by each existing CPU template. Firecracker exits with an error if a CPU template is used on an unsupported CPU model.

Deprecated

• #4948: Deprecated the page_size_kib field in the UFFD handshake, and replaced it with a page_size field. The page_size_kib field is misnamed, as the value Firecracker sets it to is actually the page size in bytes, not KiB. It will be removed in Firecracker 2.0.

Fixed

• #5074 Fix the SendCtrlAltDel command not working for ACPI-enabled guest kernels, by dropping the i8042.nopnp argument from the default kernel command line Firecracker constructs.
• #5122: Keep the UFFD Unix domain socket open to prevent the race condition between the guest memory mappings message and the shutdown event that was sometimes causing arrival of an empty message on the UFFD handler side.
• #5143: Fixed to report process_startup_time_us and process_startup_time_cpu_us metrics for api_server right after the API server starts, while previously reported before applying seccomp filter and starting the API server. Users may observe a bit longer startup time metrics.

Firecracker v1.11.0

Added

• #4987: Reset physical counter register (CNTPCT_EL0) on VM startup. This avoids VM reading the host physical counter value. This is only possible on 6.4 and newer kernels. For older kernels physical counter will still be passed to the guest unmodified. See more info here
• #5088: Added AMD Genoa as a supported and tested platform for Firecracker.

Changed

• #4913: Removed unnecessary fields (max_connections and max_pending_resets) from the snapshot format, bumping the snapshot version to 5.0.0. Users need to regenerate snapshots.
• #4926: Replace underlying implementation for seccompiler from in house one in favor of libseccomp which produces smaller and more optimized BPF code.

Fixed

• #4921: Fixed swagger CpuConfig definition to include missing aarch64-specific fields.
• #4916: Fixed IovDeque implementation to work with any host page size. This fixes virtio-net device on non 4K host kernels.
• #4991: Fixed mem_size_mib and track_dirty_pages being mandatory for all PATCH /machine-config requests. Now, they can be omitted which leaves these parts of the machine configuration unchanged.
• #5007: Fixed watchdog softlockup warning on x86_64 guests when a vCPU is paused during GDB debugging.
• #5021 If a balloon device is inflated post UFFD-backed snapshot restore, Firecracker now causes remove UFFD messages to be sent to the UFFD handler. Previously, no such message would be sent.
• #5034: Fix an integer underflow in the jailer when computing the value it passes to Firecracker's --parent-cpu-time-us values, which caused development builds of Firecracker to crash (but production builds were unaffected as underflows do not panic in release mode).
• #5045: Fixed an issue where firecracker intermittently receives SIGHUP when using jailer with --new-pid-ns but without --daemonize.
• #4995: Firecracker no longer overwrites CPUID leaf 0x80000000 when running AMD hardware, meaning the guest can now discover a greater range of CPUID leaves in the extended function range (this range is host kernel dependent).
• #5046: Retry KVM_CREATE_VM on EINTR that occasionally happen on heavily loaded hosts to improve reliability of microVM creation.
• #5052: Build the empty seccomp policy as default for debug builds to avoid crashes on syscalls introduced by debug assertions from Rust 1.80.0.

Firecracker v1.10.1

Changed

• #4907: Bump snapshot version to 4.0.0. Ensure Firecracker v1.10.1 is incompatible with snapshots from other releases.

Firecracker v1.10.0

Added

• #4834: Add VIRTIO_NET_F_RX_MRGBUF support to the virtio-net device. When this feature is negotiated, guest virtio-net driver can perform more efficient memory management which in turn improves RX and TX performance.
• #4460: Add a call to KVM_KVMCLOCK_CTRL after pausing vCPUs on x86_64 architectures. This ioctl sets a flag in the KVM state of the vCPU indicating that it has been paused by the host userspace. In guests that use kvmclock, the soft lockup watchdog checks this flag. If it is set, it won't trigger the lockup condition. Calling the ioctl for guests that don't use kvmclock will fail. These failures are not fatal. We log the failure and increase the vcpu.kvmclock_ctrl_fails metric.
• #4869: Added support for Aarch64 systems which feature CPU caches with a number of sets higher than u16::MAX.
• #4797, #4854: Added GDB debugging support for a microVM guest kernel. Please see our GDB debugging documentation for more information.

Changed

• #4844: Upgrade virtio-net device to use readv syscall to avoid unnecessary memory copies on RX path, increasing the RX performance.

Removed

• #4804: Drop Support for guest kernel 4.14. Linux 4.14 reached end-of-life in January 2024 The minimum supported guest kernel now is 5.10.

Fixed

• #4796: Fixed Vsock not notifying guest about TRANSPORT_RESET_EVENT event after snapshot restore. This resulted in guest waiting indefinitely on a connection which was reset during snapshot creation.
• #4790: v1.9.0 was missing most of the debugging information in the debuginfo file, due to a change in the Cargo defaults. This has been corrected.
• #4826: Add missing configuration of tap offload features when restoring from a snapshot. Setting the features was previously moved from net device creation to device activation time, but it was not reflected in the restore path. This was leading to inability to connect to the restored VM if the offload features were used.

Firecracker v1.9.1

Fixed

• #4824: Add missing configuration of tap offload features when restoring from a snapshot. Setting the features was previously moved from net device creation to device activation time, but it was not reflected in the restore path. This was leading to inability to connect to the restored VM if the offload features were used.
• #4829: v1.9.0 was missing most of the debugging information in the debuginfo file, due to a change in the Cargo defaults. This has been corrected.

Firecracker v1.9.0

Added

• #4687: Added VMGenID support for microVMs running on ARM hosts with 6.1 guest kernels. Support for VMGenID via DeviceTree bindings exists only on mainline 6.10 Linux onwards. Users of Firecracker will need to backport the relevant patches on top of their 6.1 kernels to make use of the feature.
• #4732, #4733, #4741, #4746: Added official support for 6.1 microVM guest kernels.

Changed

Deprecated

• Support for guest kernel 4.14 is now deprecated. We will completely remove 4.14 support with Firecracker version v1.10

Removed

• #4689: Drop support for host kernel 4.14. Linux 4.14 reached end-of-life in January 2024. The minimum supported kernel now is 5.10. Guest kernel 4.14 is still supported.

Fixed

• 4680: Fixed an issue (#4659) where the virtio-net device implementation would always assume the guest accepts all VirtIO features the device offers. This is always true with the Linux guest kernels we are testing but other kernels, like FreeBSD make different assumptions. This PR fixes the emulation code to set the TAP features based on the features accepted by the guest.

Firecracker v1.8.0

Added

• #4428: Added ACPI support to Firecracker for x86_64 microVMs. Currently, we pass ACPI tables with information about the available vCPUs, interrupt controllers, VirtIO and legacy x86 devices to the guest. This allows booting kernels without MPTable support. Please see our kernel policy documentation for more information regarding relevant kernel configurations.
• #4487: Added support for the Virtual Machine Generation Identifier (VMGenID) device on x86_64 platforms. VMGenID is a virtual device that allows VMMs to notify guests when they are resumed from a snapshot. Linux includes VMGenID support since version 5.18. It uses notifications from the device to reseed its internal CSPRNG. Please refer to snapshot support and random for clones documention for more info on VMGenID. VMGenID state is part of the snapshot format of Firecracker. As a result, Firecracker snapshot version is now 2.0.0.

Changed

• #4492: Changed --config parameter of cpu-template-helper optional. Users no longer need to prepare kernel, rootfs and Firecracker configuration files to use cpu-template-helper.
• #4537 Changed T2CL template to pass through bit 27 and 28 of MSR_IA32_ARCH_CAPABILITIES (RFDS_NO and RFDS_CLEAR) since KVM consider they are able to be passed through and T2CL isn't designed for secure snapshot migration between different processors.
• #4537 Changed T2S template to set bit 27 of MSR_IA32_ARCH_CAPABILITIES (RFDS_NO) to 1 since it assumes that the fleet only consists of processors that are not affected by RFDS.
• #4388: Avoid setting kvm_immediate_exit to 1 if are already handling an exit, or if the vCPU is stopped. This avoids a spurious KVM exit upon restoring snapshots.
• #4567: Do not initialize vCPUs in powered-off state upon snapshot restore. No functional change, as vCPU initialization is only relevant for the booted case (where the guest expects CPUs to be powered off).

Deprecated

• Firecracker's --start-time-cpu-us and --start-time-us parameters are deprecated and will be removed in v2.0 or later. They are used by the jailer to pass the value that should be subtracted from the (CPU) time, when emitting the start_time_us and start_time_cpu_us metrics. These parameters were never meant to be used by end customers, and we recommend doing any such time adjustments outside Firecracker.
• Booting with microVM kernels that rely on MPTable on x86_64 is deprecated and support will be removed in v2.0 or later. We suggest to users of Firecracker to use guest kernels with ACPI support. For x86_64 microVMs, ACPI will be the only way Firecracker passes hardware information to the guest once MPTable support is removed.

Fixed

• #4526: Added a check in the network TX path that the size of the network frames the guest passes to us is not bigger than the maximum frame the device expects to handle. On the TX path, we copy frames destined to MMDS from guest memory to Firecracker memory. Without the check, a mis-behaving virtio-net driver could cause an increase in the memory footprint of the Firecracker process. Now, if we receive such a frame, we ignore it and increase Net::tx_malformed_frames metric.
• #4536: Make the first differential snapshot taken after a full snapshot contain only the set of memory pages changed since the full snapshot. Previously, these differential snapshots would contain all memory pages. This will result in potentially much smaller differential snapshots after a full snapshot.
• #4578: Fix UFFD support not being forward-compatible with new ioctl options introduced in Linux 6.6. See also bytecodealliance/userfaultfd-rs#61.
• #4630: On x86_64, when taking a snapshot, if a vCPU has MSR_IA32_TSC_DEADLINE set to 0, Firecracker will replace it with the MSR_IA32_TSC value from the same vCPU. This is to guarantee that the vCPU will continue receiving TSC interrupts after restoring from the snapshot even if an interrupt is lost when taking a snapshot.
• #4666: Fixed Firecracker sometimes restoring MSR_IA32_TSC_DEADLINE before MSR_IA32_TSC. Now it always restores MSR_IA32_TSC_DEADLINE MSR after MSR_IA32_TSC, as KVM relies on the guest TSC for correct restoration of MSR_IA32_TSC_DEADLINE. This fixed guests using the TSC_DEADLINE hardware feature receiving incorrect timer interrupts after snapshot restoration, which could lead to them seemingly getting stuck in sleep-related syscalls (see also #4099).

Firecracker v1.7.0

Added

• #4346: Added support to emit aggregate (minimum/maximum/sum) latency for VcpuExit::MmioRead, VcpuExit::MmioWrite, VcpuExit::IoIn and VcpuExit::IoOut. The average for these VM exits is not emitted since it can be deduced from the available emitted metrics.
• #4360: Added dev-preview support for backing a VM's guest memory by 2M hugetlbfs pages. Please see the documentation for more information
• #4490: Added block and net device metrics for file/tap access latencies and queue backlog lengths, which can be used to analyse saturation of the Firecracker VMM thread and underlying layers. Queue backlog length metrics are flushed periodically. They can be used to esimtate an average queue length by request by dividing its value by the number of requests served.

Changed

• #4230: Changed microVM snapshot format version strategy. Firecracker snapshot format now has a version that is independent of Firecracker version. The current version of the snapshot format is v1.0.0. From now on, the Firecracker binary will define the snapshot format version it supports and it will only be able to load snapshots with format that is backwards compatible with that version. Users can pass the --snapshot-version flag to the Firecracker binary to see its supported snapshot version format. This change renders all previous Firecracker snapshots (up to Firecracker version v1.6.0) incompatible with the current Firecracker version.
• #4449: Added information about page size to the payload Firecracker sends to the UFFD handler. Each memory region object now contains a page_size_kib field. See also the hugepages documentation.
• #4501: Only use memfd to back guest memory if a vhost-user-blk device is configured, otherwise use anonymous private memory. This is because serving page faults of shared memory used by memfd is slower and may impact workloads.

Fixed

• #4409: Fixed a bug in the cpu-template-helper that made it panic during conversion of cpu configuration with SVE registers to the cpu template on aarch64 platform. Now cpu-template-helper will print warnings if it encounters SVE registers during the conversion process. This is because cpu templates are limited to only modify registers less than 128 bits.
• #4413: Fixed a bug in the Firecracker that prevented it to restore snapshots of VMs that had SVE enabled.
• #4414: Made PATCH requests to the /machine-config endpoint transactional, meaning Firecracker's configuration will be unchanged if the request returns an error. This fixes a bug where a microVM with incompatible balloon and guest memory size could be booted, due to the check for this condition happening after Firecracker's configuration was updated.
• #4259: Added a double fork mechanism in the Jailer to avoid setsid() failures occurred while running Jailer as the process group leader. However, this changed the behaviour of Jailer and now the Firecracker process will always have a different PID than the Jailer process.
• #4436: Added a "Known Limitations" section in the Jailer docs to highlight the above change in behaviour introduced in PR#4259.
• #4442: As a solution to the change in behaviour introduced in PR#4259, provided a mechanism to reliably fetch Firecracker PID. With this change, Firecracker process's PID will always be available in the Jailer's root directory regardless of whether new_pid_ns was set.
• #4468: Fixed a bug where a client would hang or timeout when querying for an MMDS path whose content is empty, because the 'Content-Length' header field was missing in a response.

Firecracker v1.6.0

Added

• #4145: Added support for per net device metrics. In addition to aggregate metrics net, each individual net device will emit metrics under the label "net_{iface_id}". E.g. the associated metrics for the endpoint "/network-interfaces/eth0" will be available under "net_eth0" in the metrics json object.
• #4202: Added support for per block device metrics. In addition to aggregate metrics block, each individual block device will emit metrics under the label "block_{drive_id}". E.g. the associated metrics for the endpoint "/drives/{drive_id}" will be available under "block_drive_id" in the metrics json object.
• #4205: Added a new vm-state subcommand to info-vmstate command in the snapshot-editor tool to print MicrovmState of vmstate snapshot file in a readable format. Also made the vcpu-states subcommand available on x86_64.
• #4063: Added source-level instrumentation based tracing. See tracing for more details.
• #4138, #4170, #4223, #4247, #4226: Added developer preview only (NOT for production use) support for vhost-user block devices. Firecracker implements a vhost-user frontend. Users are free to choose from existing open source backend solutions or their own implementation. Known limitation: snapshotting is not currently supported for microVMs containing vhost-user block devices. See the related doc page for details. The device emits metrics under the label "vhost_user_{device}_{drive_id}".

Changed

• #4309: The jailer’s option --parent-cgroup will move the process to that cgroup if no cgroup options are provided.
• Simplified and clarified the removal policy of deprecated API elements to follow semantic versioning 2.0.0. For more information, please refer to this GitHub discussion.
• #4180: Refactored error propagation to avoid logging and printing an error on exits with a zero exit code. Now, on successful exit “Firecracker exited successfully” is logged.
• #4194: Removed support for creating Firecracker snapshots targeting older versions of Firecracker. With this change, running ‘firecracker –version’ will not print the supported snapshot versions.
• #4301: Allow merging of diff snapshots into base snapshots by directly writing the diff snapshot on top of the base snapshot’s memory file. This can be done by setting the mem_file_path to the path of the pre-existing full snapshot.

Deprecated

• #4209: rebase-snap tool is now deprecated. Users should use snapshot-editor for rebasing diff snapshots.

Fixed

• #4171: Fixed a bug that ignored the --show-log-origin option, preventing it from printing the source code file of the log messages.
• #4178: Fixed a bug reporting a non-zero exit code on successful shutdown when starting Firecracker with --no-api.
• #4261: Fixed a bug where Firecracker would log “RunWithApiError error: MicroVMStopped without an error: GenericError” when exiting after encountering an emulation error. It now correctly prints “RunWithApiError error: MicroVMStopped with an error: GenericError”.
• #4242: Fixed a bug introduced in #4047 that limited the --level option of logger to Pascal-cased values (e.g. accepting “Info”, but not “info”). It now ignores case again.
• #4286: Fixed a bug in the asynchronous virtio-block engine that rendered the device non-functional after a PATCH request was issued to Firecracker for updating the path to the host-side backing file of the device.
• #4301: Fixed a bug where if Firecracker was instructed to take a snapshot of a microvm which itself was restored from a snapshot, specifying mem_file_path to be the path of the memory file from which the microvm was restored would result in both the microvm and the snapshot being corrupted. It now instead performs a “write-back” of all memory that was updated since the snapshot was originally loaded.

Firecracker v1.5.1

Added

• #4287: Document a caveat to the jailer docs when using the --parent-cgroup option, which results in it being ignored by the jailer. Refer to the jailer documentation for a workaround.

Changed

• #4191: Refactored error propagation to avoid logging and printing an error on exits with a zero exit code. Now, on successful exit "Firecracker exited successfully" is logged.

Fixed

• #4277: Fixed a bug that ignored the --show-log-origin option, preventing it from printing the source code file of the log messages.
• #4179: Fixed a bug reporting a non-zero exit code on successful shutdown when starting Firecracker with --no-api.
• #4271: Fixed a bug where Firecracker would log "RunWithApiError error: MicroVMStopped without an error: GenericError" when exiting after encountering an emulation error. It now correctly prints "RunWithApiError error: MicroVMStopped with an error: GenericError".
• #4270: Fixed a bug introduced in #4047 that limited the --level option of logger to Pascal-cased values (e.g. accepting "Info", but not "info"). It now ignores case again.
• #4295: Fixed a bug in the asynchronous virtio-block engine that rendered the device non-functional after a PATCH request was issued to Firecracker for updating the path to the host-side backing file of the device.