Method to reseal encrypted partition keys after PCR7 update (was: Unexpected "sealing policy digest does not match")

[jfroy]

Bug Report

Description

I applied a seemingly innocuous machineconfig edit (modified a machine.files.op: create patch for /etc/cri/conf.d/20-customization.part), which required a node reboot. After a while with the node not coming back up, I tried to get to its dashboard with talosctl, which failed. So I went to its IPMI and I saw this:

I have no idea why suddenly this node is unable to decrypt its STATE partition, but now I have to basically to go through DR and reset the node. I can't provide any more information since the node's state partition is effectively lost.

Logs

Environment

• Talos version: 1.9.1
• Kubernetes version: 1.31.4
• Platform: bare metal

[jfroy]

Ah! Damn it, I know what happened. I updated that node's dbx database some time ago but had not rebooted the node until now. I may be able to recover it by removing that entry from the firmware interface.

Is there a method to update/reseal the disk?

[jfroy]

Yep, that was it. I was able to recover by deleting the dbx variable from the firmware interface.

Since Talos does not have a systemd-measure -like system, at least for PCR7 (which is where dbx gets measured), I am not sure how I can update the dbx on that node. I only have a tpm key slot in the machineconfig and IIRC you cannot add new key slots after the initial install/seal operation.

[frezbo]

Since Talos does not have a systemd-measure -like system,

Talos internally uses similar code as sd-measure, and it omits PCR7 AFAIK, since it's the secureboot state, I think it uses pcrlock for the same

[jfroy]

Well AFAIK systemd-measure also only predicts PCR11. I took a quick glance at the seal code in Talos and it seems to read the current value of PCR7 and uses its hash in the policy. So there's no way currently at a glance to update the policy without rebooting with the new PCR7 value, which I can't do because that will prevent STATE from being decrypted. I could do it if I could retroactively add a passphrase key slot -- I will look into that. IIRC you can't do it post install, and it relies on having a META partition (which I don't have edit: I do have one edit2: there's no filesystem on it edit3: ah, it's just custom raw bytes stored in there?) to store the key in plain text, which I don't want to do long term but could do temporarily. There is no interactive prompt option AFAIK.

[frezbo]

Well AFAIK systemd-measure also only predicts PCR11. I took a quick glance at the seal code in Talos and it seems to read the current value of PCR7 and uses its hash in the policy. So there's no way currently at a glance to update the policy without rebooting with the new PCR7 value, which I can't do because that will prevent STATE from being decrypted. I could do it if I could retroactively add a passphrase key slot -- I will look into that. IIRC you can't do it post install, and it relies on having a META partition (which I don't have edit: I do have one edit2: there's no filesystem on it edit3: ah, it's just custom raw bytes stored in there?) to store the key in plain text, which I don't want to do long term but could do temporarily. There is no interactive prompt option AFAIK.

i don;t think this situation is also not possible via systemd with a dbx update (see fwupd/fwupd#6318) since there is no way to predict what value would that be to generate a lock file (correct me if I'm wrong here)

I took a quick glance at the seal code in Talos and it seems to read the current value of PCR7 and uses its hash in the policy.

this is same as sd-cryptsetup, since we don't want someone to be able to unlock disk by disabling secureboot, sadly there's no easy way predict PCR7

[jfroy]

Yeah I am not expecting a solution where PCR7 can be predicted -- as you stated and linked, that is not trivial.

What I am looking for is a method to temporarily reduce the security of the policy protecting disk encryption by not including a PCR7 hash, or a way to temporarily enroll a plaintext key (a META key), to allow a reboot with the updated dbx, followed by a reseal of the disk encryption key with the new PCR7.

This reminds me, would this also require providing the PCR signing key? I imagine it is never copied out of the installer image...

[frezbo]

What I am looking for is a method to temporarily reduce the security of the policy protecting disk encryption by not including a PCR7 hash, or a way to temporarily enroll a plaintext key (a META key), to allow a reboot with the updated dbx, followed by a reseal of the disk encryption key with the new PCR7.

yeh we would need to do pcrlock someday and allow reseal, will see if we can get this implemented for 1.10

[smira]

IIRC you cannot add new key slots after the initial install/seal operation.

you can modify the key slots, it takes a reboot to modify them, but you can (as long as one key slot is valid).

[jfroy]

IIRC you cannot add new key slots after the initial install/seal operation.

you can modify the key slots, it takes a reboot to modify them, but you can (as long as one key slot is valid).

Ah I'll give it a try.

[jfroy]

OK, so reading the code, it seems like I could

1. add a plaintext key to machineconfig and reboot
2. remove the tpm key from machineconfig and reboot
3. add a tpm key to machineconfig and reboot
4. remove the plaintext key from machineconfig and reboot

talos/internal/pkg/encryption/encryption.go

Line 91 in 09758b3

func (h *Handler) Open(ctx context.Context, logger *zap.Logger, devicePath, encryptedName string) (string, []string, error) {

calls syncKeys

talos/internal/pkg/encryption/encryption.go

Line 182 in 09758b3

func (h *Handler) syncKeys(ctx context.Context, logger *zap.Logger, path string, k *encryption.Key) ([]string, error) {

which can remove and add key slots. When TPMKeyHandler is asked to create a new key in NewKey

talos/internal/pkg/encryption/keys/tpm2.go

Line 50 in 09758b3

func (h *TPMKeyHandler) NewKey(ctx context.Context) (*encryption.Key, token.Token, error) {

It will generate a new random key and seal it in the TPM, using the current PCR values and the public PCR key, which is present on the node.

[jfroy]

OK, that procedure did work. I had to reboot more than I thought I would. I don't have concrete evidence, but it seemed that VolumeManagerController would only delete or add one key at a time. So with STATE and EPHEMERAL I had to reboot 8 times total to: add a static/plaintext/meta key to each, delete the tpm key from each, add a new tpm key to each, and delete the static/plaintext/meta key from each.