Flaky e2e test

[yitsushi]

What happened:
E2E test is not consistent, as I see because it hits rate limit when it tries to pull the image.

What did you expect to happen:
E2E test is consistent. Potential fix: if it hits rate limit, wait 2 minutes (or I don't know long, check service rate limiting docs). This extra would occur only if it hits rate limit, so it shouldn't cause significant slow down on average.

How to reproduce it:

Anything else you would like to add:

https://github.com/weaveworks-liquidmetal/flintlock/actions/runs/3745123487/jobs/6359213483

time="2022-12-21T00:04:03Z" level=debug msg="do request" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.14+unknown request.method=GET size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="fetch response received" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-type=application/json response.status="429 Too Many Requests" size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="do request" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.14+unknown request.method=GET size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=debug msg="fetch response received" digest="sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control=no-cache response.header.content-type=application/json response.status="429 Too Many Requests" size=529 url="https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6"
time="2022-12-21T00:04:03Z" level=error msg="failed executing plan" controller=microvm execution_id=01GMRZX0ZVFSBF1TZYFT689F28 execution_time=2.244781908s num_steps=2 plan_name=microvm_create_update
time="2022-12-21T00:04:03Z" level=info msg="[1/10] reconciliation failed, rescheduled for next attempt at 2022-12-21 00:04:23 +0000 UTC" action=reconcile controller=microvm vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=debug msg="saving microvm spec ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B" controller=microvm repo=containerd_microvm
time="2022-12-21T00:04:03.909277766Z" level=debug msg="(*service).Write started" ref=works.weave.flintlockd/microvm/ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=error msg="failed to reconcile vmid ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B: executing plan: executing plan steps: executing steps: executing step runtime_volume_mount: mount images docker.io/richardcase/ubuntu-bionic-test:cloudimage_v0.0.1 for volume use: getting image docker.io/richardcase/ubuntu-bionic-test:cloudimage_v0.0.1 for owner ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B: pulling image using containerd: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/richardcase/ubuntu-bionic-test/manifests/sha256:371792335fe2360e0fd30108e42eb565c990140aeb9bcc796de10a59da54beb6: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" controller=microvm
time="2022-12-21T00:04:03Z" level=debug msg="Getting spec for ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B" action=reconcile controller=microvm
time="2022-12-21T00:04:03Z" level=info msg="Starting reconciliation" action=reconcile controller=microvm vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:03Z" level=info msg="Generate plan" action=reconcile controller=microvm stage=plan vmid=ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B
time="2022-12-21T00:04:23.930717502Z" level=debug msg="event published" ns=flintlock topic=/microvm type=microvm.services.api.events.microvmspecupdated
time="2022-12-21T00:04:23Z" level=error msg="handling events: getting vmid from event data: name is required" controller=microvm
    e2e_test.go:67: 
        Timed out after 120.000s.
        Error: Assertion in callback at /root/work/flintlock/test/e2e/e2e_test.go:55 failed:
        Expected
            <string>: /var/lib/flintlock/vm/ns0/mvm0/01GMRZX0Z5J7KPAM8MMJ88W60B/firecracker.pid
        to exist

[github-actions]

This issue is stale because it has been open 60 days with no activity.

[yitsushi]

E2E nightly runs are disabled, but that does not mean it's fixed.

[github-actions]

This issue was closed because it has been stalled for 365 days with no activity.