objectstorage-provisioner moving to crashloopbackoff when the cephobjectstore is edited.

[amshankaran]

Description:
cephobjectstore has been configured and the objectstorage-provisioner was started and it was running properly.
when I edited the cephobjectstore configuration, objectstore and rgw pod got restarted.
during this time objectstorage-provisioner moved to unknown state and then it got stuck with crashloopbackoff

~> kubectl get pods -A -o wide | grep obj
rook-ceph objectstorage-controller-7d5c7874c6-wrpr6 0/1 Unknown 3 (8m51s ago) 115m pool1-bm08e5
rook-ceph objectstorage-provisioner-8bdd4d77d-wthsj 0/2 Unknown 2 (8m52s ago) 24m node1
~> kubectl get pods -A -o wide | grep obj
rook-ceph objectstorage-controller-7d5c7874c6-wrpr6 1/1 Running 4 (55s ago) 115m 192.168.0.38 node1
rook-ceph objectstorage-provisioner-8bdd4d77d-wthsj 1/2 NotReady 4 (55s ago) 24m 192.168.0.37 pool1-bm08e5
~> kubectl get pods -A -o wide | grep obj
rook-ceph objectstorage-controller-7d5c7874c6-wrpr6 1/1 Running 4 (59s ago) 115m 192.168.0.38 node1
rook-ceph objectstorage-provisioner-8bdd4d77d-wthsj 1/2 CrashLoopBackOff 4 (5s ago) 24m 192.168.0.37 node1

Logs:
~> kubectl logs -f objectstorage-provisioner-8bdd4d77d-wthsj -n rook-ceph
Defaulted container "ceph-cosi-driver" out of: ceph-cosi-driver, objectstorage-provisioner-sidecar
E0911 17:45:30.697608 1 server.go:51] "Failed to start server" err="listen unix /var/lib/cosi/cosi.sock: bind: address already in use"
E0911 17:45:30.697681 1 main.go:46] "Exiting on error" err="Failed to start server: listen unix /var/lib/cosi/cosi.sock: bind: address already in use"

Lease:
NAME HOLDER AGE
cosi-ceph-objectstorage-k8s-io-cosi objectstorage-provisioner-8bdd4d77d-wthsj 26m

Workaround:
Manual deletion of the objectstorage-provisioner pod recovers the pod from crashloopbackoff