Local Storage not mounted at host reboot

[itobin]

CS Version 4.22.0
Management Server OS Ubuntu 24.04.3 LTS
KVM Host OS Ubuntu 24.04.3 LTS
QEMU Version 8.2.2
Libvirt Version 10.0.0
Storage
1 x OS NVME
1 x local NVME for VMs
1 x local SSD for VMs

We have a strange issue that when we reboot the KVM hypervisor host, the 2 x local storage drives for VMs are not mounted in CloudStack. When browsing the primary storage and using the browser option, there is no data shown in both datastores.

To get round this problem, we have to place both local storage drives into maintenance, then take out of maintenance. Then we can browse the datastores and start the VMs.

We have had this issue since we started doing a POC in previous versions of CS.

We also have a NFS share mounted as a primary shared storage that exhibits a similar issue. If we place the datastore into maintenance, then take it out of maintenance, there is no data available. To get round this issue, we have to completely reboot the host then the data is available again.

Ive seen comments about virsh in other posts but not sure how this works with CS and not really helped much.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[DaanHoogland]

@itobin , this is the first time I hear of such an issue. Can you also see the problem from the KVM hosts themselves? If so this would sound like a libvirt issue. (It would still need fixing!)
Can you add the qemu and libvirt versions to your description for people to check against their envs?

[itobin]

@itobin , this is the first time I hear of such an issue. Can you also see the problem from the KVM hosts themselves? If so this would sound like a libvirt issue. (It would still need fixing!) Can you add the qemu and libvirt versions to your description for people to check against their envs?

ive added the vesions into the description. Just a bit more info, the 2 local drives are mounted at /mnt/ssd and /mnt/nvme on the KVM host, and the NFS share is mounted at /mnt/nfs-storage/cloudstack/primary

If i reboot the host, login to SSH, cd to the /mnt/ssd /mnt/nvme and /mnt/nfs-storage/cloudstack/primary i can see all data

virsh pool-list --details --all (2 storage pools missing)

Name State Autostart Persistent Capacity Allocation Available

243cf784-7ebc-37e5-933a-ad7911dca2ec running no no 13.34 TiB 98.51 GiB 13.25 TiB
5d51a3d7-7807-4921-bbb0-c969fa49b766 running no no 97.87 GiB 15.00 GiB 82.87 GiB

In the CS GUI the storage is empty, its at this point we need to place them into maintenance and take out again

virsh pool-list --details --all (4 pools now showing which is correct)

Name State Autostart Persistent Capacity Allocation Available

243cf784-7ebc-37e5-933a-ad7911dca2ec running no no 13.34 TiB 98.51 GiB 13.25 TiB
5d51a3d7-7807-4921-bbb0-c969fa49b766 running no no 97.87 GiB 15.00 GiB 82.87 GiB
64727ad9-c3ee-4e8c-b474-3db4bb0073ce running no no 3.44 TiB 17.86 GiB 3.42 TiB
da5bd125-3afc-4bd0-9edd-def3f7310809 running no no 3.44 TiB 16.13 GiB 3.42 TiB

its an odd issue but has held us back going live with it as obviously its significant enough to cause problems such as VMs getting stuck in a startup state.

[itobin]

ok, made some progress, it seems we were missing the extra UUIDs and paths for /mnt/ssd and /mnt/nvme in the /etc/cloudstack/agent/agent.properties file, so after adding the following and rebooting the local storage is available after reboot.

local.storage.uuid=5d51a3d7-7807-4921-bbb0-c969fa49b766,64727ad9-c3ee-4e8c-b474-3db4bb0073ce,da5bd125-3afc-4bd0-9edd-def3f7310809
local.storage.path=/var/lib/libvirt/images,/mnt/ssd,/mnt/nvme

For the NFS shared storage, this isnt resolved but we found doing the following is a work around without rebooting.

After enabling maintenance on the NFS primary storage
run service cloudstack-agent restart
Then cancel maintenance and the NFS storage is now showing the data.

So not sure if this is specifc to us or a bug?

[DaanHoogland]

@itobin , I think the local part is expected. The NFS part I would say should just work, i think.

[itobin]

@DaanHoogland that would make sense, i have looked over the instructions for NFS and doesnt say anywhere about having to add entries to config files for so to me this does look like an issue somewhere with the agent/virsh

[weizhouapache]

ok, made some progress, it seems we were missing the extra UUIDs and paths for /mnt/ssd and /mnt/nvme in the /etc/cloudstack/agent/agent.properties file, so after adding the following and rebooting the local storage is available after reboot.

local.storage.uuid=5d51a3d7-7807-4921-bbb0-c969fa49b766,64727ad9-c3ee-4e8c-b474-3db4bb0073ce,da5bd125-3afc-4bd0-9edd-def3f7310809 local.storage.path=/var/lib/libvirt/images,/mnt/ssd,/mnt/nvme

For the NFS shared storage, this isnt resolved but we found doing the following is a work around without rebooting.

After enabling maintenance on the NFS primary storage run service cloudstack-agent restart Then cancel maintenance and the NFS storage is now showing the data.

So not sure if this is specifc to us or a bug?

maybe not a bug

the doc mentions

[itobin]

@weizhouapache

we reviewed the local storage requirements and after settings the extra UUIDs the local storage mounted after reboot.

The remaining issue is the NFS share, after placing it into maintenance, then taking it back out of maintance (in the GUI), the NFS share does not contain any volumes/data when viewing in the browser section.

If we place the NFS share into maintenance
restart the cloudstack agent on the KVM host
then cancel maintenance, the NFS share then shows all the data/volumes in the browser