NetApp feels slow and has high latency peaks

[DanielFroehlich]

All VMs running on NetApp are way slower compare to a couple of month ago.
Inside such a VM using top, I see "iowait" up to 90%, esp. for io intensive operations like pulling images.
Netapp monitoring shows high latency values >50msec during those times:

Did we reach capacity of netapp? is the network the bottleneck?
Can somebody please investigate? I can quite easily re-produce on demand, e.g. by deploying a fleet of 10 edge VMs concurrently.

[DanielFroehlich]

Issue is still existing, causing instabilities in layered clusters running on netapp:

[DanielFroehlich]

To identify top contributors of IOPS:
https://netapp-mgmt.coe.muc.redhat.com/sysmgr/v4/storage/luns

Sort by "IOPS".
Its probably one of the trident_lun_pool_isar storage pool volumes.
If that is the case, one can idenfy the PVCs in the isar cluster by the LUN name easily.