Whilst I don't know how to fix this directly I'd know a possible workaround.

If your motherboard still has a PCIe slot free (a 1x slot is enough), put a USB 3.0 PCIe card (example) in it and pass it through to the VM like you did your GPU.

Then it's just plug&play with any USB device + if you get a USB hub switch (if you're fancy even an KVM switch). Then you can switch between those two systems (host & guest) on the fly using a single mouse and keyboard.

Only problem here could be that the card won't get it's own IOMMU group assigned but that depends on the motherboard

If your computer has a pcie slot you try something like this maybe. FebSmart 4 Ports USB 3.0 Super Fast 5Gbps PCI Express(PCIe) Expansion Card for Windows Server, XP,7,Vista,8,8.1,10 PCs-Build in Self-Powered Technology-No Need Additional Power Supply(FS-U4-Pro) https://www.amazon.com/dp/B072LS4JH7/ref=cm_sw_r_cp_api_i_9lVOCb57A7MD4 Dont have any experience with it. So cant say if it will work or not. Just a idea.

There are different options:

This one will give you more PCIe ports.

JIUWU PCI-e Express 1X to 3 Port 1X Switch Multiplier HUB Riser Card + USB Cable https://www.amazon.com/dp/B01I2Y84TE/ref=cm_sw_r_cp_api_glt_fabc_EJHKJGJ0AGHBNTASRNBH

But you can also plug an expansion card that gives you more USB3.0 ports and then plug the risers you found via USB.

FebSmart 4 Ports USB 3.0 Super Fast 5Gbps PCI Express(PCIe) Expansion Card for Windows Server, XP,7,Vista,8,8.1,10 PCs-Build in Self-Powered Technology-No Need Additional Power Supply(FS-U4-Pro) https://www.amazon.com/dp/B072LS4JH7/ref=cm_sw_r_cp_api_glt_fabc_RQ95T2TF10G0SXMRY9Q8