I used a usb switch with a usb hub since it only had one input. Works perfectly with mouse, keyboard and additional peripherals plugged into the USB hub. This is a 4 port usb switch that works better than my setup.
its this one. https://www.amazon.com/gp/product/B07TS5JNT3/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
It never used to do this a year ago.it started after some update in Vb. which one I cannot pinpoint. I am unsure if it has anything to do with the kvm but the combination of that and my system.