I will probably end up getting something like this 3 monitor KVM switch:

https://www.amazon.com/gp/product/B089NNJ6DP/

It's pricey, but might be the easiest/cleanest solution.

This will take care of switching the entire setup from one computer to the other.

And then for my "goof off" monitor, I could just RDP from the work laptop into my personal computer and keep it on one of my smaller monitors.

The downside being that when you RDP into a computer it screws up the DPI settings and stuff, but I guess I'll have to deal with it, I don't think there is any perfect solution here.