I'd say pick up a dedicated 5/64" ball end allen driver for your mortise and lori set screws. Here's a cheap one or just search for 5/64 driver and pick one you like.

https://www.amazon.com/Eklind-91105-64-Ball-Hex-Screwdriver/dp/B000189SM2/ref=sr_1_1?dchild=1&keywords=eklind+5%2F64+driver&qid=1609185712&sr=8-1