If you get one prescribed by your doctor your Orthotist will typically dispense something like this one:

https://www.amazon.com/dp/B075FCKXTW/ref=cm_sw_r_cp_apa_glc_fabc_BBC1H7HW6WPGFG4N8YYP?_encoding=UTF8&psc=1

I haven't had any complaints about it falling off yet but if the pain on the side of your foot continues to bother you I would consider trying a type of Strassburg sock, a lot of people do have success with them.