Not the residue, rather any lingering dust and dirt that may compromise the adhesive

You don’t have to remove the whole pillar, you can get by with just running it down the length of the rubber seal. I’d recommend investing in some trim removal tools like these

KLTECH 4Pcs Auto Door Clip Panel Trim Removal Tool Kits for Car Dash Radio Audio Installer Pry Tool https://www.amazon.com/dp/B00HNMLQAG/ref=cm_sw_r_cp_api_glt_fabc_6YGHA8KJSWNB9K05MPY6?_encoding=UTF8&psc=1