That speed controller is too small and i think you meant 5ah batteries. The aH stand for ampere hours which is battery capacity not amps. The speed controller is 30A but really thats what the motors draw at max current. Its about 30A with 18v. So youre running that thing at its very max rated amperage for extened periods and causing it to overheat.

This guy works a little better. It gets warm but not too hot. Its 60A so it has a nice safety buffer. I decided to wrap a squishy piece of foam around mne to try to protect it from vibrations.

https://www.amazon.com/dp/B086HFXM85/ref=cm_sw_r_cp_apa_i_wPmjFb2R8RRFJ