Did you add "replacement" to your search? Because it turned up a bunch of results for me;

https://www.amazon.com/BRHE-Replacement-Nintendo-Original-Joystick/dp/B08D6L94R6

https://www.amazon.com/Skull-Co-Replacement-Joystick-Nintendo/dp/B07W6VSJ8P

The little teeth on the underside of the thumb caps aren't meant to be removed in the first place, so decorative caps are usually the more popular choice. As cleanly fitting a replacement cap on can be quite difficult.

https://www.amazon.com/dp/B07W6VSJ8P?ref_=cm_sw_r_apin_dp_KRGVVP8Y5NA4QG9RYADE

You can get replacement caps that don’t have the pegs in them (which is what’s causing it to be bumpy because they aren’t aligning with the holes completely)

I’ve heard that it’s very difficult to get them back on so this might be your best choice