I've upgraded a stock Vevor 3040 CNC to an air cooled 48vdc 500W brushless ER16 spindle a year or so ago, and it worked out great. There are a few considerations that may limit what modifications you are willing/able to make. Since the spindle was a much larger diameter, you must change the z-axis mount to accomodate the new spindle. On my machine, I had to literally saw off the existing mount in order to mount the new spindle. On top of that, I drilled new mounting holes and tapped them...all the while keeping everthing plumb and accurate. The motor alignment is critical...and generally not a project for inexperienced hobbyist.
However... if you can find an ER11 brushless 500-600W kit with the spindle the same diameter as your existing mount (~55mm) then that greatly simplifies the upgrade.
Something like this kit should work out fine.