In my experience, wheels make the biggest difference.

What you want are "High Rebound" wheels. The urethane they use makes them bouncy. Don't ask me the physics of it, but this makes them faster and grippier.

To give an example. I tried out some old cheap wheels, which made a dull thud noise when I drop tested them, barely bouncing at all. l've been putting together a "rain wheel" setup. I tried skating on them and they were sooo slow compared to my other wheels. I also got these super cheap bearings that I don't care if they rust. I'm testing a hydrophobic bike grease in them. But that's a different story. Anyhow, since they were so slow, I put the FR Street Kings back on them and I was back to going fast again, with the same cheap bearings.

So, the magnets will slow you down some, but if you want to test the wheels themselves, is it possible to take out the magnetic spacers and put regular spacers in them? This would allow you to see how they do on their own and then compare to the magnetic resistance that is required to light them up.