Not necessarily. If the computer responded to a static zap in that manner then something could be wrong with the grounding. The entire case of a PC is grounded any zap to it should go straight to the electrical system ground through the case without having any significant route to go through the actual boards because the case resistance is so much lower. If there's a broken ground and the only return path is through the neutral they may have a floating AC voltage on the case which could be causing this.
The very first thing I would check is using a basic outlet checker like this. I happen to have one of course, but these are extremely handy for a basic sanity check and are cheap.
https://www.amazon.com/Receptacle-Tester-Klein-Tools-RT110/dp/B01AKX3AYE
I'll point out that the turning off the power step should happen before the taking the plate off. I strongly suggest plugging in a light, radio, or some noise generating device that you can tell switches off when you're trying to kill the power to the outlet.
While you're at the hardware store you should pick up an outlet tester to confirm that you wired it correctly. For under $10, one might not be had to have on hand if you have older or questionable outlets.
Outlet tester. Make sure your outlets and GFCIs work.