There is no way it's because you touched the lens. The potential oil from your fingers would just wash off after a few uses anyways.
I've used several masks and I hate to say it, but even the best ones fog. It's simply the nature of water temperature condensing against body temperature / exhalation.
First, make sure your mask has a proper fit and seal, so that water doesn't easily get in, but I highly recommend anti-fog spray.
Anti-fog spray is absolutely essential. Apply before diving out, rinse off the excess spray, and it typically prevents fog from forming for about 30 minutes. After that, I sometimes have to re-apply. It's cheap enough to use pretty liberally.