Don’t wipe them. It just makes things worse, or you risk scratching the glass. If you use a small brush for flicking away the dust that is sufficient. Think about this: if you are taking flats you will remove it all from the photos anyway. Some people don’t bother cleaning a thing.

Edit: In a year of shooting I’ve lightly dusted mine with this brush maybe 3-4 times.