Let me clarify. When I say I do not dry them that does not mean I do not wipe them clean. To truely make them dry you would need to remove all traces of oil. This I do not do. I wipe off the liquid oil and any dirt. That is it. In doing so there is a film of oil on the gaskets. There is no need to remove that film (you would need an oil solvent to do this, which would dry out the rubber). I have done it this way on thousands of valve adjusts, and never a leak.
That said, if it makes you feel better, wipe them dry, clean them with solvents, and change them every few years. I have never replaced a set of gaskets except the donut gasket when improperly installed.
PS I also do not follow the exact process for valve adjustment as prescribed by the factory, among other steps that make more sense other ways.