If I exercise I try and do it in the beginning or end of my day to coincide with cleanser/regimen. But if that doesn't work into your schedule I'd just rinse with water and if my skin feels tight or dry I'd apply a light moistuizer, with a light hand as well.

is it ok if I just use a damp towel

Are you on The Regimen? If so, Dan really does not recommend adding in extra washing or rinsing during the day. Here is his advice on managing working out while on The Regimen. http://www.acne.org/faq.html#sweat

If you have acne prone skin I think it would be a really bad idea. If you put moisturizer on your skin without cleaning it thoroughly you are trapping oil, sweat, etc. underneath the moisturizer which is just asking for clogged pores and acne.