Definitely wash your face if you have been heavily sweating. You need to get the sweat off so it doesnt contribute to bacteria on your face and clog your pores.

What time of day do you do sports? If it's early morning/night just wash your face like normal. If it's the middle of the day or a weird time I would at least just wash with water/a towel to get all the sweat off without having to actually cleanse 3 times a day.