I wouldn't worry too much. Sleeping on your pillowcase or touching your face won't cause breakouts. It could irritate acne that is already present but the effects are negligible. What you should worry about is the acne bacteria in your pores. That's what's causing you to breakout.