You wash it off in what is called "short contact therapy", this is where you introduce the active (in this case Benzoyl peroxide) to your skin for a short period of time. That's essentially what you're doing, and for some people with very sensitive skin this is the only way they can tolerate using benzoyl peroxide to treat their acne.
You can get the best results from leaving it on, but not if leaving it on causes you irritation which may lead to irritation breakouts.
My skin is sensitive personally so I used SCT to get it to acclimate to the benzoyl peroxide. I started by leaving it on for 1 minute increments, to where now I can leave it on overnight.
I agree that it is fine. esp if you're just introducing it. take it easy. use as little as possible at first. use every other day if necessary. give yourself time and build to everyday.
if BP is stinging and not getting better despite minimum use, then consider not using it. i don't see the point in continuing it esp if it may worsen skin. Decide once you believe you've given it a fair trial. Like the prev poster said, start small, build up. If you're making sure not to over do it, and your skin reaction is really bad, stop.