...However, I do seem to be more oily when I wash more. Maybe washing removes skin that would cover up the oil?
I'm not completely sure why your skin would appear to be more oily when you wash more, although I suspect that one possible explanation is that when you wash a lot, it causes a lot of your skin to be exfoliated, leaving behind a smoother, more even layer of skin which gives the superficial appearance of being oilier, even if it's mainly just a tactile effect. You might test it more scientifically with something like Sebutape Skin Indicators, which aren't "fooled" by such issues as how smooth the skin is.
Back a few years ago when I did my own test of the "feedback theory", I spent a few days washing my skin vigorously several times a day, then spent several more days not washing it at all. When all was said and done, the Sebutape Skin Indicators showed no difference at all in how much sebum my skin was putting out at that location, whether I washed it or not. I bet if you tested it that way yourself, you would get the same result.