I had the same situation as the OP and I think many of us do. My problem is, I genuinely am allergic to a few different foods, specifically soy, gluten and dairy. I was hopeful that maybe I had imagined the dairy allergy (I noticed it causing me problems after I originally went gluten free) but unfortunately, after testing it with a truly open heart, I discovered that I do indeed react terribly to it and the reactions are bad enough to make me not want to eat it again. If it was just a little pimple that would be one thing. I'm talking terrible, deep cystic/nodular acne that is very painful and takes months to go away. If one bowl of yogurt can do that, it's not worth it to me at all.
That said, what really led to my eating disorder wasn't avoiding the foods that I react badly to, but avoiding other foods because I thought they would increase the likelihood of me having reactions. Basically, it was the leaky gut theory that did me in. Because I believed these allergies were a result of leaky gut, my focus became on making leaky gut better so my allergies would then (supposedly) stop. The problem is, no one really knows about leaky gut or how to make it better, and depending on where you will look, there are huge lists of different foods you are supposed to avoid, from beans to okra to sweet potatoes. So I just kept cutting these nutritious foods out of my diet, not seeing results, getting more and more distressed thinking everything I was eating was worsening my gut...and cutting out more foods. It's a scary, fast tumble down a very slippery hill.
Now, my rule is that I will eat anything in moderation IF I don't notice any immediate, violent acne reactions. Things like beans, which are supposedly not good for leaky gut, are back in my diet a few days per week because 1) I love Mexican food and felt so sad without it and 2) I don't notice any terrible acne right after eating them. There is no point in being so restrictive if it doesn't help anyway.