Your concerns are normal but they are unfounded, and lemme tell you why! =)
I am a woman, 23, and I choose to remain a virgin until married. I made the decision a few years ago, when I was single, and yes, i was completely honest with guys in not only have I not had sex, but I am not going to with you at this moment. Let's just say that made for some interesting/amusing reactions over the years. Even before I had made that decision, I was never okay personally with casual sex.
A bit of a background about me...as I said I'm 23, I'm engaged, and I've been in a relationship with a guy who is a virgin at almost age 24, for over 2 years...and we aren't getting married for another year and a half. Now, to some people that could be crazy and I'm not pushing my beliefs on you or anyone, the choice about who to share your sexuality with is your choice, not anyone elses. It's not a race or a competition, even though I think boys (and some men) see it as one. I refused to play that game, and to counter what you say about being undesirable, I think I became very desirable at times because I was a challenge. Which isn't a good thing, but you know how men are, since you are one. =P Kidding aside, making a choice about yourself on this level and sticking to it really shows a positive portion of your character, and THAT is very desirable and beautiful.
I know very few people who are still virgins, and even less who are as strict about boundaries as my fiancé and I. Those boundaries are determined by you, the other person in the relationship, and what your moral beliefs dictate. It makes no sense to compare yourself to other people or to feel like you are less of a man because of it. And it surely makes no sense to worry about a potential girl coming into your life and abruptly leaving because you aren't experienced enough for her. If that happens, there clearly isn't a deep enough connection (in my opinion) to even contemplate having sex with her.
Bottom line: Your virignity is solely your own, and you can choose who to give it to. That choice shouldn't be influenced by other people outside of your relationship.
PS: I don't find my virgin fiancé lame at all...in fact he's pretty much the coolest guy out there!
Although my opinions and beliefs are very different from yours you are spot in saying its an individual choice.
Can salute and respect anyone who choose to not buckle under the social pressures associated with it.