Not necessarily. Not if you're a guy anyway. I can honestly say there's not much I haven't been able to have, or places I couldn't go because of children.
The only downside I've found is that they make it difficult to do things with your Mrs, because most of the time one of you has to look after the kids, and it's her.
As long as she's happy to do it you can still do pretty much what you like.
See, I was always a "No kids for me thanks" guy but then my Mrs got herself pregnant by accident (apparently). I wasn't keen but couldn't face the alternatives so we carried on. As it happens she had a miscarriage at 5½months (Not the most fun day I've ever had) but by then we were both quite looking forward to having a kid so we had a planned one this time and everything was fine.
TBH, I'm still not a 'baby person'. I still don't like other people's kid's but your own are different. A bit like farts; your own seem fine but other people's are horrible.
Anyway, my point is that you can have it all. They don't have to ruin your life.