I assume age: anyone can look after a child old enough to eat solid food, but a baby needs a wetnurse and one may not be available, willing, and/or affordable.

The only other fantasy I can think of that brought kids along for the adventure was Patricia Wrede's "Caught in Crystal" (one of a series but the only one I read and I thought it stood alone). Of course it happens all the time in real life. Getting the tension right probably is a challenge and why it's not done so often but I think it's worth the effort - I'd certainly read the heck out of it.