I've met plenty of families travelling around the world. It's another level of complexity, but it certainly can be done.
I bumped into this family driving the length West Africa, one of the toughest routes in the world! They had an amazing time, and last we chatting they're busy saving money to go on another massive adventure!