There are definitely downsides - a gigantic git repo is a big one. Lengthy clones, far slower git operations, etc. Many projects I've seen go this route will exceed 10s or 100s of gigabytes pretty quickly, and git is very unhappy about sizes like that.