At the same time, nobody has accumulated a billion dollars in personal wealth with a totally clean conscience. Millionaires, sure, thats an order of magnitude less wealth though. Billionaires are such a distortion of economic allotment that just to be one necessitates having such a profound influence on so much of their economic sphere that to have gotten into that position required someone else being taken advantage of at some point, and in practice its a lot of someones and a lot of harm caused. Its no different than diverting a river into a reservoir - you can make a small lake with minimal ecological damage, but when you divert into a sea or an ocean the entire environment shifts around it for some good and some ill with there always being some consequences for it happening.

For a lot of people (myself included) it is not mainly about the ethics of individual billionaires, its about the ethics of the existence of billionaires to begin with.