That's pretty much it. Any penny I won at gambling did nothing to harm the broader economy. It might harm some other guy who gambled it, but he knew what he was buying into.
My problem with Goldman and their peers isn't that they won money gambling from other people. It's that in so doing they knowingly destabilized the global economy. The actions of investment bankers have harmed every American, most of whom couldn't even spell CDO let alone know why the existence of so many of them led to Joe Average getting laid off.
I'm not some sort of moral crusader by any means. I really did mean I had less against crack dealers (in fact I think drugs should be legalized, but that's another topic entirely) because the person buying crack knows what they're getting. I don't think everyone can or should be saving lives, they just shouldn't be harming people who aren't knowing and willing parties to the transaction. If my playing poker could have caused economic collapse (in fact the opposite happened, poker became a huge industry that provided lots of well-paying jobs) there would have been an issue.
Like many people, Goldman went long on housing. This did harm the economy. But I take issue with your use of the word "knowingly" - why would Goldman knowingly lose billions of dollars?
Also, many of the people being laid off (e.g., construction workers, realtors) are also complicit in harming the economy.
I'm not being intentionally obtuse, but I still don't understand your specific objecton(s). As I read it, I think it comes down to "a difference in degree is a difference in kind."
I would argue that you can't really know that your winnings never caused a lay-off, for example. Perhaps you beat Frank, the owner of an RV dealership in Eau Claire, so badly that he had to let one of his salesmen go to keep his company alive.
You can raise all sorts of objections at this point about the personal responsibility of Frank. He shouldn't be gambling with money he can't afford to lose! He should understand all of the rules before he sits down! He should have a firmer grasp of the probabilities! He shouldn't be playing with his heart instead of his head!
And those are all objections that market participants can (I believe in good conscience) make about their activities. So, I'm not sure exactly where your objection lies.
Is finance too important to be left to the market? (Let's ignore, for now, the moral hazard of current government interventions... unless that's your objection.) If not, should the markets be limited to qualified investors who have demonstrated some level of knowledge? Should market participants be required to diversify their holdings? Perhaps only a certain percentage of one's net worth can be in play?
So are taxes for building bridges to nowhere etc.
A penny in tax is one penny that cannot be allocated for something else (which is far more likely to be productive).
The money supply was inflated because of the easy money policies of Greenspan and co. Attributing that solely to bankers oversimplifies things.
And let's remember that Fannie Mae and Freddie Mac, two of the groups that originally lobbied for 'increasing homeownership' and thus accelerated the real-estate bubble, are still roaming around scot-free.