At least in my (and my colleagues') areas of CS, you usually ask the original authors of a paper you want to compare to. A large percentage of them will be glad to provide you with their code, for it goes a long way towards ensuring that their algorithms aren't grossly misrepresented. If they don't (because the code is a mess, they can't find it or get it to work any more, or for whatever reason), you'll have to re-implement it. We take great care to match their performance and have, on several occasions, beat it (compared to the published results, running on similar hardware). It's a lot more work, but in our group we really try to make fair comparisons. Maybe not everyone is so inclined, but I'm sick of reading these stereotypes over and over again.

>> Maybe not everyone is so inclined, but I'm sick of reading these stereotypes over and over again.

I'm sorry if my comment upset you. I'm really not trying to advance any stereotypes and rather just report what my experience is so far, as a Masters student (rather than a researcher). I've also discussed this with one of my professors, who does research mainly in NLP and he told me the same thing.

Personally I find it hard to understand why it's not compulsory for academics to point to a public repository where anyone can find the code that goes along with their papers. I've heard arguments for and against sharing code and particularly data, so I'm not saying that I am necessarily right, but it's really just weird to read "our system beats the best results reported so far" with nothing but a few figures in a table to back that up. And with so much infrastructure and tools around to support sharing code (just think of github) I really don't see what's stopping people.

I didn't mean to attack you personally - that sentiment gets echoed a lot on HN and I replied to your comment, meaning to address all of those that I read before but didn't reply to, so please don't take it personally.

My area is in algorithmics, which is a lot less of a "hot topic" and there's less competition and more cooperation all-around. That may be an important factor in the willingness to share code.

I do agree with you on publishing code and have open-sourced the code to some of my research. Unfortunately I never heard back from anyone, maybe my topics just aren't hot enough ;) I'm a bit unsure whether to publish code for ongoing projects (where some aspects have been published, but others are still in the works for future publications) - in that case there is some incentive to keep the source closed until the project is finished.