I had one of these recently but didn't realize I could just decline it. I stuck firm with my "Nothing personal, just moving on to something new" rationale for leaving and basically answered all the questions in a "10/10 would work here again" kind of way. I figured any actual constructive criticism would be ignored anyway.
I've used "peer-led exit interviews" to great effect. Essentially get one of their co-workers to take them out for lunch to talk about want went wrong. Folks are much more comfortable talking to their colleagues. The person conducting the interview is generally happy to pass along the criticism (usually because they agree with it) without having to since they don't have to put their name to it.
I've gotten some real painful, but incredibly meaningful, feedback from these interviews.
Steer well well clear of them too! http://www.asktheheadhunter.com/haexit.htm