This seems as good a place as any to post something I've been thinking of. (It doesn't really have anything to do with the story, just is on topic generally.)
I'd like to do a small study personally, that should take me about 10-20 hours, and I'd like people's feedback here about what they think about the methodology.
- First, I'd put together ten generic keyword-laden resumes using machine learning from a large dataset - which means it should somewhat reflect an "average" programmer, without representing anyone in particular. It will literally therefore be the creation of a non-existent person.
- Next I would edit it by hand so that it doesn't seem grossly machine-generated. At this point I'm still blinded, and I'd have ten programmer resumes.
- Next I would generate a separate list of common first names and surnames. Since most people are white, likely most names are somewhat white but this shouldn't be that relevant, also given that people have the right to change their name legally. The importance of this is that with such generic names, it is almost certain that there are a large number of programmers with that name, and so googling by the employer won't point to anyone in particular. At this point I have ten resumes and ten names.
- Finally, I would find stock art of a generic black man and generic white man. I would try to pick neutral images that if I were such a person I could actually have taken of myself, and may actually use on a resume. I would only just find two, since the goal of the experiment would be to do A/B testing on just the effect of this image. This is what is being studied.
- For the experiment itself, I would create email addresses for the ten names, and write cover letters that match the ten resumes.
- From each email I would email twenty-thirty companies based on a keyword search from the associated resumes for positions that seem a match. I'd be careful to pick all different companies.
- I'd carefully rewrite the cover letters (in my own voice) about how excited I'd be to work there and what a great match the position seems to me, as they can see from the attached resume.
- Finally, and this is the tricky part, when actually attaching the resume, in a blinded way I will use a script to insert either the black or the white man's picture. I must not see it in order for this to be a doubly blinded experiment. This is actually easy to do technically: .docx files are just .zip files, you can rename them, change the image in the zip file, and zip it back. I don't need to record anywhere which of the two the script chose (randomly), because as soon as I send it it will be in the sent folder.
The same name and resume must go to a mix of companies, some being sent the picture with a black man, some being sent the picture with the white man. Since I'd be careful not to email the same companies, it doesn't matter if the same name/resume is sent to some companies as a black man and some companies as a white man.
What my prediction is, is that the picture will have a statistically significant effect on emails back. (I don't want to bother setting up ten real phone numbers, what a pain, so the phone number will have to be omitted from the resumes.)
Now, this method isn't perfect. For example, real programmers frequently have github profiles or a large online presence. Still, by sticking to the most common of common names, perhaps there should be enough of a profile to be worth an email back, especially given an enthusiastic cover letter, even if they can't find this programmer in particular.
Finally, after the experiment the recipients could be informed that they were part of a study. But perhaps this is not so important. After all, if they don't email back they likely have forgotten about the resume, and even if they do, if the applicant does not answer their email then the applicant must simply be busy with other offers.
Of course, for authenticity purposes, it would be better not to share this methodology here, where some people might read it and be tipped off.
But given that I am not certain that I am doing things right, and haven't run an experiment in the social sciences before, while at the same time I've heard many people here report on experimental methodology, I thought I would run this methodology past the HN crowd. What do you think? Is it scientifically well-constructed? Is it possible for it to show an A/B effect?
Short of literally reusing people's real resumes with a fake image, I don't know of any way to improve this proposed methodology.
-> Shall I run the experiment?
-> Is there anything I can do to improve it?
-> Would the results be meaningful in either case? (If it does show a statistically significant deviation in email-back percentages, and if it doesn't.)
Thanks. I think I read that study (or a report about it.) But I didn't like it so much. (Maybe that study inspired the idea for my study - I don't remember.)
The reason I don't like that study as much is because it's about people's names. But people's names do in fact reflect the level of education of their families, and in an extreme case they can choose a new name if they want. The poster we're reading about, for example, is called "Richard Smith". (Whether by birth or a name that he adopted doesn't really matter.) I like names. If on a dating site I were given, sight unseen, the chance to view an Octavia or a Hermine I would choose it over any Susan or Latisha. Because it seems to me much more likely that Hermine would be educated and interesting.
Does this make Hermine a racial name?
Why couldn't Hermine (or Octavia) be black? In fact they could.
So I would say that the effect of a person's name, and a picture showing them as black, are, as this blog post already shows, somewhat orthogonal. Richard Smith isn't a "black name", but this poster had a black experience.
It's also quite important to point out that parents have almost complete choice over what to name their children. As some linguist pointed out, it's one of the few times that people get to name something (someone) in the world!
So I don't like it as a signal. A picture is much more pure. And also not something anyone can change.
Finally, in my case I'm specifically interested in programmers and tech. So my small "study" would be very targeted.
I'm still very interested if anyone has any comments about the methodology. I've never done any social science research before (well not formally anyway), and I'd be very curious if I'd be making some mistake in the methodology that I could avoid.
> But people's names do in fact reflect the level of education of their families, and in an extreme case they can choose a new name if they want.
Good point -- I hadn't thought of that. But wouldn't the resume reflect the actual education of the applicant? Maybe the discrimination against names you describe is cultural, which could also be of interest in the tech world ("this guy wouldn't fit in with us").
I did see a study on the success rates of people that had changed their names to be less "ethnic," and it was high, but of course there's a self-selection problem, since someone would have to be a very motivated person to change their name to get ahead.
One disadvantage of using pictures is that it makes race immediately apparent, and people might deliberately change their behavior to not appear racist. Maybe they offer an interview to the black candidate to check off a box on the HR form, even if they'd never hire him/her.
Anyway, just some thoughts -- I'd love to see the results of the study here on HN if you manage to pull it off!
thanks! oooh, excellent point about deliberately changing behvaior to offer interviews (which can be patronizing.) Now I really, really, really, really want to see the results of my study. I am 100% putting it on my plate, this should be doable. I've got this :)
I'd like to do a small study personally, that should take me about 10-20 hours, and I'd like people's feedback here about what they think about the methodology.
- First, I'd put together ten generic keyword-laden resumes using machine learning from a large dataset - which means it should somewhat reflect an "average" programmer, without representing anyone in particular. It will literally therefore be the creation of a non-existent person.
- Next I would edit it by hand so that it doesn't seem grossly machine-generated. At this point I'm still blinded, and I'd have ten programmer resumes.
- Next I would generate a separate list of common first names and surnames. Since most people are white, likely most names are somewhat white but this shouldn't be that relevant, also given that people have the right to change their name legally. The importance of this is that with such generic names, it is almost certain that there are a large number of programmers with that name, and so googling by the employer won't point to anyone in particular. At this point I have ten resumes and ten names.
- Finally, I would find stock art of a generic black man and generic white man. I would try to pick neutral images that if I were such a person I could actually have taken of myself, and may actually use on a resume. I would only just find two, since the goal of the experiment would be to do A/B testing on just the effect of this image. This is what is being studied.
- For the experiment itself, I would create email addresses for the ten names, and write cover letters that match the ten resumes.
- From each email I would email twenty-thirty companies based on a keyword search from the associated resumes for positions that seem a match. I'd be careful to pick all different companies.
- I'd carefully rewrite the cover letters (in my own voice) about how excited I'd be to work there and what a great match the position seems to me, as they can see from the attached resume.
- Finally, and this is the tricky part, when actually attaching the resume, in a blinded way I will use a script to insert either the black or the white man's picture. I must not see it in order for this to be a doubly blinded experiment. This is actually easy to do technically: .docx files are just .zip files, you can rename them, change the image in the zip file, and zip it back. I don't need to record anywhere which of the two the script chose (randomly), because as soon as I send it it will be in the sent folder.
The same name and resume must go to a mix of companies, some being sent the picture with a black man, some being sent the picture with the white man. Since I'd be careful not to email the same companies, it doesn't matter if the same name/resume is sent to some companies as a black man and some companies as a white man.
What my prediction is, is that the picture will have a statistically significant effect on emails back. (I don't want to bother setting up ten real phone numbers, what a pain, so the phone number will have to be omitted from the resumes.)
Now, this method isn't perfect. For example, real programmers frequently have github profiles or a large online presence. Still, by sticking to the most common of common names, perhaps there should be enough of a profile to be worth an email back, especially given an enthusiastic cover letter, even if they can't find this programmer in particular.
Finally, after the experiment the recipients could be informed that they were part of a study. But perhaps this is not so important. After all, if they don't email back they likely have forgotten about the resume, and even if they do, if the applicant does not answer their email then the applicant must simply be busy with other offers.
Of course, for authenticity purposes, it would be better not to share this methodology here, where some people might read it and be tipped off.
But given that I am not certain that I am doing things right, and haven't run an experiment in the social sciences before, while at the same time I've heard many people here report on experimental methodology, I thought I would run this methodology past the HN crowd. What do you think? Is it scientifically well-constructed? Is it possible for it to show an A/B effect?
Short of literally reusing people's real resumes with a fake image, I don't know of any way to improve this proposed methodology.
-> Shall I run the experiment?
-> Is there anything I can do to improve it?
-> Would the results be meaningful in either case? (If it does show a statistically significant deviation in email-back percentages, and if it doesn't.)
Thanks for any thoughts.