Working at Amazon does mean that you need to work hard, long and smart. There is a lot of technical debt to deal with too. But the technical debt did not come from bad design, but from the way business grows and changes. There is a 20% growth YoY. Inventories run into millions of products across multiple warehouses. We ship through multiple means and we ensure the product shipped reaches the customer on the promised date at over 95% of the time.
In real life, this means that you are going to have trade-offs and some of the trade-offs do turn into technical debt. What we do with the debt is more important and amongst my groups, we are decreasing the debt by leaps and bounds.
If you want to see how it is, give me a shout. We are hiring, as usual :) I can take resumes in any format too :)
Judging from things Steve Yegge has written about Amazon, and from friends of mine who have worked there, I don't get the impression that Amazon is a company dedicated to technical excellence.
In particular I seem to recall something about 40 million lines of C++ code in their core product, and about mediocre engineering talent and high levels of burnout due to the technical debt they're required to deal with...
In fact, ever-increasing engineering costs would be just as indicative of mounting technical debt as it would of an increasing commitment to R&D.
I worked at Amazon from 2004-2007. On one hand, Amazon does some really excellent and novel work like Dynamo and AWS (S3, EC2, EBS, etc). They organize developer conferences and bring in really good people to lecture. They hire really solid distributed systems people like Pat Helland and give them reign to work with engineering teams to improve their designs.
On the other hand, when you're in the trenches of a product team at Amazon, good engineering takes a back seat to features, deadlines, and keeping everything running. There was an incredible amount of pressure from the top to maintain high levels of service, but not a lot of support for taking the time to refactor or rework those systems to address the underlying causes of service disruption. Everything was always an emergency, but we always had to make the shallow fixes.
Overall, I'm much happier at Google where there is a lot of support for good engineering at all levels of the organization, not just as a means to an end, but as an end in itself.
I bought an xbox and a game for it at the same time from amazon. The xbox was fine the game disk was defective. I contacted amazon via my account, got a call from a human in less than a half hour (mainly I'd say to confirm the xbox itself was OK), received the new game a couple of days later and a refund of my return costs for the game itself.
The only (very) slight quibble I'd have was that the postage on the return exceeded what they refunded. They made the return process itself simple by emailing me barcoded return labels and frequent reminders. I feel very confident buying off amazon a lot more so than ebay for example
Good engineering can't take a backseat to features, deadlines, and keeping things running. Good engineering IS features, deadlines, and keeping things running.
If you aren't making features and meeting deadlines you aren't engineering anything. You're just messing around. Features and deadlines are the difference between science and engineering. There's never enough time, that's what makes it hard.
Of course, Amazon might be pushing time tables too far. I have no idea, and it seems like you do.
I really didn't get the impression that the parent was disappointed he wasn't allowed to 'mess around'. If management doesn't give a flip about technical debt that's a problem.
There is a balance between shipping a good product with good technology. If you are constantly pushed by deadlines and make things just work, it breaks stuff in the future. Any engineer that is continuously working under pressure of tight deadlines cant ship anything creative, solid, simple and open to change.
Good engineers also have to be good analysts. What often makes a programmer better is his/her opinion to say yes and no to products people/designers. But yet again there is a balance. haberman mentioned Google but I personally don't like the over-engineered Google culture. It's too left-wing, if tight deadlines are too right.
I would say the opposite is true. Amazon is a collection of largely independent product teams that make their own decisions and share very little code. Infrastructure services are used based on its merit. Stuff that works stays, things that don't die out. Unlike most tech. companies that depend on large, common infrastructure, Amazon is carefully designed to evolve.
Main problem is that a new family of species called AWS is so well-adapted and successful that its population is rapidly outgrowing some of the older, less dynamic species on which it depends. You can call that technical debt. I'd call it progress.
In my experience, their tech job listings list tech degrees as a must-have. That seems to be a common attribute of companies that care more about HR box ticking than technical excellence, alas.
I thought about going to Amazon when I decided to get out of the startup world about 6 months ago. I went through the standard application process, and promptly closed the tab when I found that they would only accept my r�sum� in MS Word format -- rejected PDFs, HTML, and even plaintext. It may seem like a silly reason not to work somewhere, but I think that the hiring process says more about a company than just about anything else.
Of course, that goes both way. Any engineer so fixated on not using Microsoft Word, and unwilling to submit a resume in the format requested by their potential employer, might not work out well in a number of environments where they might gasp be required to use operating systems, applications, and methodologies that they were unfamiliar with, or downright opposed to.
Works out well for everyone - so, to some degree, a valid filter.
I don't have anything that can (reliably) create .doc files that consistently view the same in MS Word as they do in the application I've created them (I create and update my resume in Pages on the Mac). Creating a PDF means I know what it'll look like when they see it.
I once sent a .doc file to a potential employer, and when I arrived for my interview I saw that the font it had fallen back to (since it didn't have the exact one I'd used) was an aliased version of Courier for some ridiculous reason. It was ugly and difficult to read, the spacing was entirely incorrect, and it was generally a mess. If I were an employer and received a resume that looked like that, I'd surely count it as points against them (though it's possible that accepting Word document resumes means that this happens frequently and you get used to it).
I don't mind using MS Word at work, but I don't use it at home and have no intention of paying for it, so being able to provide someone a reliable, working document is not a guarantee unless I'm using PDF.
Google Docs? LibreOffice? Just make it simple, if they are about box ticking then it does not matter at all how well your resume looks (as e.g. a PDF version would).
I mean I feel your pain, Word Docs give me the cold shivers, but there are some tools.
Word doesn't produce a document that looks exactly like the document you created unless only open and view that document on the same version of word you created it on. Everyone, even HR/business types, inherently know this from their years of dealing with Word.
In short, if your format messes up a little bit no one cares.
And yet, having your format messed up is completely unnecessary. We have this format called PDF which exists solely to make the same document look the same for everybody. It works on every major platform and is supposed by lots of software.
Unless the HR department intends to edit your resume, why on earth wouldn't they accept PDF?
it's a filter. just not a filter for qualities I want to see in a potential employer, or, in potential co-workers. i want to see intelligence, flexibility, efficiency, substance, creativity, modernity, human-friendly systems and an emphasis on ROWE, not warm bodies in seats like clockwork, or conformity. Word? I mean, in a world where PDF and plain text exists? And non-MS/agnostic systems, and HTML? Seriously?
A computer scientist against clockwork and conformity?
All technology is about making stuff work like clockwork. That's like.. what it's all about. This applies to human processes just as much as machine ones.
Conformity is one way to create good clockwork machines.
Your reasons strike me as childish. Word is still the best word processor by far. Personally I respect people who use Word because it means they're willing to use the best software despite it being associated with Microsoft. Word for Mac is about a billion times more sophisticated than Pages.
Sure, but most of them aren't 100%, and tend to lose formatting nuances. My resume looks awful when exported to MS Word, so I always have to load it up in a pirated copy or on someone else's computer, tweak it there, and then send it on. I hate the idea of using commercial software that I haven't paid for, so it just irks me that this is required.
Yeah, but the MS Word format is vague enough, that the only way to ensure that someone else sees the same document you're seeing is to ensure that they're opening the document on the same version of Word, with the same OS and the same fonts that you have.
I've never applied for a job at Amazon but I have done the same thing with other companies. When I was last looking for a job, I found a couple of interesting positions whose HR systems would only take resumes in MS Word. I closed the tab and kept searching.
For me, it's a practical and ideological thing. My resume is written in LaTeX (with a plain text version) and unless I'm missing something, there is no good or easy way to convert from LaTeX to MS Word. I also have doubts about a company that insists on word documents - it smells like inflexibility and a heavy-handed top down management approach.
Word should be perfectly happy reading that. And a lot of places that say they only accept Word won't notice the difference. (I don't know about Amazon.)
Actually, a lot of the places that require Word format use a tool which extracts relevant information from the resume to automatically populate their database. For some reason those custom tools don't work with text documents (go figure).
I think the source of this is that they use third party tools that automatically store/index/rip apart your resume for keywords. Theres obviously no reason why the third party tools don't support plaintext files, but it seems that they don't.
Considering Amazon didn't develop the system, I don't think the choice of software by the HR department is really something that is safe to make generalizations from.
<quote>
All the effort we put into technology might not matter that much if we kept technology off to the side in some sort of R&D department, but we don’t take that approach. Technology infuses all of our teams, all of our processes, our decision-making, and our approach to innovation in each of our businesses. It is deeply integrated into everything we do.
</quote>
According to Bezos you're wrong. If technology infuses all of their processes that also includes hiring.
I honestly think its a bit of a stretch to think that Bezos was talking about the HR department when he said that. Do you think he was talking about the people that vacuum the floors and sell sandwiches in the cafeteria too?
You're talking about HR here; they have no idea how to use anything other than Word. The idea of opening a PDF is foreign to their way of doing business, not just because a lack of know how, but because these are people who are very rigid in their processes. If I were to disregard companies that require Word I'd probably be unable to get a job where I live. I just disregard it, and what their qualifications supposedly are.
Word-only resumes indicate that the HR department is in control of the hiring process. This means that the company is passing up on excellent people for stupid reasons ("We're looking for people with "Unix" experience but this candidate only lists "Linux" and "BSD") and hiring incompetent people who happened to have won the Bingo game.
So the HR department is in the driver's seat on hiring is a good proxy for determining how good their engineering is.
Personally I think a better indicator than if a company is they Word-only or not is the quality of their job postings. HR is not competent to write job descriptions for engineers, and the idiotic descriptions that get passed around the internet are almost always an indicator that hiring is "HR's responsibility."
Those job descriptions are stuff they find on the internet. It's divorced from the reality of the position; you only find that out when you do an actual interview. I'm not saying there aren't places where HR is in control of the hiring processes, but I also know from person experience that it can be the case where HR is in charge of posting positions and taking applications to simply hand off to Programming.
This is... odd. I started at Amazon in January, and as of November, plain text was the preferred format for resumes. Then again, mine got into the system through the employee referral program, so perhaps it takes different inputs.
I pretty much do the same - mostly since I decided to typeset my resume a while ago, and converting it to anything, particularly MS Word, just feels like a travesty.
I am truly surpised by the rigidness displayed regarding Word docs. Limiting opportunities in your life because an HR department uses the most common document app in the world seems self-defeating. Why not rule out cmapanies that use a certain type of printer paper?
I have no problems with whatever document app the HR team is using. If they send me a doc file, I will read the information contained therein. In creating a document the author puts in more work than the reader; so as long as the format is not too onerous the author should be able to choose whatever tool is convenient to him/her.
I hope that you agree that txt and pdf files are just as convenient formats to read as doc ones. IMHO it is irritating when someone insists that any document you send them should be in the format of the application they write documents with.
Furthermore, these are no absolutes. If the State asks me to send a document in doc format, I will. Despite pretenses, these companies (and HR depts) are not Republic of Greater Timbuktu really - so I will skip, thank you. :)
Bayesian filtering that works both ways. In fact we currently take PDF/Word/txt/RTF at my job. I'd actual considering narrowing it down to just Word. The type of person who decides that they don't want to submit in Word is probably a type of person that's not a good cultural fit. And I guess we've already ruled out those that prefer to send their resume via smoke signals only.
As an employer it is more important to find good cultural fits than it is as an employee, since it is typically easier for employees to leave than it is to fire them.
My CV is a PDF because I made it in LaTeX. I did that because .doc is not a standardised format, does not make the promise to look right on any system, and doesn't support some of the nicer typesetting I have going on. In my opinion, forcing people to use a proprietary locked-down format is the kind of thing that should be fought. What really gets my goat is that most of these places still don't accept .docx, even though it is standardised, well-supported, and absolutely TRIVIAL to pull plaintext out of for indexing purposes.
However, because I took the initiative to teach myself an industry standard text markup language in order to make a CV that looks really nice and will always display and print properly (and is distributed in the most widely-agreed upon document format for that purpose), you want to exclude me? I'm not submitting in PDF to be difficult or to take a stance, it's a convenient, standardised and well supported format which I am using for its intended purpose.
Comparing smoke signals to PDF is completely disingenuous, as is bundling together "people who happen to submit their CV to you as a PDF" and "people who would flat-out refuse to submit their CV as a .doc".
I don't want to send my resume as a Word document because I wrote it in LaTeX, and it looks correct as a PDF. I don't understand why you wouldn't do me the courtesy of accepting the PDF.
This seems like a good cultural filter to me (on both sides).
I wouldn't rule out an employer for this, but it's annoying.
I use Ubuntu at home. As someone who will need to work with Ubuntu Server at work, I think that's a good thing for my employability. Using Microsoft Word in Ubuntu is not an option for me.
So it's a Giant Pain in the Butt to produce a Word-formatted resume (no, OpenOfice and Google Docs don't make it reliably look good), and there's no reason they need that because they're not going to edit it.
On the other hand, it's easy for me to produce a PDF, which will look better anyway, and it can be done using the OS I will be using at the job I'm applying for.
Being told, effectively, "we require you to install Windows so that you can submit a nice-looking resume for this job working with Unix" is annoying. It's a small factor in how interested I am in the job, but a factor nonetheless, because it shows (to me) that technical people are not running the company.
I was hired with no college degree, when Microsoft and Google would not even consider me on that basis.
Also, for Amazon and other companies, there is a world of difference going through their public-facing recruiting process and getting to know someone on an interesting team through your network, and getting hired through them.
I'm sure the same rules of networking apply anywhere, though, including Microsoft and Google. Degrees are an HR filter, but networking can bypass HR. So it goes.
More than that, it is very easy for only one programmer or researcher doing a bad job to create technical debt that will take disproportional time to pay off.
I've seen some negative stories about working at Amazon. I don't recall seeing anybody write that it's a great place to program. Anybody have any stories or links about the upside?
The upside is that the company has excellent developer tools and infrastructure (I mean aside from AWS), very accessible principal engineers, a huge amount of paying customers so your work has real impact, and it seems like most teams have autonomy to work and deliver in their own way. Within a week or two of joining I was given ownership of an important internal product while also getting up to speed on the main event (the retail website). The technical challenges are there for the taking.
The opportunity to have an impact cannot be overstated. I worked at A9, the search subsidiary of Amazon. Given that a substantial portion of purchases go through search, we were always keenly aware that even improvements that appeared to be modest could have very substantial impacts to the bottom line.
I know a lot of people who work/have worked there, and it seems to be a very love it or hate it company. They work people hard, but the people who like that kind of environment thrive. They seem to be pretty ruthless about improving in certain areas and are very metric-focused. I'm not sure it's a great place to program, but it's certainly a great place to learn about programming.
I get the feeling that you're both right and wrong. Right in that a lot of the legacy Amazon code is probably burdened with huge technical debt. I also get the feeling they're pushing the boundaries in other areas. A bit of the "chase the shiny coin" syndrome.
One of the issues is that none of us really know what goes on inside Amazon. They are notoriously secretive about most everything.
I don't see technical excellence having a lot to do with technical debt in terms of if you are excellent, you have little or no debt.
Business needs change too quickly for a system as large as Amazon; they simply cannot maintain net zero technical debt and not cripple their reaction time in the market.
The key is to know what level of debt is necessary and acceptable to take on and deal with going forward.
Maybe not working at Amazon, but their suite of companies are still technically innovating : their http://www.a2z.com subsidiary is doing some amazing stuff, with top-tier talent.
My feeling about it was that it was written to appeal to top programmers, not investors except so far as "You can read this letter, in which I prove that I know the language one needs to speak to hire smart people".
Investors want higher margins but obviously Amazon is investing substantially in itself. So far, though, Bezos seems to be winning the tug of war. Many analysts are glad to see higher revenue and feel that Amazon is cementing its position at the top of the ecommerce world. AMZN shares are up over 5% despite the earnings miss.
Its funny how people seem to be whining about Amazon.
In fact Amazon is one of the best places in terms of innovation. In addition to having a strong core business, they also have AWS, kindle, android app store.
Who cares if their legacy codebase is 40 milllions lines of C++ and not your hip Ruby or LIPS nonsense. The truth is that they get work done. AWS is one of the best cloud computing system out there. Nothing else comes even closer. Dropbox, Heroku and Reddit all rely on them.
Who cares if they take resume in word format, they pay well.
Its sad to see HN turning into a language fanboi and apple fanboi club.
The AWS EC2 outage did not even affect 0.01% of their revenue.
The end result of all this behind-the-scenes software? Fast, accurate search results that help you find what you want.
Has Bezos ever actually used Amazon's search functionality?
I know it isn't the easiest problem in the world to solve, but their search is by far the worst of any of the major shopping sites that I use on a regular basis.
Specific examples? It seems pretty good to me. Which shopping site has a better search, and what makes it better?
Exact keyword match is obviously fine. "harry potter and the sorcerer's stone" gives me various options for buying that book and a bunch of other harry potter related books, and i can sort by some useful options like reviews or bestselling.
Harder problems, Google shopping vs amazon:
Misspellings: Hairy Potter - Amazon suggests 'did you mean Harry Potter?', but does not initially put that book in front of me. When I check again later they are now providing a second list, "Results for "harry potter" (corrected from "hairy potter")"
Google shopping shows me the Did you Mean question, search results are the lego sets and one movie.
Concepts/descriptions instead of exact thing: "Boy Wizard Series" - both fail to suggest harry potter to me, but they do start with "Harry, a History" which might be enough to jog my memory. Still, no real points here, but this is a hard problem, and I don't know any shopping site that does it well. Even Google's regular web search is not fantastic at this, hitting mostly news articles and not reading my mind and popping out with "I bet you want to see information about Harry Potter..."
In particular, searches for computer equipment is generally frustrating. I've also seen sorts result in a large number of irrelevant outdated used items getting filtered to the top.[1]
I'm willing to cut them slack since it's good enough, but look at Newegg to see how it should be done. It's not a completely fair comparison (since Newegg specializes in this), but they do it so much better.[2]
Thanks, great example. It looks like they don't account for the importance of certain terms in tech searches well enough. Like they're far too willing to throw away the "16gb" part of "16gb sdhc" when sorting by price, and sorting by relevance shows you expensive stuff, so there is no good way to say "show me cheap stuff but I'm not kidding about the 16gb part".
Also note that sorting by price doesn't actually sort correctly. In this search, a $6.99 card comes before a $5.99 card (both with free super-saver shipping). This is so broken (and has been for so long) that I almost think it must be intentional somehow.
I agree that it's mostly pretty good, but here's one flaw that always annoys me: when I tell it I want something under $50, and I only want stuff sold by Amazon, they'll include an item even if Amazon's price is $60, as long as at least one seller has it for under $50.
You can say that again. By far the worst; quite frequently I have to make several tries and occasionally resort to Google. I used to play with this technology in the early-mid '90s when it was a document imagine adjunct and I'm truly impressed by how bad it is.
I would be very surprised if they were not losing a significant amount of business from this deficit, especially outside of the well cataloged area of books.
There is probably over a million products on Amazon. Is there any store in the world that offers a bigger selection? The only problem I have with the search is that they need something more NewEgg-like for searching for computer stuff.
This is only slightly on-topic, but regarding the service-oriented architecture that he mentions at the beginning of the article, how would one implement this internal? Specifically, how would you communicate with your internal services? Just an HTTP API? Is that fast enough? Another thing I'm wondering is what's the best way to handle internal authentication between your frontend and some backend service for example? Sorry if this is too specific for this discussion, but I've just always wondered this and I figure that HN knows.
DotCloud [1] is built from the ground up around distributed services. We use a custom rpc mechanism called ZeroRPC. It uses ZeroMQ [2] for transport, msgpack [3] for serialization, and some extra glue for RPC mechanics, discovery, event dispatching, etc.
Among other neat tricks, it allows us to handle synchronous RPC calls and asynchronous message passing in a unified way. It also maps transparently to any ZeroMQ topology.
Combine this with the recent release of gevent support for zeromq [4] and you get a fun, fun playground :)
Oh, awesome, thank you!. I never thought to use ZeroMQ, but that would work perfectly. Ironically, it doesn't seem like DotCloud supports these sorts of architectures very well yet, any plans to fix that? ;)
Absolutely - our job is to make developers happy and productive. Whenever we find a sweet tool that makes our lives easier, we're practically tripping over ourselves to share it with you
Our network team is working on some major goodies which should make you happy. Personally they bring tears to my eyes :)
In the meantime, my recommendation is to start with http. If you keep the RPC semantics simple, you can easily migrate to a different transport later. And you'll need HTTP to authenticate and encrypt at the boundaries anyway.
Interesting how AMZN jumps 5% after missing earnings, yet GOOG drops nearly 10% with nearly the exact same results (increased hiring and expenses resulting in lower than expected profit).
Stocks don't react to how results in announcements compare to last year, stocks react to how results in announcements compare with how Wall Street priced the stock yesterday based on their guess to what the results would be. Efficient markets 101 - otherwise you could make money by buying, one day in advance, stocks that obviously probably did well compared to last year. If this is not obvious to you then you should never invest in anything besides index funds (seriously, I've seen people do really badly by not understanding elementary implications of the inexploitable markets hypothesis).
I thought it generally meant that it was less than the company's own predictions, not the analysts? In any event it shouldn't matter which estimate they're falling short of as long as it isn't the market's implicit estimate.
I think that is because investors believe in Jeff Bezos more as CEO than Larry Page. Bezos comes from Wall Street, so Wall Streeters probably have more confidence in him. Plus he has been CEO for a long time, whereas Page comes from academia and just started the role (again).
I think it's really silly to try to draw out any meaning, or even assume any meaning exists in the first place, in single-day movements or reactions like these. What happens in a few hours of trading after earnings announcements means nothing after a few days.
As a general rule, short term movements on Wall Street are "dumb" because there's a lot of emotion driving trading on that scale, plus lots of weird automated/hedge fund trading going on. To give just one example of something that probably contributes to noise: naive trading programs that see an uptick on "Hathaway" in the news streams, assume that's a signal indicating some new item about Buffet's company, so they buy up BH -- but in reality the spike was caused by a wave of news about the actress Anne Hathaway. I'm sure there's a lot of dumb/naive algorithms like that out there. Plus who knows how many pre-scheduled trades or triggered trades. In the short term, lots of dumb noise and froth. I think overall movements on Wall Street are the most meaningful and valid in the longer term, and across a large class of things.
I was mulling over that exact trade, but figured the relatively small amount of AMZN I had didn't justify getting off my tush. Sold a couple hours ago with a solid 60% up in 10 months (usd -> usd) and 41% in CAD terms. I'm fairly bullish on AAPL, but I already own 200 shares (selling medium term call options, hence the 100s multiple) so buying some more probably isn't worth it. Just going to keep it in cash until I see something.
Do you have any experience in hedging your US investments by shorting the USD? I'm wondering if the additional fees are worth it.
Exactly. This is all about their earnings. Given a general trend towards decreasing or outsourcing R&D, Amazon is still doing it in-house, because it's their competitive advantage. Explaining that to a bunch of panicky stockholders who just saw a Big Ugly show up on the cashflow statement is mandatory, no matter how much they care about the long-term.
I feel like I'm missing something. As of this moment (9:42AM PDT) Amazon's stock is up 5.6% for the day and near a 52 week high. That doesn't seem like panicky shareholders to me.
I'm not a stock maven, so it's possible that I'm missing some important detail.
for liberal definitions of "competing" The only similar cloud services are from Microsoft and Google and they are about the same price, definitely not "a song" territory.
I meant that given your app works within the restrictions of GAE you get the benefit of a lot of brainpower from men with long beards.
Maybe the association was temporal as I'd just finished watching an IO presentation about how to scheme data for appengine where the detail was primarily the same as Bezos' first few paragraphs.
I wish the company would reveal some information on the internal systems he outlined in his letter. Amazon's architecture is impressive and its infrastructure teams build great stuff to help developers get their work done & deployed.
Exposing these things would be a powerful recruiting tool.
This is instructive. Google's stock might not have tanked as hard the other week if LP had come out and placated the street with some jargon like this, re: Google's also ballooning expenses.
Bezos explicitly said it in this letter - he expects all of this stuff to make the company gobs of money in the future. For a lot of the stuff Google spends money on, I don't think investors are necessarily very confident that it will have the same impact on future profits.
Working at Amazon does mean that you need to work hard, long and smart. There is a lot of technical debt to deal with too. But the technical debt did not come from bad design, but from the way business grows and changes. There is a 20% growth YoY. Inventories run into millions of products across multiple warehouses. We ship through multiple means and we ensure the product shipped reaches the customer on the promised date at over 95% of the time.
In real life, this means that you are going to have trade-offs and some of the trade-offs do turn into technical debt. What we do with the debt is more important and amongst my groups, we are decreasing the debt by leaps and bounds.
If you want to see how it is, give me a shout. We are hiring, as usual :) I can take resumes in any format too :)