The article explains a few network protocol reverse-engineering techniques that were used over the course of 12 years to write Samba.
The techniques are:
1. Use of publicly available documents on the CIFS/SMB protocol
(not reverse-engineering but the documents are incomplete);
2. Use a network sniffer to listen in on conversations between Microsoft clients and servers;
3. Use of a protocol scanner: a program that tries all possible "words"
in some section of a protocol and uses the response
to automatically deduce new information about the protocol.
4. Use the "differential" technique: You write your own implementation
of the protocol with what you know, then send pairs of identical messages
to both the real implementation and your implementation and verify that the responses are the same.
Trivia: Andrew Tridgell's techniques here catalyzed the creation of git. He applied them to reverse-engineer parts of the Bitkeeper protocol, a proprietary version control system then hosting the linux kernel. Bitkeeper CEO Larry McVoy revoked linux's license to use BK, Linus went off and wrote a replacement, and the rest is history.
Although supposedly the BitKeeper protocol was documented, and to find it, Andrew Tridgell connected to a BK server via telnet and typed "help". No idea how much the truth has been romanticised, but I suspect that it at least helped the reverse engineering effort, even if the documentation was incomplete.
I'm currently reverse engineering a USB VoIP handset in my spare time, as the only driver that exists for it is for 32-bit Windows. I'd recommend the experience to any hacker. I'll try and write up my findings when I'm done - there are a bunch of HOWTOs out there but I found them woefully incomplete.
The techniques are: