| blob | a86ca71db07c6d2b2544f2cdf3c6eebc80630c18 |
1 1. What is retchmail?
3 Eric Raymond's Fetchmail is a full-featured, robust, well-documented
4 remote-mail retrieval and forwarding utility intended to be used over
5 on-demand TCP/IP links (such as SLIP or PPP connections). It supports every
6 remote-mail protocol now in use on the Internet: POP2, POP3, RPOP, APOP,
7 KPOP, all flavors of IMAP, ETRN, and ODMR. It can even support IPv6 and
8 IPSEC.
10 Oh, you were asking about RETCHMAIL. Sorry.
12 Retchmail is the world's most stupidly fast POP3 retriever.
14 It is almost completely unlike fetchmail: retchmail lacks features, isn't
15 particularly robust (although it won't delete a message until sendmail says
16 it was delivered okay), has nearly no documentation, and is actually fast.
18 Thanks to Patrick Patterson's WvSSLStream, retchmail also supports
19 POP3-SSL, which is much more secure than the RPOP, APOP, KPOP and other
20 password obfuscation routines currently en vogue to hide your password. SSL
21 has the nice feature that not only is your password secure going across the
22 wire, but so is the rest of your mail.
25 2. Where can I get it?
27 http://open.nit.ca/retchmail/
30 3. What exactly do you mean by "stupidly" fast?
32 Well, retchmail was written because I was annoyed with waiting for my e-mail
33 to download. I have a pretty active email box (several, actually), and I
34 often have to retrieve my mail through a rather high-latency Internet
35 connection from whatever odd places I might be in at the time. Traditional
36 POP retriever programs were slowly driving me crazier.
38 Here are some reasons that retchmail seems to be and actually is faster than
39 any other mail retriever I know:
41 - Retchmail downloads from multiple mailboxes simultaneously; if you have
42 several email accounts, like I do, you don't have to wait for one
43 download to finish before starting the next one. If you want, you can
44 deliver mail from each mailbox into a different local mailbox. If one
45 of your mail servers is slower than the others, it can download in the
46 background while your faster servers go faster.
48 - Retchmail can be waiting for one message delivery to finish while it
49 starts the next one. Since mail delivery programs tend to be
50 ultra-paranoid and fsync() and run the disks a lot, you'd be surprised at
51 how much faster this can make things, without making them less safe.
53 - Retchmail downloads smaller messages before larger ones, without
54 disturbing the message order too much. This means you can start reading
55 your small messages before the giant attachment has finished downloading.
56 However, retchmail only reorders messages when it's actually useful, so
57 your mailbox stays in mainly chronological order.
59 - Special bonus feature: absolutely nowhere in retchmail's code is there an
60 arbitrary 3-second sleep(). Wow! What other mail retriever can say
61 that? (Hint: not fetchmail.)
64 4. That sounds fast, but not "stupidly" fast. Are you forgetting to tell me
65 something?
67 You caught me. The main reason retchmail is faster - often more than 5x
68 faster than fetchmail - is that it pipelines message retrievals. That means
69 it asks for several messages from the server at once, rather than requesting
70 a message, waiting for it to download, deleting it, waiting for the delete
71 request to be acknowledged, and requesting the next one. For a mailbox with
72 lots of short messages (the common case) the request-wait-request-wait style
73 of transaction is _hideously_ slow, because no messages are being sent (the
74 link is idle) in between the previous message being sent and the next
75 request being received.
77 If you do the math, you'll find that for the common case of a relatively
78 slow or overloaded Internet connection with relatively small messages,
79 _most_ of the time it takes to download your mailbox is actually spent
80 just waiting while no data is transferred.
82 Instead, retchmail requests multiple messages at once, so it always has
83 several messages "in flight" at once. For example, it requests messages 1
84 through 5, and the server starts sending those in order. When retchmail has
85 finished receiving message #1, it then requests message #6, and so on. The
86 mail server won't have finished sending message #5 by that time, so the
87 download part of the connection is _never_ idle after the transfer starts.
89 All that makes retchmail go REALLY fast. Almost everyone will find
90 retchmail to be at least 3x faster than most POP retrievers, and probably
91 more than 5x faster.
94 5. Is retchmail standards-compliant?
96 Quick answer: HA HA HA HA HA HA no.
98 Longer answer:
100 I've read the POP-3 RFC. As far as I can tell, nothing in there
101 says we can't do what we're doing. That said, I know perfectly well that
102 what we're doing is definitely NOT what the authors intended. So we're in
103 compliance with the letter, but not the spirit, of the standard.
105 Just to be extra clear about this: yes, it is morally wrong for us to have
106 written retchmail, and it is morally wrong for you to use it. But try it,
107 it's really fast!
110 6. Will the lack of standards-compliance cause problems?
112 Maybe, if your POP server is badly written. It wouldn't surprise me
113 terribly much if this were the case.
115 The most likely type of bug will be "failure to split packets" or "failure
116 to merge packets." Most network programming libraries (not WvStreams) make
117 it kind of hard to do simple, obvious things like reading a single line of
118 text, and most people writing network code tend to screw it up. Your POP
119 server is likely to screw up in one or both of two ways:
121 - failure to split packets: if extra commands arrive while data is being
122 transferred, or two commands arrive at the same time, stupid POP servers
123 (since they only ever expect one command to be sent before they send a
124 response code) might ignore or give an error to the unexpected commands.
126 - failure to merge packets: retchmail sends several commands at once, so
127 the data it produces might take more than a single network packet to
128 transmit. If this happens, the packets will probably be split in the
129 middle of a line, so that (say) "RETR 5" is at the end of packet #1,
130 and "6\r\n" is at the beginning of packet #2. Stupid POP servers
131 might not be able to merge those two requests.
133 The problem you're more likely to experience is the first one, since
134 retchmail mostly doesn't produce huge packets. However, your POP server is
135 probably more likely to actually _contain_ the second problem, since it's
136 harder to test against.
139 7. Do you support IMAP or any non-POP3 protocols?
141 No, although it wouldn't be very hard to add. Thanks to the handy-dandy (if
142 I do say so myself) WvStreams library, retchmail is only 800 lines, so it's
143 pretty easy to extend and understand. (Fetchmail is more than 20000 lines
144 - 25 times bigger - but they didn't have the benefit of WvStreams or C++.)
146 In fact, IMAP properly supports pipelined requests, so it would even be
147 standards-compliant to do what retchmail does if we used IMAP.
149 If you want this feature, write it and submit a patch.
152 8. Why do you call the local MTA instead of implementing the entire SMTP
153 protocol inside retchmail? The fetchmail design notes said...
155 Because that's what the local MTA is for. If you don't want a "full" MTA,
156 you should still install a basic one so other packages will work. "sSMTP"
157 is a nice, simple, basic /usr/sbin/sendmail implementation that does exactly
158 what fetchmail's does, but other programs can use it too.
160 9. Okay, then, what do you think about "syntactic noise" in config files?
162 !@#*@!#(*#%^!%$@#';,!@#
164 10. Are you going to write a long-winded essay about the relationship
165 between this program and sociology?
167 I actually rather liked "The Cathedral and the Bazaar," and it doesn't have
168 serious latency problems, so I don't think there's any need to duplicate
169 that effort.
171 In fact, I'll just link to it here:
172 http://www.tuxedo.org/~esr/writings/cathedral-bazaar/
174 Eventually I _will_ write a long-winded essay about fixing latency problems.
177 11. Are you at least tracking the number of users of retchmail?
179 As far as I know, retchmail has exactly three users.
182 12. And you'll put that in your essay?
184 No.
187 13. Who's responsible for this madness?
189 First, Eric Raymond wrote fetchmail, which was really the inspiration for
190 retchmail. And I mean that in the nicest possible way.
192 Avery Pennarun <apenwarr@nit.ca> wrote the first version of retchmail and
193 started this FAQ. When I say "I", I mean me, Avery.
195 Patrick Patterson <ppatters@nit.ca> added the SSL support and keeps track of
196 public releases, since Avery is too busy/lazy.