bynkii (bynkii) wrote,

POP, IMAP, and Yahoo Push Email, a primer

So over on The Apple Phone Show, Liana Lehua posted an article entitled POP and IMAP, what's the Buzz?, that while well - intentioned, (wants to explain such things in an easy to understand manner), ends up looking like a badly done, (has real factual errors) article on IMAP that was really, in theory about Yahoo Push email.

If it seems confusing to you, well, it is to me as well. The fact that when I pointed out some of the errors, I got snarked at by Scott Bourne, my comment was deleted, and he managed to throw in a few comments about how I need to learn about targeting my audience. Evidently, I am incapable of relating to a non-technical audience. The people I've supported over the years would be amused by this, but oh well, Scott's more concerned with saving face than being right.

However, I am far more bothered that an article with real factual errors is being held as something it's not, so in the spirit of of the intarweb, I'm going to to do an overview of POP and IMAP, then talk a little about Yahoo's Push email. If you're fairly technical, this isn't the article you want, it's really for people who are perhaps dealing with things like IMAP for the first time, and want to know just what the heck is going on. I will do my best to avoid waxing technophile but it's kinda hard for me to avoid ;-)


Of the three, POP, aka POP3 is the oldest email protocol. POP is really quite simple to understand, and conceptually, it works rather like going to the UPS store to get a package. When you connect to your email server, all email in your inbox is copied to your email program on your computer or iPhone, and it is deleted off the server. Just like when you get your package from the UPS store, it's in your possession, and no longer in the UPS store. POP is very simple to deal with from the user and the programmer perspective. It does have some disadvantages. If you have a slow connection, and a lot of email, well, you're going to be waiting a bit for all that to download to your computer or iPhone. If you have a lot of attachments, then you'll be waiting even longer, as the entire message has to completely download to your system before you can read it. If you aren't at the computer you normally read your email from/don't have your iPhone with you, then you can't read any of your old email. By the same token, any email you download to the computer you are at won't be easily available to the computer you normally use. Now, if you only have one computer that you ever use, and only read all your email in one place, this isn't a real problem.

But you have an iPhone. Now what? Well, as it turns out, POP has an option to kind of work around this. This options leaves the email on the server after you download it. In fact, this is the default for the iPhone, which makes sense. So now, as long as you have your email program set up to leave the email up on the server, there's no problem right? It's always available, life is good.

Well, only kind of. Remember, with POP, you have to download all the available email when you connect. If you leave your email on the server, that can start taking longer and longer. On a slow connection? It's even worse when you consider that spammers are now trying to use PDF attachments to spam you with. So if your email provider doesn't have a good anti-spam setup, then you get all those attachments downloaded to your iPhone. Then you get them all downloaded to your main Mac or Windows machine. To keep using our UPS analogy, this would be like every time you went to a different UPS store, they gave you another copy of every package you'd ever had delivered. No one needs that many packages. Another issue with POP is that your email storage is only as reliable as the machine it lives on. With POP, if you aren't careful about backing up, a hard drive crash could easily destroy all your email archives. Of course, we ALL back up, right? ;-)

So if you're using multiple computers or devices to read your email, (and if you're an iPhone user, this is probably you), and you have a slow connection, POP is still simple, but it's no longer so great. However, there is another answer: IMAP.


IMAP was created to make up for POP's deficiencies. The most important difference between POP and IMAP is where your email lives. By default, with IMAP, your email lives on the server. Assuming your mail server is run by competent folks, this is actually a good thing. If nothing else, servers tend to have redundant drives, better backups, more reliable hardware, better power, etc., than your typical laptop/desktop/phone. Face it, you don't see a lot of Xserves taken out by small children, pets, or people tripping while carrying them to a meeting.

IMAP is also designed to be more efficient over bad connections than POP. In a perfect world, when you connect to an IMAP server, (Yes, I know Liana said IMAP = Push email. That's not correct as we'll see), you download a list of headers. Headers are the parts of the message that let you see who sent it, when, the subject of the message, and the like. Email headers are, as a rule, very small, and so a lot of them can be downloaded very fast even over a slow connection. As an example, way back in the dark ages, when I was running Netscape Communicator 4.5 on a Powerbook 3400c, I could download over 3,000 IMAP headers in under 5 minutes on a 33.6K modem connection. For all you people kvetching about EDGE speeds? EDGE is blindingly fast compared to that. The important part here is that by only downloading the headers, our perfect IMAP client lets you see how much email you have very quickly, without making you download the whole kit & kaboodle.

So now you want to read a message. You select the message, and our perfect IMAP client then only downloads the body of the message. Since the body of the message is the part you read, this makes sense. Rich text, HTML, plain text, doesn't matter. Even a fairly long message isn't going to have that much stuff in the body. So over a typical EDGE connection, this is not a long delay. Even over a modem, it's not really long. But what about attachments? What if you have a 2MB or greater attachment that you don't want to deal with? Well, this is one of the beauties of IMAP. You don't have to. The attachment isn't, or at least shouldn't be downloaded until you explicitly choose to. So you can read the message and blow off the attachment until you get to a faster machine, one with more storage, a faster connection, whatever. Keep in mind, I am talking about a "perfect" IMAP client. Not many clients are that perfect.

Now, when I say "download" here, I'm not talking about how POP does things. What I'm really doing is copying the information in the message to my local mac/iPhone/whatever. It's all still "physically" on the server. Why is this good? Because if someone sends me a big horkin' WMV file as an attachment, I don't want that on my iPhone. For one, I can't view it, for another, it's too friggin' big. With POP, I'd have to set my client(s) to always leave mail on the server. With IMAP, that's how it works until you move it off, either to a local folder on your client, or you delete it. This means that by default, that same message I saw on my iPhone will still be there when I get to my Mac. However, unlike POP, since IMAP is a smarter protocol, when I bring up my email client, it will show the message as being read already, since I did read it...on my iPhone. If we go back to our UPS example, this would be like the "Star Trek" version of UPS, where no matter where I went, I could get to any package in UPS's system nearly instantly, but I'd always have access to that one package, instead of making copies. I could get just the information I wanted on the package, or even open it up. Then when I was done, just walk out, and know that the package would still be available for me whenever I wanted. IMAP is the email equivalent of having your cake and eating it too. Only with less calories and fat.

If I really want to get all complicated, I can create email folders on the server and use rules to move my email to those folders out of my inbox. That way, even when I file my mail, it's still available from well, any IMAP client. If I have a really kick-butt ISP, then they have a server that lets me create server-side rules. These are rules that live and run on the server. They work just like rules in Mail, Entourage, Eudora, etc., but since they run on the server, they don't require me to leave my email program running to use them. So even my iPhone can reap the benefits of server rules. Server rules just leave POP in the dust.

Now, there are some disadvantages to IMAP. First, if you have an email provider that is stingy with space, IMAP makes it really easy to run out of space. This is why I have 2GB with over a GB dedicated just to email in my .Mac accounts. Secondly, deleting email can be more complicated with IMAP in that you can either say "Move it to a 'deleted items' folder" or 'Mark' it for deletion. (For a point of reference, the iPhone does the latter.) With the former, you have to be careful, because even though it's in a deleted items folder, it's still up on the server, and can still count against your email quota, if you have one. The latter is more obvious to see in an IMAP email client, (for example, in Microsoft Entourage, items 'marked' for deletion have a line through them), but you still have to take another step to really delete the email, aka "purging" items marked for deletion. On my Mac, I just have things moved to the deleted items folder, and I empty that when I quit my email client. For stuff I 'delete' on my iPhone, I just manually purge them in the email client. Off-line access can be tricky too. After all, the only way to have off-line access with IMAP is to cache the messages (and optionally attachments) locally. This would seem to make IMAP about as useful as POP, but remember, unlike POP, you haven't removed anything from the server, and most email clients don't require you to cache the entire message, attachments and all. So even IMAP caching can be more efficient than POP. Finally, a lot of email providers either don't do IMAP outside of webmail, (which almost requires IMAP), or they're so stingy on your server space that it's almost worthless.

Now, while IMAP is most definitely not "Push" email, it can emulate a lot of the features of a push email service/client, such as Goodlink or Microsoft Exchange. The way it does this is via the IDLE command. The IDLE command, at a VERY basic level, lets the client tell the server "Hey, when something happens, tell me". The server, when something happens, i.e. new email shows up says "Hey! Something happened, and here's what it is." So, as new email shows up, you can get it without having to do an actual email check. This is technically not Push email, as the client has to start the process, whereas with true Push, the server says "Yo, here's your email".

However, not all IMAP servers support IDLE. This is not a big deal, it's not required by the standard. Amusingly enough, Apple's .Mac servers don't support IDLE. (If they do, they hide it well.) Using Microsoft Entourage's ability to get the features an IMAP server supports, we see the following:

Now, there's a lot of stuff in there that's really only of use to email administrators, but what we don't see is the word "IDLE". What should we see on a server that does support IDLE? Check out the same information for my email server:

Looking at that screen shot, we see the second word from the right is "IDLE". So this email server does support it, whereas .Mac doesn't. If I had to guess, I would say that .Mac has a lot of email servers, and upgrading a large server farm is not something you do often, (or at all if you can help it). I'll also hazard that this kind of upgrade is not minor at all.

So where does this leave Yahoo!'s push email?

Yahoo! Push

As near as I can tell, Yahoo! is just using IDLE, although based on some of the comments I've seen, not very well. However, in Yahoo!'s defense, based on how the iPhone deals with my two accounts on servers that support IDLE, it may not be all Yahoo!'s fault. So, until I get better info, which can always happen, I'll go with Yahoo! using IMAP's IDLE feature to emulate "push" email. I would also REALLY like it if the iPhone actually used IDLE better. That would leave me with only my .Mac account being the one IMAP account I have that doesn't support IDLE.

So that's a very basic look at POP, IMAP, and Yahoo! Push email on the iPhone. Hopefully, it helped you more than it confused you. If not, leave a comment, and I'll try to make it more clear.

On a personal note, this article was started as a bit of a finger to The Apple Phone Show, particularly after a few emails where my evidently odd insistence that one be correct even when talking to the less-technical crowd was viewed as "nitpicky" and an attempt to drag the article down into "pixel - pushing". I cannot possible disagree more. Like everyone in my field, I started out knowing, well, nothing. I had to learn, and make the journey from ignorance into knowledge just like everyone else. Without accurate information along the way, that journey would have been impossible. I find it rather insulting in the extreme that anyone would defend inaccuracy for any reason, but especially in an article aimed at beginners, who, because they don't know any better, need to be able to trust that what they're reading is accurate, no matter how it is written, or at what audience it is aimed at.

While my normal audience is well able to catch any errors, I don't feel that a non-technical reader is any less deserving of accurate information than someone with a decade of experience. It's for that reason, along with the attitude I got from the Apple Phone Show that I have stopped reading that site.

okay, it was mostly the snark. Dude, when someone points out real factual errors, don't get your damned hackles up. Just admit it, correct it and move on. Yeesh.

  • New Physics!

    Arachnivistic Velocity The speed at which an arachnophobe retreats from a spider. Can be expressed as: A v=1/D a Where A v is the speed of the…

  • Fuck Jaws

    He was a pussy...i give you true seaborne terror.... HOVERSHARK WITH FRICKIN' EYE LASERS!!!!! RUN AWAAAAAAY!

  • Wait, wait, wait

    I thought those darned illegals were taking away jobs that "Good Americans need and want", that they were denying us paying jobs in hard times. It…

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.