On E-Mail, the Deletion Thereof
Posted on April 12, 2007 Posted by John Scalzi 36 Comments
I don’t want to expend too much thought on what a lying sack of crap the White House is regarding all those missing e-mails of theirs, but let me just say this: I have in my personal possession about 60,000 e-mails dating all the way back to January 9, 1996, so if you’re a real live person and you’ve sent me e-mail since that date, the chances are pretty good I’ve got a record of it somewhere in my database (minus the occasional accidental deletion during a spam sweep). And I’m not actually required by federal law to keep a record.
Now, you may ask, why do I have nearly every single e-mail I’ve been sent in the last nine and a half years? Aside for the desire to pass on my hard drives to the University of Chicago archives after my death, because I’m absolutely certain future scholars will want to read all my e-mails from over the years trying to coordinate restaurant plans with my friends, it may be because one really has to go out of one’s way to delete them. You really have to actively and affirmatively decide to delete mails once you’ve pulled them off a server and put them into a e-mail reader. Even when you’re not actually required by federal law not to delete them.
I wish I could claim the same. Too many crashed hard drives, too many different email programs used.
And I won’t go into it about a friend of mine who lost a bunch of emails because… I cleaned out her Trash folder. WTF was she thinking?
I wish I could claim the same. Too many crashed hard drives, too many different email programs used.
And I won’t go into it about a friend of mine who lost a bunch of emails because… I cleaned out her Trash folder. WTF was she thinking?
This actually might prove to be very entertaining. Destroying eighteen minutes of a tape is one thing; eradicating every last trace of an email from a large organization’s network is something else entirely.
Maybe W should appoint Al Gore as part of a blue ribbon panel to investigate tehse intranets thingies he invented.
Bet he’d do a heckuva job.
Jim Winter – LOLOLOLOL!
I agree for an individual user it’s easy to keep a backup of your emails…. but as an IT professional I’ve seen too many server crashes, corrupt files, hard drive failures, tape backup configuration errors, to make me discount the possibility that the emails were lost unintentionally .
And individual mailbox restoral has been notoriously challenging on a centrally managed Enterprise mail servers such as Microsoft Exchange. I know it’s different for managing a single PST file, but enterprise network environments and Helpdesks are constantly pinged for their inability to restore data in a timely fashion. I’d have to imagine government environments are no better of than commercial networks in this regard.
There are on-going rumors that the NSA backs up *everything* that goes through the internet, let alone the White House. Maybe they could ask for a copy…. ;-)
“but as an IT professional I’ve seen too many server crashes, corrupt files, hard drive failures, tape backup configuration errors, to make me discount the possibility that the emails were lost unintentionally .”
As another IT professional, I don’t disagree entirely – except they white house was warned by Fitzgerald in 2004 (I think) to fix their retention policies because of missing emails in the Plame case. Further, that they would be missing so many is just incomprehensible to me. According to CREW, the missing emails number in the millions.
Even if they are off by a couple orders of magnitude, we’re still talking tens of thousands. I’d like to get a job at the Whitehouse. Apparently, there isn’t much to do, and the backups go really fast when the backup device is \dev\null …
Wait Wait. I have some questions…
So what happens if you’re not a real life person?
Do androids get their own special folder?
Lastly, would this not be a good time to come out as an alien, or should I wait until I win an oscar? ;)
As the Ultimate IT Professional, I…
Oh never mind, these are boring comments.
Scalzi, hop to it – Ghlaghghee demands worship!
The Official Ghlaghghee Fan Club
Aghast, aghast I am at your suspicions. We’re at war, War (with a “W”) – questioning anything the President says during wartime is downright unpatriotic and you should be locked up for saying it. You should be grateful we have a President who’s blah blah wouldn’t want Kerry would you? blah blah and so on…
At least that’s what somebody told me today. Yep, after 24 years active duty, 2 wars, 13 decorations – I am apparently an unpatriotic asshole. Who knew?
Oh, and the Patriot Act is good thing, because it’s got “Patriot” in it. Duh. They wouldn’t call it that if it wasn’t patriotic would they? Well?
Of course it’s possible that potentially incriminating e-mails related to an ongoing Congressional investigation just happened to go missing, with no way ever to restore them from anywhere. Is it likely? No.
Sure it’s possible they deleted those e-mail messages accidentally. It’s also possible that the 18 1/2 minute gap on the Nixon tape was created accidentally by his secretary.
Not terribly likely, mind you, but possible…
For corporations as for governments, nothing good can ever come of saving e-mails. It can only hurt you!
Personally, I use IMAP for all my email and routinely cut down my Inbox to under 100 messages, per account. The really important stuff, I save to folders, but most of it gets deleted. I even go through my sent mail and clean out all the crap. It’s a little compulsive, but 90% of what gets sent in email is meaningless within a couple days. I see no reason to keep it around when all it really accomplishes is making it harder to find the important things.
So I’ve got maybe 500 of the, oh 20000 email messages, that have come and gone in the last 5 years. Which doesn’t mean that the White House isn’t lying. They are. Because the missing messages are the convenient ones, not ones that show a chronic pattern of cleaning-up. But i just wanted to point out that no everyone saves all their messages.
You really have to actively and affirmatively decide to delete mails once you’ve pulled them off a server and put them into a e-mail reader. Even when you’re not actually required by federal law not to delete them.
Not really. You’re right for the individual but you’re talking mail servers it’s not so easy to wave your hands and say “well if I can do it it must be easy”
I’ll skip the hand waving “as an IT professional” foo-faw and skip right to the horror story.
Contracted at a company in Texas. They were using whatever email server that HP offered before Exchange stomped them into the ground. Point being it wasn’t a lightweight mail server that scales from a two man office to the enterprise – it was a serious mail server for big companies. A single ‘box’ handled mail for nearly 10,000 users across North America. Hosted on a HP9000 (think ‘size of a refrigerator’ server) with some large impressive disk arrays from EMC for data storage.
There were guys at that company with tens of thousands of emails going back fifteen or more years – some of it predated the then current mail system and had been imported in.
Then the mail administrator quit for greener pastures. They handed his job to the Help Desk supervisor. Months after that they upgraded the disk array. A twenty minute outage lasted three days. Then it came back up, and down and so forth for the next two weeks. When it was ‘up’ the users were startled to see that all of the mail was gone. Finally it was stabilized with only mail from the week prior to the upgrade.
What happened? The new mail admin had no idea what she was doing; backups that had been running for years were failing, and she didn’t tell anyone that prior to the upgrade she had not had a good backup for months, didn’t realize that she’d over written (in an effort to get something working) the monthly backups dating back for a few years …
Years of mail, gone. On the other hand they had plenty of room on their new storage arrays.
That’s no excuse for legislative or executive shenanigans but .. big systems are complex and require some attention to detail that is non-obvious to the uninitiated.
Brian, et al:
“Golly, big e-mail systems sure are complicated” is not actually a very good excuse at all, nor is “well, I can set up my e-mail to delete all my mail after a certain time,” particularly when one is required by law to permanently archive e-mail. I do not claim to be an IT pro, but I do know it is a relatively trivial matter to both have an e-mail server not automatically delete e-mail after a certain time, and to archive data like e-mail in a stable and (relatively) permanent format.
I certainly allow for the fact there are ways to lose e-mail. However, when by law one is required to archive one’s e-mail on a permanent basis, it would likewise follow that one would have robust and redundant archiving processes in place to avoid these problems.
My point is that from a technical point of view it should be damn near impossible for the White House to lose massive amounts of e-mails unless it is actively working to do so — the ease in which I can archive my e-mail over two separate administrations, without making a huge effort to do so, points to this.
And in fact it seems like the White House has no problem archiving all sorts of e-mails, just so long as they don’t have anything at all to do with something the Congress wants to know about. Which means all this has less to do with non-obvious e-mail administration problems, and rather more to do with an obvious administration problem with certain e-mails.
Brian, et al:
“Golly, big e-mail systems sure are complicated” is not actually a very good excuse at all, nor is “well, I can set up my e-mail to delete all my mail after a certain time,” particularly when one is required by law to permanently archive e-mail. I do not claim to be an IT pro, but I do know it is a relatively trivial matter to both have an e-mail server not automatically delete e-mail after a certain time, and to archive data like e-mail in a stable and (relatively) permanent format.
I certainly allow for the fact there are ways to lose e-mail. However, when by law one is required to archive one’s e-mail on a permanent basis, it would likewise follow that one would have robust and redundant archiving processes in place to avoid these problems.
My point is that from a technical point of view it should be damn near impossible for the White House to lose massive amounts of e-mails unless it is actively working to do so — the ease in which I can archive my e-mail over two separate administrations, without making a huge effort to do so, points to this.
And in fact it seems like the White House has no problem archiving all sorts of e-mails, just so long as they don’t have anything at all to do with something the Congress wants to know about. Which means all this has less to do with non-obvious e-mail administration problems, and rather more to do with an obvious administration problem with certain e-mails.
I imagine those e-mails were eaten by the same black hole that ate all the missing travel office documents that were “lost” by that evil witch that ran the white house before W. She had to be running things – Bill was too busy chasing interns.
The elected politicians are all bad – what we really need is a benign dictator. How about it Scalzi? You ready to take over?
Let me see how I do running SFWA before I try to take over the world.
Shocked… Jeeze, I can’t even pretend to make it all the way through *that* joke anymore. Say, remember when we were all told that it was okay that GW wasn’t the brightest bulb because the people around him were very sharp. It now seems abundantly clear that wasn’t the case either. Just another example of the HFT going on with this administration.
I find it hard to believe that the e-mails have not been archived and placed in off-site storage. The relevant disk images may be hard to find and tedious to recreate, but I suspect they are still around. (e.g., the relevant e-mails may be part of an incremental backup. So not only would that have to find that, but all the incrementals backward in time to the nearest full backup.)
If they are also conveniently missing from offsite storage, that would be suspicious, especially if other e-mail is recoverable. Actually, if they are required by law to permanently archive e-mail, the whole notion that no e-mail is recoverable is suspicious.
I should point out though that there is a dodge that they could use, but the dodge may make them look worse. If the missing e-mails in question were on the gwb43 domain, it may not be subject to the permanent archival statute because it’s not a federal government domain. However, in that case, what were they thinking conducting government business with non-government e-mail accounts? Surely that must violate a law of some sort?
Jim Wright who is a ghast.
You had to be told you’re unpatriotic? I’m not sure there are any patiotic ghasts
I think the real question is this: Was there a longstanding understanding that “risky” email be made on RNC email accounts instead of White House email accounts. I think the answer is yes. What proof do I have? Will, remember that nice fellow, oh, what was his name… Jack Abramoff! Well, when he was on trial one of the things that came up was this whole RNC vs. White House email issue. When he found out that one of his emails had been forwarded to a White House aide he said the following.
“Dammit. It was sent to Susan on her RNC pager and was not supposed to go into the WH (White House) system.”
Now why would he care so much which email account it got sent to?
Please people, Rove intentionally deletes emails because he doesn’t want to get caught doing the things he gets paid to do. He uses the RNC account because it allows him to delete (or at least it used to) emails he doesn’t want a “hostile” congress to get a hold of.
I think the Bush administration can by now be presumed guilty unless proven otherwise.
Nathan, well I guess that would depend on the ghast. Though I suspect you’re right, the affairs of men wouldn’t mean much to a ghast.
You had to be told you’re unpatriotic? Yeah, in parking lot of my son’s school by a shrill harpy who apparently was offended by the Australian Navy Hat I usually wear and the fact that I wouldn’t sign her petition demanding Intelligent Design be added to the curriculum – because as I said, “its goofy bullshit.”
Did anybody else catch the White House IT Manager making excuses on NPR yesterday. “We screwed up, and we’re looking for ways to make sure this doesn’t happen again…” Way to take one for the team.
nisleib: There are actually two email scandals going on right now — one involves the use of the RNC issued email accounts to circumvent the White House email system, but the one John is talking about is the White House itself losing untold numbers of email relating to the fired prosecutor scandal.
The RNC is also admitting that it may not be able to produce email from before 2005 (when Patrick Fitzgerald ordered them to archive everything). I can understand that a little better because (as far as I know) they are not required to maintain backups in perpetuity. And it does seem that since Fitzgerald talked to them they have been keeping everything.
What is inexcusable is the White House being ‘unable’ to locate emails that have been requested. Speaking as another IT professional, I too find it highly suspect that they can’t produce backups of their system. I realize the challenges of backing up complex systems, but I would expect the career techs in the White House Communications Agency to be plenty up to that challenge.
Paranoid: I propose a corollary to Godwin’s Law that would state ‘as any discussion of Republican malfeasance grows longer, the probability of a comparison to the Clinton White House approaches one’.
Good news! (http://www.tpmmuckraker.com/archives/002998.php)
The RNC found a bunch of those missing emails. Except…by some strange coincidence, a bunch of Karl Rove’s are still missing. Drat the luck!
You know while I was out in the shop turning bowls, I got to thinking about this (coincidently, the subject came up on NPR at the same time). When I begin frothing at the mouth about the Patriot Act (especially the parts regarding phone/email taps and un-warranted searches) neocons inevitably say “if you don’t have anything to hide, you don’t need to worry about it.”
I hate that logic, but let’s run with it. If the White House doesn’t have anything to hide… oh nevermind, I’m going back out to shop.
Jim Wright, right there with ya.
The difference is, the government is accountable to me — not the other way around.
As a non-IT professionally, my only comment is that I get email after email from “system administrator” (whoever THAT may be) telling me my mailbox is over the size limit. So I delete. And delete and delete. Until the next time it tells me that. My emails from five years ago are long gone. What I deem unimportant emails are also gone. But how could I prove they were unimportant now?
As a non-IT professional, my only comment is that I get email after email from “system administrator” (whoever THAT may be) telling me my mailbox is over the size limit. So I delete. And delete and delete. Until the next time it tells me that. My emails from five years ago are long gone. What I deem unimportant emails are also gone. But how could I prove they were unimportant now?
Back in the olden days, when you had to use two tin cans and a string to get online, the owners of the VMS server where I had a shell account trained all their users to read an email and then delete it. I won’t say that I follow that now, or even that I followed it then, but I do still have an itchy delete finger. Between that and several hard drive crashes, I have to say that I’m impressed.
Has anyone else noticed that April 2007 minus January 1996 does not equal 9.5? More like…11 and a touch? I had to subtract 1996 from 2007 using a calculator because I’m a social sciences major and wasn’t sure, lol.
Love ya, Scalzi.
John: “I do not claim to be an IT pro, but I do know it is a relatively trivial matter to both have an e-mail server not automatically delete e-mail after a certain time, and to archive data like e-mail in a stable and (relatively) permanent format.”
Bwahahahaha! Let me rephrase that in a way which will sound as comic to you as the above does me: “I don’t claim to be a writer, but I do know it’s a relatively trivial matter to type and to make copies of printed pages these days, so becoming a successful published author should be simple.”
But congratulations on spotting the top of that iceberg, Captain. I’m sure you’ll be at the helm of the Titanic for years to come.
John again: “However, when by law one is required to archive one’s e-mail on a permanent basis, it would likewise follow that one would have robust and redundant archiving processes in place to avoid these problems.”
Every taxpayer is required by law to keep your tax files for 7 years, but I doubt you’d save them before your daughter in case of a house fire (God forbid). Resource allocation matters.
JC: “I find it hard to believe that the e-mails have not been archived and placed in off-site storage. The relevant disk images may be hard to find and tedious to recreate, but I suspect they are still around.”
In my current job, I’ve had tapes with unreadable data from as recently as 2002. I’ve more than once heard others in the industry refer to tape as a “write-once” media. But assuming that isn’t a problem, don’t miscontrue what “tedious to recreate” means — it means “money and manpower they have to spend which therefore won’t be available for other necessary projects.” At some point, the time and budget costs of heroic data recovery measures exceeds the political capital spent to say “the data is lost.”