The Great Facebook Collapse of October 4 2021 and What It Can Teach Us
Posted on October 5, 2021 Posted by John Scalzi 46 Comments
Facebook and its associated services Instagram, Whatsapp and Oculus went down for several hours yesterday, coincidentally after a damning 60 Minutes interview with a whistleblower on the service. While this afforded a few hours of schadenfreude for many, myself included, others noted that there are lots of folks who actually rely on Facebook and its other services for day-to-day connection with family, friends, and community, and being locked out of that connection for any period of time is no laughing matter.
My thought about this is, these folks are not wrong, and also, this is not a state of affairs that anyone who can avoid it should put themselves into. Schadenfreude and joking aside, any single point of contact with the Internet is vulnerable to what happened to Facebook yesterday. Sites go down, DNS assignments get scrambled, servers get Fresca spilled onto them, and so on. Arguing that people rely on Facebook services is neither here nor there to the point that Facebook services will fail at some point (and have before), as will Twitter and Google and Apple and Microsoft services, and, really, any other site or service you can name. Everything goes down on the Internet. Usually not for long, and usually not with permanent repercussions. But long enough to mess with your day for sure.
The solution to this problem is (fairly) simple: backup systems and multiple points of contact for communication. You may notice you’re reading this on (or at least from) Whatever, which is on Scalzi.com, my personal site which has existed for 23 years. It’s outlived several social media giants, from AOL to MySpace, and hundreds of other lesser sites. No matter what happens to Facebook or Twitter or Instagram or TikTok in the future, Scalzi.com will persist as long as I continue to pay an ISP to house it. But if it goes down temporarily — I’m on Twitter and Facebook and Instagram. I can be found. I have backup.
I think everyone should have their own space not reliant on a rapacious social media giant intent on commodifying one’s existence to house it, and I happily pay to have my own. But I understand that’s not feasible for everyone. But almost everyone (and every business/group/association) can have multiple points of access, and — importantly — can let others know where they be contacted/where the group can go when the primary access point goes down.
So: If you’re a group who mostly connects by Facebook, also have a community space on, say, Discord, or a dedicated Web site that allows comments. If you have email via Gmail, have a backup email address via an ISP (or, in my case, the other way around), or through another service like Outlook. If you rely on Whatsapp, keep Skype or Google Meet in your pocket for emergencies (or, you know, text and phone).
Point is: whatever it is that you do on the Internet, have a second way to do it when the first goes down, and make sure people who need to, know how to get to it. No, it’s not necessarily going to be a 100% equivalent experience, but then, Facebook or Google or Twitter aren’t likely to be down forever (or if they are here in 2021, we’re likely to have larger issues to worry about). They don’t have to be equivalent, they just have to provide access and connection for a little bit of time, even if all one does with it is send a “don’t panic, I’m fine” message to others.
What having multiple redundant points of contact on the Internet does require is effort, which people don’t like to do — the whole point of social media and especially Facebook is that it is mostly frictionless (which is why your grandmother uses it, and why terrible political memes are so easily spread on it). But these are the breaks: You can make an effort, or you can be locked out for however long it takes your favorite social media provider to break into their own data services and remove the squirrel that has electrocuted itself in one of the servers, knocking out the service worldwide. Your choice.
— JS
How many phone numbers do people actually know? If they didn’t have their cell phone and its memory, who could they call?
How many people have paper maps in their car in case the GPS doesn’t work?
Technology is great, there is something to be said for having backups.
I decided to download my Facebook contacts in order to have alternate contact info saved offline.
I found that being Facebook friends with me and having good privacy hygiene is well correlated. Two out of 100+ had emails.
Fresca? Nobody allows Fresca near a server.
Mountain Dew or coffee, maybe.
/s Not a Facebook user.
When my kids went off to college, I prepared a paper address book for them. They may not have thought it was necessary, but if their connections go down, it’d be a good thing to have. I also taught them how to read maps. I haven’t made inroads on cursive, however, I have to read my mom’s letters to my daughter still.
I have my blog on WordPress. I also post some if them to medium. I’m on Facebook, twitter and Instagram. My primary email is Gmail but I also have one via my ISP. I have met with people on google meet and zoom. I’m all over the place for someone who doesn’t really talk to people, but it’s good to know I have backup plans
Facebook shut me out after somebody hacked my account. I couldn’t reach a human at Facebook to try to remedy the situation.
The situation has messed up my ability to keep in touch with family somewhat, but phones and email still work.
I’d really like to have a service that would allow me to aggregate all my different social media sites into one feed. I use Instagram, WordPress, Livejournal, and Tumblr. It would probably take regulatory legislation to break through their proprietary barriers and make it possible to interact with my friends on each of these platforms from one page, but it sure would be nice. It would help to break the FB monopoly too.
This is why I particularly hate that so many small businesses have not bothered to get a website in favor of a FB page. I get that websites cost money to maintain, but not that much and requiring your customers to use FB to interact with your business online is just terribly short-sighted.
Somebody wrote “Send ransom or I’ll turn Facebook and Instagram back on.”
A lot of small businesses have a FB page as their only public internet contact. They may think they are saving money by using a “free” service. They may think EVERYONE is on FB.
No, many people find FB toxic and will not have anything to do with it.
A business that relies on FB for customer support will find me doing business with their competition.
Numerous venues have lost my business because they exclusively use Facebook.
Bob: For the record, I have a paper phone book (that I carry with me) and I just ordered the latest Rand McNally Road Atlas to tuck in the back seat on an upcoming long car trip. But then, I am widely regarded as something of a technological troglodyte by many family members; I also still have a landline!
If you remember in the 1970s SNL skit “The Pepsi Syndrome”. Only cola products will cause a meltdown when spilled on the control panel at a nuclear power plant.
what makes me weep?
before 9/11 folks at megacorps refused to test their DRP — disaster recovery planning — because it would be too embarrassing if any of it failed… then the day after 9/11 they had to face how far out of reality their DRPs were and still tried to avoid the updates… fast forward to Superstorm Sandy in 2012… and guess what lots of megacorps got flatfooted… then anyone care to gauge the readiness of megacorps in FEB, 2020 to suddenly needing to switch over to remote work? …anyone wanna bet me their DRPs are still out of date as of OCT, 2021!?
Notice how they say “SOME PEOPLE use FB exclusively to contact family etc” not I use FB exclusively…”
Some people just need to be contrary. I’m old & on disability, and you know what? FB, Twitter, all the socials are FREE. Everyone in my family has more than one. While we use FB Messenger because everyone has it, you know what we did yesterday? Texted. Even my cheap-ass MetroPCS phone has that.
Folks need to stop advocating for other folks who didn’t ask you to. In Shirley World, “devil’s advocate” means “I just like being a dick”. “What if they don’t have a phone?” Stop. Just stop. My mom is on my brother’s phone plan, and I’m on my sister’s.
All my friends bought phones for their older relatives, because they worry, and like to call them once in a while. “What if they don’t have family?” Well, then they don’t need to keep up with them on FB, do they?
My home page is hosted by a community-run co-op-style ISP based here in my current home city. I have points of contact on Facebook, Blogspot/Blogger, Dreamwidth, Twitter, DeviantArt, and a few other places that I think I’ll keep up my sleeve for now. I also like to collect road atlases for fun and research value, even though I’ve doomed myself by refusing to own a car.
An alternate means of communication? That wouldn’t be my phone would it? But I mean really Facebook was down and I didn’t see my sister’s post about their new puppy till today? Oh the horrors. If only I was on snapchat or instagram I could have heard about the puppy yesterday, today it’s to late…it’s old news.
“coincidentally” ? But was it a coincidence? My wife is convinced there was a connection between the whistleblower’s testimony and the timing of the outage.
As someone (and many of my friends are the same) who does NOT use FB or other social media, I don’t think that I am the one who needs to learn anything. I’ve said for years that you don’t need a nuclear attack. Just destroy the internet connection and the world would shut down. I said get Trump off Twitter and his Presidency would be screwed too, but they waited way too long.
Yes, Mark Zuckerberg & co. deserve it (and probably much worse).
I think the thing that bothered me is that people kept saying “The Internet is down.” As in, the top trending tag on Twitter was about “the Internet”, not Facebook, and a meme shared relatively frequently was about how Twitter users were laughing while “the rest of the the Internet is burning.”
Facebook is big, bit it’s not the whole Internet, and it’s not even “the rest of the Internet.” And it was concerning to me that, at least as far as I saw, there was very little pushback on that connotation.
On my side, Facebook and Instagram are my preferred social media, Facebook because it’s kind of a centralized profile that connects my disparate communities to basic life announcements (I don’t share article, opinions, memes, I just say stuff like ‘I got a new job!’ no biggie) and Instagram because, sincerely, I really using that one. My feed is 100% art and I am unfollow anything that strays from that.
That means that other than when I checked Twitter to see Facebook downage jokes yesterday, I had a relatively pleasant five hour break from social media to remind me about how little I was missing.
In a lot of the world, WhatsApp is effectively the only affordable messaging platform, because Facebook has made deals with telecom providers such that WhatsApp messages don’t count against data (or SMS) usage, and not all parts of the world have affordable data plans. So having alternatives takes more than effort, it also takes money.
If only it was that benign. Facebook is basically a conduit for political propaganda that keeps you on the service by holding your family hostage. History will remember Facebook as willful bad actors in the same class as Exxon and Philip Morris.
I think where the “The Internet is broken” comments were coming from is just how many sites use Facebook for logging people in. I know that at least five non-social media sites I use I’ve taken that option – and of course, all my usual phone games were acting up yesterday. Despite that, I didn’t particularly notice the whole Facebook down thing until I saw the comments on Twitter.
But, I can understand where a lot of concern was coming from – half my facebook feed is local “lost animal” groups, small craft businesses (think Etsy-style or the hobbyists that straddle that line) and similar interest-group-based groups.
The fully-commercial businesses ought to have the resources to have other options, but the craft-fair groups the pandemic displaced to Facebook for lack of in-person events…
Facebook engineering just posted their post mortem. Self inflicted damage. https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
I use facebook so intermittently, I did not notice it was down, until it was back up again. So happy I have other avenues of contact.
Remember when the telephone company used to distribute telephone books every year. To your house!
I miss them.
I don’t use Facebook, but I was experiencing Internet issues on the morning of 2021-10-4: DNS wouldn’t resolve for many sites, and my work Slack stopped functioning. Presumably these were side effects of the large resource usage for the Facebook attacks. So it was pretty reasonable to say “the Internet is broken” even without conflating Facebook with it.
Back in the 1990s, when my boss and I constituted what would today be called DevOps for a medium-sized database software company, much of our spare time was occupied with working out “What do we do to mitigate $FOO failure?” for every possible value of $FOO, and one of the early, obvious ones was “What if our border routers’ BGP tables get blown away?” We made sure we had a plan, and, in particular, made very sure both the internal and customer-facing services, starting with the DNS, was spread across multiple ASNs, in case that happened. It’s what happened to Facebook yesterday, they were caught completely flatfooted, and just about everything possible broke.
Apologies for the jargon above, but my point is that “What will we do if $FOO breaks?” is Operations Dept. 101 thinking, the same mentality that mandates not only multiple offsite backups but also periodic test restores as part of a disaster recovery plan. These guys seem to lack the requisite Ops mindset, and yesterday discovered that Cargo Cult IT doesn’t work.
But have they learned? That’s what the pain stimulus is supposed to be good for, after all.
My own small self-hosted Internet operation resides on the opposite side of Menlo Park, California from Facebook HQ. I just double-checked: My own pair Internet domains is each supported by five network-diverse authoritative DNS nameservers. How many Autonomous System Numbers are there, among those I? Five.
How many Autonomous System Numbers for billion-dollar corporation Facebook, Inc.’s public nameservers, after they’ve had a day to correct fatal blunders?
Let’s get all public nameserver IPs for facebook.com, look up their ASNs, and see how many there now are:
$ dig facebook.com @k.gtld-servers.net. | grep NS | awk ‘{print $5}’ | xargs dig +short
129.134.30.12
129.134.31.12
185.89.218.12
185.89.219.12
$
$ whois -h whois.cymru.com ” -v 129.134.30.12″
AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name
32934 | 129.134.30.12 | 129.134.30.0/24 | US | arin | 2015-05-13 | FACEBOOK, US
$ whois -h whois.cymru.com ” -v 129.134.31.12″
AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name
32934 | 129.134.31.12 | 129.134.31.0/24 | US | arin | 2015-05-13 | FACEBOOK, US
$ whois -h whois.cymru.com ” -v 185.89.218.12″
AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name
32934 | 185.89.218.12 | 185.89.218.0/24 | IL | ripencc | 2015-02-27 | FACEBOOK, US
$ whois -h whois.cymru.com ” -v 185.89.219.12″
AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name
32934 | 185.89.219.12 | 185.89.219.0/24 | IL | ripencc | 2015-02-27 | FACEBOOK, US
$
One. Zero redundancy. All the network-services eggs; one ASN basket. They’re exactly as vulnerable to BGP single point of failure as they were yesterday.
They need to hire Ops people, and/or let them do their jobs.
Rick Moen puts the hurt right where it belongs.
Amateur hour systems design. I never had to do network support, network design, etc in my IT career, but we did work to have off site backups for our database… We designed the DB to maintain a record of all updates, what was the data before the update, when was the update, who did the update, etc. A history of every datum changed.
I would never have known Facebook was down without the news reports that told me about it. Back when I first heard about Facebook I started to sign up, and stopped, thought about it, and said, nah! Don’t need that. And for sure, I still don’t.
That requires too much management.
I don’t need to be connected every minute of every day. If it goes down, I can do something else. I guess that is a function of how not-important I am, but it is also a philosophical point. Very few people are so important that they need to be reached instantly every second of every day. And for most people, if you think you are one of them, you are not.
On a related note, I just read Wil Wheaton’s latest blog post, which was apparently considered hate speech by Facebook, and that whole thing really proved your point here. Keep your own space!
http://wilwheaton.net/2021/10/white-fragility-in-action/
@Bob and Laura W
Some of us are old school enough to have paper address books!! The antediluvian horrors! Also paper maps in the car, for when you drive through the mountains and even with the car hotspot (thanks Honda!), there is zero connectivity with phones, tablets or GPS devices.
@Charissa, Dan, David Scott Moyer
Never was on FB, don’t miss it, can’t be on it due to current job and am 100% with you on businesses that only use FB for advertising. Ugh. I remember when FB first came out & it was disturbingly similar to chain letters… then in quickly morphed into the toxic pit of proud ignorance and halfwit (I’m rounding up here… heavily) conspiracy theories. And the endless pestering of friends to “like” their posts and to join so I could “like” their posts. (Or join FB so I could chase fake farm animals around a fake farm. No, thank you, I have a life.) And its predecessor, IMing, where people would get annoyed if you didn’t answer their chats immediately or earlier than that, the damnable beeper.
Maybe the reason so many people conflated “Facebook is down” with “the internet is down” is because that’s all they use the internet for.
Howard NYC
NO BET!!
Remember the BP oil spill in the Gulf of Mexico? Their DRP included protecting walruses. You know, the walruses that haven’t lived in the Gulf of Mexico since the last ice age? Hell, we don’t even manufacture our own medical supplies like N95 masks in this country. Because this country is just full of 50 cent millionaires. Dumb ones.
pjcamp
I resisted getting a smartphone until this year because I don’t want to be all that connected. My old phone had a removable battery and limited functions. Now with the smartphone, my employer can track me everywhere. Dammit.
@Audrey: Back when I was IT minion at said medium-sized database software company, newly hired coders would sometimes spot the Rolodex sitting on my desk and laugh: “Dude, you work at a database company!”
I’d reply: “OK, genius, imagine you’re me, a major system outage happens, including AC power, and you need to make a number of telephone calls to restore systems. Your company PeeCee with company database software not only ain’t talking SQL, but is doing doorstop emulation. So, now, who’s stupid for keeping a Rolodex?”
The Japanese animated film Summer Wars is a great explanation-by-example of why you shouldn’t connect everything to a Facebook-like entity. Recommended.
As for paper maps, which some have mentioned, for me the value is that I feel oriented, knowing the territory to the sides of the route I am taking. I would feel weird not knowing the whole map. Of course, this requires a level of abstract thinking, of interpreting symbols. I enjoy the effort.
Some have mentioned Facebook. I agree it’s coincidence that this happened while they were being spotlighted. From “open culture” I found an article that includes a Ted Talk, with transcript, by an ex-C.I.A. person (hearts and minds) who was hired by Facebook and demoted the next day. (She says why) Link:
https://www.openculture.com/2020/10/dear-facebook-this-is-how-youre-breaking-democracy.html
Speaking of hearts and minds, as regards America’s other war, in my humble opinion, if we fail to learn the lessons of Vietnam then we are utterly doomed in advance to fail to learn how to win the war on drugs. But then again, some people let the word “war” fall from their lips without the integrity and urgency to act on the word. Losers.
My first thought about this post was “Spoken like a well-off American white guy,” but Brian Quirt said it better, so I will just quote:
“In a lot of the world, WhatsApp is effectively the only affordable messaging platform, because Facebook has made deals with telecom providers such that WhatsApp messages don’t count against data (or SMS) usage, and not all parts of the world have affordable data plans. So having alternatives takes more than effort, it also takes money.”
A friend of mine and my wife Deirdre passed along this assessment, FWIW:
“I am told by a mate, the centralisation of Everything Internal to Facebook Corporation onto a single platform was a strategic ploy to make it impossible to separate components, in case an antitrust court ruled the companies needed to be broken apart.
They deliberately engineered themselves to have a single point of failure, out of fear of potential future court rulings.”
One more afterthought: Operations teams normally rely on an implementation mechanism called Change Control Requests (CCRs) for all alterations to production systems — a type of detailed technical procedure document. The CCR writeup will be a proposal including at least four sections (in addition to why this should happen, scope description, etc.):
Rollout procedure: a scripted set of steps to deploy all at once, to implement the change.
to be followed immediately by:
Rollout test procedure: something you do to see whether the desired effect has been achieved.
In the unlucky event of that failing, proceed ASAP to:
Rollback procedure: a scripted set of steps to deploy all at once, to revert to the exact prior situation.
and:
Rollback test procedure: something you do to see whether things are back to as before the change.
A vital part of the CCR-writing task is making very sure that steps #3 & 4 are complete and correct, in case a hasty retreat proves necessary. The admin is tasked with anticipating any blocker and addressing it. CCRs must be reviewed by competent peers, with special attention to that part of them, averting the dreaded “production is broken, we can’t rollback, and will have to improvise under pressure” scenario.
And, point is, if step #4 necessitates “Have a junior sysadmin standing by in the server room ready to run the rollback script if ordered via telegram or smoke signals”, then you do exactly that. Steps 3 & 4 never can be “I guess we’ll just figure something out, probably using an angle grinder.”
Why wasn’t this done? Best guess is the syndrome Bill Woodcock described, which rings depressingly true to this ageing sysadmin: https://krebsonsecurity.com/2021/10/what-happened-to-facebook-instagram-whatsapp/comment-page-2/#comment-539971
Very good advice. Yes nothing lasts forever. It’s also good to have a paper copy of contact info of people. I had once recommended someone about having a backup for her freelance job because it probably wouldn’t last forever and she blocked me on facebook. It’s always good to have backups.
When it came back online, I told my husband that Zuckerberg finished sulking.
A local fan run con I attend has a bridge (main room for communications) and a different room — the battle bridge — already designated. If the main one goes down (unusable, no power, etc.) everyone already knows where to go.
Smart planning.
“WhatsApp is effectively the only affordable messaging platform, because Facebook has made deals with telecom providers such that WhatsApp messages don’t count against data”
That would be like saying “phillip morris at least provides cheap cigarettes to children”. Facebook provides free messaging because they have developed their algorithms to be as addictive as possible. And their algorithm feeds off toxic, deadly, misinformation.
Facebook is freebasing free speech.
Facebook is actually more evil than phillip morris because phillip morris hasnt developed a cigarette that eavesdrops on smokers, extracts personal information on those smokers, and then sells that information, allowing ohillip morris to give those cigarettes away for free.
The fact that any service owned by facebook is free is only because facebook has made itself an amoral spreader of toxic misinformation based on an algorithm designed to make facebook as addictive as possible. And then they figured out that every click tells them something about their users, and that is information they can sell to someone else.
Free whatsapp is like free cigarettes.
But at least we regulate cigarettes.
Whatsapp is owned by FB, but they are not algorithm driven. They are an encrypted messaging system, period. One does not need to have a FB account to use Whatsapp, and in countries like Mexico, where data is limited and costly, a free messaging service is invaluable. I favor splitting both IG and Whatsapp off from FB, but I think you underestimate the value of a free messaging service. Maybe because you don’t need it where you live?
Some of these options are only available in developed countries, however; for many small businesses in developing countries, Facebook and other social media sites may be the only options for easily hosting and promoting their products and services.
David Scott Moyer
I hear you about the value of free messaging services but should that really be the primary consideration? So far, cheap internet access and platforms like FB and WhatsApp have largely made it easier for toxic disinformation, polarization, bigotry, insurrection and outright lies to spread faster than ever. For better or worse, when you had to pay for information or to communicate with other human beings, you generally had to be civil to them (or they would hang up or chuck your letters in the garbage or stop buying your newspaper. Even the 6pm news used to have to have actual content.) or provide actual information. No one was going to waste money on yellow journalism on a regular basis, not in a way that made it a better business model than actual, factual news. Sure, it was a gatekeeper model but the model was also “all the news that’s fit to print” not “all the news that fits we print”.
Case in point: India. Cell phones used to be out of reach of the average Indian. Grameen Mercy Bank started the whole concept of microloans, mostly to disenfranchised Indian women because their research indicated that loaning money to women meant that the loan was more likely to be repaid and that loaning money to women was also likelier to raise the entire family’s living standard. (Indian men, as a cohort, had notoriously poor repayment rates and business plans, mostly because they spent the money on themselves only and otherwise screwed up with the business loans.)
A common microloan was for a cell phone, back when cell phones were relatively expensive. The “telephone ladies” would have a phone for the entire village (usually) and other villagers would have to pay to use the phone. The income would be used to pay off the microloan and provide a livable income for the women and their families. The expense of paying for the use of a cell phone meant that people didn’t have a lot of time–or money–to spend on the phone spreading bs internet lies, conspiracy theories, racism, online stalking & bullying, etc.
Now that cell phones are cheap and data usage is affordable to pretty much everyone, all manner of crap has proliferated. Crap like “use ivermectin to cure Covid 19”. Face it, people are not using the promise of free/cheap communication to better themselves or learn about other people; they’ve used it instead to grow Internet balls, stalk and attack people, particularly women (GamerGate, anyone?) and plan attacks on democracy, including the 1/6/21 attack on the capitol. (Call it what it is. If black people did the same thing, it would be called a riot and the media would be screaming for the rioters’ execution as traitors.)
There is unquestionably value in free speech and free/cheap communication. But back when the First Amendment was being written, the founders were also writing it with the idea that you would actually have to engage with other people face to face or by mail. Not anonymously with a side of stalking. And while the internet in general and FB in particular made it easy to communicate with each other, it also created a toxic cesspit of lies, disinformation, bigotry and stalking as well as giving the current generation the attention span of a fruit fly. It says something about internet usage that colleges and universities no longer assign long form reading assignments because the students won’t actually read anything that long. Because they’ve become accustomed to the instant dopamine hits that most internet provides. “tl;dr” is a thing for a reason.
You value what you pay for. And I’m not convinced that cheap or free unregulated communication is a greater good than the problems of reduced attention span, increased isolation, mental illness and general unhappiness is worth the price of cheap internet/cell communication.
argh. Should have been “most internet content” in second to last paragraph. My bad.