Skip to main content
684

October 25th, 2023 × #coding#mistakes#horror stories

Spooky Coding Horror Stories 2023 - Part 2

Web development horror stories about bugs, mistakes, and disasters

or
Topic 0 00:00

Transcript

Announcer

You're listening to Syntax, the podcast with the tastiest web development treats out there. Strap yourself in and get ready. Here is Scott Talinski and Wes Bos.

Wes Bos

Welcome to Syntax.

Wes Bos

This is our annual rid Spooky coding horror stories episode.

Wes Bos

You may have heard actually the Hasty before this, but this is the tasty full of rid Coding horror stories. So every year, we ask people to submit their,

Announcer

my wife just texted me, is Dracula in your office? Good. I nailed it then. You were really loud. In fact, I had my headphones turned up, and you started that intro so loudly that I, like, and almost shot the earbuds out of my ear.

Wes Bos

So these are stories that make you want to put your head in the sand. Stories of Caution stories of people totally dropping the ball on customers websites with their tech, with their database. It's just awful stuff.

Wes Bos

So it's a good lesson both for entertainment as well as

Announcer

there's something like Valuable stories to be learned here. Yeah. These stories will make you wish that you had some brains. Right? If if you had any brains, You'd be using century@century.io to track your errors and exceptions. Because when these types of spooky situations pop up, You wanna make sure that you have a, well, a non spooky pal to help you out solving those issues. So head on over to century. Io.

Topic 1 01:10

Podcast hosts discuss annual horror stories episode

Announcer

This podcast is presented by Century. So thanks. Let's get into these spooky stories. Wes, you wanna hit that first one? Yes. This one is wild. So, by the way, we should say

Topic 2 01:40

Story about accidentally deploying a crypto copy paste bug

Wes Bos

All of these stories we are keeping anonymous whether the person has asked or not just because some of the best ones are anonymous because they're total I don't want them to see the light of day, but we don't want to put anyone's name on these because if you tweeted it, you can delete the tweet. But if it's on this podcast, you can't delete the podcast. So rid thank you everybody for submitting. The first one is a crypto copy paste horror story. I worked for a crypto company in 2020. And while building the React Native app, one of my colleagues made a PR to solve a small copy to clipboard issue Where it didn't alert the user that the copied value was actually copied, he, quotes, fixed it. And I threw a a classic LGTM, looks good to me, and it was deployed.

Wes Bos

A few hours later, 1 angry customer made a huge thread On Twitter about how he made a mistake and sent $60,000 to the wrong wallet as he thought he had copied the address rid To the right place. The UI showed that it has copied successfully, but it wasn't for some specific Android phones.

Wes Bos

Rid Oh, we patched the fix, and the company paid him after all the buzz. TLDR always test your code.

Announcer

LGTM.

Announcer

Also, I thought LGTM stood for let's get This money.

Topic 3 03:05

Discussion about misunderstanding GitHub comment acronym LGTM

Announcer

So every single time I've read LGTM, I read it as let's get this money, which It makes so much more sense as looks good to me. That is hilarious. So when I first saw this in this context, I was like, oh, that's so that's even,

Wes Bos

rid so you think every time someone, like, sends a PR and they say LGTM, they said, let's get this money, like like, that's slang for deploy the sucker?

Announcer

Yeah. Like, what? This is this is our product. It's going live. I I've I'm, like, I'm having a moment of, like, I'm so stupid right now. So this is my spooky story.

Announcer

And I honestly I'm having that moment too. Extremely red in the face right now. I wish y'all could see because,

Wes Bos

I'm very embarrassed to admit that, and I feel really dumb. So rid Alright. This next one, I think, might be the best one. I'm gonna let you read it, but, like Oh, yeah. Buckle up. Maybe if you're driving, pull over. This is a big one.

Announcer

Re Buckle up, strap yourself in, and get ready.

Announcer

Big Brother bug. In my early career, mid 2000, my 1st week on the job as a software dev Was for a local big brother production.

Topic 4 04:10

Voting algorithm bug caused wrong Big Brother contestant to be eliminated

Announcer

I won't say what country. Big brother is in the, the TV show. Right? So is a TV show where people are in house, and it's a competition reality show in case you haven't heard of Big Brother.

Announcer

I was Tasked with writing an algorithm for that week's house vote.

Announcer

The logic was intricate. Something like The public had a percentage of the vote determined by, text message count. The house contestants had a percentage.

Announcer

The previous week's winner had a percentage, and Big Brother, the production team, had their own percentage.

Announcer

I wrote the algorithm and was pretty happy about it. However, come Sunday, as the votes were tallied and the auditor checked the results, He wrote down the name of the loser and then walked on stage to hand the envelope to the presenter.

Announcer

The contestant to leave the house was announced.

Announcer

And the announcement was made, I double checked my code and spotted a bug. Rid After correcting it and running the code again, the results flipped, And the person announced as the loser was actually the winner.

Announcer

Essentially, the most popular house guest was leaving.

Announcer

I kept this to myself for years.

Announcer

I had night sweats for months.

Announcer

I still think about the butterfly effect it would have caused As it was the number one show in quite a number of countries.

Announcer

I can only laugh about it now. Oh. Oh, man. Oh. Like like that's

Wes Bos

Like, how much do you win at Big Brother? Isn't it like $1,000,000

Announcer

if you win? Oh, so this is glad that this one is anonymous because That is a very rough situation.

Announcer

You know what? You know, we've given some of these scores in the past. Rid. Now I'm going to be giving this, like, the golden the golden spider award for their being the the spookiest story I've ever seen. Oh. Oh my gosh. Brutal. You know what? I guarantee

Topic 5 05:57

Only 6 laptops could fix NPM when it crashed

Wes Bos

the bug was, you know, when you're trying to, like, find a percentage or something and you have to, like, take 1 minus that percentage, I bet rid. I bet that was the bug. That bug. Yes. Because when you're working with percentages,

Announcer

it is easy to have that flip flop happen. I wonder everyone must have been outraged with the that I I wrote myself. I wrote That's crazy.

Announcer

For a voting platform before for a breakdance competition, where judges' votes were tallied and then put through a thing, and it would reveal the winner. And the 1st time they used it was in, like, a, A battle in South Korea. And I remember, like, thinking to myself, I am so pooched if this messes up. Rid.

Wes Bos

Luckily, it was fine. So, yeah. So many of these stories, the stakes are so high and they have to do with People's lives or a significant amount of money.

Wes Bos

So next one. For 2 years, I had one of the 6 laptops That could fix npm when it fell over. So this one was submitted anonymously, but from somebody who was a major player at npm in the rid. Not the early days, but in the mid days, the feeling of dread when I would occasionally misplace that computer in my apartment was true horror. So I asked him, like, that's wild.

Wes Bos

How come there's only 6 laptops in the world that could fix NPM if it fell over? And he said there was rid. SSH keys on those computers that were not backed up anywhere for security purposes.

Topic 6 07:45

Autocompletion mistake deleted home directories

Announcer

Oh, the dread. The feeling of dread. Both both losing it, but also somebody getting their hands on that. You know? Yeah. You'd have to know what you had, but still, Yeah. That is, that's that's spooky.

Announcer

Alright. Next one. I once implemented a useEffect wrong and was auto submitting users' payments before they could submit before they click the buy now CTA.

Announcer

Rid. Dear gosh. Auto submitting payments. Here's another case. When when, just just for those of you out there, if you're working rid. Ever with a payment system, and before I read the rest of this, and you do have an action to submit a payment, the very first thing that should be done on that action to submit the payment should be to disable the button, especially if you're you're working in some sort of click, but I guess this might have been outside of that. So, rid Okay. I once implement implemented a useEffect wrong and was auto submitting user payments before they clicked the buy now CTA.

Announcer

It was a full purchase flow. And near the end, they put their payment info in, but there were some other steps for them to manually Click a button where they would actually make the purchase.

Announcer

But instead, they put their payment info in, then got redirected to the final rid Confirmation stage without them manually confirming.

Topic 7 08:57

Payment form submitted before confirmation

Announcer

I think, like, 100 orders got through. Our traffic wasn't super high.

Announcer

There were there was a big meeting with the business people to be made aware about it, and the higher ups were basically like, well, let's just see how it Hands out and see if these people contact us. Oh, no. That's not right. That's not the right thing. I think all but 4 of those orders activated, And it was our highest conversion month. I like to brag that my bugs are run of you revenue generators. Oh my gosh.

Wes Bos

No, people. Man.

Announcer

The thing is the right thing to do is to refund everybody without them having to contact you. That's

Wes Bos

Oh, that's wrong. Next one. So I accidentally turned on a feature before it was ready. Nobody noticed for 4 days. By the time we did, forty rid.

Wes Bos

$1,000. No. Not $40,000.

Wes Bos

40,000 orders had been shipped to the customers, but then also refunded.

Wes Bos

It took another week to notice that nothing we can do about it. 40,000 people just got a whole bunch of free stuff. A few days later, I was fine. The total cost of the company was £700,000.

Wes Bos

That's that might be the 1st $1,000,000

Announcer

mistake. Is that is that a 1000000 dollar company? We've had a lot of expensive mistakes on these spooky stories. Dollar mistake. But that one is you know that, did you watch that TV show on Netflix, The $1,000,000 menu.

Announcer

It's like oh, there's a $1,000,000 man million pound menu.

Announcer

That that's right there. Holy cow. $700,000.

Wes Bos

£13,000. That's a $1,000,000

Announcer

US. Pounds. You're right. Oh my gosh. I I, like, I recoil reading these. This is This is simultaneously my favorite episode of the year because I feel like it's the most laughs per second,

Wes Bos

just because of how Bad. These are but they're cringey laughs. They're they're laughs that come from a deep place of Yeah. I also I spent all morning going through these, and I'm so stressed out right now just from, like, Oh. Oh. Oh. Oh.

Announcer

I have so much tension in my body reading these. Alright. On the on the 1st week of my internship, I dropped the analytics database for the past 6 months.

Topic 8 11:25

Intern dropped analytics database

Announcer

I was trying to build a view with just the data I needed, And I dropped the source data instead of the view. Honestly, I blame them for giving the intern full access to the database. Yeah. Right. You are. Yeah. Don't give the intern The ability to drop it. Yeah.

Announcer

Data phase.

Wes Bos

2nd week on the job was testing SQL injection attacks on our dev site.

Wes Bos

Rid. Example. So SQL injection is like you have like an input and and you're like, what's your name? And you say my name is drop DB dev. So he typed in, like, semicolon t equals t drop d v dev.

Wes Bos

I got a support email saying they couldn't access the site anymore. Rid. Turns out our dev DB was named dev_ dev. Dev was actually production.

Wes Bos

Happened at 11:57.

Topic 9 12:15

SQL injection attack dropped production database

Wes Bos

The next DB backup was at 12. Oh.

Wes Bos

My own.

Announcer

I wanna say, I I also feel like, accidentally having That dev to production thing, I accidentally did that with PlanetScale when we did our coupon voucher site because they have, like, an interface for swapping them, and I accidentally rid Swapped them myself. I fixed it, and I put it back to where it was. Oh, yeah? But that is easier than you might expect, so I feel for you. This is from a past guest. You'll know who this is because of of the website, but past guest on the show. Yeah. Oh, from the website.

Announcer

Yes. Okay. When updating the DNS for, polyfill dot I o. There's your hand. I got 1 digit incorrect and took it completely offline for 2 hours.

Topic 10 13:00

Typo took down polyfill.io

Announcer

As after updating the DNS, I immediately got on the train and had no signal.

Announcer

So I was unaware that people were trying to contact me about it being offline. Now this is a service folks, if you don't know about polyfill dye, this is a service that people are regularly hitting. So It being just straight up down for 2 hours.

Wes Bos

30,000,000 hits a day or something like that.

Wes Bos

Like a good chunk of the Internet. It's like things polyfill.

Wes Bos

Io to polyfill APIs that are not available in some browsers.

Wes Bos

So rid. It not being available means people's entire applications break. That's like AWS going down without, you know, being a hosting thing. I wrote a large application performance management software similar to Century.

Wes Bos

This happened about 10 years ago, But one of our offerings was so called synthetic tests, basically a way to ensure for a company that Their website works when accessed all over the world. A customer can configure a URL and optionally some JavaScript to click a few buttons. And if not nothing breaks, it test rid. Passes. We would then periodically run these tests on all our machines all over the globe, which we had hosted on AWS. If The test would fail. The customer would get an alert. Everything was fine until 1 weekend when a team came back to find lots of the AWS alerts on high cost. Rid. Turns out a trial customer configured lots of synthetic tests to hit his website and stay there for some seconds.

Topic 11 14:32

Bitcoin mining exploit drained AWS budget

Wes Bos

Well, in the background of his website was a Bitcoin miner running. So basically, he used our AWS machines to mine bit We lost $50,000 due to this exploit on 1 weekend and quickly had to add some detection rid To fix the exploits as well as we could simply not turn off the feature altogether.

Wes Bos

Oof. That those are the worst bugs where rid It's being used legitimately in a specific way, but also it could be used nefariously. And it's always Bitcoin miners. If you let somebody run code anywhere. Even if you let somebody hit a URL and like, sit there for 3 seconds or 15 seconds.

Wes Bos

Then they have 15 seconds of your compute running, and they can do

Announcer

Nefarious things with that compute. Yeah. Bitcoin miners are like, it's like life. They will find a way. Right? Did you know That Mongo's dollar sign out aggregation stage clears the collection you're saving too.

Announcer

That's That's a good start. What a good hook, this person. This is a person who's, like, writing a story.

Announcer

I didn't I didn't. Not the 1st time I used it. I cleared 80,000,000 records in one swift move with 0 backup. No. Stop.

Announcer

It wasn't user data, though, But a dataset to train AI that took me months to build and recover.

Announcer

Oh, please. 80,000,000 records With no backup, if you got 8,000,000 records, he gotta make a backup somewhere.

Topic 12 16:06

Command deleted dataset for AI training

Wes Bos

I asked him if it was time consuming or expensive, and he said rid Both. Yeah. Yeah. Oh. Next one, web chat DDoS. I'm a front end At a company that does software for customer service many years ago is working on a prototyping web chat feature. Once it seems rid. Once it seems stable enough, we would let our customers put chat on their websites. That's when we realized our code was now loaded onto a lot more computers than our main app, And it was making way too many calls to our server. Basically, we were DDoSing ourselves with our new web chat product. We took it down as soon as we could, but it was already loaded on a bunch of pages that were still hammering us. So like, I imagine they have rid A piece of JavaScript somewhere that people load up a chat, and every 5 seconds, it's pinging an endpoint to rid. Say, hey. Are there new messages? Are the new messages? Are the new messages? So even if you take it down, that JavaScript is still running In somebody's browser and pinging them. So it goes on to say, I'm sure there were people out there with a bunch tabs open in Chrome, not realizing it was making a bunch of people sweat in a conference room somewhere.

Announcer

Dear gosh.

Announcer

We we see kind of the same stories over and over again in a in a different way. Right? Accidental DDoS, rid drop the database. Accidentally charge a 1000000000 people. It's, like, so funny how the these same patterns in in I mean, in software.

Topic 13 17:48

Chat feature DDoSed own servers

Announcer

Oh my gosh. And I'm I'm I'm seeing the next one right now, and I'm already in tears looking at part of the next one. So okay. Rid. The this one is a a URL shortener.

Announcer

This is great. Okay.

Announcer

A while later a while later, we made a is this the same person? I gotta ask. Yeah. Because they say a while later. Submitted too. I chopped it in. No. That's great. This is a double dose then.

Announcer

Good for this person. A while later, we made a URL shortening service to our URL.

Announcer

Someone from the marketing team was going to demo this and Slacked us About our prank, none of us knew what he meant.

Announcer

So he sent us a screenshot of the URL that the app had generated for him. The URL was and then there was a URL shortener forward slash, Capital f, capital u, lowercase c, capital k, capital m, lowercase e.

Announcer

I'm gonna let you, spell that out yourself, but it is beep me.

Announcer

I know you can't read it on air, but you get the idea.

Announcer

This wasn't a prank. It was actually randomly generated.

Announcer

But then we had to do some extra work to make sure the shortened URLs that didn't contain profanity.

Announcer

Fortunately, this only happened to someone at our company and not to a customer during a demo.

Announcer

Yeah. I think that's Gotta be a concern with any random generated word. You think about, like, the name generator on Heroku, Dinos, or anytime you have, like, dynamically generated anything.

Topic 14 19:08

Profanity generated in URL shortener

Announcer

Got to put a profanity filter on that.

Wes Bos

Ontario puts out a list of rejected personalized license plates.

Announcer

Oh, yes. And

Wes Bos

it's hilarious because they have to try to decipher. Are you trying To sneak 1 by me and spell something out on your license plate that I don't know. So it's this, like, game of trying to figure out, Is this a misspelled bad word? Or and also is it like a new slang that I have not yet heard of? And It's hilarious. If you go on the Ontario website, you can look at the list of denied plates, and there's so many good ones on there. Speaking of that,

Announcer

there's a podcast I listen to. It's like an Australian comedy podcast, Bunta Vista, and, That is, like, a segment of 1 of theirs is reading rejected plates.

Announcer

So the they just have a segment where they're reading rejected plates from various places. Butt of everything. That's such a good plate, chicken butt. There's so many worst ones in the out west. There are they get so rough, and it is very funny. Have you ever considered getting a personalized license plate? I have

Wes Bos

many times.

Wes Bos

I don't know. I I don't wanna be, if I cut somebody off. I'm a pretty good driver, I I think, but Every now and then, you make the wrong You don't want to do that easily. Yeah. You identify the wrong person mad. You know? Maybe I would guess something funny, but every time I look at it, it's, like, $300.

Topic 15 20:45

Office servers containing source code stolen

Wes Bos

And I was like, that I've never found one that's $300 funny. You know? Like, it's gotta be

Announcer

Yeah. Really, really Honey, we have, like, new black and white license plates that are really slick in Colorado because the other ones are, like, green with mountains on them.

Announcer

But I I thought of, like, a black and white one that says syntax. It's the perfect amount of characters. Our license plates are 6 characters.

Wes Bos

It would look pretty cool. Be cool. Like on brand. What's his name on Twitter? I forgot his name right now. He has no JS. That's a custom Iowa plate.

Wes Bos

Next one.

Wes Bos

Rid I sent an email to 20,000 users with the wrong username and password. Basically, Excel import rid had a 1 row shift. The client was not happy. We had a terrible meeting with shouting, and we had to write an apology email to all 20,000 users.

Wes Bos

For some reason, my boss was chill the entire time and didn't say anything to me. Everybody on Twitter replied to that, like veteran.

Wes Bos

You know? He's seen it all. Next one, dodged it.

Announcer

My 1st web dev gig set in the heart of downtown Vancouver.

Announcer

I was a newbie navigating the tech jungle.

Announcer

When it came to making my 1st change to my code base, my boss mentioned I needed to commit my changes to CVS.

Announcer

They say not the pharmacy. For all you new folks, CVS is a, version control system that's not Git.

Announcer

Instead of drowning in CVS confusion. I had a light bulb moment.

Announcer

Let's move our code base to GitHub.

Announcer

My boss gave the knob. He was planning to make this move for years now but did not have the experience himself.

Announcer

Timing, they say, is everything.

Announcer

A weekend later, our office got hit by real world thieves who swiped our servers. Oh, no.

Announcer

Rid Little did I know our CVS server held our entire code base.

Announcer

I became an office hero, sans cape. I'll never I'll forever be the code savior who spared our code from literally being robbed.

Announcer

Oh my gosh.

Announcer

Rid. Yeah. That is, off-site backups. Man, off-site backups are a thing.

Topic 16 22:55

Wrong database connected, charged cards repeatedly

Announcer

CVS, by the way, stands for concurrent rid Versions systems. Concurrent version system. Concurrent versions system.

Announcer

I I've never never used anything other than rid get myself. So, shout out to all of you folks who have had to deal with non Git version control. Whether it's Mercurial And

Wes Bos

what does WordPress use? SVN? That's the big one. Subversion was the big one before Git. A lot of people still use that. And then I know, like, Google and Facebook have their own versions of it because it's much bigger than Git can even handle.

Wes Bos

Next one we have here is called Lorem Sale. Okay. So this one time back in the day, I was showing a new dev around a dev instance on a website we had built and managed For massive national airline. I love how people have to redact. And it's kind of funny because the people that emailed it to me, if you look at Where they've worked in the past, it makes these even more like, oh, the site also ran Company's APIs. It was a Drupal monolith. The site is basically a sales funny funnel for the company's separate booking engine site. Rid There was a feature where you could put a sale entity live, and it would push a notification to all of the company's double digit percentage Of the national population.

Wes Bos

Double.

Wes Bos

So, like, even 10% of the country at at at a minimum.

Topic 17 24:20

Turned on unfinished feature, customers got free items

Wes Bos

Anyways, showing the dev around, I put a sale live. Keep talking.

Wes Bos

Put it off live then on again.

Wes Bos

Rid As I'm talking through how it works and what happens when a sale goes live. So I just kinda explain, alright, you turn this thing on and off, and this is what happens if a sale goes live. All of a sudden, I get a handful of push notifications from the company's sales app indicating a sale went live. Ugh. Client Call incoming.

Wes Bos

As it turns out, the ops guy had refreshed in air quotes, rid. Dev with prod without changing the DB connection script back to the dev DB. So the dev site was connected to the production database, And I was putting sales live with the dummy data in production.

Wes Bos

Still one of my worst mistakes.

Wes Bos

1,000,000 push notifications to customers saying Lauren Sale is now live.

Wes Bos

A few points of policy were developed.

Wes Bos

Oh, yeah. How much is sending a 1,000,000 push notifications cost? 1,000,000 push notifications.

Announcer

Rid Do do push notifications cost money? I don't know if they do. I guess it that's a good question. I don't think they do. I think it's just SMS. I could be wrong because I I don't work that much in push notifications. Frankly, I find them to be awful. I think it depends if you have your own push notification

Topic 18 25:32

Autocomplete mistake deleted directories

Wes Bos

service or not. Of problems. Like, if Clari is doing I mean, something this large has their own infrastructure.

Announcer

Hokey. Yeah. Yeah. Hokey is right. If you get a Hokey out of West, That's one hokey. Oak. That's that's pretty important. Yeah. Hokey. There might even be a couple hokey doodles in here. Yeah. We get into hokey doodle territory. You know you know it's serious biz.

Announcer

Rid I wrote a bug that sent an SMS to a customer of my client 2,000 plus times in the loop, And we took 3 hours to find out. So another massive amount of notifications. Although, As we just mentioned, I think SMS, this one's gonna be quite a bit more expensive. Holy cow. Imagine getting 2,000

Wes Bos

rid. Push notifications to your phone or text messages. That's a mess. That's that's wild. Like yeah. What what does that do? When I sold my stickers, rid. I got, I think, like, 5,000, 6,000 notifications in, rid. Like, 6:6 hours or so, and it was just constant. My phone battery was dead. Oh, okay. Yeah.

Wes Bos

Crazy.

Wes Bos

Rid. We, I actually strongly opposed myself to that, shipped hundreds of kiosk hardware and software in the US From France without any remote control solution, boss had to ask a cousin living there to travel and check rid For a crash computers and to manually reboot it.

Wes Bos

That would be so scary to me. Like, imagine You are pushing an update to ecobee or some sort of hardware that is in someone's thing. Like, it has to download the firmware, put it on itself, and then reboot itself. Like, what happens if you break it rid At that point, you know? And, like, hopefully, there's a recovery mode, but then you got support telling people how to do recovery mode. But if they're rid. If you can't get into recovery mode or you need to do some weird, like, USB stick thing, can you oh, that's sweaty moms doing that kind of stuff. Hardware is another level. Yeah. I know. Whenever it's like

Topic 19 27:24

Shipped hardware with no remote access

Announcer

whenever you you hear an alert of, like, iPhones are being bricked or this is being bricked, The first thing I do is think about all those poor devs. Whoever whoever pushed out the software that's bricking hardware.

Announcer

Okay. Next one. While at a booking engine company, I made a change on prod with no staging or Git back then just as the boss walked in for a meeting.

Topic 20 28:30

Semicolon typo crashed production site

Announcer

Afterwards, I found out that I had missed a semicolon and crashed prod For a whole hour, losing 1,000 of euros.

Announcer

Broke out in a cold sweat, offered to resign, and he said no. Yeah. We've all been there. Rid. You know what? If you haven't taken down prod at some point in your career

Wes Bos

I took down Are you a real dub? Instant Tax podcast feed

Announcer

rid. Yeah. You did. With DNS or what?

Wes Bos

You know what I it was? I was using Cloudflare to do the syntax meetup URLs.

Wes Bos

And I made the redirects. I was like, these aren't working. I was like, oh, it's because we're not you. I had Cloudflare gray clouded because we weren't using any of the other Cloudflare features. We were going straight to Purcell. Okay. So I was like, oh, we need it on. So I flipped it on and I tested the website and immediately the website broke. And I was like, I know what this is. You got to like the CloudFlare default is like flexible SSL, And that causes the HTTPS redirect loop.

Wes Bos

So you just flip it to full strict SSL and it fixes everything.

Wes Bos

But I had failed to remember that we also host we don't host it, but we proxy the RSS feed at feed syntax.

Wes Bos

Fm so that if we ever change podcast providers, we own the URL.

Wes Bos

And something happened there with the SSL, And it was doing the redirect loop.

Wes Bos

So I Quickly threw a Cloudflare rule on there that says if it's the feed, change the SSL, the flexible, and that it fixed it right away. But our podcast was a couple of hours late and from going out, I was worried that Spotify wouldn't pick it up, but literally, like, 3 minutes later, it popped up in our feed on Spotify. So it was smooth.

Wes Bos

Rid. They they must another weird thing, like, talking about RSS for a second, is that Spotify downloads our RSS feed Probably every 5 minutes. Right.

Wes Bos

And it's 9 megs, and we have 600 episodes.

Wes Bos

RSS Podcast feed doesn't get paginated. There's no spec for that. So you think about how much bandwidth these companies do. Just rid Parsing RSS feeds must be unreal. Yeah. It's not like we're the only podcast with 600 episodes. Oh, there's a lot that are are much larger. All right. This next one again, I'm keeping them anonymous, but this guy works for a very large ticketing company.

Wes Bos

Rid. I charged credit cards and updated balances within the same database transaction, all inside of a batch job with retry logic.

Topic 21 31:20

Charging and updating card balances together

Wes Bos

Customers had their credit cards charged dozens of times until they eventually hit their credit limit. Created a terrible mess, But a few critical learnings to retry logic that kept retrying.

Wes Bos

Rid. Oh, man.

Wes Bos

Yikes. It can you imagine checking your credit card and seeing rid $10 worth of charges run up. Yeah. No. Thank you.

Announcer

That is, Yeah. And the last thing you wanna do is have to get on The phone and do chargebacks for all that stuff are oh my gosh. Yeah. I swear I didn't order $10 worth of one thing over and over again.

Announcer

Next one. I accidentally I accidentally implemented an infinite redirect loop that sent a 110,000 emails to a group of about rid 6 users causing their IT department to think they were under attack.

Announcer

6 people got absolutely

Wes Bos

Buried under a mountain of email. Oh, man. That is and if they host their own, like, rid. Mail servers, like, they're probably choking under all of that. You also can't

Announcer

you can't put that one back in the bag. There's no one due for that. Undo a 100 and like, yeah. What do you do? You're just gonna have to swipe them away or whatever rid Oh, man. Even I I was curious. I looked it up. A 110,000

Wes Bos

emails depending on, like, what they're sending. But if you're a medium sized company, It was $100 and just sending emails.

Wes Bos

Next 1, 1st commit at a new company brought down the entire site, rid. And I had just boarded a plane for a cross country flight.

Topic 22 33:05

First commit brought down site before flight

Wes Bos

How how many times rid. Have we heard this where someone says, I did something on a Friday afternoon, and then I became totally uncontactable?

Announcer

And, turns out I pushed to prod, and then I went to get into a submarine to visit the Titanic.

Announcer

Next one here. Back in the day, I created an augmented reality game in Flash AS 3, that's ActionScript 3, for a customer to use in public places such as malls.

Announcer

Every winner receives a small prize. All of the data was already randomly generated in an XML with the exact time when people can win. And I was saving the index to the last 1 item into a shared object, which is like local storage or cookies these days. But the problem is is that you need to call Flash for the data to persist on the hard drive, which I obviously didn't.

Announcer

So whenever the computer crashes or they take a break or they reload the app, the index will be reset to 0, And all the previously won prizes will appear again. So the customer is handing prizes all day Long. Yeah. A lot of big winners that day. Hey.

Announcer

That one's actually good for the user for a change. All of these ones, like, They they cause havoc to the user, but this one, hey. Free prizes. I'm pretty sure in in Canada, we had

Wes Bos

roll up The Rim, which is Tim Hortons. You roll the rim up and you see if you win, like, a doughnut or a car or something like that.

Wes Bos

And they have rid. Stupidly moved it to some app based that you you do, like, a fake role. During COVID, they nixed it. Now they have an app so they can track you and rid. Oh, stuff is awful.

Wes Bos

It's like the one, like, nostalgic thing I have about Tim Hortons. But, ready.

Wes Bos

At one point, all these people were winning. I think, like, I think it was a car or something significant, you know, like rid. 1,000 of dollars of prize at TV, and, they had to, like, claw them back because they were accidentally, rid. You know, like, for every one of these stories, people will tell me there's probably 10 more that people will never they're tight lipped about.

Wes Bos

Next one, a $20,000 hour. I borked a logical expression in a condition rid. For some credit card processing and caused easily $20,000 of damages in a few hours by making everything free.

Wes Bos

Rid. If you're not if you're not following along, an if statement was goofed up, Probably some sort of multiple and or the parentheses in the wrong spot And literally

Announcer

everything. It would be very interesting for somebody to tell tally up the amount of dollars lost over every one of these stories.

Announcer

Oh, okay. Next one is 3 years ago, I did a major release, pushed hundreds of commits from Staging to production after extensive testing.

Announcer

The site went down for 3 days. Oh. I had no idea what went wrong Like a needle in the haystack. I mean, that's the worst feeling. Right? Things down, you have no idea why. Then I found a single line regex was bottlenecking All queries.

Topic 23 36:25

Regex bottleneck crashed site for days

Announcer

Did not sleep for 48 hours straight trying to look for the issue. It was a different time and place. Rid Man. Man. Yeah.

Announcer

48 hours.

Announcer

Just I you know, imagine, like, the amount of Stress you're under in that amount of time. I've I've I've had some sleepless nights where I'm up coding trying to fix a bug myself. Yeah. And there's really not a whole lot. Like, You know, your your partner can be like, hey. You doing okay? Is everything okay? No. It's not okay. I'm I'm on fire.

Wes Bos

Rid. I I don't wanna be a a century, Schmuck here, but Yeah. Slow DB query detection, you know? And like, there's lots of tools out there that will like I'm not sure if the regex was in the query or if the regex was somewhere, but It probably was something weird where someone rejects a piece of text and that text was not rid. You can't do that just in the DB. So it probably had to query every single record into memory and loop over them. You know, That is crazy.

Topic 24 37:35

Truncated production database accidentally

Wes Bos

But, yeah, you certainly should have some tooling around telling you when rid a query and what the query is that is slow.

Wes Bos

Next one. Accidentally truncated the production database one. We had a weird setup where we needed to connect the fraud because debugging was near impossible.

Wes Bos

So, yeah, that happened. We changed all the rights of the database User so we didn't have the problem anymore.

Wes Bos

Brutal. And then somebody followed up, said, didn't you have a bunch of medical equipment delivered to a customer At one point, and he says, yes, lots of bed hooks. However, that wasn't my fault. Someone made the test test form point to live.

Wes Bos

That one like dropping a database. Yeah, we've heard it. But like we're customers.

Topic 25 38:25

Wrong form sent medical equipment to customers

Wes Bos

We had that one with toilet paper a couple of years ago. Somebody literally shipped, like, like 15 bundles of toilet paper to somebody's random house, and it just started showing up. When product actually arrives at random people's houses, that is just that's hilarious to me. It's unfortunately hilarious, especially if it's large amounts of things like toilet paper.

Announcer

Next one. I once made an off by 1 error writing a high efficiency load balancer for a platform as a service So all domains served the wrong traffic domain sites index for food.com was actually served rid Sites index for Food.com

Topic 26 39:08

Load balancer served wrong sites

Wes Bos

plus 1. So you can imagine, like like, Netlify is a platform as a service. I don't this probably wasn't Netlify. I don't know I don't know who submitted it, but imagine Netlify had has a load balancer, and every single request They get in for westboss.com is off by 1. Is off by 1, so I get, like, diapers.net.

Announcer

You know? Yeah. Oh, yeah. Diapers.net.

Announcer

Rid Oh, yeah. There was a I I know this isn't the same thing, but there was, like, a thing with, like, Steam, the gaming platform, Where they have, like, a similar issue that was serving up the wrong cache to the wrong user, and it was a similar bug, I believe.

Announcer

So yeah. Broke everything, and people thought Every site had been hacked or discontinued for a day or so. I was ex was extremely confusing as it wasn't a simple index lookup. Yeah. No kidding. Yeah. And, also, you know, you give people somebody else's anything, and, like, that that that hurts customer trust pretty badly. Right? Yeah. No kidding.

Wes Bos

Rid. In 2007, the author, who was 15 at the time, worked as a freelancer, and developed a web store for a local music company.

Wes Bos

The client's tech wizard son who served as the project's reviewer and administrator inadvertently removed removed exception handling rid Handling.

Wes Bos

So causing an error message containing the database credentials to be displayed on the website's home page. Rid Unfortunately, this error was indexed by search engines during a maintenance window, exposing sensitive credentials in search results.

Topic 27 40:46

Leaked credentials indexed by search engines

Wes Bos

Despite The Sun's responsibility for the issue, the client unfairly held the author accountable, Demanding a refund equivalent to the amount paid for the freelance work. Regrettably, due to their youth fear and lack of knowledge of their rights, the author rid complied with this demand. Subsequently, the author never had any further contact with the client. Oh, that sucks. Bro. Oh, that's not even funny. That's sad. That's sad. You know what sucks about that is, like I mean, you think about the horror of committing

Announcer

Your secrets to GitHub. I mean, this is like I mean, GitHub's permanent. You can just, like, GitHub, change your seat, but to have it indexed by search engines. That's why it is so hard to

Wes Bos

like, in remix, Next. Js, Gatsby, all these frameworks, For you to explicitly put an environmental variable into your template, you have to prefix it with Next underscore because they don't want anything ever accidentally happening where you possibly leak, especially with these, like, full stack. You're not sure if your back end or front end, you have to explicitly put them in To make say, yes. I want this to be on the client side. Yeah. It's the same concept as behind

Announcer

dangerously set inner HTML or whatever. It's like, We're putting these words here or putting this extra step in here so that you think about it. Just so you you think about it. Rid. Alright. Next one here. I just put a note here. I've heard this story a several times over the years, and this is This is a cautionary tale. I wasn't sure if this was an author's note or a West Boss note. So this is a West Boss note. That's a West Boss note. Yeah. This is a cautionary tale. A few years back, I SSH'd into a prod server. And when trying to delete a temp directory, I accidentally executed r m hyphen r f till date forward slash. Till date forward slash for those of you who don't know, it's the home directory.

Announcer

The temp directory was supposed to autocomplete, but before it did, rid. But before it did, I pressed enter too soon, and the data was gone. And, you you know, people Keep all sorts of stuff in their, home directories.

Announcer

Could be keys. Could be I mean, some people put, like, sites and stuff. The entire computer is in tilde.

Announcer

Rid. That's the root of the hard drive. Oh, the So No. Till day till day isn't the root of the hard drive. Forward slash is the root. Sorry. It's the root of the user. Rid. Yeah. You're a user home. Yeah. And especially if you have, like, a droplet or something, I never put anything in my user directory. But I know what I'm saying is I know some people put a lot of stuff in their user rid. And on a, like, a Mac computer, there's a lot of stuff on your user directory, including system files. But yeah.

Topic 28 43:01

Home directory deletion cautionary tale

Wes Bos

Rid Woof. Yeah. If you were to do that on a Mac, your entire computer folder. Would be gone. All of your apps, all of your data, All of your set like, pretty much the entire computer unless somebody else had another account on that computer as well, but that's rid. That's unlikely.

Announcer

Yeah. Totally. Oh.

Wes Bos

Oh. That is it for today.

Wes Bos

Thank you, everybody, for submitting your spooky stories. If you Still have a spooky story. A lot of the ones we have here are from last year. People submitted them after listening to this thing. So send me an email, wes@wesboss.com, And I will put them in the queue for next year. We love doing this, so please send us your horror story.

Wes Bos

Yeah. If anything, The world can learn from your mistakes,

Announcer

and, you can have a little chuckle. I think that's the key of this. This isn't just us laughing at other people's misfortune, But, like, hey.

Announcer

You know, what do they say? You know, people who know the History Channel, don't repeat the History Channel. You know what I mean? So, rid I I think that is an important important thing that we all can learn from each other's mistakes. We can laugh at these things. It is a unifying aspect of being a software developer.

Announcer

However many times you you really think you have everything dialed in, Everybody makes mistakes, and it's important to really celebrate those mistakes in a fun fun, laugh y way where we all get to cringe together. So, yeah, That's it for this year's horror stories.

Announcer

Spooky spooky stories, all that stuff. My bones are rattling. I'm ready to get out of here. West, do you have a sick pick for us today? I do. My sick pick is

Wes Bos

100 pound magnetic hooks.

Announcer

Oh, wow.

Wes Bos

Rid. I've been using it for 2 things. So first, in the gym, I have all these attachments that I need. I got handles, I got bars, rid. I've got what else? Other like bands and things like that. And you need to be able to store them somewhere. Right. And The whole rack that I have is metal. So I bought these a 100 pound. They sell really cheap ones, like 25 pound ones.

Wes Bos

And I'll say, I don't think those are good for anything that you would be.

Wes Bos

They get knocked off all the time. I hate One of my biggest pet peeves in life is a weak magnet where things don't stay on properly. You know, come on, make this like, I want to be able to pry this thing off. I want to be able to pinch my finger in between it.

Wes Bos

So I got these 100 pound ones and they are awesome.

Wes Bos

So I literally am holding up like a I don't know. It's probably a 15 pound bar. It says a £100.

Wes Bos

I don't know if I believe that That's like Amazon. All this crap is just a bunch of lies, bro. Bunch of lies. I guess you probably have a story about that. I do. But They're super handy. And I also put them on my whiteboard in our kitchen and I hang my keys on it. And it's like the best little hook to hang keys on. You can move it around and Put them on the side of the fridge as well. So if you are looking for decent

Announcer

magnetic hooks, check these out. Yeah. At first, I laughed at your weak magnet thing, but then, like, as I was thinking about it, I bought, like, this, like, magnet based pop socket competitor.

Announcer

I'm gonna return it because it stinks.

Announcer

And my biggest complaint is that the magnet on it's weak, so it doesn't you know, it's not you can't, like, trust it if you're using it as a pop Socket, which the PopSocket branded ones actually do have a stronger magnet. So, yeah, you're right about that. My my, Amazon story is that I got a a waterproof speaker that was 300 watts on Amazon, and, it stopped working. And when I opened it up, obviously, the thing's full of water. There there's no waterproofing in it. In fact, the only waterproofing even being done like, there's no there's not even a gasket or a seal around the plastic, which, You know, you can't tell from just looking at it. So when I opened it up, I was like, alright. There's not even a gasket here. And the only bit of waterproofing was, like, quota was, like, a It was like a, really crappy kind of, like, cotton ball material in there that they were probably just hoping would suck up some water. I have no idea.

Announcer

And or it could have been an urge for prevent riling. Either way, there was not an ounce of waterproofing done to these speakers, and it said they were 300 watts.

Announcer

And the amplifier I I googled the board. The amplifier was 40 watts. It was a 40 watt board. And And it's just like I went to Amazon. I was like, listen. This is just completely misleading. It's, like, it's not that it stopped working. I don't care about that. Yeah. I mean, I do. But, like, rid. It's not even close to the product that it says it is.

Wes Bos

So much of that Amazon stuff are just, like, straight up lies, You know? And you can straight up lies. Yeah. Sometimes you see one with, like, like, like, the magnet hooks, and you'll see one for £105.

Wes Bos

You're like, yeah. Right. Rid Yeah. Right.

Announcer

Yeah. I know.

Announcer

I have a sick pick that is a documentary on Netflix, rid. And it's exactly my type of documentary.

Announcer

It is the mountain climbing, mountaineering documentary. You know I love those.

Announcer

And rid. This one is really fascinating because it's just like any of these other, like, really modern mountaineering documentaries, this is a 2003 one. It's called Race to the Summit. Just like any of these modern one, your jaw's just kind of on the floor the whole time.

Announcer

This one is about speed climbers, and It's about, like, 2 2 different climbers who are doing several different alpine climbs as well as Himalayan climbs, but they're rid. Trying to break the records to see how fast they can do them. And some of these climbs like the are, like, historically Very scary climbs that, like, take people a long time, and these guys are doing it in, like, a couple of hours. And is he's like, this section has ropes, but, like, If I really wanna move fast, I, you know, I I don't wanna use the ropes. I wanna do this entirely without ropes. So not only is he doing the scariest climb of all time, he's doing it Without ropes entirely and basically running up the mountain to the point where, like, there there's times where rid He's, like, walking around. I mean, there's 2 different guys. They're both crazy. Walking around corners really fast, and you're just thinking, man, I couldn't even you couldn't even catch me rid Walking around a corner, like, 30 feet up that looks like this. And he's Wow. On the side of a mountain, and he's basically, like, running around it. You're just like, rid. This is absurd.

Announcer

And, you know, with any of these things, you kinda have these wild kind of personalities. So the people involved in this documentary are super good. It's an hour 30, so it's a really nice nice watch. We put it on last night and had a great time watching it. So if you see this on Netflix, throw it on race to the summit. Alright. Thanks everybody for tuning in. Hopefully, you have a spooky

Wes Bos

Halloween.

Announcer

Head on over to syntax.fm for a full archive of all of our shows.

Announcer

And don't forget to subscribe in your podcast player Or drop a review if you like this show.

Announcer

Rid.

Share