Spooky Web Dev Horror Stories - PART 1

Welcome to syntax syntax. On this Monday Sanity treat, we're gonna be Tolinski spooky stories.

Yes, folks. It is time for our annual spooky story episode.

We're gonna be talking web development spooky stories. We have rounded up some of the best spooky stories from our community.

Maybe you dropped the database.

Maybe you sent a test email to 1,000,000 people.

Well, we're gonna be talking all about those things in this episode. But if your code is spooky, you best believe you need century at century.io.

Sign up and get 2 months for free. Wes.

Spooky.

Bones boss. What's up, my dude?

This is one of my favorite episodes. We do 2 of them every single year Wes you submit your spooky stories of just the most put your head in the sand stories of web development.

And they're awful in the time, but I'm glad that we can laugh about how it went and also learn a lot about how to, like, not get into the situations that these folks are in. If you have your own spooky story, go to syntax.fmforward/spooky and pop it in there. We collect them all year round, And then once a year for Halloween, we read them.

I'm gonna have to take this mask off because it's just it's cutting into my eyes.

I can imagine it does not look like a phone call.

That hurts.

Yeah.

Can you use favicons in URLs, like, as a URL path path name or not favicons emoji?

Emojis.

You Sanity a certain ESLint.

Be they to get converted into what's called punnycode.

I own a domain name that is just a 1 a 1 letter Node, but, yeah, you you can put them. Like, I have a fab.farm, and you can do forward slash any emoji, and then that turns it into an SVG or PNG and serves it up as a fav Scott. So you certainly can.

Yeah. So we should do that. We should have a little ghost emote. You could Yeah. That sounds like a annoying that sounds like a fun joke that will be a pain in the ass to have to support for the rest of our lives.

Hey. That's the best kind of fun joke here. Yeah. Let's get into the first question. Some of these stories are longer. Some of them are just 1 or 2 sentences.

We are keeping everybody anonymous for the sake of nobody getting in trouble.

And, there's a couple where we're able to say the company or at least tell you what we think it is.

So the 1st spooky story is Node A Coffee.

I made a website for a very big company selling coffee all around the world. It was a website promoting an online game where people could win a trip. In other things, the minimum prize for a coupon was 20¢ off a pack of coffee.

There was a QR code on every pack of coffee, millions of it sold around the country.

So this is this is pretty big. I was in charge of back and front end, I was supposed to connect to an API to generate coupons and send them to email to people who played the game. Supposed to. Interesting.

We launched the campaign, and everything seemed to be going well. Client was happy and so on. After a few weeks, the client called and said a customer JS complaining he didn't receive his coupon.

I went to the source code to check what was happening, and I looked at my send coupon email function. Oh, no. It was empty.

Totally empty.

I forgot to code it.

Client was mad. We sent the coupons weeks late to thousands of people and got some very angry messages in response. I never sweat so much in my life. Yeah. You didn't take the test that it actually sent a email?

Sometimes, Wes, you just think, I already did that. I already did that. Is that has that ever happened to you where you're just, like, alright. Now that that's Deno, and then you move on to the next thing, and you're, like, oh, it's half done. You didn't even really get there. That's that's kind of how I work sometimes. Just put a to do in there.

Yes. Oh, man. That is actually one of the big reasons why I started.

I know people poo poo did to dos. They poo poo to dos, but I have started just littering my code with to dos anytime that I'm mid project. Just so Yeah. I do a command f for to do before I push anything and then make sure, there is nothing that needs I know that's a whole controversial thing about that in general. But, hey, I I like to dos for that.

Alright. This next one is, from the founder and ex CEO of GitHub. This one is straight from Twitter. It was a gorgeous Sunday morning. Birds were chirping and squirrels were swirling. Coffee in hand, I began upgrading GitHub's testing infrastructure.

When I was done, I ran a quick test and deleted the entire database.

This might be the spookiest ever. Yeah. And there's been some spooky ones.

There was a lot of things that went wrong. Our tests should not have had access to production. Yeah. That's that's 1. Our production Node shouldn't have been wipeable. 2, we should have been able to restore the DB faster.

We should also have known our events table would be a doozy.

Man. But the main thing that went wrong, our GitHub application assumed it was running in production mode unless told otherwise.

When I ran my test, I forgot to set the test environment.

So a connection was established to the production database, which was promptly deleted.

That's a good way to put it. Yeah. Assume

don't assume production by default. That's a good tip. That is a good tip. Always assume either that it won't work or that it's connecting to a local or something. Yeah. I remember exactly where I was sitting in my apartment next to Dolores Park. The Wes rang quickly at that point. So when I started my test and it just hung, I immediately knew something was wrong. But I thought it was a connection issue.

I thought that's weird. Only when I visited GitHub Scott seconds later did I realize how bad things were. Nothing worked. As I explained in the blog post, we always wiped our test DB before running tests. Turns out people don't want their data wiped on a Sunday morning for no reason.

Everyone who was around helped out, but our Dogek really saved the day by quickly restoring the DB and punting on the events table. We also immediately locked down production so it wouldn't happen again. My main takeaway JS don't let anything access production except production.

Save server upgrades for the weekday when your team is working so they can help you.

It's a whole don't deploy and try anything.

Wes, deleting and restoring data regularly, and 4, it can happen to you. Yeah. If, it it is funny. Whenever I I do have a database and I need to do something, you know, major on it, whether that's a migration even though it's not that major, but still a migration or something where I'm modifying the database, I always make sure I click back up to, like, a rollback type of situation. I know that's a smaller scale than GitHub. But, like, for me personally, yeah, that is a a fear of mine. So I'm always always be backing up. Yeah.

There's something very poetic about GitHub having lost data. You know? The truest sense of it can happen to you for sure. Next Node we have here is called rejected. I once sent 50,000 people an email saying, thank you for your application, but we decided to move forward with someone else. Most of them didn't even apply.

Those are awful because you can't stop it. Once you send 50,000 emails, you're gonna get 3, 4, 5000 emails back being like, what's this about? What's going on? You know? And even if you send up a follow-up, hey. Sorry. That wasn't meant for you. You're still you just cannot stop it, and the poor support people are just gonna be slammed for days. Yeah.

Yeah.

That that stinks. I I may have told this before on the show, but I had applied to Y Combinator Startup School, which Wes, like, a really you got a lot out of that program if you were accepted, and they were accepting, like, 20 people or something.

And they did that, but it was the opposite where they admitted everyone. So I got the email saying I was admitted to Scott up school. This Wes back in the early days of LevelUp Tutorials, and I was so stoked. I told everybody in my my house right away. I Wes, like, really? And then sure enough, I got the email, like, 15 minutes later. I'm so sorry, but we accidentally sent that out. Like, that's yeah. That was rough. No. Thank you.

If you're rejecting or accepting people for something important, yeah, be extra cautious. Careful with people's lives.

Yeah. Careful with that. Yeah. Infinite loop. I once sent the same email and text notification to a single user 200 times within a few minutes on both channels.

Damn those infinite loops.

That's a hell, like, f you in particular, type of move right there. That's hilarious. Oh,

that sucks.

Next one is my squeal horror. During my 1st years as a software developer, I had to rewrite a login register form for one of our biggest customer CMS portals. Part of it was the password forgotten page Wes I implemented a whole submit your email and reset your password flow. I then went on to write a nice little SQL query to reset the user's password whenever they requested a reset link.

We released the whole thing, and everybody was quite happy, especially me JS it was my biggest project to date. After a few weeks, started getting emails from users telling us they had to reset the password every time they wanted to log in to the CMS.

I had forgotten the user reference in the where clause.

It caused a reset page. Every time a user resetted their password, Wes were actually resetting the password for all users of the company's account.

The the fix was quickly implemented and deployed to reset the passwords once again, and everything was fine. Since then, I triple checked my Wes causes.

This is awful because, first of all, that's a security issue, meaning that you could set reset your password, and then your password is then the password for everybody's account.

Yes. And, like, honestly, it's kind of like a a scary part of SQL Wes if you forget the where, it will literally update every single thing in the database. And probably we'll we're not gonna read all of them, but we've probably about half of the stories we got submitted were stories where this had happened. Someone forgot the where clause.

Yeah. Classic.

Next Node, pnpm company. This could also be pet cemetery. That's a scary Oh, that that would have been a better title. I like that. No. It's alright. Forgot a Wes clause on an update and changed every customer master record to redacted large pet retailer.

It was my 3rd day on the job as CIO.

I almost got in my car and drove away. Luckily, there was another field that had roughly the same information, ran a second update, and told the CEO that the quality of data in their database was very poor.

And I asked how they ever did accurate sales reporting. It's like a master yeah. No kidding. Right? Started a master data management initiative. Wow.

Spooky.

Sucks. It those are some of the worst stories we've had over the years where people had to piece things back together with Oh, yeah. Alright. What do we have? Like, often, we've had people go through logs, and they've been able to piece it back with logs or some other piece of information they've from in another field, they've been able to figure it out, but awful. Awful. For real. Get corrupted. I once lost 3 weeks' worth of work without noticing.

Oh my gosh. 3 weeks while using version control.

The reason I later discovered an automatic r sync backup, with my NAS was corrupting my Git folder, which wasn't excluded due to a system date set in the future on my NAS. I pushed a remote 10 times a day now. This is this is honestly one of my fears is even if you are git committing often locally, it's still only local. You know? You you push it somewhere else. Or Vercel Node now has that, like, automatic backups. Even between commits, you can lose stuff. So scary stuff.

Yeah. Scary stuff indeed.

Next Node, circular horror.

We had to deploy for the send money app at PayPal Wes the server memory would spike almost immediately and then restart the server over and over.

In seconds, 100 and then thousands and then tens of thousands of users were affected before Wes rolled back successfully.

Serialization logic with an out of memory error.

Added a custom ESLint plug in to prevent that from ever happening again ESLint addition to handling circular references better. Man Oh, man. Have you ever logged, like, a giant object and even your local host is, like, struggling with it? Yeah. Yeah. It chokes or even, like like, some things if you try to JSON something, like in SvelteKit,

if you try to send specific objects back, it catches it and says that's not a POJO, which is a plain old JavaScript object.

But sometimes, references like, objects aren't just data. And if you try to serialize JSON stringify an object that refers to another object and it's circular, then you can get into some big trouble there, especially when it's you're writing your own log serialization logic.

Yeah.

Jeez.

Man, these are great. Next one is the haunting of the forgotten MX records.

The first one is simple. I moved a client's name servers to my hosting provider and forgot to add the MX records to Google Workspace.

I moved the domains right before a holiday, and we only found out about it days later after his client started messaging him wondering why he is not answering his emails. Oh, 2 years later, I had to close the account again, and I forgot to move the name servers.

He did it again.

It was a holiday.

It was a holiday again, and it took 3 days until it was resolved. Both times, the client didn't have the credentials to the registrar, so it took a full day to track down the guy who originally set it up.

Oh. That always scared me. When people, like, move, like, oh, we're gonna launch our new website, what people would often do is they would just change the name servers of the domain, which changes all the DNS management. And if you have anything else on your domain name, like MX records for email or, SPF authentication for sending one off emails, transactional emails for spam detection, like, you can get into some big trouble. I I once did it where I had moved a domain name, and I thought that I had deleted all of their email records. And it turned out that because I had moved it from their hosting provider to my own, and then their hosting provider deleted all the email addresses in the cPanel. And I said, oh, crap. And I was I was sweating for a couple hours, and then I realized the lady's like, we're still getting emails.

And I realized, oh, they actually weren't using the cPanel email. They were using external MX records. So, thankfully, I was spared there.

I yeah. That stuff always freaks me out. I I know so much of it JS not a big deal, but, yeah, I email in general just because I know how to Never wanna goof it up. Yeah. Between spam and Node spam. That that's a that's a hard hard hard task there. Man, these these have been well, this is a good good kickoff, Wes. I'm, like, pretty stoked about Yeah. These spooky stories here. Well and and folks, if if you if this is your 1st spooky stories episode, we have a whole hour of spooky stories coming for you on Wednesday.

So we're gonna continue the spooky stories in part 2 of the annual spooky stories episode. And, again, if you have any stories of your own, we want to hear them. We will read them on the show next year and share Node delight with all of the horrors of the things that we do at work all day. Alright. We'll catch you in that one on Wednesday.

Spooky.