Spooky Coding Horror Stories 2023 - Part 1

competencies for their degrees.

It was an exhausting few weeks working around the clock to rectify this monumental screw up. I have to say discovering that your Pull request was the root cause of the problem after opening a ticket searching through the logs and finding the issue Yeah. Was the worst feeling I've had in my professional career.

After processing about 500 students, we encountered a rate limit issue.

As my colleagues and I tried to unravel the mystery, we discovered that the system actually required those duplicates to award historical competencies

Pushing data to a national registry.

Yeah. Yeah. I'd say that's And, like Like, possibly people's lives. You know? Like, if someone's degree I'll I'll tell you right now for everybody looking for lessons learned on this one is sign up for would have fixed that. Although, in the next episode,

I I intentionally didn't read this one because I could Tell that it was very well put together,

You're like, oh, well, I just moved to Denver. I can't go back. I can't go back. I had a a nontechnical Oh, I know. I know what that is.

Probably for nothing. Just yeah. Who needs them? Right? Okay. We gotta clean this up. So we pushed it to production. After another review, I Celebrated my 1st professional poll request had been deployed.

They admitted they should have provided more oversight.

I deployed a disallow everything

was to be awarded to learners as they progress through their degrees.

On the surface, it might not seem like a big deal because we fixed it, but the worst part about incidents like this is knowing that your code caused significant

was to add new competencies Alright. Next one. The embarrassing test page incident. Many years ago in my 1st web developer job,

Had our client review and approve the differences

initiated

like, a a project where I I created a bunch of music for him, and he just gave me the credits I needed. So it worked out.

And reintroducing the duplicates to the lookup table was straightforward, but now we needed to write a script to review approximately 60 50,000 individuals in the system ensuring they had received the correct national Yeah. Pushing production to learners.

are like, I broke something, and we immediately realized how awful it was, and some of these stories are like, we realized on a slow burn, you know, over 8 months. This next one I'm all done with it. Slow burn.

them entirely to clean it up in quotes due to the substantial

to this table for learners.

Oh, that our integration used to determine what national Being blissfully unaware.

Expired? Poor girl this URL

gosh.

Okay. So the duplications

We had a script that converted a CSV into a structured JSON tree

My responsibility

scrub. My My, adviser in college once told me that I had enough credits to graduate, and then, like, 2 months before graduation, he was like, oh, oopsies. You were you're missing this one thing.

Without a complete understanding of how it worked, I noticed duplicate entries in the lookup table. I discussed this with a colleague, and together, we decided to remove

students had incorrect competencies awarded to them, and some even had their degrees frustrated.

8 months later, I received an email from our client stating that over Yes. Right. Yeah. You see the word national in any of these, and it, definitely gets your ears perking a little bit.

Regardless,

and then nervously

As it turns out, about 7,500

But some of these

After in the end, we resolved the issue. I didn't get fired, though my managers were understandably

Competencies had a meaning. Right? Believe it or not.

stories

to NER.

aspire to missing information.

3,000 400 files were deleted wrongly instead of tens of thousands, a blessing in disguise. Okay.

disruptions to other people's work weeks.

Increase in lines of both the CSV and JSON files. The differences between the 2 became significant.

Yeah. Hey. Probably nothing.

When it came time for my review, my manager Glance it over without much concern. Alright, folks.

Reverting the changes Needless to say, they didn't like me anymore.

Only me and the QA leader are aware of this, and thanks to her, she had not told this to the morning stand up the next day. The next day, I wrote a Bash script that double checks the git branch before pushing an update to production environment. And since then, nothing like this has happened as the script Access and intermediate. Yes, folks. This is why people often have,

to 100% Is it 2,000,000?

Wait. That's 10,000 So I went into the bank. I was like, look.

of users. Hey. Nice. Because you can do a, stage rollout on Android like that. I'm sure you can on Ios. Luckily, we realized this soon as in the production testing, users were automatically to be logged out, or anything after,

emails to 200 people each. Oh.

continuous deployment, continuous integration setups because A few days later, we found out about the wrongly deleted files. Thanks to that early rollback,

The rollout was also

They do this stuff for you. It's like, yeah. If you push to main, it will run production. Now I don't understand exactly how that would work in an Android world. I don't do that kind of stuff.

dev to production is definitely something I think a fair number of us have done at some point in our our careers by accident.

And the user does not exist in the staging database. Token return error. I checked the branch name and it was developed.

pushing

I still remember this moment.

We immediately halted the rollout the integration to correct the students.

and pushed a new update with the production code to all users. It was the worst day of my life as developer.

And so so I took the

by far the most stressful part of the job. You're you're you're walking out of the bank. You're filling out that piece of paper and having to do it, like, once a week. I remember the 1st time I did it. I felt like,

We were preparing for one of their peak days and needed To make some changes to their payment gateway integration Brain brain the way Braintree works,

And I walked to the bank and I sat down and it was just like, I had never seen this teller very young, and I sat down and I said, hey. Like, I need to wire $10.

everything was fine. But the next time I had to go in, I got the same teller and she was you could tell she was just like, You know, like, you you plan or or if you're like, oh, do I have all the credits? You log in. Oh, I got I got them all, and then they're clawed back.

Oh. To a random and a Chinese bank account that I do not know.

3. They're making sure every and, like, it's so funny because a copy paste for 3 months. This is for a huge ecommerce platform, and it tanked so I can't stop laughing and that others would have to stay late on Friday. That's not funny at all

The little piece of paper that they give you when you make a wire transfer at the bank. Yeah. I I scanned it in. I sent it on. Like, look. This is proof of payment. He said that's the wrong account number. So I looked at it, and I go, oh, no.

And I was wiring it to China. So, like, I wasn't wiring this to another Canadian bank or maybe in the States. I was wiring it to a bank that you could not read the name of.

Alright. Last one here. I was once working for an agency on a high profile customer.

know? quick little if statement in there, but Who knows? Yeah. Woof.

We didn't get anything. Like, we contacted the bank. Mhmm.

Like, it took, like, a week and a half. So there was, like, a week and a half of just, like, did I just send $10 into the Yeah. Into the ether? It's gone forever. It's gone forever, But luckily the bank returned it and the first time I ever did stickers,

I just sent

a, yeah, a great one for the headline there. 60,000 copy paste mistake. I used to have to, when I worked at a record label,

That's 10,000.

oh, I cannot believe this company is putting me in charge of sending out their royalties like this.

7, 4, Can you triple check this? You know? And we sat there, and we're like, quick fix that. That takes forever

royalties out all the time, and that was

I used to have to wire files was a blessing in disguise because the alternative is is is much greater than that. It's 10,000.

$10 We received a support ticket reporting a concerning issue.

all of a sudden, I don't like, a couple days later, I asked the printer, hey. Did you get the money? And they said, no.

and hits the submit button.

copy paste mistake. So listen to that one.

was shaking.

we have a $60,000

And then on Wednesday coming out, we have the big one, which has some

some doozies in it. You certainly don't want to miss that one either. These are going to be that one to me had me rolling. It it, yeah. No. It gets really bad. $1,000,000 in, like, potential revenue. Like, literally He lost $1,000,000.

soft skill, web development, the hastiest, the craziest, the tastiest web development treats coming in hot. Here is Wes, Barracuda,

Boss, and Scott, El Toro

Oh, syntax.

Am I talking about spooky stories? Double data? I I don't think it's running anywhere. I'll just unplug it. Alright. I mean, I don't need it. That's like when I build something, I have all these extra bolts

On this

fantastic stories of The printer $10,000 Because the URLs were whatever. And luckily,

Loco,

Wanting to stick your head in the sand, What issue?

Monday

ways people have goofed up in tech, deploying, deleting databases, all that

was working on-site updates for one of the largest furniture retailers in the country.

because maybe you're frightened. You it's Spooky Saturday, and you pushed a code on Friday, and, you don't have error in exception handling tracking. You gotta have that. So check it out at century.i0.

Wes, oh, I like that. Yes. Thank you. Freestyle. Alright. So we have 2 shows for you. This is gonna be the first one, a little hazy. Just a couple. We're going to get you wet your whistle.

history treat,

but

to the depths of

This is part 1. This is the hasty version of spooky stories that we're gonna be telling on the podcast where basically people write in. They talk about Spooky dev stories.

We're talking about Spooky dev stories. This is one of our annual episodes where we do where we talk about spooky stories. I'm not gonna do the spooky voice the whole time, Fatal flaw number 1 here.

we'll be sending you Monday. Monday. Monday. Open wide dev fans. Get ready to stuff your face with JavaScript, CSS, node modules, barbecue tips, get workflows, breakdancing,

online, from that point on, I implemented stricter review protocols

screw up, But, like, it's it's people's money at and, like, that is such a human mistake

accounts and banks, right? So

Yeah. Frog wine. I once made a, even if you turn it off, There's it's still in caches around the world, and they still ripple through. And it's one of those things where it sets off a chain reaction. It can be impossible to put the genie back back into the bottle. Yeah. I was still getting messages about the episode that didn't work, is automated test of testing every single make sure that it logs in. Make sure that it incorrect password doesn't work. Make sure that a blank password doesn't work.

Yeah. Right? Can you imagine manual data entry of these bank account numbers, and then you accidentally GDPR deletion of customers for a few months and was ready

So I had to take the invoice and the payment details from the printer.

you know, you you just goof up looking at the wrong? learners nationally were affected.

not only was this a major times 200 people is I don't know how much, but a ton Wait. Ton of emails. 10,000 times 200?

I had to wire brutal, brutal, brutal. Hopefully, it was just a, like,

And fang type companies.

That is how people in the US send Money between

updates to. However, I had no null checks in the JavaScript bundle to run on every other page. So if the element didn't exist on the pages, JavaScript would break.

I was still very inexperienced and wrote some JavaScript. We worked on the page I was making Online sale of the year.

or roll back a deployment.

and this issue was not Notice until late Friday afternoon.

Very common thing. You got to check if the element exists there before you go ahead and use it. And of course, it runs if the element's there. But if that JavaScript runs on another page, thing will break. Somehow this code passed PR review in internal testing as they were only testing the page updates had been made on.

developer, and my manager said I must be wrong. There's no way something small as a missing null check

to attempt to resolve

Uh-huh. I realized what had happened and told my manager, but I was a junior those things

What's worse is the deployment to production happened on a Thursday,

across the site right before the retailer's biggest breaking JavaScript

So the code was deployed to production completely Oh, woof.

thank goodness, they're all web developers, so I was able to say, hey.

Replace local hosts with to print them all.

Not having smart customers you know, but they won't actually charge your credit card anything. Depends on who the provider is.

Oh, this is this is classic. Oh my gosh.

sprang into action and looked into the changes I had shipped.

And don't forget to subscribe in your podcast player or drop a review if you like this show.

Oh. Man. Hey. Guess who's done this? I'm raising my hand right now. I used to, awful, awful stuff. We should talk about what you do if you're scared, We created this script to simulate these updates,

Well The 1st $1,000,000 mistake 1st $1,000,000 mistake is syntax,

I don't I've I don't think I've ever done that. I've almost accidentally done it. I I think we had this last year. Somebody submitted a a thing where they deployed it. And if you put real credit cards into a test environment, for an item, it will cause clients to automatically redownload the associated

you actually have to have a specific You're missing. So I did a a special, Dev stories that are very spooky, spooky, spooky, spooky stuff like dropping the whole database or maybe charging 10,000 users at, You know, who knows? After negotiation because, yes, they weren't checking the end.

when I before I release a course, I usually let, like, 100 people buy it and and

We let it rest and waited a few days before shipping it to the website. Next morning, we deployed and started working on other features.

They'll work, and you're good to go. But so I didn't read this one. And I'm cringing at the very first thing, which is 2 systems put together You

imagine that times

Alright. Well, buckle up, folks. It's about to get really spooky on Wednesday. So Thanks so much for listening, and we'll catch you on Wednesday. Peace. Peace.

What you will get on Wednesday is a barrage of of some of the spookiest stories we've ever had on this podcast, including,

and they weren't but they had paid real money for it, but they weren't able to, like, activate it

one of them that cost,

again.

so you can hit me up at a syntax email. That's pretty sweet. I I do wanna say, just like you said, Wes, at the jump of this episode, this is just a taster for what you're about to get on Wednesday, because

Yeah. Nightmare. So if If you enjoyed this, hopefully you're not too stressed out. We got another one coming for you on Wednesday. Some doozies doozies in there, so make sure you tune in to that one as well. Please submit your own. We'll we'll put them on the show for next year.

the moon.

Who couldn't pay? We If this is, like, a a public company that's, you know, money off

I had made a hard coded switch To make sure I was testing with the test gateway, which I had forgotten to remove again. OMG.

and they all got emails with, like, a local domain in the email,

Working on the payment integration

is Always scary. Very. But, yeah, as we have just talked about.

go through it and whatnot. So I had released it to the like a 100 people that wanted, like, early access unique identifier on Very popular feed that lists a ton of large audio files. It turns out that your entire back end dies a horrible death, and Amazon sends you a huge

who told us that customer service was getting calls from a customer

This rejected all payment from the production site.

Fortunately, we could make the correction and deploy it, but this was back when deployments could easily take half an hour. Oh, yeah. So by the time it was out, they had lost several 100 orders.

Email Scott. Tweet us. Whatever you gotta do, We love hearing them. Yeah. And in fact,

so they could offer better service to their customers.

But all went fine and worked as expected.

Like, times a 1000000 customers.

It certainly depends on who the provider is. I know there's some of them that do a range, so there's a certainly a chance it could work. My situation, Wes, was I had a Staging site that had used the sandbox. Oh, yeah. The staging site was an entirely different code base. And when I was ready to swap over, I was like, oh, yeah. I'll just make the staging site live. Oh, shoot. I forgot about the most important part, which was v and v variable change. Oh, I did it once Where Android app of my company.

I have a new email, [email protected], That's like a awful time. Then Yeah. What? You can break that into task for a certain amount of people to do, but,

type of credit to get a certain response. Oh, man. He's trying to the provider is. Yeah. Yeah.

I made a hard coded switch to make sure I was Testing with the gateway.

spooky story history. So if that doesn't get you, hyped up, I don't know what will because that I decided to clean this up Without fully understanding what it does. That's like looking at, like, a box full of wires and then, like, I'm just gonna pull all these Yeah. In your does it. Yeah. Probably nothing.

you know, level up tutorials. You're managing those the sandbox environments. You're managing the production environments. You're testing your payments. Next thing you know, you know, there there's some sort of bug. You gotta get it up and fixed. Next thing you know, that Production key and, sandbox key gets swapped somehow in your e n v, and you're you're pooched. Yeah. Because

Email me at [email protected].

and Yeah.

Within an hour, we suddenly got a call from their CMO

Our hosts.

Oh, you're saying if we move our host, if we move our podcast. If we move our host. Yeah. Yeah. Totally. Something I'm sure guarantee we're not the 1st people in the world. Luckily, we own the RSS feed. We just have to look into it. Yeah. And and what I I know is that at least the some of those services that we're looking into have specific you you tell them what service you're coming from, and we will Hopefully, white glove migrate you. So Yeah. If you accidentally change every

We had to restore the backup

for the area code. And instead of a prayer line, it went to an adult And so she I give her the piece of paper. She sits there and punches all the numbers in And then

Hard enough to find anyone who had done it. Another more technically consequential one was accidentally clearing out a varnish cache

gosh, anything that ends up in manual data entry, Brutal. Not my favorite type of thing. We should say for non Americans, Head on over to syntax dotfm for a full archive of all of our shows.

Back end wasn't happy. Yeah, because a Varnish cache will

guaranteed unique identifier the bank sent it back.

RSS feed might be a massive database query,

or some resources, or you're gonna get a bill that is massive Interesting.

That's a great

I need to look into that because That scares me of waking up one day. Somebody says, hey. I got 600 episodes of New episodes. Yeah. Yeah. Well, aren't we just we're just reserving

file.

ACH database making all of the routing and account numbers bogus.

One time, I ran an update statement on a production

or maybe it runs once Hey. Who knows? Right? But if you if you clear them all and they all need to rerun at the exactly same time, you're probably gonna run out of CPU My name is Wes. My dog eats food on

the exact wrong place you'd want that call to go.

Bill. It was just like you said, both of the, both of the instances there. Oh. That's hilarious.

I didn't know. Maybe they had some support there. You know? Yeah. Right. Yeah. No one admitted the mistake, and we didn't try, Next one here. Biggest tech fail. I forgot to take out a test condition and sent around 10,000 and sought assistance from the team to prevent such oversights in the future. Honestly, don't at the time, my bank didn't have the ability to do a wire

from a previous night, and the data entry team had to key in all of the HCH,

that's a LMS to c SMS My name is Scott Tolinski. I'm a developer from Denver, and with me as always is Wes.

this served

This is kind of we want to move our podcast RSS feed, but This thing and everything will be fixed. But could have caused a

once every 10 minutes,

Yeah. Your solution to the problem is having to Key stuff in by hand, and if you have, you know, 1,000 users if you have more than a 1000 users I mean, even if you have a 100 users, that's ACH info again.

866

millions of requests daily.

from your cloud service. Anyways, he has he goes on to say related to RSS feeds, you may or may not know, but if you change the GUID,

guaranteed

ACH is automatic clearinghouse.

Across every single instance, which had thousands of large RSS feeds cached,

Feeling overwhelmed, I briefly skimmed the change and tested them on my end. Satisfied with the results, I merged and deployed the request. However,

That's a bad one too because that take a straight up security incident. The email is gonna be going out.

Users were able to log in with incorrect passwords

Tips. Upon investigation,

One particular request had extensive Luckily, GUID,

that's much better than the last one we read, which was LGTM.

Do we do to forgetting their own? still haunts me to this day. Oh,

I discovered that the authentication code had been mistakenly altered. The developer responsible confessed that the disabled passwords

a learning management system with a student management system that would later pass data to our national education registry.

Looks good to me. Well, a few days later money.

that the only thing that can save you from that hotline.

the stock price or whatever, could you imagine the the value That would actually have an a real, like, big scenario. Yeah. Luckily, hopefully, this was a a small scenario a small time company.

occasions, spending multiple files and lines of code.

Oh, at stake here. You know, you don't wanna accidentally send that to the wrong person.

Oh, man. Yeah. Uh-huh. Sometimes Sometimes you think, you know what? I I'm working on this feature. Let's just disable this other thing that gets in the way so I can focus on the feature.

the rest of the team realized their credentials were on my desktop,

Mistake number 2 is

Next one. I accidentally pushed staging code as an update through code push to production

Only having 1 person with the magic keys, I came in still a little sick on Monday to a bunch of tired faces and dozens

And they couldn't deploy without contacting Amazon support.

deleted a hard drive, which had a Postgres database on it.

While I was unable to respond on Slack that whole weekend,

of DMs on Slack and no clue what had happened.

Yeah.

Gave me a production Android app of my company.

I dropped the backing disk for the production Postgres,

Then fell horribly sick. So basically, you deleted entire Postgres

I didn't manage analytics, so it went unnoticed I was told to go home

Definitely a big trouble at that point. You know what?

To come back from.

robots dot txt. So robots. Txt is the file that search engines use to say what pages am I allowed

Oh, that it and, yeah, messing with search

or something like that. Yeah. And they will email you anytime

yeah, that's that that's good to know. Yeah. Holy cow. Alright. Next one. I had worked On implementing

web messages, anything anytime anything weird comes up. Like, I got an email this morning of a website that I have. I have, like, a little demo video on one of my websites, and it's, like, And, yeah. Just just let them know that you you heard about it from Syntax. That'd be that'd be pretty rad of you. So let's get into it here. Alright. First1 we have here, my dev horror story was I was only 1 month into my 1st job as a junior web developer. These are always the most awful ones. There's so many that are, like, on my 1st day on the job, my 1st week on the job,

search rank

to launch.

And I went, oh, like,

can be devastating for a number of reasons. And like you said, it's not a quick fix. You can't just call up Google and say, oopsies. I made a whoopsies, and, can you fix this for me? Yeah. You're you're

for a while after. Oh, you cannot

that's helpful to know. Like, I don't want it indexed. It's not like a video that should go in as a video thing. It's just A video for the website to show a demo, but,

to crawl.

Like, 500 pixels by 300 pixels. This is tiny. Right? And I got an email from Google saying we can't index this video because it's too small, So you cache the results of that. So maybe that query runs I solution

Their search rankings

Yeah. No kidding. That's wild. Yeah. Wow. That is Crazy. Yeah. You think those things are kind of funny where they set off a chain chain reaction. You know, you think, oh, I'll just delete

Publish a podcast before it should be out.

We had We had to seek help from another SEO firm to remove the embarrassing tweet. Oh, dang, folks. You gotta get control of your own Twitter. I think it's a little bit of an own goal here that you can't delete tweets from your own Twitter account.

Although I promptly deleted the page, the tweet remained.

code changes.

published a test page with a silly message to a WordPress site. Folks,

like yesterday even after you had fixed it. Yeah. The Quest oversight. As a lead developer for a web established SaaS product, my role included reviewing and deploying pull requests to production. During a busy period, I Received

numerous Pro requests with substantial and even access other users' accounts.

Do we hear about it, man? Like,

for for Slack. Never. Never. I've I've done it myself many times. They will get out to to real people. We we've all done it. Yeah. Little did I know, the website Site had a live Twitter feed that displayed the latest tweets from the company's account. To my horror, the test message appeared on the feed Oh, man.

Visible to 12,000 followers.

keep your silly messages

Luckily, we had another problem with GDPR Deletion, so we had to roll that back quickly.

Be

etcetera. In other words, deleting 1 customer could remove files from other customers.

Only about

You know that you messed up When you can say that accidentally deleting 400

People who are experienced can see where this is going. Right?

and 12345 Oh, wow.

Yeah. This next one comes from, again, somebody who I we can't say. We keep them all anonymous, But this person was worked for some very large

but I accidentally forgot the trailing slash.

One of the things to delete was Customer uploaded files in AWS s three.

starting with, for example, customer forward slash 123, Where 123 was the customer ID,

I meant to delete files

This means that not only Don't forget. I fixed it. Pushing it high.

What is it? It is 2,000,000. No. It's no. It's not. 10,000 times 200.

200. It is 2,000,000. Now I am mister Magoo. I got the big old mister Magoo award.

Next one. A previous employer, we made white label mobile app. So White label means where, like, you make, like, I think it's like Google search indexing unravel and they go like even sometimes we as whatever they want. One of the features we had a I never was blamed or called out for causing the issues, but the guilt of causing the team members to work late on a Friday night for my mistake

Make a pool of yourself. Make a mister Magoo out of your

Oh, yeah. I was thinking Oh, no. Try to call me out for math, mister. 1, 2, 3 Nothing to do but own up to my mistake and make sure it wouldn't happen Alright. Next 1 is this is a developer horror story from my 1st professional role. I was tasked with making changes to a JSON file, essentially a lookup table We're integrating with the systems provider, they temporarily lifted the limits allowing us to proceed.

companies will use that app and just brand it themselves

an app that's, like, kind of generic, and then different

times files starting with customer 123 were removed, but files Starting with customer 1234

One day, they emailed support to change it. I guess someone misread 8884 You typed this number wrong, and this

Never try to do math in front of people. You always make a

calling a configured phone number. Well, 1 client was a religious group with a toll three number for a prayer line.