Hasty Treat - Hasty Horror Stories

Should we do a couple more? Yeah. This one right here. I once IP blacklisted my own balancer. It was dumb and hard to figure out as you might expect.

It was dumb and hard to figure out is, and we were able to trace him With FBI help, his dad was angry at him big time. Wow. So FBI getting out here in the, forum space.

That turns out to be a a big theme here over the next the next episode.

the best way to describe a lot of our errors. Oh, that's so funny. Blacklist the thing that lets you in. That's good.

and that's that's the way it's shown to everybody.

Oh my gosh.

We were about to release a feature for our Bible app called Stories.

this developer

These are ones that we've been submitted over Twitter, so this is the first one. I've been laughing about this one all morning. So

And I created a CMS right before we launched public testing. I remember a verse that I chose, and it goes like this. Listen to me, you fat cows living in Samaria. You women who oppress the poor and crush the needy and who are always calling out to your husbands, Bring us another drink. So that's Oh, man. Here's a good one. I think this one got, I think, more favorites and more retweets than my actual tweet to this is. Yeah, ratio. I had to chase down A bug that caused very broken styles on a banking platform, but only for 1 client and no one else. After a lot of head scratching, we discovered watching watching this big product, And that's the reason that they couldn't log in. Normally, this happened on a big day where we spent 1,000 of dollars on marketing campaign to bring customers into our brand new location just to see them leave frustrated and never get back in again.

Oh my gosh. I remembered at 3 AM and fixed it immediately.

These ones are just short little ones. In the next episode, we have much longer stories.

So, like, kinda like Instagram stories, but in the Bible.

CSF. You're watching And I just use the email as the user signed up with, To import stuff, they're highly available.

Classic bug right there. I was once fixing a bug where some items weren't showing the values of stored in the DB. Turns out someone had hard coded a check directly into the view That tested it if an item was created before the hard coded date and displayed the wrong value if that was the case. I spent a few hours digging through controllers and testing queries. Couldn't believe my eyes when I saw it. It didn't cause any problems. It was just yikes.

Well, cool. This is a nice little introduction that was supposed to be replaced before released into production. They were, except for 1, which needless To say the client didn't appreciate being told where to stick their input data, which, again, folks, come on. This is this is something that we see over and over.

It took 7 years for this bug to manifest itself, but the whole app was instantly down when it finally did.

That's classic. Classic. Next one here. My 1st web dev job in the early aughts, I was bringing in Google Analytics for a university. I managed To bring down the whole SIS, the student information system, which wasn't my responsibility and made the guys from tech support tell users that they were just mistyping their password, worked For the largest Bible app available on iPhone and Android, and they were releasing a he says,

taking down something for a lot like, the the more people that are involved, the worse they are every single time.

It was because Yeah. I was storing numbers as leading decimal zero. So we had to go through every single episode

Let's see. Here we go. Weeks after a database migration, I realized the target at DB was case sensitive, but the source was not. Users ended up with new accounts via the same email address upon next log and lost access to their original data. Merging the data between the 2 accounts was quite a puzzle. Oh, oh, yeah. You should be using Git and this and that, whatever, And he's feeling like that was, like, super overkill. This was way back in the day when I first started. I was like, but I'm fine saving the file and pushing to FTP,

with leading zeros,

That one, I feel that one deep in my bones because that seems like something that I could do. I did that very early on in my course platform where it was case sensitive, Hello.

001, 002, and then I would just parse int on those, and it would give me the actual number. And then at a certain point, Because of decimals or something, I forget the actual issue. But at a certain point, I think it was, like, episode 192, the whole thing croaked.

Because if it's affecting 10, 12 people, not a big deal. Affecting 30,000 people, boy, hold on to your buds. No. Thank you. Okay. Next one here is I ran a huge forum with several 100 thousands of posts and pictures in the early 2000. Yeah. They were they were everywhere back then. And to save space and money, I changed for a full archive of all of our shows, and don't forget to subscribe in your podcast player or drop a review if you like this show.

Be a poorly written chat program, which we had linked to in the forum, but it he wasn't smart enough to spook it spoof his IP.

Oh, so that sucks. I wonder how you got in the 1st place, but that's what happened at Facebook is they just would literally not able to get into physically into the the building

and didn't get Google Analytics because it was misconfigured.

Everyone knew it, but they didn't know the password either. So this can be subtitled as dev and customer immobilization

Oh. Wait. Wait. Is it actually f IE 11 or, like No. He did this the whole swear. The whole swear. Yeah. The whole swear. It's

test.

Yep.

when Yeah. That one sounds awful, especially because a lot of people are Yeah. But, also, it's, like,

backup from daily to once a month. Then a week later, a Fourteen year old hacker managed to issue a drop command in the DB, and we lost 28 days worth of posts

I changed the DNS and logged out of our hosting service account, and I didn't know the password.

99%.

The client had a very aggressive swear filter, and it was blocking our CSS file that had the comment f IE 11 in it. Oh, yeah.

Change the episode number to a number and then just format it with leading zeros for display purposes.

Don't take down anybody. Don't take this guy's forum down. The FBI is gonna be knocking at your door.

Yeah. So very thankfully, being a developer, you get somebody who emails you being like, I figured out why. It was because That's a classic bug. Yeah. Very easy just to to lowercase everything. Yeah.

Oh my god. Oh. Yeah. That's messing around with some stuff because how do you unblock a phone number? You know, you can't do that. You know, that is that is a tough one to come back from. I used an integer field in PostgreSQL

we should have used a big int.

Just yikes. Back in the early 2000, one of the team f. Had put in a funny error messages people as well. It's so slow. Talk about a moose boost. This right here is the little, like, pre course to what

And when it saw the new cookies, it freaked out and just crashed.

Head on over to syntax.fm

with a whole bunch of these really great little dev stories. We have some bigger,

for when I restored. The hacker managed to get root access CSF. That's like a regular y two k kinda situation right there. That was,

I wrote an infinite recursion in our alarm system. Me and the CTO got about 20 1,000 text messages overnight, and both of our carriers blocked the phone number. The Twilio bill was high, and we had to switch everything over to a new phone number because the whole one got blocked.

Can you imagine, like, an entire university is down because of something you did? Yeah. That's at a lot of these. Like, the worst ones are, like,

They wrote spook, which I like to think is a really nice little typo for this episode. He wasn't smart enough to spoof his IP,

More elaborate stories in the next episode, and they're very spooky, and they will make you very they'll make you shake in your boots. They will make your git commits quiver. I don't know what else. I don't have anything else there, but we will see you on Wednesday for the tasty spooky treat. Peace. Peace.

so funny because we're PG podcast, so, And, like, trying to dance with the fudge, the whole the whole, yeah. That's funny. Oh, there's a lot of really good ones lower down. Yeah. I'm I'm going down. Somebody mentioned the HBO test integration email, which we did talk about. That's very funny. HBO sent out a test email, but luckily, it was not anything other than just, You know Did you read this one? No. No. It's it's a similar a different DDOS one. Yeah. So there are 2 self DDoS ones in here. I I was, like, halfway through. I was like, wait. Wait. Wait. Wait. This is the same one. No. The whole office lost connection for a few days. You lost And then you d you DDoS ed your whole office, and and people had to that's some They turned the Internet off. That's some money lost right there. Woof. Woof. Woof. Woof. Not to mention, like, corporate Stuff is often usually not unlimited, you know, like, especially if you're Yeah. Paying for Bandwidth.

and they signed up, they were logged in, and then they try to log in later, and they type it differently. And they're like, I swear my password is correct or, like, I reset it. And, like, the the password reset was normalizing it, but not the sign up process.

So no one caught it until we purged the staging bucket.

prod versus production and Hundreds of thousands of other syntax when we first started the website. I labeled the episode numbers that updated records with username and date time when they were updated. Users complained that changes were taking too long. I spent a Hours debugging the app, and I found it a week later in the database. Oh. So every time you update your username, it updates You could automatically manage your DNS.

to staging,

that sent all the images

It defaults to staging bucket. There was a mismatch

is and then saves image to s 3 brook. It's a Lambda's a script, and he saved it to s 3. That's where you save images. It returns the s three URL and saves it to the database. Okay. So if someone uploads an image, uploads it, and and you put the URL in the database,

And let me tell you, this one is so believable because my dog loves to give me her paw.

She just goes, paw. You know, paw? And if I'm trying to develop and I'm sitting there and she's right next to me, she'll just keep giving me her paw. But since my hand's on the keyboard, she's just slapping my keyboard over and over again, and I've had files be, like, just all sorts of extraneous text being added to them. Never deleted anything off FTP, but that feels very, very possible to me.

API FTP.

Yeah. That'd be a big big deep horror story right there. No. Thank you. Oh. Oh, man. Okay. Why did I Issues that I have no clue, but it's pretty funny.

Oh my. Yeah. That's a good one. Starting off with a bang. I'm gonna be starting off with this, moving on to one that is less of a bang and more of like tippy taps here. My cat walked across my keyboard and deleted a client's entire production I accidentally deleted a client's Custom website that I've been working on for 2 weeks. Yes, people. Back up. Back up. Back up.

CSM.

Tolinski.

soft skill, web development, the hastiest, the craziest, the tastiest TS, web development treats coming in hot. Here is Wes, Barracuda,

Monday. Monday. Monday.

Open wide dev fans. Get ready to stuff your face with JavaScript, CSS, node modules, barbecue tips, get workflows, breakdancing,

Oh, welcome to Syntax.

Boss, and Scott, El Toro Loco,

Oh, that's that great.

I I just can't stop laughing at some of these. I I'm sorry. Like, obviously, it's in the past, and it's all been resolved, but just oof. I think the laughs are like it's coming from a place of love here. Yeah. The laughs is like the equivalent to seeing somebody like Have you ever watched, like, rope swing fails on YouTube? I love fails. It's that kinda lot. Yeah. It's like a We used to have a neighbor that would like, they didn't have a TV, and And they would just, like, knock on our door, and they'd be like, can I come watch America's Funniest Home Videos? It's on right now. It's just like, okay. When I was in college, They just come down into our apartment and watch America's Funniest Home Videos and just sit on the couch and just be like, oh, for testing

Yeah. That's exactly it. It's America's Funniest Home Videos laugh. I love the,

I left Twenty boxes running an infinite download loop test overnight to downloading a Linux distro.

The Office ISP cut us off due to a d DOS attack on the distro.

the, like, the dance compilations where they just show people dancing at weddings and falling over and stuff. Love Me is America's Funniest Home Videos. I gotta give a shout out to America's Sound of Funniest Home Videos up there. Alright. Next one is I was working on a set top box that used JS.

Next one here, says, I used to develop marketing emails for previous job that sent emails to millions of customers.

data over and over and over and over and over again

himself. Oh, yeah.

CSF. That's a good one. That sucks.

Century.

squash those bugs. Fix those bugs and push those bugs up to production. But, hopefully, you run your tests and all that stuff first because as As we get into these stories, you will see that there are a lot of people who do not run their tests before pushing to production.

And it well, they're gonna end up with more bugs in their application, and then Sentry is going to have to find them for you. So head on over to [email protected]. Use the coupon code at tasty treat, all lowercase, all one word. You'll get 2 months for free,

Yes. This episode is sponsored by Oh my.

and All over again. How many of y'all ever been out there and have been cowboy coding on a server with FTP? You're just you're updating a file and then yeah. It never works. And I remember, like, some point in your career, there's, like,

there were some hundreds of thousands of emails that weren't opening.

It turns out that Grammarly Marley was injecting code into the inputs in a service that we use to send the emails.

There

So, Linode, check it out. They're gonna give you a $100

and you can, you know, you can squash those spooky creepy crawlies in your application. We are also sponsored by Linode, and one of our our spooky Key stories of of the next episode that we have is somebody at Facebook accidentally blew out an entire DNS and took Facebook out.

Also, Facebook should just be using this thing instead of trying to do it themselves because They could've called up Linode and say, hey. We goofed up. Let's roll that sucker back.

Sick. Alright. Let's Get into the spooky

So Or Facebook was trying to run their own DNS. Should they be doing that? Probably not. They should be using Linode. Linode is cloud computing. You probably know them for Hosting Linux servers, but they have all kinds of other products and solutions. Specifically, one is they have an entire product to manage your DNS, So all the images were being saved, and then they just wiped them all out at once. Hopefully, Were you able to get that back? That sucks. Next one here. Before we started using Git,

episodes.

And, like, also, like, you had to pay people to reply to these people and say, hey. It was it was fine. Like, sometimes you sit on hold for 45 minutes on a thing, and You don't think that, oh, maybe some developer accidentally sent out an email, and all their customers are freaking out right now. Here's another one. Once upon a time, I was giving styles via FTP,

and the file was corrupted, and I have to do the styles

the full course that we're gonna be doing on Wednesday for our Spooky treats. We have a lot a lot of cool stories planned for that one as well. So buckle up. Let's get into some spooky stories. But before we do, I should introduce myself. My name is Scott Tolinski. I'm a developer from Denver, Colorado, and with me as always is the spookiest West boo. Oh, hello.

Hello.

Uh-oh. Uh-oh. Next one. I sent out 35100

overdue account emails 3 days after they had already been emailed Hosting or or any of their products, which is pretty sweet. So check it out at linode.comforward/syntax.

While trying to make a change in our automated system, apparently, the depot got a lot of phone calls from angry customers who are trying to explain that they already paid their Yeah. It's it's so funny when these include numbers, like big numbers, they instantly get that much worse. Yeah.

Uh-oh.

In this Monday spooky treat, We're gonna be diving into some spooky stories from Twitter that we heard on WebDev Twitter. We asked you, we need your WebDev horror stories. For every year, we do an annual At first, but also, like, they literally couldn't get into the servers because they were not on the network. You know? Yeah. That's a big one. Well, I would like to say that we tweeted this 10 minutes before Facebook went down. I feel like we had a bit of a 10 minutes before Facebook down. Seemed like there was, like, a ton of outages and little goof ups over the course of that day. So that was, like, super funny. I I was just, like, watching this all this take place. I'm like, oh, yes. I'm gonna make sure really good content on that spooky episode.

guess that's what every year means, an annual horror stories

And this actually is a little bit of a Hasty Amuz Boosh here. We're we're doing we'll be releasing before the

where we read off scary dev horror stories, Things that have happened to real people, these are not made up events or anything. These are things that have happened to real life devs, and we read their stories to you So that you can sit in your seat and cringe a little bit and feel all tense all over for other people's mistakes.

so I can only let known users into the beta. Okay. So that makes sense. He's coded things up and don't wanna let people in until you mainly approve it. I forgot to remove the code when the app went live

Or private repos for your private code if you need it. So, make sure you're you're backing up ASAP. Next one, coded functionality to manually approve users After registration,

And it's always so funny because I wish I would have heard stories like this earlier on in my career where I've been like, oh, yeah. I need a better I need to I need to invest that time ASAP to learn this stuff. Because people, backups and having your repo somewhere else, whether that is a public or private service, you know, we have public

Next one here is I managed to DDoS our production server 1 night at my last job by pushing an update to 40 clients that made a request

Oh, that is hilarious.

User will have a botnet of computers that request

Of apps not loading. So a DDoS is a distributed denial of service. So the way that that works is often a malicious

To our API server towards From a thing, and if you do that too much, you can overwhelm a server. So this guy

that would endlessly retry on failure. I discovered it the next morning, and I receive a mountain of emails

CSF. You you heard the main episode because it turns out a lot of people use the where claw or forget the where clause. There are SQL queries.

Oh, yeah. Paying for Bandwidth. No. Yikes. Big yikes. Next one, I misplaced a closing tag on a comment, and it removed the where clause in an SQL trigger

out of your control, but still your fault because, ultimately, you use that tool. Yeah. That is that's a rough one. Doing a mistake in an email is the worst because once those emails are sent, there's literally no way other than sending a follow-up email saying, sorry. We did it to stop f. People from replying saying, hey, did you know that this happened? I've done it a couple times myself. Even a little spelling mistakes, You get hundreds of emails from people being like, hey. Did you know you did this? Next one we have here is, I wrote a Lambda. It process