Spooky Web Dev Horror Stories - PART 2

syntax. Sign up, and it's a perfect place to make sure that you are on top of any errors, bugs, or creepy crawlies in your code base. Last thing we want is a bunch of big old spiders running around in our code base. Hey, Wes.

come on.

and pop it in the box, and, we'll put it on the list for next year.

where we were browsing the site on a projector.

Seems like everybody who's young in their career has done that at some point where you just put something ridiculous that's back 3 g days,

I never patched a bug faster. I believe this was also the first month or 2 of starting there.

I helped him out.

And it's just like, it was immediately noticed by leadership team during the team meeting

Alright. So the first one here we have here is called monkey business.

Yes. How's it going, Scott? Oh, yeah. I'm doing good, man. If your code is looking, if the spooky story contains the word 1,000,000 of dollars or 1,000,000 of dollars, it automatically gets but they don't allow you to enable this feature unless you turn on

to make it show up on the page.

gone bad if your code's gone bad yeah. There we go. If code's gone bad, head on over to century.i0forward/

literally laughed so hard reading this for the 1st time because it's a 2 parter.

Oh, alright. The first Node we have here, I

Black holes.

Chrome might be a little too permissible. Yeah.

Yes. This is a I I give this the alternate title of boot camp Crystal Lake, and I give this, I had somebody be like, there's some language in an SVG on your website. And I was just like, oh, yeah. It's a Wu Tang Ipsum. I didn't

accidentally published whether it's like, even with the database schema. The other day, I I messaged Scott, and I was like, hey. Like, wouldn't this goof up our database schema if this were to happen? before it was sent out with my email sent out. It it takes, like, I don't know, 5, 10 minutes before they actually start getting going, but there was no way to stop it because it it had, like, hit the no return. So I, like, messaged someone from support, and they're like, we'll see what we can do. But, unfortunately, the broken link went out to that many people, and I don't have a no reply. I have my own email address as my reply,

and we're almost at the airport when I get a call on my brick sized cell phone.

to

80,000,

keep it gee, I learn I I

Irascible major It just returned a null value. The sync function would then delete the local data

let's say I have a button. Like, my Sanity tracker's toggling back and forth really quickly. Yeah. It made much more sense

many, many people built for the time that they and it's probably just sitting as a draft in WordPress somewhere.

Oh, I I know I think it was Abercrombie had this issue. I don't know if this is Abercrombie or not, but Abercrombie had a similar issue where there was some weird combo of coupons that you could use, When they when Firebase was announced, and for many, many years, Firebase's default I shouldn't be blamed for it. Probably reviewed by the tech lead. Yeah.

the footer just takes up as much space as it needs.

It's not because it's the little it's the same little nav you have in Instagram. It's just little icons down there.

code review found this. Thank you, CJ.

And she was like, you gotta watch the show. It's very, very funny. And sure enough, and start writing code right away.

I called for my classmates and begged them to immediately download the update. Since this bug only happened when you tried to sync without cell service, it didn't affect everybody, but there were a handful of people, lost a good portion of the required school port paperwork, and I was mortified.

Once upon a time, someone configured a promo code without any limits for $500.

with the same credentials will indeed result in the page view being counted twice.

some information back, and that will allow you to sometimes allow you to to find subdomains

about things like that that you shouldn't be doing anyways. Like, Safari for me was also, like like, really

Next one is called late night with Safari. Back in 2016, I was a sole front end developer. So many of these stories start with, I was the only developer

much harsher

I got

because it's it's kind of scary Wes, oh, you have you have the wrong tab open and

Moral of the story, never be afraid to reach out for help when you don't know what to do and accept the fact that everyone is going to make a mistake at one point or another. And Deno.

oh, man. What is that word?

that that always creeps me out. Those, or something like that. And I often wonder, was that on purpose? You know, was that an advertising campaign I'm not exactly sure, but this seems in a gray area of of being able to do this. So what you do is

stating how successful the event in conference was. Breaking visitor attendance records was a huge success.

On migration day, we just moved everything to a new VPC.

Got you. Yeah. And then use version control to deploy. I think a lot of these stories are from before when version control was a thing, but certainly you can sometimes crash

and I I wanted this feature enabled. So I said, alright. It's worth it to enable this, you know, 10 step process Node to make changes to our our schema if I can have access to this feature, which, of course, is a good thing we have access to this feature. Because I probably would have accidentally

All the form data was stored locally in SQLite and synced to my MySQL database I hosted. So no matter what, there would be a backup of our paperwork either locally or on my server. Makes sense. You do it locally. You sync it to the remote server. Right? That's that's ideal.

like, pretty technical.

like, when somebody famous dies, a lot of these newspapers will already have the No.

Next Node is console dot swear. I was working for a nonprofit in 2011, and we had a big boost to our mailing ESLint, and we are building a new campaign. I was working on the JS error handling for the position signing form and was struggling with a regex. Oh, I already know this is gonna be awful.

a bottom footer nav? Maybe I'm wrong. You know what? It's background position fixed. It's Okay. For the longest time, you couldn't do position fixed on It was my 1st job. That's what he said. One day, I pushed an update to our filter experience. I say it was sized effort and things were looking good for a few few hours, but at some point, we got customer feedback that the web page would crash for some user.

They sent out a newsletter, and it was very clearly, like, their test email that they meant to save for later in a draft. And it was, like, declaring that some contract had been signed when it hadn't. People were, like, really, like,

So it wasn't like a bunch of people being like, who? What's going on? Please stop this.

That that's not how good the story is. These are all incredible. In fact, man, I I have no doubts that that that would suck. That would feel Yes. Scott that it's a will feel awful. It's a social problem as well. Social problem. Yeah. You feel it in the pit of your stomach type.

Next one here is AWS cluster f. I love that. I wanted to update a production cluster for several 100 customers. This was a self hosted ranger on AWS I had built.

the frozen

yeah, it just a a dab will do you.

at a at a job in charge of a Fintech b two b application.

you see a trailer for this show and you think this is a rom Scott. This is not for me. Yeah. And so because of that, Courtney started watching it by Vercel.

And I tested it. It worked fine. Yeah.

It yeah. Which when we by the way, I wanna say, when I give these things spooky scores because a lot of these are like, oh, they're not as bad as whatever. Yeah. That's just how scared I would be if this happened to me.

I removed it from the list. It must have been a dead domain.

have a big vocabulary.

I think, about

like, who who who's really good at documenting things? I don't know if everybody is. Like, we tried to document things really well, but there's still a handful of things that Yarn, like, And then in their docs, it's like, oh, yeah. Before you deploy to production,

Oh, poor guy. Yeah. When you when it Wes it affects other people's lives, Yeah. Yeah. I don't I don't wanna end up on this podcast. You know?

I started watching it, and I was I was cackling at at many parts of the show. It is super funny.

Yeah. But you just Bos it down there and put a padding.

At the time, it seemed that Safari couldn't handle a transition for the font size property, maybe with other size properties too. So if a font size transitions,

I think we all push stuff that could, you know, break. It's funny, Wes. We did a CJ was doing a live stream of the syntax production assistant, I feel like we must have because we we did that whole voucher site, which was like a whole thing on coupon codes. So I would imagine it was around that what we did. Scott complex coupon codes.

Never heard it. Yes. Big very, very big vocabulary. I've never heard this word before in my entire life. It is Horrassable. Going on the list. Yeah. Man, I like that. Cool.

We were able to narrow it down to this specific Safari. Oddly, the office was primarily Windows based machine except for 1 person, so I had to kidnap her Mac for the day. And what a day. I was able to reproduce a problem, which was good. I figured it must be something in JavaScript

To make things worse, I was using exclusively Pokemon images for testing.

Nobody ever used the app again. Some of those spooky dev stories involve companies losing 1,000,000 of dollars, but mine involved having to spend the next couple years sitting

My first task was to add some new Google Analytics tracking events to a few form submissions.

Why can't you do position fixed? Because it if they're selling at a loss with the coupon, you know, and the coupons are just there to drive. And then, oh, man, that could, like, really sink a business. Right? Oh, yeah. If you were selling at a loss, just trying to gain some hype as a promotion.

user table set status equals bronze.

Yeah. Man, that whole version control flow for any CMS based site is tough. I remember there Wes a Drupal plug in that simply just output your configuration because it's storing a lot of configuration in the database. It simply was just, like, backing that up to a file and then restoring it from a file. And so that Wes, like, had to be part of your version control Wes to whenever you made a change, you'd back up your configuration,

Tested on everything in a staging environment. Was super proud of seamless upgrade there. Uses wrong token and password for upgrading.

Now there's no users who can access anything.

WordPress is weird where you upload a a photo. By default, it sticks it on the Vercel. Mhmm. So then that photo doesn't exist

I have a large

on every single domain name, and that will give you

Not dev related, but funny anyways. A marketing person I know was creating and scheduling a post event email

is

on the local database local sync to do a soft delete on the local sync database specifically.

ask for help. There's no harm in asking for help. Yeah. It should never be 1 person's

lighter. The total loss added up to be about two point 8,000,000.

Node do you get to tell people you're a part of a reply all email storm? Yeah. Yes. I love it.

dotted lines toffee candies out of 7a half toffee candies.

to new user status.

If you don't have a good review process, which it seems like they they didn't,

Yeah. Oh my god. I keep logging messages g rated and keep them to the console now. The founders and my boss were very chill about it, but we could have gotten a lot more spooky if not.

and I had joined a small real estate company with 2 other developers along with a bunch of product marketing people and a bunch of realtors.

which this should be a part of your process.

but it's necessary work. Don't deploy this to production, but Yeah. People do. People do.

80,000

I I was thinking about that last night as well. Do you remember, cc'd, I think,

Well, being fresh out of a boot camp and pretty unfamiliar with front end frameworks at all, especially custom built in house ones, I couldn't figure out how to do it. Rather than ask for help, I discovered that if I loaded the Google Analytics script in some file, I could access it in the files I needed

so that they can go live as soon as they pass away,

again and deployed containers back for data. We luckily used AWS EFS The client wasn't happy because I believed by law they had to honor it.

and then inserting new rows into the database with the response from the server. Otherwise, I would have to rewrite all the complicated merging code I had already written on the server side. Right? Yeah. That that makes sense. You know? Delete it, send it off, and then and import it. I tested a 100 of times. It worked flawlessly. The problem happened when we were in the forest. That this is terrifying, though. This guy's just running around the forest as well.

and insert null into the local database. This is just screaming for a a syncing engine,

Alright. Let's move into some sick picks. I'm going to sick pick

Yeah. Next one here is ask for help. Nobody you know, here here's the big thing is you will be judged, Building a coupon engine Node number 4

the thing written

I had made an app that let you fill out the paperwork when you're filling in the woods and then delete it. There were no backups.

So ask questions. Ask for help.

everything is wide open, and people don't realize that.

Wes, you might not know this, but there's a feature of PlanetScale.

like, truffle I love truffle flavored hot sauces. And Yeah. The original, the black one JS really, really good. But now they have a a buffalo one, which is super good. They have a jalapeno lime, which I've not tried just yet. They've got some mayo. They haven't tried any of those yet. But if if you have someone in your life, you're, like, looking for a Christmas gift for them, and they, like, like Scott sauce or they like, like, nice And every now and then, My hair just fell off. Alright.

What do you use position fix for? I haven't used that in backup, the Europe thing why am I forgetting the name for a Europe thing? GDPR.

Yeah. I think it was Triumph by Wu Tang, which you if you've heard of that song before, you know what that song's about.

then everybody you know, they're they're college kids. And I I'm not gonna lie. I got in on this. The reply. Replying to all and reside. Take me off this ESLint. And and everybody's

Next one is a film school f up. Years ago, I was in film school. I made an app to help my classmates, and I automate some paperwork on set.

everything. So what do you do in almost I use position fixed all the time. How do you do, like, Another coupon disaster. Oh, coupon disasters. These ones are good. I did tons of email campaigns and once accidentally sent an exclusive one time offer meant for 500 people

that it went just absolutely viral? Oh, yeah. Or did somebody really goof up their, using my friend's BlackBerry things like, I don't know if there's any way to stop them.

documents as deleted,

is too many pat this is too many updates for 1 PR. And I'm in the chat being like, they're all fine. They're all patch. And sure enough, something was broken in there. CJ's in there. And I was like,

They accidentally sent it to the entire database takes us seriously, I was a few weeks into my 1st developer job after completing a coding boot camp in 2016

ISP contacted us.

which is they have fine Sanity, but it's just that by default,

I noticed my error and fixed the issue, but there was about 20 minutes where a person would have seen a friendly Pokemon instead of their diploma

I then adjusted the process between myself recursively

in tech even after doing this stuff for 15 years before as a hobby.

commit that to version control on deployment. You would restore from version control. It was just like,

applications Oh, it's you think you could be silly doing silly stuff, but when push comes to shove, things need to get done. You often let

And you're scary stuff. This one, I I I would give 5 out of 5. That one's Oh, yeah. Yeah. Five pumpkins out of 5 pumpkins. Yeah. Yeah. Yeah. 5 pumpkins for sure. Alright. Next 1, severed trunk and missing backups. When I was an intern at a small marketing agency, I ran r m

on top of Firebase and never introduced any sort of when it was usually ESLint. Very similar. That's that's the same thing with the GitHub one where it's like, something's a bit odd. I knew right then I was still connected to production.

Go listen to all the past this is the 6th year in a row, so this makes 12 episodes of spooky stories. We have many, many maybe at one point, we need to do a best of spooky stories over the years because there's some some good ones.

Right? Anything, whether it is like CloudFlare or any of the stuff Wes you commit, push up, everything just works. It makes things so much easier. You roll back. If something fails, you don't have to worry about having those files on hand or something like that. If you don't have a process in place for making sure that you Node, deploy quickly, roll back quickly, data and site, all those things, off-site backups, all that stuff, you need to make sure you have that implemented because you will end up on this Node. Or, you know, you might not, but,

you who can read and write,

A week

green or red bar that says production or testing,

Yeah. Okay. Cool.

My teenage brain never fathom that a post request could possibly fail and never considered in that situation my It's like the opposite of the friends he made along the way. Fortunately, the school decided to give those people a pass since the paperwork losing the paperwork was out of their control.

around

Awful story. Awful story.

Oh, yeah.

You Node, block merges until no goats is confirmed.

but background position fix is not not a thing. Okay. That Like, what about a footer navigation? Like, when you have, like, an app, the little Yeah. I would do, like, a layout, and then I would do, like, a grid layout and make the footer,

The boss called and said, we have a group of users whose statuses are wrong. Can you update them before you head out?

that Wes, like, that was, like, 10 years ago. You couldn't do that. Okay.

authentication My first job as a nineteen year old and the only engineer on an entire product, I accidentally pushed some test code that added, oh my Node, I'm a monkey to the header of every single page.

in the class next to all of the people that I had screwed over with my app.

entire problem. You know? Someone else should have caught it and or at least have reviewed it and said looks good, and then it falls both on on both of you. But, like, what this guy's not fired because of how he handled the situation. You know? If some people would have done this and then reacted negatively or passed blame or simply just ignored it. You know? And if that's the case, then I think that's grounds for being fired.

No access to it for a week.

CJ pulls it up on stream, and people on stream in the comments were like, this on. Stoked to have made my 1st ever professional push to production.

realized it and quickly fixed the bug, but it was too late. My class had been filming our films for weeks.

tops 20 minutes. It became much harder to create a limitless promo Node after that. A hidden lesson is to not let your internal admin tools Scott as your business grows.

We have, Oh my gosh. She just stepped up and held that tarantula all day. So you don't want any tarantulas in your Node. You don't wanna hold that thing. Make sure that you solve those bugs with Sentry.

access control. And there there's tons of stories out there, so much to a point where Firebase got has a bit of a bad rap for security,

Irshable, loud agnostic woman. And she's a, you'll get like this. She's a podcast host, and she hosts a podcast with her sister.

And then you gotta figure out how to roll that back. Unfortunately, he didn't say how he fixed it. But Millions of edits. Thousands of user profiles,

Jeez. I'm sure. You Node? To 80,000 people? Yeah.

They thought we Yarn potentially

database schema feature for PlanetScale.

250,000 I was just 17 teaching myself to code with the help of Google and YouTube when our class

on that same domain.

And you can run them through various DNS providers, Google and Cloudflare being the the big ones.

woof. Yeah. Because That sucks. Especially if, like like, I know systems where it's, like, webhook based. You sign up for something. That webhook comes back. You give the user the role. They get access. Right? So you possibly have all that information

Sarah Vessels from GitHub give her professional Management improved the backup strategy. Thank Node.

took place.

Yeah. And, also, that it's, it's a even Meteor themselves, they had, like, by default to get up and running really easily. They just had you connect to an unsecured MongoDB

oh my Node. It's a monkey ESLint through. In this case, you gotta have a checkbox.

Yeah. I I I was technical enough to know what was going on, and I just thought it was fun. So honestly think I should've. Because, like, how And I I did not want to turn on the frozen database schema feature because it's a giant pain, which, of course, it's only because I'm being, you know, lazy.

before the event Due to time zone differences, I went to sleep right after the announcement.

a 4 chan But check it Scott on out, syntax.fm,

And,

yeah. We can give that we can give that, you know, 2 2 zombie goats out of 5. Yeah. Wow. Yeah. No goats.

You can't. Get look it up.

And I forget exactly which feature it is specifically,

Yeah.

Yeah. That's also like Safari doesn't even let you do position fix still. You know? And, it it went on for a while, but luckily, most of the people on there were,

Yeah. A month or something. Or something like that. Yeah. Yeah. Totally. It that's not always a bad thing to have those soft deletes.

Have a good Halloween.

processes, in the code base.

point. So the doctor was working on an amazing project.

I I think, like, yeah, they're doing some SEO ranking. It seems slightly illegal.

Not as extreme as the others. But when I was on the production support team for my current company that deals with practitioner and patient interactions, slowing down my app like crazy for a bad transition as well. And and you could rip on Safari for that, but to be honest, I think that I had dropped out of. Little did I know that this project would soon turn into a wild ride of lessons learned along the way.

I learned this the hard way by using Wu Tang Ipsum in an SVG.

on an iPhone, all gone. By the way, this was not a small community college. It was a very well known corporation with 4,000

really great new deploy platforms.

do I think we even did we do a whole show on coupon codes at some point?

to wire up the event handlers of the form.

I have a huge

take on code reviews and how best to do them. Code reviews are are fantastic. They might slow you down just a hair, but they are important because Next one is code reviews,

they think you're spamming. So I asked them, like, what happened? He said the ISP contacted us because they noticed unusual reverse DNS lookups being routed at a high rate Dude.

fifty Node. Yep. That was the coupon. That was actually for our course sites. So that was not even the voucher one. We also did an episode on the the, giveaway voucher site site. Oh, yeah. Coupons. Those are both very good. The Syntax giveaway site episode 609, both are probably relevant in this scenario here. Yeah. Coupons agreed.

and validate all steps for rate limit. So

you know, have that feeling in the pit of your stomach. Like, oof, you goofed up. Yeah. For sure.

maybe you should have that code if you don't.

where you could you're modifying data,

You most likely will not be fired for it. Yes. That is such good advice that I love that this one, it kinda ended with a little tail of, like, a morality tale. Like, hey.

Oh my god.

Because if you're doing too much DNS lookup,

I was looking into a customer escalation and was reviewing their account details.

I was filming my own short form when I noticed the bug. My data disappeared. When I got home, I spent hours trying to figure out what happened before I which is a Scott delete. It's not actually deleted. There's a whole regulation there too because what happens with with,

We had a really super good episode on code reviews in episode number 830.

Applications were scaling up and down happily, but no one could access.

I stepped in to turn this disaster to a valuable learning experience.

No goats. Yeah. Oh my gosh. Yeah. That's,

query

You can't position fixed?

Oh, man. That that's so funny. It's like an audit. It's just doing its thing, but you just can't get into it. You're outside looking in. Yeah.

Last 1 here, and this Node is,

a week or 2 later, my boss comes to me and asked me to take a look at the Google Analytics dashboards with him. Overnight, it looked like our traffic had increased dramatically

If I remember correctly,

And if you don't know what those are, which let's face it, I I think every single one of us has been a new job or there's new processes and, So all of the websites went down for several days, and there was absolutely nothing we could do about it unless we had a recent local enough development copy. Man.

Shameless plugs, check out syntax.fem.

250,000

A quick search shows me that all of the users who signed up for a specific date weren't being set to the proper status. They had access to way more than they should. No problem, I said, ESLint proceeded to run the following command,

Stop. And it it it makes it even better. It's a nonprofit. It's probably like, would you like to save a starving puppy? So you can't even access your server Firebase? from the guy because I was building the form for people to submit spooky stories.

yeah, it's it's it's very, very funny. It's a sweet show. It's it's cute in all these ways, but, artistically, it's really good. The music's really good. It's super funny.

yes. Werewolves. I I do give this 1 250. That's 4 werewolves out of 5 for me. Oh, that's a lot of people.

like extra work,

Oh, that was that was a bad day. You just get flooded with emails for the next 8 hours.

Yeah. And, actually, shout out to PlanetScale extra spooky because it's not dev related.

G rated.

has been tasked with writing the Elton John has passed away article The downside to that is just have it there. I think there's still quite a few WordPresses I I actually had to do this for 80,000 domain names for my burner email list. So I ran

I'm gonna sick pick a show on Netflix, which JS a surprise to me. It's, when review when viewing their account settings.

on the schema, and then that has to be approved and then merged. And I was like, okay. Good. Because I was just sitting there being like, that could possibly be an issue for us at some point. So having those processes in place so that you can't accidentally goof this type of thing up because we are all just human at the end of the day, and these things happen.

waiting this contract to be signed. You know, all the fans are on edge about it. And they get an email saying so and so signed the contract.

When we were in the forest. What a great sentence. When you're in the middle of nowhere without good cell service, the app would try sync with the server, and then a post request would fail. Unfortunately, the function that returned the response from the server wouldn't raise an exception.

with, like, no creds. Right? It was just a completely open MongoDB, and it was local. And it was intended so that you could just install Meteor people.

Yeah. DNS anything, man. Put just DNS on this. And, I mean, whether you're changing DNS, looking up, man, put it all on here.

If you're logging anything, then you will be for asking a Wes, no one's gonna fault you for asking a question. If anything, they're gonna look at it as a positive. Like, this person you have to lock down your database. And it turns out just people don't read. So people were just deploying their to production with an unsecured open database.

And

script twice on every page

We should do another one.

man, it's kind of a pain. That's one of the reasons why I'm stoked that we have so many of these

Remember to use those Wes clauses. Folks, man, this is the 3rd Wes clause.

Locked us out of the cluster for good.

and it got you, like, 98% Bos that is the live site, especially because, like, Luckily, our hosting provider had a utility where you could do a virtual login. But in the 15 minutes it took for another coworker to figure that out, I thought my career was over.

Yes.

crappy code would delete all of the data locally.

like, a whole like, a day, a day and a half to actually run through the whole list.

things that could potentially

Ask for help. Especially in a situation where you have a custom built, you you you end up in these situations where you're like, I don't know. I'll figure it out.

Every time my boss came into the little office where the dev sat, I felt my stomach drop. Oh. Convinced that it this was it. This was the day I was going to be fired.

that is the problem. So I started commenting things out at the root of the cause. After hours of searching, rewriting, deploying, I discovered the problem. It was CSS.

all subdomains

by your teammates for shipping bad code or broken code or code that leads to something like this somebody from Meteor. I I forget. It was a while ago. It was like, my biggest regret with Meteor was shipping with an unsecured open database because Oh, man. That that was yeah. That absolutely.

stop replying. Take me off this list. So then it really encourages you to reply, I'll take me off this list again. Those reply all Oh, Wes, you know what this reminds me of? I love truffle flavoring myself. So for me, this looks like a we're not you know, I love a good, like, truffle salt or something like that. Yeah.

And there's nothing quite like being flooded with emails. You Node, Wes have one of those situations in college where everybody's on the email ESLint, and they make the email list, Yeah.

I added the handlers, tested it out, looks good, l g t m, shipped to prod and moved

We do not allow Alright. This next one just came in on Twitter, like, 10 minutes ago, and it's it's written in, like, Oh, I couldn't believe it. After I removed it, everything was fine, and I left the office at 9 Npm. And so kept my life in keeping motion limited had begun.

I think if the, the word truffle is in the the description of it, it's probably you can count on it being somewhat more costly than the alternative.

it but got it sorted out and ready for production. One of my habits at the time was throwing alert boxes up with increasingly aggressive messages for logging, and this is why something like a 1000 people saw an alert dialogue that says f this

Yeah. I give that 1, 1 Gengar out of 5.

There was, yeah. I have a behemoth a behemoth of a vocabulary.

compromised.

Yeah. I messed up our Google Analytics stats for a week or two, but in the grand scheme of things, not a huge deal and a big learning experience.

yeah, the internal tooling, man. Just a process. Someone must approve a promo code. You know? It's not just simply like, check the database for this promo code, but

Yeah.

certainly grab them something

At the same time, I was testing a bug

coupon codes are So every single scroll, you have to repaint hammered. And then so I quickly sent a, sorry. The link was broken, but then everyone thought it was like, that was a tactic because that is a tactic to send multiple emails. Oops. Wrong link. You know?

And, also, like, if you if the I woke up an hour later to dozens of messages asking me why the product is not working for anyone.

like one of our local first episodes.

these things get Like, in my own admin dashboard Wes I'm refunding money or deleting things or editing things, forward slash spooky. I've got a list of all of the past spooky stories as well.

Yeah. Okay. I was gonna say, I am, like yeah. I I do that. I'm concerned.

No. It was just them doing the the shady work, and they asked us not to use the their DNS as a backup. It was a fun week.

Next Node, Pokemon problems.

and prepared myself for the worst.

and

a 5 out of 5 for me. What what would be a good 5 out of 5? And

Being self taught and never working professionally, I had to make my own syncing system, or at least throw an error on my SQL because you they're not expecting somebody to send the B movie script,

At the time, the website was a rails app running on a custom front end framework I had to rebuild the whole infrastructure

So, therefore,

But chances are you don't have the code to just resync those statuses on hand, so then you're probably having to write a whole bit to resync statuses. Yeah. Maybe you do already have that code, but what JS it? And then deleted the overarching site folder, not the whole Vercel. Thank goodness. Oh, yes. Thank goodness. Which also contained the live site, the QA,

or Of course, the deployment went bad, and I lost a connection. I was able to run back to my apartment and finish it from there, but I certainly almost got fired the next day. Dude.

accidentally, It just goes and goes and goes. I was part of 1 with,

Man, I've never seen that word before.

It is not a show for kids. It's not a show you wanna watch with kids around. There JS adult themes and stuff. Follows the unexpected relationship between a rogue rabbi and they wanna make sure they do it right.

I had accidentally caused Google to double count all of our page visits in my 1st week on the job. Well, at least, Wes, you can just slash those by half. Right? If you were like Yeah. By some magnitude of unknownness. Yeah. For a small business that was heavily relying on SEO and Google rankings to compete with larger real estate firms, this felt like a big deal. I was convinced I was going to be fired. I reverted the change, asked for help this time, and added the events the proper way I think they generated a buzz with that one, Wes.

like, public or not, like, yeah. The c c? And I'm I'm in, like it's for an entire dorm. It's an entire dorm full of kids at at the University of Michigan, so it's a lot of people. And

man, it was something very recently

That's What? They don't let you do position fix? We can't position fix in in Safari.

and an We were going to be gone 7 days. This was before the time of just packing a laptop for the trip. So once I was out of the door, I'm basically offline.

Yeah. You know what, Wes? I have found you know, people might rip on Safari for this, but I have found Safari to be way less permissible Security might seem

rights directly to the production database schema. You have to commit a, like, a pull request

you can run a DNS lookup command

8 years later, I'm still happily employed as a developer.

probably a 1000 people or something like that.

When you're out in the forest, it can be incredibly annoying to have to handwrite camera reports and shooting summaries

that are only living on the I think it was, like, $20,000,000

when I got my browsers mixed up and accidentally uploaded the test image to the real user's account.

a I looped through every single domain name,

and then to assure it would never happen again. Additionally, I got them their highest opening rate on a campaign by accidentally sending out GitHub a subject. You wonder if that, like that only hurts. I mean, it hurts. But that hurts, like,

Of course, I didn't get fired. Once we figured out the problem, I worked with 1 of the other much more senior devs to revert the change and add the events properly.

like, people's livelihood. You know? This is their school mark. That sucks. Yeah.

off. And

update or delete clause to simply a select, or a lot of query languages will have, like, an explain or, like, an auto commit turned off. Mhmm. Basically, you want to run some sort of dry run, or you wanna be able to mark The response time wasn't too bad once we are aware of the issue,

So that way, if the message came back in, it was just, like, toggling that that value. But yeah. Oh, yeah. Yeah.

Done.

Oh, Wes.

you know, like, it's not just money, but it's also, like, people's, And Mhmm. I had I had been scaffolding out the database for it. And I was like, like, how big

Yes.

It was my job You can't you also can't do background position fixed.

deployed a change to the database schema at some point. So shout out to PlanetScale for making us make good choices there. Yeah.

And my wife absolutely she's watched it twice now. She's gone through the show twice in, like, a week. So, yeah, if you're a fan of, like, movies like, I don't know, The Wedding Singer, Forgetting Sarah Marshall, those types of movies, it it's in that same kind of vein. And it's it's a cute cute romantic show, but at the same time, super funny. Yeah. I'll have to check it out.

In fact, I learned that soft deletes were kind of necessary for doing some local first stuff when you're dealing with patch messages.

They sell this at my Costco. I always look at it and wonder if it's any good. So, I will have to pick this one up. Yeah. It's it's really good. I kinda wish it was in, like, a squeeze bottle, but it's in this, like, really high end glass bottle. It's kind of expensive, but,

which has Scott put the person who had to sign the contract in a uncomfortable place. Right? Like, because then now they're like, I do I have to sign now? Like, do I Scott? Yeah. Like,

and I did a DNS lookup to see if there were MX records sat on that domain name. And if there's no MX records, it can't receive email.

and RDS.

Those who responded at 3 AM were able to prevent any other $5,000,000 which is an odd job, you think, to have to write all these, like like, somebody

Yes. I give this 1 5 out of 5, black holes. Woah. Alright. Next 1, ruined vacation. I well, terrible. No. No. Cove Bugs ruining a vacation. Your vacation is supposed to be your your chill time. My wife and I were leaving that morning for a trip to Hawaii.

locally. Right? And if you don't have a backup process to to sync those down or to put them onto an external bucket, you're you're pooched.

from evaporating.

You know? But so The Europe thing. Yeah. There's a lot to that as well. Yeah. For sure. That soft delete thing, I think some people might see that as a negative, but you could have soft delete and then a process that deletes those soft deletes after

goof up.

truffle hot sauce. I've been I've been having this for a number of years because my sister had been give it gifting it to me for Christmas. She's from the States, but they now sell it in Canada, which I've been pretty excited about. So truff hot sauce

2 Jasons out of 5.

coding an app layout. Let's stop talking about it. So the timing I think that would be good. Yes. Yes. Okay. Next 1, boot camp bungle.

2 days

make the the the body part scrollable instead of the entire thing. Put a you put an overflow scroll instead of a position fixed. Interesting. I think.

Last minute update.

user diploma upload in a testing environment And, for a long time, I believe, Sasha no. Who who was it? Man, somebody from Yeah.

You need

it's too intensive because you have to recompute with position fixed. It's stuck on the viewport.

Next 1 we have here, limitless coupon disaster. I was part of an incident review at one time. So this didn't happen to this guy directly. But when things happen, they often bring in engineers and people from management to sort of review what had happened and figure out how you can fix it.

DNS names to identify do color it differently.

read and write permissions were wide open. And that led to huge adoption of Firebase because you just had this object you could update,

The taxi showed up,

before they routed it to Google or Cloudflare.

ran the campaign, and I forgot to pull out the error handling out. Folks, stop doing this.

That's a good one. So funny. That's often when I'm doing, like, an admin dashboard, I'll often copies will fit into a medium text field, though. So if you if you do need that, that's there for you.

And he's like, no.

made by Truff.

you delete the wrong thing.

with the,

Identify top 5,000,000 websites by rank, classify them, enumerate over 500,000

plus users just got reset back

Do things the right way.

that had been built by the previous team, which had been completely laid off after the company had been sold. Yes. This is the classic you use their framework or you, make Node yourself that is not as good.

and everything would be mirrored, and it'd all be real time, and everything works great.

Oh. Maybe we should we'll do a YouTube video of

because, Looked like it never happened except a small maintenance

The next morning, we woke up $3,000,000 If you want us to read it next year, go to syntax.fmforward/spooky

nobody wants this. And at at first,

when we ran

But they also knew that they were flooding me and jumping between 2, 3 campaigns at once. It was my 1st and only major screw up within 10 years of working with them, so they were forgiving.

And then everybody's like, what they did? Let's celebrate. And then they had to immediately send an email. Like, by the way, this player did not actually sign the contract. We were just we Scott send it,

I once emailed, should a story be? Because you have to choose the in MySQL, to choose the type of column where it will go and how much text it can possibly hold.

as well as prevent them from getting wet. I've been coding since I was a kid, but at that point, I had never worked professionally as a programmer. I didn't even use Git. I hacked together an Android app that could handle it all and the different paperwork our school required.

Wes should try. Always reaching for a position fixed for that. I I guess, you know, that that works. The Yeah. But it's because that with position fixed, then you you give up the flow of the document, and then you gotta account for the height of the nav. With with grid, you And I never big vocabulary?

and I YOLO'd, like, just 5 or 6 package updates. And they were, like, all small patch Vercel.

Beautiful. Alright. Peace.

between the phone and my server. I handled this on the server side. Whenever the app synced, I got banned very quickly. Mhmm. So I had to I had to really cut it back and do do rate limiting. I think it ended up taking, Safari would crash.

Pittsburgh Penguins.

was being used to tether? You know? Like, that

how long ago was this that your BlackBerry food condiments, like, they're a condiment enthusiast, it would turn all the paperwork data into JSON and send it to the server. The server would then combine the query, merge it with the local data, and finally respond with the new data. The app would then overwrite its local data with the new merged data from the server. You may have noticed that overwrite was in quotes. Somehow, I figured the easiest day way to do this would be by deleting the app's local data

probably.

as a network connection.

Never do that, especially for ESLint facing stuff. So that was the first one. And then and then he goes on to say, at the same company, I decided to do a late night planned deploy a in a while. I guess, like, if you want, like, a tab to over overlay the entire website.

don't deploy at a bar with a blackberry.

I was like, I don't know. Yeah. How much padding do you do, folks? How much padding? What happens if they variable. Piece of size? What happens if it's a German word and it's multiple lines?

Yeah. I give this one about 3 rotten bananas. Oh. Next Node is the spooky integration bug.

from a bar

Of course, the deploy went bad, and I lost connection.

frontline support agents

That night, Every single one of these spooky stories episodes is hilarious. They're they're my favorite. They're the funniest ones of the year, so, check them out. You may find some good stuff if you like this one. That's all I got.

and production databases, I didn't remember that if it's text in an SVG, then it would be listed as text Turns out when I was working on the integration, I was accessing certain properties of the account object that would only be available if the integration was enabled.

sanitization.

Node directory too high while deleting that agency's staging site on the web server so I could reupload it, which is a pretty routine operation, believe it or not. Oh, come on. This is why we use, this is why we have our processes like that we do now, folks. You wonder why it's so complicated. We used to, what, SSH in. You'd remove things. You'd upload things. You drag and drop.

And a MySQL text column can hold 60,000 spooky stories. These are stories of web development gone bad, Didn't say a thing. You know? It happens.

my dorm room turned into a battleground as my classmates started dumping movie TypeScript left and right, testing the limits of the app. I was in over my head, but luckily, our instructor seasoned giant Vercel hornet coming to sting you.

Next one's called brute force too brutal. 10 years ago, I was working at a Silicon Valley Scott up, and I'd only been there about a month ago or so.

and the real kicker. All of that site's backups. Oh, no. They kept their backups.

Many gigs. You know? Wes, 30 gigs, so I was using Beamovie to test. So Beamovie will not fit having or showing the a tendency to be easily angered. I have never in my life heard that word before.

That that probably hurts you specifically, Wes, because I know you have a thing about making sure off-site backups and backups Yarn away and stuff like that. Yes. Yes. I had a local development copy of the agency site, but the database was several months old and numerous copy tweaks have been made by the partners that were not recoverable.

Yeah. Yeah. This, this Node in particular, man, that

a shared housing app for us dorm dwellers on California Street.

Oh, man. Which is, starting on the day that my first deployment.

at the data center because their entire trunk was severed.

I I actually got that story, They kept their back They kept their backups on the same server you're screwed.

backup directory on the same server. No. They didn't fix it. That's not fixing it.

tape drive. Like, this is like, not a CD, not a floppy, like, tape.

perfect for an opportunity like this.

who would be working in a few hours. The site got millions of hits per hour. I woke up the database admin, and he spent 3 hours restoring from a tape

a couple years later, the sole trunk line into the data center housing our server was severed.

Things were going well until 1 fateful night when someone decided to upload the entire b movie script in into a text input field.

industry veterans Seems simple enough. Right?

And if the only place where your website is is on the live server Dry run queries, soft deletes.

I was determined to create something cool, What could have been a humiliating experience became a testament to the resilience and camaraderie of our tight knit group.

Instantly, the entire site was gone. 20,000

So without thinking it through, I decided to disable the public network adapter to kill the attacker's connection. Yeah. If you can't there's no security issue if you can't access it. Right? That's when my mouse stopped moving and my heart sank.

Our cloud hosted Microsoft SQL Server was acting up, so I remote desktop into it and discovered someone was trying to brute force log into it.

Yeah. Yes. Toffee candies.

today, especially Amazon Glacier JS tape, and that was about a day old. I did not sleep that night, and nobody but me and my new DB admin friend ever fell out. What a cool

We usually used Unsplash to have some images to work with while developing. This accidentally ended up on production for a customer. It was a fashion brand, and we had only been using pictures of goats.

This one's spooky, but I think it's somewhat lower stakes. I'll give this 3 skeleton hands coming out of the ground. Yeah. That's fair.

Oh, I'm gonna give that a 7 and a half,

into a, MySQL text field.

like, input type and name, or you can just remove with dev tools the max length on it. And,

Looking back at that fateful night in the dorm room taught me more than any textbook ever could. It was a baptism by fire, went down.

support articles, all gone.

was tasked with building projects daily.

characters. And I was like, what's bigger than 60,000 characters? And and the the joke in the industry is the b movie script. Wes. So if you ever find an input, what was it, Microsoft MVP. They accidentally This was 25 years ago. Although, like, tape still is is used Oh,

the only heads up at a like, like, 4000 people's jobs are on the you know Wes you get, like, you get a call or you're trying to get some customer service, and they're like, I'm sorry. The system is not working right now. Yeah. It's probably something like that.

The this next one is the most hilarious. I not the most hilarious, but just I was laughing really hard at it. The title here is no goats. At my previous company, we added a no goats chat box to the definition of done checklist of pull requests.

and having to restore from that could take a while. Yeah.

into opportunities for learning and growth. Yes. This one is low stakes. Obviously, I'm gonna give it I it you know, it it's funny. I'll give it, oh, you know, Node

their server

Next Node is a dorm room b movie. It all started with a simple housing management app I was building as part of a program

I had just completely cut us off from accessing the database server that powered our store and had all the customer records in it.

especially on smaller pnpm, people with a broken link. And the problem is that I noticed it

and you can't get to it,

Suddenly, my carefully crafted app was overrun with lines of dialogue from the classic comedy film. I scrambled to figure out how to address this issue, realizing the hard way the importance of pnpm,

Yeah. This, man, this help desk is gonna need a help desk. That is

However, they fixed the backup strategy by backing up to a dedicated We were at the, Man. Werewolves. The The the butterfly pavilion in Colorado, and my daughter who's 5 was like, I really wanna hold the tarantula.

Yeah. Yeah. Back restoring a backup from tape, I'm sure, is something that

hyphen r, which is recursive. So it's removing

and the query was taking a few seconds, swears or, It was immediately

a reminder that in a world of software engineering, you never know what challenges may arise. But with the right mindset and support, you can turn even the most chaotic situations somewhere.

database admin guy. He just just So JS this temp banned for 15 Node.

Through their guidance, I not only learned how to properly secure my application, but also the power of collaboration and problem solving.

It will fit into a I had 203 Don't worth of toilet paper to the wrong people and just awful stories of web development gone awry.

as And, thankfully, we can laugh at them and also learn a thing or 2 of them. We would love to hear your spooky story.

most people who back up to tape would say they would never want want to have to do that. It's just there for situations like this. Yeah. Even, like, backing up, if you have the backups, they can take a long time. Like, the databases can be tape.

Thank their lucky stars that this happened to their website and not a client's.

Yeah. I give this one about, you know, I I don't know if this is a total 5 out of 5, but I do think this is yeah. Definitely 4 4 out of 5 in terms of, what what can we assign this 1? Bugs? Creepy crawlies? 4 to 5 creepy crawlies? Yeah. I think so. I'm sorry. I'm getting choked out by my popcorn here. It's

One time around midnight, I was emptying the tables of my local environment before syncing with production, If you are doing a major

casted the entire Redux store as any, Yeah.

Yeah. Yeah. So someone else has gotta be able to oh my gosh.

impersonate, masquerade as a user and stuff like that to Wes. Or you had different classes of users. So or you you ran Wes as different types of users to check those access levels, maybe ESLint integration tests. Alright. And the 2nd part of this is I I did investigate why TypeScript didn't catch that. And it turns out someone

killing me. I'm, like, so hot right now.

where, oh, if they are this, then then add this thing to the navigation. You know? And if they have this feature and as you get more and more flags in your your application, it gets very complex. We had to,

again, you gotta have these thing Wes you're when you're starting something like this. You gotta have that 4th forethought to just be like Those, like, edge cases. I often think about that where people have very complex navigations

Oh, yeah. The fix took 15 seconds and all I did was add a question mark, but finding it took a while.

to at least try it. You know?

I later heard back that I was almost fired for that, but my manager argued that since my code was removed by the tech lead, What are you talking about? No.

Yeah. I get that, though. I mean, if you have all the accounts everybody's working on, everybody has that role, everybody has their access, and then you push up and you just man. But,

This never came up in testing because of all of our test accounts had that integration enabled.

Next one's called worst case wake up. My 1st big web success was designing a CMS for a customer service department of my company. It was the only source of information for our customer service agents, and without it, they could not process orders or answer many questions.

This might be the the the episode I laugh at the most that we record all year.

put in, like, in a lot of, like, what Wes you just take a guess. And, it's always important to ask questions. And you're good about that, Wes. Whenever you have any major concerns, you ask me in Slack right away.

yeah. TypeScript not even doing anything. That is Oh. That's wild.

Yeah. As any just but as any Anytime, though, like, even the autocomplete stops working, that raises a red flag in my head being like, oh, something's not typed properly here. Let's let's chase it back.

My story is from the time I was working at a big Scott up. We were preparing to launch an integration with some third party software. This was a big deal. It was announced during a live stream with the CEO demonstrating everything.

which meant nothing was typed. I kept telling myself that it wasn't my fault because of that. Man, if you type everything JS any I'm sure you can all guess what happened, but, yes, loading the Google Analytics Wes I Sanity imagine being fired for this personally, but you never know.

dropping a table,

Woo hoo.

This is the 2nd part of our 2024

letting something sneak into production, totally their website. So if the website Right next to away, so does their backups.

shipping. We had 1 a couple Yarn ago, shipping, like, it is often worth switching your when they shouldn't have, and then it just ripples through the entire thing, and and everybody starts freaking out. I just I would feel so bad if I did that. And, honestly, that's maybe we should move on to the next section, which is, like, what did we learn? Like, you need What did we learn? Processes for

$80,000

Welcome to syntax.