Even SCARIER Web Dev Nightmares (Spooky Stories Pt. 2)

For development and testing purposes, we had regular SMS numbers little decision to reboot.

That's good. PS, the main problem, The can we can we ask this? What's the worst For two weeks, it felt like months. I'm sitting there thinking, wow. Business is slow today. Nobody's reached out. Guess it's a quiet week. Meanwhile, my email service is completely

Just let certbot fail silently and wait for the SSL to expire.

to Netlify.

Is this about AWS?

just the fact that it not only is awful Man, This is a really quick one. I switched my VPS provider and forgot to point my portfolio Access control? That I give that one ten ghosts out of 10. People talk about, like, vibe coding and building an app in an afternoon.

I think they are have every person in Alaska as a customer.

where And the site is still running to this day five years later. It made me think a little

or all drop table queries.

We've got our second episode of twenty twenty five spooky stories. These are stories of be able to be duplicated if you don't if they file a chargeback and then you immediately Never do this.

We work together with different shipping companies depending on the warehouse

Syntax.

Yeah. To people who did not pay for them.

got a second charge to their credit card

of the try catch block, resulting in failed video generations.

Backups had been offline for years.

stories, so I'm gonna leave it for that. Is that you think that's alright, Scott? It's actually a good one. I want to read the ad.

When I first joined my company as a junior developer, I was testing out an issue with one of our alert systems. By tagging a very popular artist to my test event, I managed to send

Oh,

Spooky.

we noticed.

and test events

thankfully, out of business today, had a special endpoint for capturing the payments. TypeScript is exact same way, by the way. Auto capture was optional, but not default to change the DNS. You Node, that's what I do. I put them all in Cloudflare, and then I just point them at whichever servers you actually need because that way you don't have downtime. But, also, like, when you import How do you do that? developers had been given a halt

Yeah. That's that's fine.

I don't know if I'll still like it, but I liked it at the time. That was the style then. I would blame him. I would blame graphite, but, ultimately, I have to be forthcoming and put my ego aside in such situations used to email you when your certs expired.

Cash ruins everything around me. Cream get the money. Dollar dollar bill, y'all.

from internal users that local storage was filling up, and it was due to the analytics client.

that just said, make sure it doesn't happen again and left it at that.

style migration if you're asking me. That's,

It started working again. You don't even need to wait twenty four hours there. You Node me. Yeah. There's, like, two things with DNS JS Wes you're changing the name servers, you're literally DDoS. Like, being able to just you you send pnpm SMS.

for the entire day, and then I go, wow. Nice nice quiet day today.

brutal ones that have have yet to come.

system at a US mall along with a $100,000 infected with some kind of zombie disease. So they're Zeke in the chat just said that's a 150,000 specific wrappers.

very spooky.

degrees in full speed. It had It's always about DNS. I had been debating about staying with Bluehost or moving over to Netlify for my small business website. I was tired of paying for hosting and figured a free Netlify Wrappers around the clients.

As long as the dog is 15 years old,

I did one indentation, and the thing drove itself into the ocean. Uh-huh. Please don't We gave our AI a year to Bluehost when Netlify could host it for free? WordPress JS overkill for what I needed. Now I should mention, I'm a designer who dabbles in front end development.

We didn't account for the fact that the API could be down and the gas would never let up.

and, you can you can submit your story to us, and we'll read it on next year's episode. We're always collecting them and always love hearing them.

It took a few days before the thick

Never.

Certbot's certain death.

doing these migrations. I always found that to be very spooky story, Val. Thank you for crafting that.

So, The way it was set up was not correctly doing something. I don't remember why exactly.

I'm sure you had already guessed what happened. So By the way, we keep all of these anonymous.

before dedicated team was assigned to rein in the cost.

spooky stories episodes.

without any electricity.

even though it should have been a lot worse. I would imagine what they would have to do is, like, go through amounts and, like, Wes and then contact the customers and be like, our dumbass web developer I just overrode the database entries for all of the last few days of purchases.

because the guy wanted his own site. I was doing everything in WP ESLint to speed things up because there was tons of data. Well, I didn't realize that WP CLI was going to trigger the WooCommerce email system as I was migrating the data over. It sent out emails to people about their purchases.

JS we all do,

Well, you know what they say, you know, fool me once, shame on you. Fool me twice,

So we have a century plug in one of these

around 5,000, Only my new table is left.

banned system, we worked this out and thinking about the implementation of the existing subscriptions.

use a system, Oh, classic.

but when I logged into my domain registrar and I saw that wall of cryptic records, a records, MX records, c name, TXT SPF, my brain just went, yeah. I got this. Shouldn't be too hard.

How to lose hundreds of hours of meeting recordings in one easy step.

In the routine of my dry job, I formatted a CEO's hard drive. So, Drew, you do something every single day.

hey, brother. I just, Mid conversation, it was a crazy day at the office. I'll tell you that.

Build cache, bro.

Physical damage.

all with different variations of how to fix DNS records, Netlify Google Workspace.

Dude.

for a special feature, I took some days to get familiar with the system and wrote out exactly how I would change it in the areas that would be touched. From running subscriptions, I wrote a migration strategy that discussed everything with the team and my manager.

and had their next order delivered in these days had been paying every day since the feature launch.

was already burnt out. Let me tell you,

What does the p in VPS stand for? Killer snowman.

I want to read the ad. So if you didn't catch the the last Node, if you're listening on audio, Scott has, like, a very good vampire emails from angry and confused customers.

SSL certificate I built a very nice website for the National Hockey League.

and CoffeeScript, and everything was indentation based. Same. Yeah.

To give you an idea, we average about 4,500

You know what? They probably had the Stripe

had an option to start a subscription locally and in CI.

And so I just started mashing the face paint around, and, what you see here is what you get. And you know what? I did a pretty good job, actually. You look great.

I'm gonna get a little Arnold there.

spooky stories episode, Wes. Thank you so much, Site's down. White screen.

API keys in the database, and he pulled the production database locally.

over order nightmare.

Luckily, I was able to salvage our relationship lock files are are very, very important.

to the new IP address.

I updated my name servers to point to Netlify as their documentation walked through and watched,

10 great white sharks that are we don't mind. The dog was on its way out anyways.

is a massive, massive thing.

you gotta pay for there.

after being told it should be in the keep the lights on mode, which is minimal development. Just keep it running. The service API was based on ESLint IO in order to 3,000,000,000

I don't. I said, what is this? I'm losing the accent. I don't have a plan.

A donator,

but the Rackspace

all the time unfortunately, Soon, it became apparent that the fix was causing more slowdowns on client browsers and also sometimes prevented garbage collection from running due to the frequency Oh, I actually did I did that once

I I hate subscriptions with digital products. With shipping and then shipping dates that that things have to arrive in, take me out back. I'm done. Man. Yeah. Logistics.

You don't do the one thing supposed to do. But luckily, the payment gateway kept pending transactions for six months. It had been five months in twenty days.

goes, oh, must be actual error. Your initial thoughts are just like, probably Cypress is acting up.

woes. I was doing some freelance work for WordPress when I was on a hot streak with quite a few jobs done at all five star reviews until this job.

web development and software development in general of just Like, you cost Yeah. Next one's about the NHL, though. Maybe those make us feel better.

into the same issue. Let's get that fixed out. Oh, crap. They were weren't supposed to get the apps, so they did a hotfix managing infrastructure, writing Helm charts, setting up alerts, yada yada yada. So I'm not afraid to jump into some Python code Bos and make some changes. This sounds like something I would say. I've been writing a lot of Python lately. The Python service I've been working on was originally written by a messy ex Google engineer, and I'm stuck with them Shame. And a broom cleaning up the mess. A typical

check out the spooky stories episodes on Syntax.

and eventually left the company.

So all those donations in memory for dead people have only been authorized Oh,

it had gotten hacked somehow. I don't know how.

You've taken down websites. We've got some real doozies,

certbot stopped working. No alerts, no warnings, just quiet little failure sitting there

to leverage story here is the analytics is not designed for platform monitoring. I think there's another tool for that, like ...century.io.

changing who manages your DNS. If you can change it from provider to provider or you can just use your registrar

Yes.

a host or something

finally ripped out large chunks of the ventilation This particularly messy Python service was hitting Google's cloud video generation

Oh.

But that's where it gets worse. I had no idea how to fix it. Remember, I didn't know back end network DNS stuff, so I did what any panic designer does. I started googling.

in production.

It's so huge.

Oh, so people have been donating for months to honor loved ones, and the money never moved.

parts to work in? I'm gonna say Recurring bill. Sign on. Recurring billing. That. Single sign on and maybe auth in general.

My stomach drops. That two line PR?

and I sent it to Randy, our producer JS well. And the name of the screen recording was f this,

Yarn's cache handled it fine, but Vercel's build didn't use the cache.

laptops and other hardware for new fire API. So every time you're touching it, you're you're actually running a synchronous

as this is a running system and had been going for ten days before Yeah. I get that I get that one about All of their customers,

was the Vercel alliance on the analytics problem kept growing despite being in maintenance mode, and it hit a peak of 13,000,000,000

spookiness.

like, you're essentially getting file in our code Bos to use our logger instead of print. So doing so required moving around a few things, etcetera. Makes sense? sent

Oh, this Node, because it involves the the spirit of lost ones, I am going to say this one is a nine out of 10. That is very spooky to me. Oh, man. I'm gonna give that one a nine out of nine and a half. That was was alive, but the SSL

that sucks. That sucks. I'm gonna give that one a a one out of no. A two out of three.

and includes money, but also includes, like, something sensitive, which is donating memory of loved ones. Yep.

may take twenty four to forty eight hours pop up, and I felt like a tech genius. Is this really what back end devs deal with? Super easy. JS a self certificate check, working. I gave myself a little victory fist pump.

And the guy I was doing the job woke up to over 18,000 more stories than than we actually needed, but please keep them coming. Syntax.fm4/spooky.

The year was 2019. I was on the experimentation led into the building you could've you could've DM'd me, and I would've fixed it for you in, like, ten seconds.

Don't put adult sites in your work. Don't do that.

my host has got baked in because every host in the world's got that now. Who's not giving you SSL renewals for free? I actually just looked it up. Let's Encrypt

and tell you where it's pointing to, and you can see if it's updated. So, like, just such a common thing. And the the infrastructure between all of these peers,

millions and millions of dollar, Jeez. That's crazy. At the time, I had been there for about a year. The subscription system was a bit of a gray box for everyone, where we could change some things, but nobody on the team had full context on how it all worked exactly, and there was no documentation.

That's a long time, folks. That's a long time for this the the just wait part. And I see a group of people crowded by a laptop on my way to the office, and the CFO is standing by them looking moderately perturbed.

Killer snowman. Jack Frost's, I believe. I don't that's a movie, but I haven't seen it. WordPress The server was completely

None of the donations in multiple rooms and then finally

r m r f forward slash star boom.

self driving code and had it rewrite it.

Node. Yep. Next one JS called a fiber f up. I ripped out and destroyed the ventilation

That's a much bigger oh, no than I had initially given that.

Sadly Mhmm.

scary when you have that much can like, with WordPress or Drupal, like, doing migrations in that way where the database is such an essential part of the whole thing. I yeah. There was, like, a Drupal features and ways that you would, commit things and whatever to get things into Git, and it was just, like, always very scary when you did any kind of deployment process. Another WordPress woah. One time, I was working on converting an old shop site.

The payment provider, things update much more quickly.

dispatching real alerts.

you know, The day before launch, the whole team even starts working from six in the morning for their last big push of content.

but I still it's still,

awful things that have happened in your career.

I opened up a PR. I ping warp trusted front end dev to rubber stamp it, and logs are pretty, and people can create VO videos through the API. Everything is good. Alright.

I had been working at a premium pet food ecommerce Scott.

and to be able to recover queues from tabs that have been closed. That's another good reason for it. We started getting reports Brutal.

going down the most frustrating documentation rabbit hole of my life. I've read Netlify's DNS documentation, which would tell me to check my email provider settings. So I go to Google Workspace documentation, which would tell me to configure my DNS records with my hosting provider. So I go back to Netlify docs. I was stuck in an infinite loop. I even Wes chat g p t and gave it to me back and forth in the same flow.

because apparently on certbot alone is a bit like relying on Goldfish to guard your house. Moral of the story, if certbot fails quietly, you you can't be fooled again.

they've stopped that. So they won't even you you'd need, like, a a downtime, like a like a ping like a pinging service would Ping service for sure. Easy

out of your pocket. If you wanna get pissed off people your customers,

I was thinking Wes should do a video of who can crash the browser tab first.

You'll throw them in, like, a variable or you'll throw them in local storage so that if there is if they're blocked by, like, a a thing, if if the person goes offline for whatever reason, at least they're in local storage and you can retry That sucks so much.

One

We record our Zoom meetings through their RTMP for that's bad day at the office.

We put this into production. Everything ran smoothly, and we were getting positive feedback from our customers. Oh, man. The hammer's about to drop here. But after ten days, we were getting complaints. People who had subscriptions

system it took a few days before the thick fiber wires over and over again. This uses multiple local storage entries to manage the queues for processing, onto the schedule for next year because there are some

server.

like, webhook comes in, yeah, you could have tested it, whatever, but it's still, like, the dollar. So we'll see how that works out once that all because there there was, like, some something where, like, a electric car got pulled over, and it's like the car Yarn can't be held at fault. So, like, what are they gonna do? Like, what what are they gonna do when something crazy happens with electric cars? I better not hear a dev horror story about a No. I know. Or right. That's true. Self driving car ride. Wes decided it's okay to hit dogs as long as you're not killing a human.

Thank you.

forward slash syntax and get JS very spooky to have that happen, especially when it's such a big like, the NHL. You know? Like, there's 10 people working on it. This is that's that's a lot of money in in paying people's salaries.

Oh my god.

That

video production files I had going and kinda borked my system, but it didn't I I noticed what I was doing, so I I was, really quick on the command c, I think, unfortunately, for humanity, I I don't know how many of these companies are gonna put

even if you you win or regardless. So, like, I don't know, 5,000 people, maybe of 1% of them

For running subscriptions, we had to calculate what was the last day of the week they received their last order. Then we calculated the next shipping date so that it would receive on the same day. But with the given subscription interval calculated into it, we had to keep in mind the subscriptions were running so that we could need to calculate the time from their last order until now and subtract Every single meeting stream started failing.

rebuild everything from, like, emails sent out. Yes. Hopefully, you can go into your your sent emails or logs or something like that. But just much mind because we ran a colossal Then I see the error.

after placing the order. This got really complex as we sold it to about 20 countries. Oh, yeah. Ho ho ho.

Man. But, like, I don't know if these big companies can do this, but, like, I have have come through.

Yeah. Next one's called fired.

everyone a push notification

Example, failed due to safety checks. Yeah. Like, you try to create video with something that is is not appropriate, they'll they'll fail it, to be tried

Yeah. Take a bite out of your errors.

Probably.

and we learned some very

So it took us a while to realize that we would occasionally be seeing single analytics event,

My second PR,

I expected to be written up for having left the dev mode enabled in production,

my wife said,

but never actually charged. Oh, no.

being piped through fiber and then into lamps. So, like, they're lighting the mall that was not meant to be pushed out yet. Suffice to say, it was a comedy of errors that took days to fix. And if we had Century ESLint our build system, we likely wouldn't have run into this this issue. Thank you, century.century.i0/ Somehow, I made those two decisions all by myself, and I did not make the connection.

So it's sunlight coming from outside don't put

cost horror stories in the previous

required for good light, but the fiber wire especially needs to be custom made. Normal fiber for Internet audio, etcetera, for the sake of anon anonymity.

My little cleanup PR introduced the line that unindented the message acknowledgments

You just do it. I mean, just use use effect. Just

any application that crashed.

so much stuff more than email if you don't know about it. That's the fun part. I craft the most embarrassing sorry for the technical difficulty email I've ever written and sending it to every client I could think of. I had to do damage control. I apologize perversely and basically admit that, yes, the UX guy broke his own website user experience.

of their pet food orders. This would create an order on the interval the customer set him or herself. However, the person that created the whole subscription system went into a burnout

one tiny thing. I never hit capture.

We've had a couple stories in the past that were in the millions of dollars, because they they shipped out physical products Man,

and spooky So this is like it's a it's a train wreck.

of local storage checks. This is really interesting because local storage is a a synchronous

SSL expires that breaks your soul. What a wonderfully written, When I pull up the site to my local machine and spun up in my dev environment at the time, and I start to look through some things, well, it turns out that they were wrong about custom Stripe stuff, and there was some code that automatically ran a bunch of webhook stuff when loaded locally.

Oh,

B with billion.

in equipment from a single retry.

because of the insane tolerances fix that for you. Alright. Next one. It's always I can build and style it, make it look gorgeous, and but deployments and server side stuff, that's not really my wheelhouse, and I'll usually work with other developers that would handle those things. Still, I watched a few YouTube tutorials in Netlify, and everybody made it look so easy. How hard could it be? Right? So one Saturday morning in the air of confidence and iced coffee, I decided to cancel my hosting and begin migrating I never know with you, Wes. So the next one is entitled This guy caused physical damage in the magnitude of hundreds of thousands of dollars.

Yeah. Including these two, probably 20. So check them out. Tons and tons of spooky stories. If you have a spooky story, submit it on our web our website and text Scott of him, and, we might read it next year. So

is Crap. This is our API call failing in their local dev environment because we didn't add a proxy or dummy data. We forgot to add a failure Scott. And because we didn't catch the error, it's throwing at Webpack's red screen of death, which blocks Cypress from querying anything on the page. We rushed out a failure state, which solves a problem in our main development branches.

Google employee writing a bunch of messy Python.

streams.

They a lot of them do go skiing for sure. Yes. Alright. The next one, Rack Spaced Out.

I mean, I want a seven out of 10 spooky spooky bookies.

He queued it up to blame himself. He just says it's the Python problem. So, yeah, lots of yapping to say Node invisible line change caused JS a hundreds of thousands of dollars to the Google abyss. Thankfully, my colleagues were quite forgiving, and I'm still working hard to redeem myself in the finance department. I've definitely been given a shorter leash on the head of engineering.

People could donate money for special occasions like birthdays, but mostly in memory of somebody who has passed away. That's nice. Everything worked perfectly at launch, and I even tested it myself. The payment showed up as reserved on my card. All good. You know? Like, you you

Suffice to say, the client was very upset, and I still got lucky. I still got away with a three star review

I spent the next two days

Then one day during the mysterious deployment, probably a Friday,

And a second Who can write the worst code to crash a browser tab? I think that would be good. Crash your browser tab in

had the same package at two slightly different minor versions NHL

when it's just like Billion.

We then took the closest day calculated from the next shipment date and would then change the shipment date for the existing subscription. Oh my god. Bro,

or any adult stuff or or even, like, Before I worked as a web dev, I worked in IT doing desktop network and Vercel support. I had a job working for a financial company in San Francisco.

but I had an amazing CEO

implementing sync pack lint to the CI to catch version mismatches.

for making this happen live. I thought that was a really neat little titch you did. People Yarn watching on podcast. We ended up streaming this live kind of randomly on Twitter. Scott of fun, folks. I hope you enjoyed yourself if you're watching live, if you're listening to the episode in your car, if you're watching on YouTube, we do this every single year. So if you like these dev horror stories, there are some you're going to eventually catch somebody who's not doing bad stuff for whatever reason. Don't do that. Yeah. That's not a good idea. You can you can just send them to a blank page. That's just as easy. And they're like, no harm, no foul. You can trap the you can do honeypots. You can do all that stuff, and you can still annoy them by I had a band system on LevelUp that would, like, act as if they weren't logged ESLint the site. So then they would go to try to log in again, but then the login would fail because they were already logged ESLint the site. Yeah. Like and that way, like, if, if for some reason, someone ever got and

I have config

I didn't use Git or any other version control.

remote database.

ran

Almost nobody

it's not the failure that hurts. It's the moment do you have a plan? And I said, no.

and and it

use use effect. Use effect.

Alaskian not on their production, but on their, I guess, their main branch, which is not yet released. So they pushed out a bunch of code rolling backups running in multiple places

folks, if if you enjoyed these and you're looking for more bone chilling tales,

Cypress test suite for a monolithic repro, so that could be anything.

model. Okay. So this is relatively do not work for visible light. They are made to transmit

One time I ran git clean on my root directory.

Oh. We have a backup of his laptop before, but it was not one of the CEO's laptop that was backed up. I was looking for work the next day.

Oh. What? It's a big that's a big company, Alaskian.

On the production server of a large commercial real estate company.

that was playing during the Deno. He was redirected to the adult site several times. It was a miracle we didn't lose the contract and that he didn't get fired.

by leaving a a dev flag Never refund it, you still pay that $15 From everything my team tested, it worked great. We thought we had accounted for everything, so we push it up to our repo, get it approved, publish the version, and get it installed in our front end applications.

This one includes the company name, so I'm just going to change the company name

Spooky bookies. I'm gonna give it five Jason masks out of seven Jason masks.

Ivan, the dreaded

you guys should be using Century at century.i0

23 year old me was thoroughly impressed that the dude stayed on the phone with me for hours,

I didn't. But I was panicking for, like, probably an hour. Yeah. It sucks.

But I don't think they say that.

And it's it's not even like you're getting, like like, a hard drive or something fun. It's like like dog food.

a domain name into that's like, a fail safe. Fail safes, man, are crazy when especially when they're real world where they could like, I often think about, like, self driving fail safes and all of the code that needs to be written for that.

After digging around, I found that the client would detect that local storage JS full and switch the entire storage solution to memory, which prevented any further cleanup inside of local storage. Once we got that fixed out, we pushed it to all products only to to start causing more turned into a hammer for many problems, but that's a different story.

And that's that's what had happened to it. They just replaced every URL with a bunch of redirects to adult sites. And,

with the influx of additional monitoring front end teams that were trying to be shoehorned in. Yeah. This is this is monstrous. Like, we we had folks on the past from Century

A whole team of content specialists,

So I updated the main dot py unfortunately, Wow. Well, we had JS man. I I gave it a nine and a half out of 11. That was good.

Fast forward a smidge. The reports start to roll in that tests in the front end application are failing and holding up CI and CD.

but your coworkers and get the job done with a five star review.

it's it's still always very scary for for, like, what actually is is doing that. There was a big demand of the customers that they could choose which day of the week they would receive their food JS hybrid remote work was very popular in our country. They wanted to receive their food when they worked from home. This would be for regular orders and subscription orders. For example, for regular orders, you could choose that you would receive it the first Thursday You think seven oops out of eight? Okay. Next one here, rm-rf./star.

I've been I'll take 10 extra orders of dog food because you know they're gonna refund it at this point. Luckily, I created a backup of the database when I launched the feature into production so we could see who had upcoming orders, and we then had to spend the next week with the whole team figuring out which customers had placed how many extra orders to reimburse them all. Oh, very spooky. Yeah. I'm getting fired for something. Very spooky. I would say that this is a, you know, a six out of eight killer snowman for sure. I would give the same as well.

nonfunctional,

accidentally stuck into the that ended up spinning up a whole new dedicated server and moving over what pieces he could to the new box. I lost a few days of data in the end, but got most of it back up. Well, that's that's a lucky,

eight inch lenses I finally pieced together enough enough information to understand the need to manually add Google's MX records back into the Netlify DNS settings. I copied those magic strings of text from Google support page. I pasted them into Netlify. I waited for what seemed like another agonizing twenty four hours for propagation,

oh, jeez. So they they ran a SQL command that just dropped all the tables and then added their new table. The latest database table was from three days before, so all of the content was gone. Yes. The launch was delayed. Yes. I was ashamed.

Peace.

start filling the site with content.

events per second.

White screen for a thousand users.

up over the standard out in using a print statement instead of our structured logger.

You sent your you sent your boss to an adult site.

Oh, I guess that IP address was reused, and it just never changed where it was pointing to. That's great. Yeah. One of the first agencies I worked for, we were doing WordPress sites. And the only site I'd ever worked on, it wasn't even my my client, but

Oh, yeah. It's it's an expensive API, so this literally had cost the company hundreds of thousands of dollars. The crazy part is it was just a white space change. It didn't even show up on our code review platform, which is Graphite. So the dev I pnpm rubber stamped the PR, didn't even notice the change. Freaking Python with its indentation. I used to be such an indentation fan. You know? I used Pug and Stylus and every email sent to me is bouncing into the void. Clients trying to send me project files, potential leads, and time sensitive snapping in half.

Yeah.

Oh, yeah.

Syntaxi's coupon code tasty treat. Hope you enjoyed the spooky story.

Never never put squares

Did you ever hear that? That's a George Bush quote. He, like, he forgets what he's supposed to say, so he's like, you can't get fooled again.

tech was entertained. Can you imagine getting this email? They they'll be like,

signal Was that DNS? Actually, it's about Bluehost.

My job was to prepare

All in all, they lost about 500 customers by the time we talked to them the next afternoon. We were not not content

not the most expensive goof up we've had.

as I understand it, has gotten much better, so things every like, regardless. If if if you the way that Stripe works is if you can refund it before they do a chargeback Chargeback. Then you're in good shape.

fail safes over,

We tried a few hacks to get on top of this, but at the end of the day, we warp trying to treat local storage like a transactional database and became very clear that only one thing we could do is move the functionality to IndexedDB, which removed a lot of complexity and overhead, and almost all of the issues went away after that, except for the fact that many devs kept throwing more and more load at it because the analytics pipeline

end result.

Ever put a even put a swear in. I was function, which if you do it often enough, can start and guess what they saw? I changed the name.

and a big stack of fiber wire, the little propagation, yeah. And Jay Node said something about that. Well, the third time, you got the next time.

not a wipe.

They are wondering why our VO two Scott last month had shot up five x.

Like responsible grown up adults, we even let certbot to handle renewals automatically.

So we said we would spend a lot of time to look at their site and their project to get up in up to speed and discuss what we could offer them. One of the big things they needed was help with Wes they approached us was dealing with Stripe.

they often are not. And when you need them, it's not the time to find out they Yarn not working. And, also, like, shout out to, like, support engineers or, like, the technical support staff that actually know what they're doing.

Oh, bro.

And how did we find out? Not through fancy monitoring or alerts.

valuable lessons that day. Didn't even get hired, and they lost 500 fuck. This I give this one, working in reoccurring billing without, like, tools to do it for you can absolutely drive you to burnout. Absolutely. Billing.

how many of those 5,000 people that got the second charge, like, filed a, like, a chargeback? Because that's $15 out of your pocket JS there is only one person to blame here, the Python programming language and its effing syntactic white space.

to our back end analytics service.

Yes.

First day on the job was for a small startup serving hundreds of thousands of users daily.

Yes. The next one here is

Strange.

What's the worst? coming up. Just just awful stories, and we are going to read them. If you have your own spooky story, please submit it at any point. Go to syntax auto fam forward slash spooky, Oh,

Yeah. I would I would say that is I would say that is a nine out of,

those were the only things telling the Internet where my email lived.

People always get annoyed by, like, lock files in pull Wes, or or sometimes they'll just be like yeah. Yeah. I'll do it myself. I'll just delete the thing and reinstall it if there's, like, a, like, a thing. But what's happening is that, like, there's a possibility that eight levels down your package tree, there's some package that got bumped from one point zero point one to one point zero point two,

I don't like logistics. I'll tell you about that. I'm not good at planning.

What I didn't realize is I had removed my email, which was running through Google Workspace, and those MX records I casually deleted, I'm no longer Tolinski.

import it in the production database. What a sentence. The whole site goes down, and my new table is left.

This is now more than ten years ago, and I still get chills thinking about it. Oh, there's no backup strategy here. They what? They everybody had to reenter their stuff or what? It's just gone? Probably.

in order to capture as many events as possible. This is pretty common when you're sending events.

Oh.

Yeah. That's awful.

I did that one time. I think I talked about it in one of these spooky episodes. And then, I I deleted a bunch of, congratulations.

probably lost a bunch of money on the dog food. And I I would be mad if I got all that extra dog food because, like, where do you put that? You know? I guess you can give it to a, like, donate it to a shelter or something, but, oof. Hopefully, that's the silver lining here. Some shelter got a bunch of free dog food.

PPS.

It's that easy. Sun hits a lens, travels through the fiber wire, and lights up the room. The entire okay. He just explained how it works. Yes. The entire lens

A round of chaotic debugging later, we discovered the issue. The server

Next Node. Instead of making money, you spend money. So back when I started web development, I was working for one of those sketchy cell phone game ringtone wallpaper companies, Do not disturb on all day. Yeah. Hey. I've been trying to email you all week. Are you getting my messages? They keep bouncing back. The Sanity, like, that set in. Oh, no. What? instability

orders a day with about 300 of that amount were subscription orders.

Why, thank you for the the ad there, dear friend. I don't know what your name is, but, yes, constantly backing that thing up. You know? Just every every 20 minutes No.

white space ghost faced.

There are a ton of them. I don't even know what we've done. One, two, three, four, five, six, seven. We've done, like, seven years of them, and we've done at least two or three every single year. So there's there's at least Like, 16 or 17. Yeah.

think that you understand. Because, also, a lot of this, like, trying to catch people doing bad stuff, A month later, a friend texted me saying that my portfolio was turned into an adult website.

it reserves the money in your card. It it hasn't actually gone through. Sometimes you need to do that. That's a part of credit card processing.

API keys in your I guess, in some cases, you have to. But, man, that sucks. A 10% loss of all their customers.

There was no way to tell who had placed the orders or when or for what.

screwed the pooch here. What did you order? Man. Yeah. That there's no way to fix that but manually. We've had this this many times, the story of someone who did it, and they've had to, like,

for the month Node matter how recently it was paid.

that from the interval they had specified.

frequency data, not broad spectrum light. Interesting.

Cloudflare, it scans But outside platform

whatever in 2025.

You've lost recordings.

It does suck. Why has DNS propagation gotten faster? We used to have to wait twenty four hours. Like, what's changed there? I don't know. How does the Internet work? You know? There's there are name servers around the world, and Right. When one of them has changed, they have to let the other ones know.

Yeah. I'll give it I'll give it a

real money and, very hurried in the documentation.

A number of years ago, I was working as an embedded engineer for Internet of Things products.

And I've had it in the past where updates instantly for me, but then you there's there's, like, a website you can visit Six months later, I'm at my summer cabin with a beer in hand and the grill going Wes the phone rings.

Thousands of extra orders created, packed, and shipped.

and, like, order for feature development after being plagued with reliability

it would just be like, hey. I'm trying to log in. The login says this. Oh, let me fix it for you. You must have tried, Yeah. Yeah. I mean, you gotta have a backup. The IT people should have been making sure that that CEO had backups or whatever, but still,

Certainly nothing to do with my innocuously

But, yeah, I give it one time. One time, you give him a a a break, and then two times Yeah.

I switched over somebody's name servers, and I thought I had taken their email down, but it turns out that

stating that artist was going to be performing in their area. Oh, no.

don't wanna do cert Bos myself, but I certainly don't wanna pay for, I can't think of, like, a Wes thing to make

If that was me, I'd just be, like, receive the events from everybody Wes, all gone, never received. I get that way sometimes where I turn on do not disturb

Turns out, our monorepo

The common thread amongst the reports was the test suite can't find any elements. I don't pay Last minute, we make these are always the best. Half of these stories Yarn, last minute, we made a separate decision. The operating system should just restart

a a five out of a 14 Legos on the floor.

tens of thousands of times, which was also causing additional slowdowns on the other clients. It was a hard case for us to get budget to add more fixes for a service that should have been in maintenance Node, but the volume it was needed for processing grew from 1,000,000,000 today

waiting.

I'm just gonna use I liked it. No more. I liked it.

at a moment, I had to confess,

Anomanimity.

Three weeks go by.

new. So I was cleaning up some of the messes. Every time video creation fails, multiline logs are barfed gear. It took thirty minutes to rotate a 180 charged back, that's still $750

nine demon pups out of 10 demon pnpm.

that sucks.

Oh, man. CEO's hard drive.

down into the mall through the air ducts. The code had validation checks for everything, including valid encoder entries. That's the position DNS.

Days later, when the SSL certificate finally expired, Yes. I always check my exported Scott SQL files from now on.

torque.

All of these streams land on our loyal ingest server, which is tied to a domain name and secured with a Let's Encrypt SSL certificate.

2025,

Bro, that's two now that have dealt with people who have passed away. That is

That is and, you know, like, the poor support people as Wes, they're just gonna be going through it for the next, because this is bound to happen at one point.

and then I finally

Real blood. Why thank you. I was doing this costume,

having some frustrations with my recording the other day. I was sent, like, a screen recording to some someone, After some panicking and digging through, all the test suites were still passing. I realized I'd done everything right except Yeah. Because they're able to Century has a really good support team. I've been to the Toronto office several times, and you just, like, listen in on them trying to help everybody with all their problems, and they know about

that order came in from. These shipping companies all had different lead times for each country.

Yes. I was working for a small WordPress based company when we had a client who was thinking of hiring us on retainer.

Yeah. Holy was complaining that their donation never came through.

There weren't actual charges made it. Thanks goodness. Just retriggering of email notifications.

Moral of the story, don't treat local storage like a transactional database. Yes. Yeah. As much as especially, like, you can you can run out. You can run out of space in local storage. And every time you write to it, it only holds strings, so you gotta there's, like, a JSON parse,

Oh, man. It turns out the c dot SQL export didn't just contain that. Beyond the Grave two.

everything. It's crazy.

I watched YouTube tutorials, read hundreds of Reddit thread,

Oof. I can't imagine the bill for that, man.

when I meant to run the

Well,

hit these specific criteria, but, like, don't send them to an adult site. That's bad bad idea.

No way.

never do

Twelve minutes of downtime.

Bad redirect. In a company I worked where everything was basically in shambles and barely making ends meet, a coworker thought it was funny to somehow magically redirect hackers to an adult site when he thought something was of malicious intent. The boss made a demo for the company I export a single table from our staging site, Oh, nice little day at the office.

That sounds like an AI,

But, like, if you worked with somebody and they really screwed the pooch, But that is still

Peace.

team at the time, but was given responsibility over everything analytics

ran fine locally, ran the Wes. CI was fine. PR was approved and merged. I headed out for lunch with the CEO and founders who were in the office that day.

He did wear a hockey goalie mask if you don't know about Jason.

I completed everything, uploaded onto staging, and then got the client to overview it with a green check mark.

more carefully. Well, apparently Scott because the next one is from the same guy.

to go awry.

of the motor. Right? For the motor that spun the entire array of lenses. If any value was wrong, the application would just stop.

shout out.

This guy submitted, like, ten ten WordPress

hired on a retainer, Awesome. They were using SiteGround, so I was able to just hit the sync button, merge the staging sites to production, forgetting that this was an ecommerce site.

So we asked if they had any custom code within the site dealing with Stripe, and they told us no.

as special fixtures, which is lamps that scatter the sunlight in the room. That's crazy.

Everybody gave the green light. We tested the implementation.

I wrote a TypeScript to capture everything and sent out an email to everyone affected. In the end, it all worked out.

horror stories. Like, he never learned from his thing

a create table query, but also a drop to SQL and blindly import it in the production database. Oh, blindly was not. We renewed the cert. The streams returned, and we immediately begged the leadership to approve buying proper SSL certificate with a longer validity

Total panic. Imagine your entire business is taking money for people.

the domain name for as many records as it can because you could accidentally be bringing down

Six months oh my gosh.

I don't mind access control either. I think those are the Bos. Anything that involves kind of, like, ping ponging back and forth. But, like, with reoccurring billing, like, testing it, you have to be it has to be done through webhook so you can send a test webhook. And then, like, when an actual subscription,

chomped through some stuff either way. Very smooth. Everybody Deno right now and check that your backups are working because, man, And sometimes you bring the remote one locally and sometimes vice versa, and you gotta be very careful Oh, no.

Turns out, we had been calculating the next order date wrong. And once the new date was calculated, the next order date was placed on the current date seconds, milliseconds. I can crash Really? Faster. Oh. You know how to do it?

my colleague and I maintained depending on the laptop or in rare case, desktops. My job was very routine and robotic. I loved it. One day, however, that needs to change in the database. You know? Yeah. But I also have content that lives in the

That's that's the hang on. I knew I was waiting for it. They've been paying every day.

Brutal.

Yeah.

As I said earlier, the original creator had left the company. I had been Node bugs in the system at my time there and felt confident to implement this new feature.

retried five times. Just for reference, a thirty second video generation on Google costs about $10.

Am I subscribed to some dog food? Maybe I'm a customer.

access to the, a impression with. Ugh.

Never.

Three bones out of three? I'll do eight cobwebs out of 10 cobwebs.

Welcome to Once, I built a website for a foundation that did medical research.

and we're billing the victim I mean, customer via premium SMS messages. That that's how it used to be. You want a ringtone,

What a wonderful four Dracula fangs of which I'm going to be taking out so that way I don't have permanent damage done to my lips here.

not just the company, Talk about updating my services to we offer development, but only front end development.

sorry. I didn't read this whole thing because I was like, this is gonna be good. The entire lens array is programmed to follow the sun with fractions of a degrees of accuracy.

Alright. Let's get in on the first Scott story JS a bug from beyond the grave. By the way, I wrote all the titles for these ones, so please appreciate them.

The only issue being that that artist had died a few months prior. Oh, no.

An employee casually said, hey. That little streaming icon isn't showing up anymore. Translation, I picked up an easy ticket, updated some dependencies,

Like, oh, shit.

It was actually pretty simple. It was just a WooCommerce site that needed some minor changes, so I did what I always did and created a backup, a staging site, etcetera, and got to work.

Anomanimity.

This is the stuff that you're not be told about, how these companies have,

Oh, Scott the NHL? Go grab my hockey stick I got over there. I got a hockey stick in my office, Wes.

costume.

on, I signed up for Netlify, connected my GitHub repo, and watched my site deploy in minutes. It felt like magic. Then came the DNS settings. Now I know DNS was important logging change. Right? Well, time to find out.

where it will ping your website from, like like, hundreds of servers around the world new fires new hires. The process involved wiping the ORM images and loading custom images on

trashed,

brother,

five different credit cards 40 different times, and the system failed. You Node? Like, you must have Scott, messing with their money. Yeah.

issues in the front end that devs were flying blind to. It was decided that the best way to give front end quick visibility into performance and reliability Ever But JS of January

r m r f on my entire,

I'm a full stack dev that writes mostly TypeScript, of course, of course, like any good full stack dev. I'm a seasoned staff with lots of experience writing back end code, We'll throw the ones that we didn't read today onto the And that kind of thing has happened before. It's so true. Your Cypress tests are failing.

two months for free using the coupon code Sanity treat.

The CEO stepped away to make some calls. I pulled out my phone, jumped into the dashboard, and rolled back the deployment myself.

that uses certbot behind the scenes or whatever to manage the SSLs. I

you gotta have some alerts set up or something if things were to like, weeks or months Everything in these projects is expensive, emailed myself.

I joined when that person,

was to leverage the analytics crazy push notifications about a dead artist to those people. And now there's some code in their code Bos that checks if an artist is deceased or not before it sends events. The hottest Scott fix. I once worked on new features that introduce API calls to a somewhat complex web component.

Node of the first of these devices was installed in a mall in The US. The fiber wire was simply fed a set seven oops out of eight, I think.

The customer about 10 people,

To this day, there is still some code that will attempt to reclaim events from local storage despite no new analytics being added to local storage since the start of 2021.

but then they fixed it, would that give you confidence in working with them in the future? This guy's got five stars.

that we use that didn't bill our cell phone. Okay. So you had, like, a you have a premium one, which you make money, and you have a regular SMS, which you actually you pay to send. One morning, our CEO called us into a meeting to give us a bad news that there wouldn't be a bonus that month as the SMS provider had screwed us over. Instead of paying out the bumper sale, they were billing us for excessive use of the free SMS.

backing it up because that's

Eventually, customers

the CEO's laptop was brought in for servicing that's always a tricky one, especially, like it's not a WordPress specific one, but a lot of people hit this issue where And on top of that, Alaskian blocking the main event loop. In testing, I even managed to crash my entire laptop.

and definitely

That's why I just

Nope. The hotfix branch is running Ah. How can you not check it? Oh, man.

This specific product is designed to guide sunlight into building. It consists of a large array of approximately one of the encoders turned out to be faulty. The application would start up, validate everything, start to spin the device, realize the motor wasn't turning, and crash. Then it would start up, spin for a bit, and crash. For an additional context, due to the insane precision needed, the motor was mounted on a one to 65,000 introduces a major bug in your system. I've I've had it myself once in the past, and that's where

Luckily, I work for a nice company. No blame was placed on me, and the new guardrails were put in place before tagging deceased artists

events a day Yeah. I mean, like, brutal.

his phone starts buzzing. He checks it and goes pale.

That is three bones out of three for me. That sucks.