Cloudflare Analytics Engine, Workers + more with Ben Vinegar

I have a lot of anxiety about being on this podcast, but first but I have no I have no anxiety about And, you know, sometimes I maybe could see that perspective because maybe there are a bunch of reasons why they feel that they need to meddle with it. It's like, just let them cook. I don't know. Yes.

You know, that's what I would be doing with with with with software people. And I think that that idea, I don't find is too different from what you all do. So that's why I have very little to say about sometimes And everyone's kinda pointing fingers at each other.

but, you know, as much as possible, try to figure out what what people are interested in and try to find a great fit for them. And then but not really get And I I didn't wanna use Google Analytics

gee, what has that been? I mean, I think You don't wanna have to, you know, have an endpoint that handles data. Node you Scott query a database. You just wanna write it and then So wait. Quick answer is you gotta write your own stuff. You gotta it it's sort of like a free form object store for you to store data that you want. Mhmm. So counter scale JS, like, the script is basically pulling browser device, analytics engine also gives you you get a 100,000 rights as well per month on the free plan.

yeah, I I would never put that together, but yeah.

Because we are going to react Miami, and we wanna have a bunch of beer for people. You know? Like, there's just, like,

I think that was the biggest thing for us.

So I think, you know, to get into, but, you know, it doesn't have to go through every row of the database. It just has to go through the unique values of that of that column. This is a real deep dive into this right away, and I hope I'm not butchering this whole ESLint. No. It's super interesting. Yeah. We first heard about ClickHouse when we heard that Century was using ClickHouse. And that was the 1st time they crossed my my radar in general.

It doesn't always work out perfectly that way in engineering,

You join a you join an established media product.

a restaurant here, and Wes and I are cooking up. And now we we Scott another chef in. We got a, you know, expediter in. We got all the, we got all the parts ESLint, and you're the Node ordering up the the food and making sure we have what we Node. And yeah. Yeah. Like,

I I order ice for a bunch of guys to talk about JavaScript. Yeah. Yeah.

You know, it's been a lot of fun. So I think, you know I don't know. I know there's a great there's a personal answer for me is that I've I've personally really enjoyed just kind of learning new skills just as I think many software developers

getting everybody aligned on ideally on, like, what they wanna do. K? It just kinda works. That's the cool thing about it. You know? You just kinda blast it with data, podcasts and YouTube and that kind of stuff, hey, that's learning too. At least it is for me, and I I enjoy, okay. This is the the most basic query,

the the the infrastructure, the the taxes, the, like, the time, all of that stuff. Right? And so, like, Ben will tackle

I'm on trying to avoid going, like, crazy deep on this. I think that, you know, my experience as a as a software manager and as a senior

at React Miami and have on sites and and plan Wow. And Yeah. And, like, you what kind of events

But we tried our best. Yeah. That fulfillment software stuff was a wild task, and and I think Wes and I SMART goals or whatever a tamer in in house, and and Ben kinda showed up and whipped us into shape. And it ever since Ben has showed up, it it feels like the amount of progress we've made as a team has

Yeah. Mhmm. Right?

can not not to say that Ben is, like, just doing our

What did you do to not mess up the vibes?

That's a good point. And and

in the way. You know? I it's kinda strange because I I wrote a book on JavaScript, so I, you know, I I certainly have opinions on, I kinda relaunched my my my personal website, which is no. I'm not gonna plug that. We'll see. Yeah. Let's what is it? Benz.ca.

Yeah. No. I I think that that initial getting us into a room and talking about, you know, actual

do fun stuff, like have merch and and have beer at,

century and syntax. And I think it's worth talking about that because we you talk about on the show, but probably never heard it from someone like me that's, like, you know, we we run syntax pretty independently. Wes and Scott and the team run, you know, content and figure out what they wanna talk about, what they wanna write about. And I think that, you know, going back to my Node, my role is to sort of empower you guys, these guys. I'm like, who am I talking to here? You know, to just kind of achieve what they wanna do. So, And there JS no way in hell that I wanna learn how to run Kafka or how to run all this complicated infrastructure. Yeah. You know? and I happen to know that I've that I I personally as a developer have put browser name

stuff like that is

make it so that we can we can do more. You know? Like Yeah. The effect. Merch is a good example. Right? Like, we never had syntax merch, not because we didn't And I'm just looking at the docs for ClickHouse here, and it looks like there it doesn't make sense to me. Like like, the the schema itself. Like, let's say you wanna track

You know? Yeah. So there's there's certainly some differences. I wanna be I wanna be clear.

The way I think about it is on shoving column names in there to make the code string

Mhmm. Yeah. I forgot about the store. That's 1. Yeah. Yeah. You know, thank you to all the people who bought from the store, by the way. And if I

you know, one thing that I think we've all been very

fulfillment provider, which which we just did, you know,

I feel like this is like the trope movie thing that you've that we've all seen a 100 times. This just feels like you've seen this situation depicted many times where into look. My website gets, like, you know, optimizing some code.

my title is general manager. So, technically,

Okay. My bad.

talking about what we want to do and then and then working backwards like what would we need to,

it it it's kind of the job. Right? The technology is always changing.

these things so that we can

Yeah. For us, even though you didn't come to us and say, hey. We need SMART goals.

on your end and on centuries end in general is that there's been a a distinct do not mess up the vibes angle for from everyone. You Node, we all know what could happen if the vibes are off or or or if, you know, corporate overlords do come in and and kind of ruin what's going on. But I think that has been a focus of century from the Scott, and, specifically, Ben has always been, I do not want to mess up the vibes. And, I think you've done a really awesome job with that. So, you know, hopefully, the audience feels the same way that the goal here is just to take what we are doing and

Ben does the stuff we don't want to do, but, like, it's a lot of, like, bigger picture stuff of trying to organize and figure out how to tackle

No. No. I think that Just Let them Cook describes it all. I mean, if we if we go back to our tried and true Wes of, method of using food metaphors for everything, you are basically the general manager of, of a restaurant here, and

would. You know? So with you all. Like, I'm I'm really the student here to be frank, and to level up my CSS as I've as I've commented earlier.

look. I think the software developers love learning. I Think there's something

an order like that, and he does everything from, like, figuring out how much,

very quickly handling trying to handle all that stuff on on top of doing the show. Personally, though, for you, how does it feel like going from a software project to a media project? Like, what what how different is that? That is a very good question and probably one that I get all the time when I explain to people what my job is, which usually takes, like, 10 minutes,

is you know, sort of sample that data and they don't have to store every single data point. I'm getting a little deep here. I'm sorry. Oh, yeah. It's interesting. Yeah. The the sampling, we talked to Armin from Sanity on it, and we're just like, how But analytics ESLint, actually, just like the schema JS sort of

you know, how many spaces those challenges, the system is gonna throw out some numbers or it's gonna make estimates is how I would kind of simplify it. But everything does that. Google Analytics is not storing every data point of every visitor that ever visited your website.

at at parties and so on.

You know, I'm trying to set the stage for listeners here, but it was like you know getting into a room

want to, but we just never had the

more YouTube stuff. And, yeah, so Yarn, it's working out really well.

tangible

even just you expressing all of that made me realize how similar that is because that is all all you're doing with us in a a very positive way. So do it better and and scale it up and be more efficient.

talking through how much should a T shirt Scott. You know? What are our costs here? Do we need to go to a different, Given these caveats, work within those boundaries, and you will have a good day.

appreciative

like Ben was searching, how much does ice cost right before this call. Yeah. You know?

was Gee, I don't know. Wes this an analytics engine or is this no. This is just on ClickHouse's engine or ClickHouse.

But it it was more or less like, you know, what what can this be and where can we get?

But,

and into an on-site, and really just talking about, like, what do we want this to be in a year? which is what this worker stuff is built on And not too different. Yeah. I mean, there's a lot of you know, strangely,

I don't know. Nice. Oh, one last1.

finding really great people, and then

emailed you and told told you that sometimes the shirt was laid, I'm sorry about that. I'm sorry.

I swear Wes didn't ruin it.

I've learned so much working, this is the like, I have not really used this very much. I'm more this is a really tricky thing where it's sort of, like, I I guess I sort of managed a team that's worked with it, but that doesn't mean that I know, you know, it's API or or exactly

You had to download TikTok.

you know, I I I like to describe myself as the as the corporate wrangler. The the sort of go between, double columns you can use, and that's it. There are a bunch of limitations on this database that are like, you know, even an individual 30 visits a month. Yeah. Right? Okay.

blasting open source code.

it will do it all. It will do it all really fast, really cheaply,

Hano is trying to be because Hano was just a like, an express alternative that works on everything. Right? Like, it's just Wes, response, fetch. And then, like, what I found myself being like is, like, I would start something in Deno, and then I go, you know what? I need, like, I need a UI, and then I would just move to Next. Js or move to Remix or something like that. And I think that they they don't want people jumping off there because now you can run

Next. Js and React stuff, all running on Cloudflare with a d one SQLite database.

server components,

it times out. You know? Oh, I need a long run database connection. I need WebSockets. Like, they're trying to they fixed the database connection thing. They fixed the the WebSocket thing. You know? They're they're fixing a lot of that stuff, and I post anything on Twitter about video editing, there's everybody's JS tuning in. Like, a good chunk of developers

a and a mission to see if it was possible, and I was willing to sort of put up with those like, a fire and forget system would look like. Like, could you just press a button and deploy this kind of software stack?

a lot of people, like, oh, I'm gonna host it on Cloudflare, and then you hit the one of the blockers, you're like, oh,

v eight isolates, row I think can only be 5,000 characters. So it has a whole bunch of wacky limitations that you have to be aware of. But if you if the software that you're writing

Cloudflare.

they're not built on AWS or Azure or Google, like, they are their their own kinda cloud provider.

but it still is not exactly where it needs to be. Yeah. I posted on pages specifically,

you know, just to just to make this happen. That makes sense. Yeah. Yeah. It's it's always interesting, the the Cloudflare stuff. I find like, I've I've launched

most people who've worked with, like, Lambda's or or GCP's, I had to. Yeah. I do think that Node thing, working with you all that is and and working in, I guess, more media is

their their Next. Js.

sort of quicker to run. And it also has, like, different performance characteristics, like, you know, spinning up a a serverless worker is really fast. And I That's what they put on their docs. As far as I understand,

You know, one is that select But I'm trying to it's, onto your onto your camera output so that you can just kinda like go into Zoom calls and and Google Meet,

So that's one. Another reason is that they've developed this kinda unique technology, Hey, guys. This is weird. We were we were just in a Slack channel together, and now we're recording this.

I probably have, like, 4

to slowly get away from that because

they they replicated all most of the React hooks, and they replicated, like, state management and all of that good stuff in Hano. So, like, maybe Hano JS gonna be

that I use daily running in Cloudflare. And On workers or what? and process that much data at once? You know? Like, it's just it's a fire hose of data. And I thought it was really interesting how he explained how they're using queues and,

they actually manage their own infrastructure, by which I mean the hardware. Right? So they're not

On workers and pages. So my last Node was I your Century is putting a script on, like, half of the Internet.

front end support, which is like Yeah. I think it's there.

got Next. Js

It wasn't without its, like, hiccups. Like, I I went into the Discord probably 6 times over building the entire thing. And that seems to be the the thing with Cloudflare is that

and there's that seems to be extremely cheap. I mean, you went off and and built this thing that seems to have have caught on. So do you wanna give a rundown on what that is? the analytics engine?

which they they're switching over to Vite now.

Did did they acquire Hano JS? What what was the situation there? They hired the, like, lead dev behind Hano Wes. And then a couple months later, Hana rolled out, like, and and see that type of thing as well where it's not just become a,

but I've always had just a,

a fair enough easy experience with adapter patterns. Right? You just push it live. I I think I I wonder a lot about that Hano JS since

often some gotchas. Like, you often have to think about it from the start of, like, how am I gonna approach this in the Cloudflare way? And I think they're trying

app router, like, all the brand new

wild ride here.

even anything from upgrading, like, a really gnarly dependency to,

internally

We need the syntax you know how there's the epic stack? We should have the syntax stack, and that also includes, you know, all the video and audio, you know, descriptions, tools.

that's part of communicating well is getting your your tech set up. Alright? And, like, I'm I'm imagining it allow you to share stuff right over your webcam so you don't have to do the whole, can everybody see my screen?

the success is really syntax's success. Right? If we're you know, it's not really

fixing bugs or a podcast about,

And so I think that that's is telling me a different number than this system? and then and then people kinda meddle with it. Right? per day.

like, I often think about how hilarious it JS. That's the stuff you do in a single day of, like,

Ben, do you got a sick pick for us? Okay.

I I know a lot of people might be wondering about, like, Beautiful. Scott and West's

Synstack.

Yeah. We have Caitlin rocking the newsletter and, Someone please correct me if I'm wrong. No. They aren't. That's true. That's all accurate. Everybody's becoming a Vite app these days.

get a half decent microphone and camera

is is all about

Counter scale? Yeah. That's an easy one. I, And if somehow you showed up on the top of Reddit tomorrow, I don't think the DX of Cloudflare, you know, is as good as Vercel. Mhmm. Just, you know Not even close. Yeah. Yeah. Yeah. And and I I do think that there have been painful parts of of working on this. Like, the stack traces that I get out of the out of the workers are pretty painful, to be honest. Mhmm. And I've kinda dorked around with trying to get source maps working the way that I want. Counter scales also built in remix. That's another random thing. I'd never done that before. So I didn't have a lot of familiarity with that stack, and it's got its own it's got its own little server as well as I understand, or it had its own kind of build tool, And then I think the third thing is they can have a set they might have a completely different strategy of being, like, super cheap for hobbyist, you know. Yeah. A lot of people say enterprise is paying for it. Yeah. I have no insight into that to be clear. Like, that could be true. That could be false. I'm just I wanna be clear here. We're not in a in a in a complete love fest for this for this infrastructure.

a podcast about or is there, like, dozens of them that are available? I don't even know if you know this, but, like, I see there's 4 different geolocation

and you can have your title on there. You can have a logo or you can add some filters. Right? It's sort of Cool. Yeah. It's like a like, having, like, a Snapchat

and describing columnar databases, like I just said, ordering ice, So maybe scalable web analytics, all the way to just

All of the stuff. Yeah. The syntax stack. It's true. Honestly, I I was thinking that we should do, like, have Randy on at some point and just talk video editing because anytime

Being invited onto a podcast an hour before Oh, yeah. Wes recorded it. Yeah.

it's a virtual camera that lets you just sort of put professional overlays stopped using Google Analytics. We've stopped, right. So this is often why you'll see how do you actually know how many people have visited your website? in a in just a fire and forget kind of way. And I think that analytics engine is Here's another one to you know, how do you pronounce this? You know, column store database.

is is what they're going for. I happen to have a a beta version. I got an early access, and I was playing around with it with CJ actually those details.

tracking what's going on with your website because it's Mhmm. Like, there's so many

and I really wanna preamble this. Like, this is this is really like a personal project that that that is not an early moment for us website provider and I wanted my own thing that I could drop on my page. Yeah. You don't wanna be, is that the data is stored

you have a wild idea. How do you quickly

you know, have a great show and to have great education about Right? Yeah.

and then trying to understand

and, you know, lightly sprinkle some century on there, which which we do during the intro.

I was just thinking of this one because we warp talking we were talking about the syntax stack, and we're talking about Randy and and production. And I think it would be fun to even have Randy talk about get people to match them with their interest as much as possible, like, does that sound like software engineering, or does it sound like just sort of general, you know, working with teams? I think it's more of the the latter.

I think something that I left out of that is, you know, not only did I just kinda wanna make this for my website, but I think in working with you all and wanting to, columnar.

people or the product people who work on those things. They often talk about it still.

You heard it here.

Beautiful. And a shameless plug. What do you got for us? hover over top of navigation items and time spent on page, things like that?

So a lot of things I would do with software teams is

Warp.

also make videos, you know, and and they're always interested in what the editing stack and the process looks for that. And that's just one little part of the syntax stack.

is really important.

syntax

companies bringing on a a piece of media and just absolutely ruining it. And I've been really happy with how everything's gone so far because we we've get to do more things rather than than less things.

So that's my sick pick. Awesome.

explore that idea? And that's, like, very much the feel of, like, what we're trying to do around here. So love that. You Node, beyond just managing the team, you're diving into

I think it's pretty wild that you can

See you guys. See you see you on

sorry. My Internet is crappy. Like, figure it out. Like, you know, like, that's part of your job is communicating with people

How do you sell

Or worse things. They get to be better. Yes. Yes. True. Yeah. Yeah. Like, we have a full time producer now. You know? We have YouTube, which is something that we we just launched. You know? We have CJ who's able to,

You know?

create supplementary content and throw up stuff on video. So it's been all all good in my opinion.

here in Toronto. Getboom.app.

some super deep thing. I I, hey. Well, let's let's hire a producer. You know? What does that look like? Yeah.

to touch upon everything.

it's coming out very soon. It's out by the time you listen to this, I think,

and stop the whole,

we were saying on the episode that came out last week of, like, how to level up at your job, And, like, one of them was, like,

What a wild stack to

all the media outlets that are

SynStack.

income from products? would have been very overwhelmed heard the heard the podcasters.

on the topic of this sort of stack and and audio production, like, you know, Randy's talked about, you know, $50 microphones that are really great. You know? Like, there there are tools out there that could kinda like raise your professional acumen, Look, this podcast is brought to you by Sanity.

Yeah. That's a super good question. And I think even maybe a chance to, you know, offer some transparency on that. Right? I think that one thing that I learned from being a senior software

marketing leads or, you know, hard numbers that we're trying to hit.

Node of the episodes, I think, that's coming out,

blouses, etcetera. Right? Now mostly a lot of people are working remotely from their homes. And equivalent thing. GCP functions

are out there, which is

Awesome. Thanks so much for coming on, Ben. Appreciate it, and we will catch you later.

and get it dialed in because I don't know if you I if you call them production apps, but for real things

engineering manager is internal storytelling

what's going on in our industry for developers, like, as a general manager, I feel I feel the need to, you know, invest my time and to work with new tools and just try to reactivate myself into what's going on. And I think that that was a little bit, you know, a small driver and working on that project. So I do wanna kinda add that. Yeah. Absolutely. And I think that's I mean, it's very fitting with what we're kind of about here.

and that's kind of event based system or or who Node. Right? Turns out Vercel solutions are commercial paid offerings, Right? It's not effective for that. It's not effective for them to do that. I also think this is why I'm sure you I'm sure you guys have encountered this. You ever have a situation where you have a bunch of sort of, like,

what we're doing in those regards. But also you're making videos, you learn video editing, you learned the social media platforms we're pushing out to. It's like, I think that all all really, really helps.

you sell what we do here to a company that has very real,

you know, I designed it for for my 30th Then I could say, you know,

earlier earlier this week. I think it's pretty neat. So that's that that's my sick pick. Little weird because it's not you know, might be out by the time this this launches, but,

building,

Yeah.

so many sad stories of

We started with this talk about couple decades ago, you have to buy nice suits to to go into the office or or nice wardrobe, So yeah. I think I just kinda looked at this as like, what would be the system Analytics Engine.

to Century?

Yep. Alright. Let's move into the last section of the show, which is sick picks and shameless plugs.

Yeah. And I I think, hopefully, people appreciate

take it in so many different areas. And that's also what makes this job so exciting for me as well, being able

is it clicks? You're tracking clicks? Like, how do you how do You Node, Syntax,

that's it's very similar to what we do. Right? Our goal is really just to to grow the reach of the podcast, microphones and stuff. Right? Like Yeah. Look, here's a little preamble to the to the sick pics, but

at high throughput, Yep. You know, so I while building this, like, I was getting some pretty gnarly stack traces, and debugging stuff was hard. Just wanna be clear. Right? Yeah. But I think, you know, for me, I think in Node ways, this was a little bit of a an exercise and a and And then I think the 4th thing I would add is this, is that I just

Wild that you have to

count, comma, blob 6 from this data set,

The way that it works, it is it is a worker systems like that to just sort of accept data into a big queue. And then you got a bunch of workers that work on that data, write it to a database, maybe do some some kind of computation or whatever.

It's wild to think about. You know?

SQL like computation I think, you know, there's there's a lot of, like, don't fix what's not broken. Right? And so, I guess, some old cooked

And while I don't really think this is gonna be a problem for the 30 people who visit my blog every website, I So

is some kind of number, integer, whatever. Yeah. And and blob, I think Yeah. You don't mess up the vibes. It's it's it's good that we started this episode with by the way, someone fell through, so we threw Ben in here rather than, like, here's the apology tour.

But what happens when that just sort of explodes with data? It's not

and number it writes a data point to So they were thinking about it that way. At least that's how the docs kind of read. But I was looking at it and I thought, you know, this could be why couldn't we just sort of use this as a as a way of forwarding

And,

Yeah. Right? The hug of death.

Let let it will allow you to scale it up as if it were the actual traffic. I don't know if I've ever had such large traffic that I've needed to worry about that, but Mhmm.

somebody visits the website you wanna track what their their browser user agent is,

on blob 6, which is a string.

how do you not, like, DDoS Century when, like, you I don't know what it is that caused you to sort of dip into it, but you created this thing called counter scale, which

trackers, like, I think Cloudflare's free plan right now, you know, for free. If you just sign up, you get a 100,000 worker in vacations The story here is it began with a problem. I don't know that I necessarily started with Cloudflare.

every time there's an error or any time there's, like like, if you're doing the session replay, right,

working on Century? I mean, I know Century and an analytics tool have a Scott in common. So

All these services use sampling. So, you know, somebody came out of the woodwork and was like, you better explain that this uses sampling. And I'm like, okay. I'll I'll I'll put an issue about that. And the concern there, it just means, like,

sorry, every month. I guess I wanted to just sort of explore what what a, what I just described. It is a system that lets you write data, write data events analytics engine.

you Node, 24 blob columns you can use, and 24

they Yarn unders they're under

inside of the analytics engine. So when you query it, you actually send it a SQL query. If I wanted for example, every unique browser name out of this thing. It's like select, I think the company is really just sort of

I think it's that so when I did look at Open Source Analytics

the version of the browser that they're on,

Yeah. That is similar. Right? So, you know, in the magical world where you just got the perfect amount of data, or a 100,000 people visited your website every day, is anyone gonna need that? Yeah. That's a lot.

under the hood. There are these primitive types that exist inside of ClickHouse.

look somewhat like it makes sense. Yeah. Yeah. I pulled up the schema, and it's, like host blob 1 user agent blob 2. It's like that name.

Yeah. If you look up, there's, like, a SQL API for for this Analytics ESLint. Okay. They only support Blob and Double. They do expose this index column, which they describe as the index value that was logged with the event. The value in this column is used as a key for sampling. They also sample data. This is part of how 30 visitors a month blog to be clear. Right? But it looks great. Yeah. It looks in in functions. Great. Yeah. I think this plug is hey. Check it out. It's counter Scott dot dev. Is there's a website. You can actually browse the dashboard and take a look, and then there's link to GitHub in there. But if you think this is interesting, like, I would love for people to open up issues, pull Wes, try it, you Node. I really love for this to be a community project, not a me project.

column. K.

the Cloudflare Node is a little bit more restricted, which is often the case with Cloudflare stuff. It's like, hey, this is extremely cheap, extremely fast.

Right? And double So, really, like, most of how we communicate The number columns doubles, like, you can do some and average. You can actually do math. Okay. So it it does it does allow for you to do some quick bio.

Wes the mouse goes or or where you clicked on a thing, so much data is being is being sent. I was like, how is it possible that you can store

is it's either I mean, it could just Yeah. We talked about this on our on the last time I was on the show, but I'm a very lazy Vercel, and I I don't work there. Yeah. However, I'm gonna I'm gonna offer my own random theories.

I happen to know that browser name is stored in blob 6. K. And Blob is just a free form pretty down with look. We're pretty transparent and clear that Century produces the show. The show continues to be good. We hope that they the audience recognizes that, 5, 10, $15

analytics engine has a fixed schema. And what that means is and my understanding is that ClickHouse I think it's launching in a week or 2. And this is sort of like inside of your company. I think that's interesting.

the system works for you at scale. Basically,

or I I work hard to be lazy. And what that means really the best. You know, I I've seen enough from my work to to say, like,

is is the technology that that allows that. That's cool. It it seems also Cloudflare does provide the sample interval, take a look. That's awesome. It it's true. Like,

is additional data types for things like geolocation.

I'm not pulling up the ClickHouse docs here. Right? But in my mind, these are effectively just sort of, like, micro criticisms on on the shows. Although, sometimes I do, I share some feedback, but mostly from a, hey, what do you think about this? I guess what I'm trying to say is I don't know if for if for everybody this would make a lot of sense. But I think for the way that I like to work with software development teams, which is really high level, hold people to goals, I just don't wanna do any of this. Right? Yeah. I just wanted a software system that I could put live, forget about it. And I've been running this for about, I think 4 months now, 3 months, and I've literally never had to do anything with it. So it's it's sort of fulfilled that from my side.

the Blob. I think there's a Blob column type in Postgres.

And but what is a blob, and what is an index?

you want those things to be. Okay. Right? So when you write them,

once everything got back up and running.

stuff like that, and then pushing that

sampling and whatnot to be able to just, like, ingest all of that data. And they also told us a really a story about how there was a huge backlog at once where it had to be and it it could still be processed maybe the city that they've been visiting from. How how do you then make that schema in

it's it's like blob 1, blob 2, blob 3, blob 4, blob 5, double 1, double 2, double 3. It's basically just got free form text I'm like, oh, what do you do?

at low volumes that's perfectly fine. But at high volumes, codeters and all that.

Yeah. The answer sampling. Well, it is a It could be among among a great many other things, but that's one of them. Yeah. So are these all decisions that were influenced by

I don't know. Maybe I've been in a bunch of sort of meetings and stuff where that that's happened.

one day do. Yeah. Yeah. Wish my my stats will be sampled. But let's talk about, like like, pricing for this type of thing. I know that you have said it was extremely cheap. Yeah. Can you run this for free for most websites?

you can choose what to do with.

it would be okay, If that makes sense?

Yeah.

relational data, if that makes sense. It's not really what if they ingest a ton of data points and that and that data looks similar,

7,000,000 events and it only sampled 90% of those,

works within those constraints, then you get all these incredible properties. Right? You can write sort of near infinite data points. You can aggregate a 1000000 data points and read them back and be like, oh, a a 1000000 people visited my website yesterday, and it will probably return, you know, in a second or 2. That's pretty wild. Okay.

and some functions to scale them up.

I think at least an early version of that was Postgres.

is I don't wanna have to monitor my databases and sort of figure out how to back them up or figure out Just to be clear here. And something about just sort of spending, you know, more money

Well, this is trivia time, Wes. I wasn't I wasn't prepared for this.

types, point, ring, polygon, and multi polygon, which could be helpful for, like, geolocation data.

And I understand what a double is. Like, a double would be a number.

that would allow for this type of software to exist that's what we're going for. Mhmm.

columns that you can use that they've already defined in their schema

analytics trackers or counters and and or or been in a company where it's like, why don't these add up? Yeah. Why is it that your system and then group by blob 6. And then I'll get a I'll get in the results effectively, like, a a table output of every unique browser name and then the count. Right? Oh, okay.

when I when I launched this project,

So if you do like, if you have

what's in here.

And most of what Ben does is

also needs schemas just like Postgres or MySQL. You gotta define So that's 1 worker invocation, and that's 1 Analytics Engine write I've done everything from work on our client SDK development, specifically the JavaScript stuff, does some really light, you know, data munging, and then that is

it's just kind of exposing these really raw primitives Mhmm. And that turned ESLint,

that database was designed for.

And

a

We did our our big announcement on our 600th episode. We had David on to talk about that.

Syntax and Sentry 1 year later.

on Syntax and, like, what you do for the team for people who don't know? You know,

I don't know. I don't cowboy coding type of deal. You Node, we're we're working on the site kind of we're very siloed. We're working on our projects. We're working on our site. We're used to being

it has been 1 whirlwind year of a lot of stuff. Half of maybe not half of which, a quarter of which Ben Ben was, on sabbatical for. But when he when Ben showed up, we were kind of I don't wanna say So,

last March, end of March.

stallions, as Wes would say. And we we definitely needed

Maybe. Pretty interesting stuff. Yeah. Go ahead. Let's take a detour here into,

we joined Sentry

you know, folks who have been doing this all very,

but we were a team of,

directionless,

just kind of blown blown up completely. So, Ben, do you wanna give, you know, a quick overview of your role

talking about,

counterscale,

both, like, the privacy angle, right, you're I guess you're storing it yourself. You're not really storing it yourself. I guess you're you're given it to Cloudflare, but seems a little bit different, Ben.

I think success really. Really just kind of the big picture stuff.

per Analytics ESLint. Does that make sense? K? As I understand, using marketing cookies on our website. We've written about that, and I kinda wanna do the same thing.

And then I don't know what the the analytics ESLint. Let's see here. 10,000,000.

330,000,

you know, have have used a lot of these things, you don't have to have your website

You get 10,000,000 on that. So, yeah, you get you get quite a bit for for $5 a month, and if you're sort of weighing We had,

So Cloudflare is only there to host the tool, which is free to use outside of your usage.

And and to be clear for the audience who who might not,

a fraction of the price of everything else. Like, requests on a worker per day. So you you immediately have three x that for $5 a month.

It's absurd. Yeah. Yeah. The

And then with that, you get 10,000,000

of of how it Scott. Like, a lot of people are

Why it seems like everybody's just like, yeah. CloudFlare is

and and a analytics engine. Right? So every time that, like, a user visits your website there should be in a tab and and making sure everybody knows what we're doing. Yes. I think even as a even to to bring this, like, full Vercel.

do Node to try to get around ad blocker, I'm sure you could do that with this. Let's talk a little bit about just, like, Cloudflare in general and and infrastructure

and you get you get full control over it. And if you want, like, we just talked about,

$5 a month, like, that's the thing about Cloudflare Workers is you can do quite a bit on the free plan, and I'm pretty sure it's a $5

and sort of request that tracking script, the tracking script makes a request to a worker.

you host it on a worker.

hosted on Cloudfly. This is just for the tool itself. The tool itself,

That is an important clarifying

ad blockers blocking analytics, and we're trying to, like, proxy it through a different URL on a previous show. Like, you have full control over it. So if you do wanna

subscription.

per month, which is

their server prices are going up. A lot of people are are going back to maybe, should I be self hosting?

Jeez. $5 Yeah. For, like, their their base plan.

you run a bunch of commands, you generate an API key, and then it's just up there and it's just working, you know, and then you drop the script in your website. So, yeah, this is just referring to just this thing. Just counter scale. This is what it Scott to host it. Those numbers are absurd. Right? Like However, I do like I do like Cloudflare stuff, so I definitely went there pretty early. And then I found this service called Analytics Engine, which was in beta at the time.

It it's basically, like, 3 commands from the command line. Like, you clone you clone this, from MySQL or Postgres or those sort of traditional relational databases The techniques are always changing, like the tools.

why why is Cloudflare so cheap?

How much does this stuff cost?

If you're a software developer and you're pushing things into production, You basically have your own code do translation for what Cloudflare stuff.

Sign up and get 2 months for free. Man, that was excellent, Ben. Thank you. I mean,

Didn't really worked out. Got into web development. Was an early employee at the startup called FreshBooks and then discussed doing embedded comments.

Yeah. There you go.

painfully, like, 20 years. I I really don't like how that number has gotten pretty high

Trying to retread a little bit of that last episode. Mhmm. But, you know, I got into software to make video games. That was my original original career ambition.

yeah, it's a nice fun. Yeah. Yeah. And do you wanna do you wanna give a shot at doing a, a century,

what, probably 4 years,

My name is Ben Vinegar. I've been working in software and web development for, you can do a whole bunch of cool math on it. And that's really what is driving counter scale. It is a very sort of like, very simple application.

presented by? Oh, man.

some interesting stuff about how Century has saves his stuff in the database and the stack there and his time at FreshBooks and all kinds of interesting stuff. So we have him on again today to we're gonna talk about some Cloudflare stuff that he's working on, some really interesting a lot of people are really interested in, like, the new stuff Cloudflare has been putting out as well as and we'll talk a little bit about syntax. We've been doing this for 1 year now, and we'll sorta look back on it. So welcome, Ben. Thanks for coming on. Thanks for saving our butts.

at least 50 episodes a year.

And for the last

it's super reasonable considering, you know, you've worked at Sentry for so long now. So you know the product.

working on the product UI, camera, but for the professional setting

Sentry is just something that helps you understand

how your software is performing in production, whether it's telling you about errors and exceptions that you have, or telling you that your website is slow, gives you tools to just sort of become informed, really, which is the most important thing. And then gives you tools to remediate those problems and and get your software back on track. I love it. Very well done. If you want to give Sanity a try, head on over to century.i0forward/syntax.

eight and a half Yarn, I've been at Sanity Wes

Oh, yeah. But I said I've heard. Oh, you've heard. Yeah. Yeah. Yeah.

Yeah. Sure.

Ben is our manager here at at Syntax, and, he holds everything together. And we've had Ben on the show previously as well beef before Syntax was even part of Sanity.

I don't know. This is getting a little a little questionable here, but I'll I'll give it a shot.

Well, this the challenges of that. You Node, what what a high throughput ingestion stat kind of looks like. And I am not an expert to be clear.

So, yeah, we've probably done, like, a century read.

You wanna give a quick introduction, who you are and what you do?

your software is gonna break. I've been building some software lately and it it broke a lot faster than I was expecting, frankly. It's a really harsh reminder of how easily that happens. And, But it does what I understand is that it does have a SQL like language.

And, he he told us all about

Ben and I think you just sort of built it on being like, This is an interesting product from Cloudflare like, is there only 3 or

and they sponsored

And I was looking at, you know, okay, if you don't want Google Analytics, counter Scott, this project, and, Bos, basically, how you work with the class is very different from how you would work with MySQL or or Postgres. It's not a situation where you can just, like I got this data and I'll just load it up over here and it will work work the same way with the same queries. You have to work with it differently.

Riffing around, I'd be like, you know, it'd be really neat if somebody if there's just some hosted service that understood this pattern, that would let me just sort of ingest data at scale and write it, and I could read it back in kind of a in kind of a queryable way. So I started exploring tools online.

you know, Scott at like, I don't know,

Cloudflare analytics engine because

in the summer of the pandemic, I think I went Deno. This could have been like 2020. Just sort of interested in in Wes new infrastructure comes out, like, what what does that make possible? Mhmm. Right? from something like, I don't know the exact number, but what we need to do between now and this kind of, like, result that we wanted. We wanted to do some more stuff on YouTube. We wanted to reach more people. We wanted to make better episodes. Right? between

I don't even think my head space was there, Scott, to be like, oh, I'm gonna use this built in thing. I was just I just totally went to what is an external Google Analytics thing that I could drop.

I was lamenting this problem with, some other, you know, early century people. And, you know, one thing about centuries is that we ingest a lot of data, and I've learned a little bit about doing great on platforms, if we're if our listeners are up, which they have been, we just sort of tell that story internally and try to explain, you know, what is it that we're doing that we think is driving this. We don't have comment that I probably should have opened with earlier. Yeah.

Kafka rolled up into sort of or or similar kind of like, It's an open source project. It was spun out of Yandex several years ago.

US a month, which seems, I don't know, a little those are really minor very minor details that are really not gonna in a row column oriented. So if you wanted to sort of figure out every single unique column name, you know, this would be great with some with some visualizations on the screen. I don't know if this is making sense, this is JS, you you know, they're spinning they're spinning something up, a different kind of environment, and it's slower.

you wanna have it somewhat portable. Right? Yeah. I I wasn't I, you know, they will, And they have basically you know,

than my hosting provider. Right? Like, I'm not paying anything to host the website, but to actually host my analytics, Plausible is open source, and they run it as a as a commercial offering as well too. And so I I, you know, I poked at that source code just to sort of understand how they did it. And to analytics engine, and then the dashboard reads back

that's the underlying technology where all the sort of, you know, all the all the event data is written. And what is ClickHouse

You know? I just I just wave my hands a lot. But I do know a couple things. Okay? Which is if you wanna ingest a lot of data really quickly, you wanna write that immediately.

that Scott money. And I don't know. I I mean, those Yarn those commercial products. Fathom, Plausible, they're good products. I wanna be clear, but they do charge money. Mhmm. And I was just sort of curious why there wasn't, you know, just something sort of easy to use. Now I understand there is some baked in analytics with Vercel and other platforms and stuff, but, you know, I was just using some

let some other job deal with it later. So usually that's a queue

I'm always interested when people make new stuff in this industry. And when workers first came out, I thought I thought those were super neat, and I played around with that. I wanna say database, you're you're kinda saying like, Vercel.

Plug away.

for anyone? Sounds fun.

I don't know, reliant

for a variety of reasons. For example, Century little company,

This is a long preamble. I'm sorry. But No. It's interesting. I was interested in sort of, like, what would an open source kinda, like, analytics thing look like, but I knew that those kind of systems need to ingest data quickly I do know the product. I've also heard you heard you guys say that say a version of that line probably, like, you know, 50 to a 100 times at this point. So You've given us notes. More than that. I Yeah. Like, how many times? Like like, Century sponsored the podcast for, be binary data. It could be a string. Right? I think it's equivalent to

managed ClickHouse

I wasn't using a baked in solution. So even if even if

Yeah. I mean, I think this I think this even goes back to syntax, which is, you know, working with you all, wanted to do more in HTML and CSS, and and

And I didn't quite understand it when I first saw it, but effectively, and at scale.

upon your host just for your analytics, at least in my opinion, especially

database that they're running. So it is ClickHouse?

Awesome. So I wanna talk first about

and and, you know, revamping the website really early way to get started with that.

and I think Posthog, like, a bunch of a bunch of products are built on this, analytics products.

Now let's say that you wanna select 1 attribute. You're like, oh, select browser name from clicks from Postgres. K?

usage tables, Yeah. I I think even some background, like, building the product. And then I also kinda became our 1st engineering manager and and had a stint as our VP of engineering for 4 years, and now I work with you all. Yeah. Now you get to, use the blob blob 6 instead of, like, naming the columns. Right? Yeah. If you look at the code that I've written, it is it is a bunch of string, you know, that are that are sort of fixed, but you can make them whatever you want. Right? I don't know if that kinda makes sense. Yeah. It's wild.

and then Right? So if you think about it, most people, if they sit down, I'm gonna write an analytics tracker, and you think about what you Node. It's like, oh, yeah. Okay. I'm gonna write this to a data write this to a database.

web browser traffic analytic events? Yeah. In this case oh, you're right. And they have an pnpm.

a columnar

databases being a thing. Is is it specific you said it is like an analytics, like, widely used for analytics companies. Is it something, like, that's practical to be used outside of analytics?

real benefit that you're getting out of using it. Like, it's not a real alternative, I think, for 95% Sanity, whether we, you know lately. And,

Yeah. So ClickHouse is That worker

ever, but I don't really have a coherent answer for this, Scott. I'm sorry.

that engine using a SQL dialect. Wow.

group it, you can sum on it, That's part of why this exists.

So And on that note, I guess, because I was thinking about this. There's a software that's yet to be released called Boom. It's a it's an independent which, this kinda great thing joins a bigger joins a bigger company.

That worker

But I've never even, like, really thought about a column oriented

Is it used outside of that setting? It probably shouldn't be. It's really Mhmm. You Node, I I I'm not an expert, but I think you really would wanna derive some

and it ingest it all. It stands up. It doesn't seem to really fall down, and then you can read it back. It does expose really encourage them to, you know, talk about their successes

you know, if you've got some And we still do that a lot internally at Sentry, and I think you can see that even on Twitter from the the software

of what web development is out there. Right. Does that make sense? Yeah. Cool. So I was checking out this analytics engine docs, and I think you know what? I think how even Cloudflare built it was more, like, hey, if you're building products on Cloudflare, you can track your own analytics from your products into the system and you can bring that back to your users. Like, you could use this for

Postgres still has to go through every row to accumulate those names. Mhmm. Does that make sense? So, you know, at the end of the day, there is still a file system. There's there's or how this data is stored in memory is that it has to rip through every single row of data. But a column store database is different because it's storing sort of all the unique column events So that's the preamble to writes affect anybody's You know, this is the part where a producer could pull this up and verify this for us.

postgres will basically just like or the underlying database engine will sort of, you know, scan every single row, collect all of those. Right? getting everybody together engineering manager person

Couldn't maybe just like 2,000 lines of code that is it is a it is a script that you drop on the page. That script forwards events to a Cloudflare worker.

It is if you think about the name, it's it's ClickHouse. It's designed to store clicks. This is it is like an analytics sort of, database. Mhmm. What's different about it

Select star from this table,

a Vercel like sort of query language that you can use to read back that data. And you can

are you are you tracking with this? Like, does it default to anything, or do you have to explicitly go and say, I wanna track page views and

in columns instead of rows. Mhmm. If you think about when you're querying a Postgres onto onto a schema that they've defined, and I can come back to that. They queue it up. They ingest it. They write it into their own ClickHouse, And It has different sort of advantages and limitations.

I think so

Ben.

Welcome to Syntax. Today, we've got

we got a crazy week here at, Syntax here. We've got we're gonna react Miami in Scott lineup tomorrow. I'm flying out the day after that, so we're just recording for the week, getting ahead of everything. And we had a guest cancel, and we can't leave you folks without a show. So we did we got you the next best thing,