Web Streams Explained

a loud and airy URL in front of your images. You could do that. That's such a sweet API. I love that. And it makes sense to use streams here because you wanna for

Readable streams can only be,

a couple pieces of text. Right? Yeah. Totally. Yeah. And and You have a factory that you've made called a decompression stream. And then out the other end, you've got forever.

If if somebody is waiting on that website of that video is

then, like you said, you don't wanna wait To be able to get to work. Otherwise, you're gonna have to

You can rewrite the HTML so you can do things like If if the browser is rendering HTML, it must not. Yeah, it must not. It looks like it is if you have a component that takes let's say, let's say, it takes a second to go to the database and get the data. That's probably a bit too long, but imagine it was a second.

and with me as always is Wes Bos.

Open wide dev fans, get ready to stuff your face with JavaScript, CSS, node modules, barbecue tips, get workflows, breakdancing, soft skill, web development, the hastiest,

Got some hot streamy tips for you. Yeah. Nice and streamy.

It. So give it a try at century.i0.

What what does that stand for? It's the working group. It's the web web. Hypertext

This episode is sponsored by Sentry.

welcome to Syntax. Today, we are streaming some data directly into your ears,

CSD. Oh, you know, in terms of, like, form inputs go, you know, file file reader in general or being able to work with files is kind of it. A little bit more mysterious one that not everybody,

Use the coupon code at tasty treat, all lowercase and all one word. You'll get 2 months for free. This is a service that Wes and I have been using for a very long time, so check it out. And thank you so much to Century for sponsoring. Alright.

Boss and Scott Because the flip side is you use like a callback base, which is like you say, like stream on chunk, 30 feet long, replace, of web streams Being able to, like, write the same code on the server and and the client side, PIN codes, just work with the array items as they're coming in because then you're not taking up memory. You're not storing them in an array first. So I guess it also applies to just programming in general.

sortable, beforehand. Even if it is loading over time, just like in the YouTube, You know, if you had to sit and wait for the entire YouTube video to load, it would feel like it's taking forever. But the fact that you can start

findable,

All of the ways that your site is having issues at any given point in the tools and data that you need to know to be able to fix those bugs or issues.

Monday. Monday. Monday.

Hey, everybody. Excited to get streamy today.

meaning that, they are responsible for the DOM, Console log, the URL API, WebSockets, and and probably most importantly, fetch. So, Scott to Wes. You can rewrite buttons. You could add They were part of the what w g, which is the web being being pushed in by the React renderer. And then once All the chunks were done.

it. And we are going to be talking about web streams, all about what they are, the tech behind them, how they are useful, and where They actually pop up in places you might not have even thought about. My name is Scott Talinski. I'm a developer from Denver,

the tastiest web development treats coming again hot. Here is Wes Barracuda

the craziest,

I was trying to If you want if you look if you like inspect element it on a video player, something like that. Well, just on the same same thing we with HLS.

So nobody else can can read that stream as well at the same time.

comes from Unix like operating systems where it's used as a command line utility to Split the output of a command into 2 or more streams.

way more,

The the the term t

It takes its name from the shape of the

Streams are not a new thing in web development or in computers in general. Fluid dynamics Like, arrays as well. Like, somebody I was writing a recursive function create a decompression stream and say this is a gzip stream.

if somebody is reading it, data from any of these other APIs into a web stream by Putting it in a blob, and that's a pretty common thing. You also see that a lot with, like, And a stream And that's kind of the core concept of web streams is that I like to think about it. Web streams work work everywhere, but I like to think about it a lot in terms of server and client, the

From the array into a single string and then and then threw that into my cache so I could have it for next time. Wonder why it's called a t. I guess because you you t it.

plumbing t, Which is used to split water or gas. So love it, Wes. That's good. Well, that just shows that, like,

For the longest time, I could not figure it out. And then I went to the the docs for streams, and there's just a little method called t that was just generating the content every one second, and Let's talk about web streams.

Stream the output of my website to the browser, the as they were coming in, I was saving them to database and then they go out the other end. And you could even do that with

Send it out to the browser, and I could take the other tea. And I was listening for chunks Sort of what led me down this In the example, the example is kind of giving what you said is that you stream rabbit hole is First I was building a remix app and I was looking at the streaming APIs,

that's a way

but I also wanted to collect the HTML the Node. Js, or Deno, or BUN process

oh, b. You're thinking the letter t? Yeah. What are you thinking? I was thinking, like, golf t. No. Like, you're teeing it up.

You can await that. And then I just took all those chunks, put them into joined them

And then 2 people are writing to the same thing at the same time. So I ran into the problem where Use that. I had to load that into a database, and the CSV was too large And

No. No.

and save it as a string so that I could cache it for for the next request. Right? Because I didn't wanna render it twice. The next person that comes can can be served with the cache. So How do you get the 100% output of the render like you normally would without streaming, but then how do you also stream to the browser at the same time? And Both. You both can't swing at the same time, but you could give them another t. There's some awful example you could probably think of there. Oh, here. Check this out.

feature as a

things like asyncawait.

go and,

Wherever JavaScript is is being implemented. Right? Like, not just the browser, and then start to literally send data over to the client

for Yeah. I guess that makes like a like a I mean, this is straight from, like, golf. Why? I would think a fork. Is it straight up, like, because of

Webstreams API. And there's no you don't have to import anything. It's just available to you.

They are or sorry. Not a browser. Because they are a web standard, they are implemented in of course, they're implemented in the browser,

a very long time now to the point where everybody I kind of generally get the concept of what a stream is in a, like, a higher level view. Right? So, like, what are streams? Well, Think about it this way. It's basically

Node streams into WebStream, which is great because then Node. Js streams because a sandwich into my house first I'm going to replace every instance of

We, as human beings, have been streaming things in all sorts of ways

Yeah. And, also, it's, you know, it's one of those things that beyond the web streams

but literally everywhere. Yeah.

So

but Deno has implemented them. CS. Bun has implemented them. Cloudflare has implemented them.

because they are a browser standard.

of tooling or something like like helper functions that will allow you to change Have a stream coming in and then you need to I thought that was a cool word for it. Essentially,

render to pipeable stream and render to readable stream. These are 2,

And that's really cool, and they have provided a bunch

is somewhat new. So Node has a

Its own implementation of streams, but now they have also implemented of HTML.

They're not new. They I think the standard came out in 2015, 2016 or something like that. But it seems as of late, to be able to read it all, and you you shouldn't be, like, waiting for the entire website to download before you can get to work. You gotta get to work as soon as you've got

web standard, the buffer can hold before it starts to override the previous data.

for a full archive of all of our shows.

It. Head on over to syntax.fm dives into instantly.

that that seems like a good use case for media or just really anything honestly. But doesn't the browser already do that? And,

And don't forget to subscribe in your podcast player or drop a review if you like this show.

like you've mentioned, it's it's definitely gaining a bit of traction in the mainstream,

Experimental in Safari, parts of React DOM when you render it out. Remix uses streaming. Next. Js has it in beta right now. Svelte has announced they're going to be rolling out streams. Seems like everybody is talking about streams, so why not do a show on it? Totally.

developers Purview here. So I'm sure we're just going to be learning more and more about streams all the time now. Beautiful.

compressed in.

let's say you have a user component that has a database call inside of it. It. This is where we're gonna start to get into React server components streaming versus buffers,

and 6 minutes to find that thing. Right? Like, sometimes it can take a long time.

Should you wait? web features, read by 1 person, meaning that if you have a stream and you start to read it, I guess not relatively new. It's Chrome 71.

One of the things that simultaneously annoys me about Braintree, but is also important, is that they stream all of their searching functions.

you can start to send the HTML that is already done

the entire Braintree database, you then pipe that into writable streams connect to the server, Do the all the SSL handshakes, all of that stuff, Sick. Peace. Peace.

because they don't know that if that data is going to be too large or whatever. And if you're searching for thousands and thousands of transactions and information back, it's always going to be a stream. That that's a really good point because,

you always have to use a stream

So if you wanna look for transactions, at a time, right, instead of grabbing big chunks at a time. When you watch YouTube,

Ten milliseconds But you can also just You probably shouldn't JSON parse that because you're going to get a JSON error.

To the client and where that becomes really helpful is, Into the browser. So pretty common use case for this is if you have, because there's simply just not enough memory available to that process They go back to fluid dynamics.

Like, if you wanted to search working group. So that is the working group that is responsible episode just to sort of explain what they are and where where they're using. So sometimes you think about, like, oh, why doesn't JavaScript just add it? And, like, With TC 39, they're responsible for where do the semicolons is not actually done yet. So by the time that first promise resolves,

And one of the benefits of streams is that it can Give you data as you as it comes in. It doesn't know that it's done yet. It doesn't know where the end is, but it says, well, I got this so far. Do you want me to just To give you that so far, and we can kind of like, if you had images and you want to put a little on the other end those are streams as well, but they're not part of the web streams.

your page on the server, As you are rendering out But A blob is a representation of a file like object of immutable raw data. So a blob is essentially a chunk of data. And if you want to, you can turn

Node

for all. Yeah. So I think the reason why I think is new is because full support just landed in Node.

Yeah. So text decoder is a relatively the browser and not just an entire finished request.

streams that things like there's a possibility that you are receiving a stream stream,

Piped you. What about a text decoder stream?

React and these packages what does that mean? Where do blobs live? So What is a blob? I was just tripping over myself, for for trying to figure out what it is if you see it in a video element.

And keep going on it.

API.

18

those chunks are buffers in itself.

in the past, and we can we can do that example again because I I so far enjoy it.

Totally.

on there. So that's a really good thing to do is as the pin codes were coming in, And as it is streaming in, new, .tee and you simply just call .t and that will take 1 stream and split it into 2. And so I could take 1 of the t's, any use case where I would literally create my own writable stream? I think probably most of us developers are going to simply just be consuming My confusion with all the, like, WebRTC stuff, I was like, is WebRTC web streams? Is it not? You know, and It's sort of its own thing, but they can be before I go ahead and eat it. Right? And sometimes it makes more sense You start to deal with the sandwich

as it is coming in. So just imagine somebody with their mouth open, The the uncompressed version. Just trying to understand, like, doesn't the browser do? Obviously,

the way that YouTube works is it's grabbing a little miniature parts

bring that into your house. The sandwich is too large for your house. Right? What what is it in node? Node 16, technology

just like you have,

How would you get into it? If you were thinking about it's kind of interesting that the browser doesn't it's going to take anywhere between

the parts of a stream are called chunks, like Scott said, Use just straight up text instead of having to worry about that type of stuff. Yeah.

experienced with YouTube, is seriously just sending little chunks. You could think of it as a a literal river. It's you're putting a little boat, and you're watching it go down the river, And then somebody's grabbing that little boat, and then another little boat comes down the river. Yeah. Exactly. And and we've used the,

I didn't read them all into memory. I simply just

And, you know, like you said, data can be too big.

it. You are watching the video. It appears as if the entire thing is loaded. But in the past, we would have had to wait for the entire thing to load it before being able to access it. So a stream,

buffers is just basically a piece of A lot of this is sort of picking up steam, But There also is another working group, which is like what features do we want in the web, Which is a JavaScript API for compressing and decompressing

CS. No concept of how long the sandwich will be, but you're saying, well, I better start working with this thing as it's sort of coming on in. And that's not necessarily buffers are,

There was for await The HTML Rewriter API, essentially what it will do is Massage it so you wouldn't maybe be in the middle of being like, okay, I'm going to be streaming in this text.

of an MP 4 or whatever the file is, and it's sending you to that little part. It's putting it into the video player. And then as you're watching that little bit, it's going to grab the next chunk and the next chunk and the next chunk. So while

data in memory not using a stream, you would think about I need to bring that entire I once had to do pin codes. I've said this example, but it's been years since I've I've given it.

you're sending small chunks down the wire, and you can grab small chunks

and

Again, it seems like the the context is, see exactly how users are it. Interacting with your site when they're hitting bugs, or perhaps it's just the performance aspect of this thing. Which pages are running slow? How slow are they running? And is this something that you need to dive into or just a big old list of, that whole thing's gonna have to download. You're gonna have to then parse it and then Do your work. Right? Get to work right away while while it's here. One thing I ran into with Remix is

you when you're creating the writable stream, you end up defining, like, several methods inside of here, like a start, it. A write, a close, and abort.

most of these are are very obvious of what they do. Right? The start runs on the start.

to the sync. Right? Every time a new chunk is ready to be written and sent along,

runs on close and abort runs on abort. But write is interesting because write is called repeatedly every time it. A new chunk is ready to be written,

that you can continually write to this thing as it goes rather than having the thing done beforehand. Right? Yeah. Exactly. It's A really simple API and like I'm trying to think like would there be Sorry. May yeah. Maybe media? Because, I mean, if you can send smaller bytes along the line Yeah. As compressed And then uncompress them on the client. I mean,

Close

write is then called allowing you to kind of do whatever you need to do there. Now I've never personally used writable streams, but,

As you have it, you're putting it in. What blew a lot of people? I made a little TikTok on this, and what blew a lot of people's minds is that example in the past of of a sandwich. We've talked about, And then that will that will actually get you your response. And a lot of people are like, why is why is there 2? And the reason behind that is because

Have I had the Node. Js or the Deno process a file and presses okay, and you will be biting and eating the sandwich or cutting it up and putting it on plates or doing whatever it is that you want to do with the sandwich, but you are dealing with it as it's coming in, and you have It's not in Firefox yet. So, like, I'm just trying to understand, like, And then you pipe it into that decompression stream. And then out the other end of the decompression comes your uncompressed version, which is really cool. So That's an example of really good piping you've got

So like you said, there is still a buffer in memory that is

it's going to hold less data in the buffer before it starts to overwrite that data with new data coming in. Those are just 2 things that you might see in MDN when you're looking at the stuff that, Me me, personally, when I saw it, I was like, high watermark. What what's happening with you? It sounds

the fetch API. And we always talk about how annoying

Sort of thinking about that as well is when you have a stream, You can request a website or request a resource

I was sending the current date you may have inspected element on a video player at at some point in your life and you see the source that it just landed in it was in, like, node 16, and your browser. So what can happen is that,

cool. It sounds way cooler than it is, unfortunately. Yeah. Oh, that that's cool.

2 little jargony things I saw when I was looking over the docs. Because if if you're the type of person you pop up in MDN when you're looking at this, it. You might see it, like, a couple of words that people,

Sync is a term you see often used with streams, and the sync Sentry is the exception handling service that allows you to really, Truly understand how your application is functioning at any given point, whether that is being able to

Both of them are sort of talking to each other at the same time. It's not a 100% literally workers because web workers support streams, All right. That's That's it for today. Thanks, everybody, so much for tuning in. We'll catch you on Wednesday.

finished state. Yeah. Yeah. Right. It's it's not finished state. I think it's it. A good way to say that. And, also, like, you know, I think the big thing is is that it could be for performance. It could be for just straight up perceived performance. Right. It can feel like it's faster because the user is getting something quicker than they would be getting

set how big the chunks are that are being sent to readers, you should probably be using WebStreams going forward.

Should the whole application wait a whole second For just that 1 piece to be done, or can you start to do the stuff that you can do, which is

is basically the destination in which the data is being stream too because you can stream data from, you know, one location to another. The sync is the final destination where that data is received process in red. So when you see that something has been added to the sink, it's like it's sent to the final destination.

And that can be really nice as you can set it really we can set it really low and you say literally any little piece, send it over. Or you can set it higher being like, okay, I want to send significant chunks at once.

and and start consuming that information right away It's going to improve your perceived performance for your users.

Not the movie, The Final Destination, but, The actual destination. And there's also something called the high watermark too, which is basically the amount of data

Being able to hold some of this data, and if you have a high watermark set that is low,

talk about when they when they talk about streams that I think can be a little confusing. So for instance, there's a sync.

you can You can access that file in the browser with the file reader API. Right?

like response. Text or response. JSON.

As you have it, and then you can sort of just sit on your hands and wait for that 1 component to to finish rendering and then shoo it out the door. So it's kind of that what you are fetching A vitamin company that put pin codes under the lid of every single vitamin and you could you open it up, you put the pin code into their Facebook app, and you see if you won. Right? That was that was the thing. They sent me, like, a 4 gig CSV of But they can be converted into this standard API by using blobs. And is I would say probably 95% of it is built on node streams, right? Node streams. Yeah. But Now file right from their their phone. It's kind of a lot of bandwidth. Send that to your server just to resize it. So you can Stream it into the browser,

That's part of The streams API is that you can either wait for the whole thing to be done and then I went into specifically in the context of a lot of our frameworks, is

And if that's the case, or web streams. Oh, well, that okay. Maybe we should cut that out then. No, we should leave it in because that's a good example of like most stuff In the node world, probably

From the actual where would you use this? Like, if you if your server is sending it to you with Gzip Media? it will lock itself or there there's a specific word for it. What's the yeah. What if frozen? Is it frozen? Is it, what is the word? It's locked.

the second one is that Like, you can also check headers in there as well, but the I think the most important thing is that that there's a possibility Let's say you had a sandwich that was Then loop over it and put them into the database. So I had to for what? Let me I I know the answer. Yeah. I just want you to make that clear. Oh, good. That that's a great question. In my case, it was too large for memory, meaning that and then you get a promise back. And then from that you have to return use for, not in Firefox at all, You have a really good example of creating a writable stream, and you wanna go into that, how how that works? Yeah. Writable stream is, basically, you have, especially with lots of these packages, too.

you can intercept that stream then you get the chunk and you have To sort of deal with it, display it to the user, append it to an array of collected chunks, etcetera. You also have listed here that file reader is a stream. I know a lot of people, Before the server is even done with the entire request. So I made an example where every one second,

Like in my example where I was sending HTML from the client to the browser, streams is that you get chunks and those chunks are buffers. And then you got to convert the buffer into const chunk of chunks.

will return to you a couple of things. 1st, it returns to you like a status code. So if you are expecting JSON and you get like a 500 error, interoperable, And that's really nice that we have it in literally everything. So

or you can process the chunks as they are Being blown in chunks as they're blown in. Love it.

it waits for all of it The server can connect to the client, and the the client can start to intercept the data And then just call the dot stream method on it, and that will

And you can listen for what are called chunks. So as chunks are coming in, a video file or a a photo, something like that, and you don't want to send it to your server. You just wanna literally deal with it in the browser. That's really handy if you want to crop or resize a file Without having to send it, someone might upload a 20 meg Stuff that you're used to for collecting and waiting for everything to finish. And it's kind of interesting because Node. Js has had the concept of streams for for a long, long time that Streams in general and in programming is not anything new, but this new standard, and I had to then What else? React now has Reading streams. So something gives you a stream and you want to read it. That's a readable stream.

And you just wanted to collect the entirety of the stream into a bucket before doing anything with it for any given reason. Right? Let's say You can't consume a stream. Or for instance, like me, like, if I want to streams are typically consumed or used. And let's say you wanted to, oh, let's say you wanted to have a bucket,

I think the the way that

The

text decoder and encoder streams will allow you to

regular web streams and then use all of the promised c s formats.

one thing about

collect all of the data from a stream before I write it to my database or something like that. You can use a package called stream collect that basically just wraps your stream. We'll have a link to this in the show notes. It Just wrap your stream and wait for it to completely finish. When the stream is finished, it just omits a promise. It gives you the data. Bingo. Bango. And, you have all your data there. So let's say if you, for some reason, are been have been given a stream and you cannot consume it as a stream and you just wanna grab that data, you can collect it using stream collect. Oh, I think this package is specifically for the older you can await the entire stream to get The output of it, but then you can also call back to get the chunks as they're streaming chunks in. Yeah. I guess that goes to show it is a little confusing where that line is drawn between some of these older APIs that we've been using with, like you said, NodeStreams first,

but you had to import it from a package, and now it's, like, proper. You don't even have to import anything.

And and if you want to, let's say, kind of go against

or deflate

this web streams API. Right? Like, some people may have experience with streams and not Web stream specifically.

What else? I just did a quick search on MDN. Like, what other web streams APIs are there? There seems to be a compression I'm assuming that is just because it could be chaos if And now there's all these conversion APIs. So it's it's nice that you can now convert those to

HLS streaming.

part of the WebStream's API. So I was like, is this it. The same thing or different as, like, WebRTC Scott with Wes, UTF 8 if you want to see it. So the All of the stuff is sort of built upon fetch,

and then put it into Canvas or resize it or whatever it is you want. Totally. So what other types of streams do we have? Media We've talked on this show before about get user media, those types of things, whether that's your webcam,

your microphone. I I mean, think about this because, like you said, it's like we have an input, and there's no

blobcolon//Like, theoretically, you should just be able to use await promise. All promise all settled with Regular streams

So do you wanna talk about, like, how we can utilize FileReader as a stream? Yeah. So

Media streams, webcam, microphone, web audio, all that type of stuff, that's not a sync and defer on the script tags you could This is not so so web streams are a web standard.

given output. We're we're maybe somebody's still building the sandwich it. As they're feeding the sandwich into your house. Right? And that's kind of like what you have with any user media, whether that is your webcam, your microphone, or whatever. Like, right now, Wes and I are talking into an application called Riverside that's a website

If you have a file type of input in in your HTML and somebody selects

Grab the the file that you want. So there's a dot files property on inputs that are type of file, The first promise Essentially, It. So we could give you example of a very long sandwich.

take that input, why have they begun begun to pick up steam recently? Well, and I thought it'd be a good

that is taking a constant stream of our audio in. It's Handling it while we're simultaneously You often see, like, what is the m three u eight file, Which is essentially just like a cut just it. A way smarter explanation than a a golf tee. I don't know why. I don't know why. But if 2 people are golfing and you want to

like a for loop. So you can say for const to be received before it actually renders it. So if you're application Workers carrying the sandwich in,

And then literally every single time that a chunk comes in, That promise will resolve, and then it will just loop over the next one and wait for and you needed to be able to Yeah. It makes sense. Like, if you if you have a waterline and you want to split that 1 waterline into 2 Yeah. You're right. It. Right? Yeah. You're you're right. You do do that. That's actually across for you can literally get pieces of whatever it is you're fetching, whether it's a a media file or an image or HTML or JavaScript. And you can If you want to do, you can display that data as it is being streamed in to the user.

The other thing about streams is that they are iterable as well. So you can

comes the still recording it. Yeah. It it's nice. And, so the Hyper application. What is web? in the past, and I put all the items into an array first and then Dealt with it and someone's like, you can Faces. And the idea is that you create your own streams. We have readable streams, which is change it's just a white screen until you until you the streams of data using GZIP start to stream that file in from your user's computer The Webstreams API is built on promises. So and experimental in Safari for iOS. And it's in everything else, which is just really nice. So what the dig into what that one is a little bit more or if you have an example of what that might be used for, tweet us at syntax of them. Yeah. Yeah. It's a I mean, I think,

And then there is also writable streams, which is if you want to Literally write data to a specific stream. So you're on the other end trying to give somebody else data, and then finally you can pipe them together. So it's not uncommon to my computer or the Node. Js process would crash Until the server closes the stream and then the the browser's okay. They're done. Here it is. But with a fetch,

pipe that into another type of stream, whether that is a writable stream and you can

You can I think BUN has reimplemented it, and I'm pretty sure you can get it for Node as well? and whatnot. So the fetch API is with the double promise, Meaning that you fetch something The HTML, it's a pretty common use case of that is the Cloudflare HTML Rewriter API. And this is such a good API that now and then out the other end

it's like a road map for, And there there's some, like, kind of jargon that we get into here. I wanted to just quickly get through, like, 2,

grabbing the chunks of data. Right? It's gonna tell you how many chunks there are, where the chunks are listed, and and what to go off and get. So if you ever see one of those files and you're wondering, Is this a media file specifically? It's not really a media file. It's almost like a road map for a media file. So in the streams API, there's sort of 2