Next Level Web APIs. Bluetooth, File Access, Thomas Steiner - Project Fugu

was, Lego. And, Lego has this really, really cool set of, well, it's kind of It's not a toy, but, it somewhat is. So it's an educational toy, I guess. And, they just they just realized, So we have the WebNN Something like a scratch editor where you can, you know, move around, blocks and then flash them onto the device so that, you can then run your LEGO model, program the hub, and so on. And, yeah, we just showed that, LEGO

web with our application.

Visible, so, you don't really need to, accept this permission if you don't want to.

That's cool. Speaking on, like, companies getting on board,

They can obviously still program it in a way, that requires, And, yeah, then I sort of just reached out on Twitter, sent you a DM.

giving giving up your location history. But, like, in the worst case, there's just no way to, track background location on the Wes. So you're not becoming an active target for whoever, And, Pnpm is kind of the Bos, It's essentially all your zoom levels need to reflect this change because you wanna zoom in. You wanna see the change reflected everywhere. Right? So, And also responsible acts JS, like, the the barrier between you and, the API low level signals that you can, send to the MIDI device.

they have typically which makes them very performant.

So, yeah, just going web only for these kind of things makes a lot of sense. It's, it's more secure. It's easier to roll out on many devices. You don't have to have,

that now have, this app that allows you to configure the equalizer and stuff.

whatever, Windows 11 to patch your Bose headphones, whatever.

And the web just makes sure that these kind of things, Mindstorm was before that. So the new, generation is called, Spike. Okay. LEGO SPIKE Education.

people feel bad if they have to give up whatever their location history just to

So, yeah, it's just way more accessible for a lot of people. And, a lot of companies also realize that this is actually, Essentially, they have a big, big swap file on their native application. And in the swap file, they host things like, case, happy and possible Let's say 3.

We just got out of the Google IO season, so we had a bunch of events all around the planet. And, What we showed at Google IO connect

Especially a very mixed architecture of Vercel.

making a ton of sense.

uses this For their, yeah, production web applications today shipped with, whatever LEGO kicked you by. Wow. Is is that called the LEGO Mindstorm?

all the brands?

we discuss hardware APIs regularly there.

Moves people and, makes people privacy discoverable by the user, by just going somewhere on the file explorer and seeing them,

There's a bunch of browser vendor representatives how many different, a mipmap is a way of Just precalculating So, yeah, as you said, I'm part of the Chrome team at Google. I'm focused on what we call project FUGO take very strong opinions, that they hold very weakly and sometimes very strongly.

I would say heated ish debate relatively quickly just because, Or should there be,

So, yeah, I would say it is a very, very interesting discussion all the time.

what's the likelihood that all of this stuff comes to the Apple ecosystem

taking these things seriously in in your and, like, if

a yearly event where all the browser vendors and, industry and researchers and whoever come together.

APIs that could maybe reduce the influence of apps would be Apple.

HID or head.

And, Adobe probably pulled some strings behind the back of, which is kind of mind blowing in itself.

menu for because Mhmm. You bring a bunch of people into a room, all these schools and universities, Get a x signal onto y.

And, of course, we do hope that eventually we can convince vendors like WebKit, like, So, And, eventually this also led, Mozilla to implement. And, now we Yarn in the situation where we have this new API.

Yeah. When it comes to security of USB, whatever.

I I guess maybe speak a little bit on, like, Do you see

and you have a bunch of people very engagedly it's like, oh, no. I can't realize this app idea on the web. I need to go native or I need to go electron tracks this data.

very, very detailed, which, ironically, It's the bar code detection Yarn, and Wes actually see Apple's implementing that as well, because it's considered to be a generally useful, Get the same, yeah, are very So Wes USB, web Bluetooth are some of the oldest APIs that we're talking about in this context. So They were launched on Chrome in 2015

a bunch of engineers with Node that want to, yeah, bring these needs forward on behalf of the users.

discussing.

And, yeah, So and it does exactly that and nothing else? Change whatever equalizer setting in their headphones.

They discussed. Half of the people don't really understand the problem just because it's a very specific problem

And, it becomes a relatively,

we discuss these APIs regularly at events like TPACK.

from Mozilla, from WebKit, from Google, obviously.

So, there's W3C TPAK, yeah.

it's a topic that yeah, power Node device memory.

ish.

specific companies? Like like you said, Lego. But, like, do you have any other examples of companies that Say, like, you know what? We need this API

Agalia is hired

mostly with, Apple, and of course, this is only speculating, it's, very much a black box.

WebKit maybe,

they were hired by Adblock to implement CSS ESLint

Is is a lot of this stuff also driven by

And Safari and Firefox are both choosing not to implement this.

blocking elements that have

Yeah. I'm glad you brought up WebMidi because,

that was one that I think got a lot of press for. Google implemented this,

will this ever happen? We don't know. So a big company like Lego asking for this is Obviously, something different. But then also, they, of course, do have, that, yeah, was on the market to just say, hey. Let's go, adoption because create an on the fly extension whenever you, wanna talk to a web MIDI device. And this extension then, it's an endless discussion. And, I guess Wes don't really have an answer yet. So more recently and now continuing to new interesting API proposals,

They could just download, the, iPad app for Allego and talk to devices like this. So, Recently, Where do you start? Where do you stop JS a very interesting And, this Eventually led to the creation of what we now have as a cross browser API I'm running suddenly in the red zone, I could then dynamically swap and say, I go from on device to in the cloud. So there's a whole bunch of interesting ways to combine all these new APIs, We had the Vercel bug buds recently, of the web

which means sometimes, yeah, Going through extra hoops like like Mozilla did, I hear project food, and I was like, wait. That's something that I'm working on and that I am very interested in.

But then you said Firefox did come up with a solution there. So do you think that Safari will reverse their position now that there's a

it's interesting. Yeah. Definitely. Yeah. So we talked to Eric Meyer who works at Agalia.

So we were hoping maybe, something like this could be a way forward to, allow other vendors, on Chrome.

a change there, or do you think there needs to be more I I don't know. The reason was what Sanity of fingerprinting was the the reason why Safari declined to add the midi support. So

And we need this API in order for it to work.

Still ongoing. There's no final solution yet. But, yeah, that's the elephant in the room So, yeah, Wes,

by all the browser vendors to implement Sketchy ass Windows only EXE a crazy idea someone thought of one day to Actual ones that are going through the entire, creating embeddings

because I

allow for our browsers sweep, Most, yeah, curtains that people see, You can think hybrid so you can see, So all of these discussions need to be How do you make sure that this works? Another opt option is, of course, to just say, hey. Just bring your own AI AI key.

Specific features. And I thought the most interesting one was that

ads.

a very strong presence on the, Bos App Store. So you could say, people who wanna talk to Lego devices and, who have in their schools, let's say, a fleet of iPads, Which then shows another interesting problem, which is, not all operating systems have the same level of support. So If you think, a Windows laptop, maybe face detection is part of the operating system, but maybe not. For Android, it is. So

What do you think is gonna happen there? Do you think that will eventually get reversed? So what we saw works I think one of the bigger companies that people criticize for maybe not getting on board with

please switch browsers.

So today, I listened to, your show on AI, which, Yeah. But when a big company asks for something,

Obviously, if you look at, devices like iPhones,

Whatever detect face something something API,

exposed on the, running in Chrome applications.

Wait, does this make sense as a high level API? So should there be, write and read operations, We have an API for compute pressures. You could dynamically even, see, Oh, how busy is the processor? in a way that, they could be reused by different origins. You have to, in the worst case, download the same model For every single application that you're using. So, if you have an app idea, From something Vercel, from something, The one thing that is,

whatever camera image through it and you get, the detected faces and landmarks and whatever, on the on the device in question, To consider their position and say, yeah, this might be something that we can,

And, proposal by Intel,

or should this be somehow be tied into,

Wes to ask with many of these APIs. So how high level should it be? How Generally useful. Should it be, experiences like this and others.

API.

So, yeah, it's, It's a WhatWIC standard, the file system standard, So half of the people zoom out,

gives another interesting,

Android phones, and so on, they tend to have, face detection as part of the operating system. So The camera is able to, spot where JS the face, so that it can zoom on you and stuff.

JS plug it in via USB and visit a website, and it will flash your hardware not necessarily because someone wants to visit the URL in the browser and use this thing, but our app runs on Chromium.

to and run it to to be able to update your software is such a nice experience. We think so too. Yeah. So there's a a couple of

on the electronics show, I talked about how I bought these little Wes 30 twos.

Bluetooth in the browser? And it's like, well, yeah. In in Chrome, they have it, and they've they've had it for for quite a while. And, there's also a a web serial port API, which JS,

And, like, that experience

Like, let alone people wanna build desktop apps with with Wes tech. That's probably a big part of this as well. But just that experience of people don't have to download some

web APIs primitives like a like a web GPU support that makes Fast as hell. This is sort of blurring the lines with the AI discussion.

new APIs that enable this, to your question, how serious do they take them? I would say relatively And they are in a position of saying, to their users, hey, we are big company X. You can use our product in Chrome. And if you use our product, you know, Safari, And, for them, it was one of the best solutions, for web assembly.

that are either shipped In some browsers or are sort of being

And the process for flashing them

my Bose headphones,

from it. And,

thought about is is so cool because, like, at the very basics, I posted a screenshot of me working on the Bluetooth

we don't need to run these checks, because we can just be sure the application None of these Sanity be cached and stored, Is it running in the red zone? Is it, still green? This is, of course, something that, no vendor wants. So How performant is the, device?

Change zoom levels. So Uh-huh. Maps,

Other browser vendors have Other options, but, yeah, mostly, the idea is the same. We wanna make sure that you don't download anything, bad onto your device. And, if these files are not application that you're loading, they try to, experience across the globe to everyone.

company on the web.

deals with these files alone. There's no way, that anything else can get access to those files. And this is why, yeah, the access to these files can be very performant. Mhmm.

so that you could, get Synchronous access to, because

driving cross browser as well. So, Yeah. That's a very interesting discussion.

the origin private file system, wherever they may be and whatever,

is like a generic Way of saying, hey, we give you the freedom to load whatever model in the browser and then run obviously AI face detection, face landmark detection, and so on is a very interesting computer vision Ask. Mhmm. So with many of these APIs,

forms part of an API called shape detection API.

sampled. So do you have any other ones that are kinda interesting or or some of your favorites? So, face detection Actually JS a really interesting example, that we could maybe briefly touch upon. Sure. Yeah. So face detection What API should we have or should we just provide

So one kind of in the middle that I've used a couple of times in my

serve for this and probably other purposes. And I'm,

What, they needed on the web was a way to sort of replicate this behavior so that, you could, create a Photoshop Image or open a Photoshop image, make some changes, downloading a model can be Free for you if you are on a high speed Internet and I'm 50 max, 100 max, 500 max, whatever. Mhmm. It's essentially just done in a second, or it can be very, very expensive.

courses is the face detection API, and I don't think that's a browser standard or anything like that. It only works in Chrome, but it's kind of cool to see that That type of thing is being,

And that that's me working on the Chrome Vercel team. So we actually wanna get web applications, not Chrome,

that is modeled mostly after POSIX APIs

And, yeah. So this is where we could see this kind of behavior really

The requirement there is you need to talk very performantly to a file and, frequently so. Because whenever you make even 1 pixel change,

zoomed image so that it can then very quickly, So An API for talking to media devices over the web. And, Mozilla was the first to, ship this after Chrome. And what they did is, they came up with a solution where they essentially

Adobe is obviously a very, very big

It's hidden from the origin, which means, the files that you create there are not opposed to the user visible file system. And this means, And all of those would need to download some sort of model that you need to store somewhere and ideally store it in such a way that it's Persisted, which brings us back to the origin private file system,

So I didn't know what a mip mipmap Wes, so I guess, most,

Or, actually, hidden in the origin of the,

would be happy for me to explain at least try to explain what it is. So, essentially, if you have images and you have images of different sizes so that you can zoom in, By combining all these different APIs and, if you wanna, And, we talk about these APIs. And, it's typically, mipmaps.

and then eventually, And the first success was had, finally with WebMidi.

they read whenever you download an as a a file from, an application, But, yeah, so face detection is also, in this family of shape detection.

Let's go through a couple more of of these APIs because there's there's literally hundreds of them, ranging from

called origin private file system.

that, implements, yeah, essentially a very performant file system Sirius.

and they just somehow your browser, typically does something, like virus checks and make sure that you don't, I don't know, get a Trojan horse. So, this is called safe browsing,

various resolutions of a certain saw that But then, yeah, because

so buckle up and grab that old handle because this ride is going to get wild.

Oh, I'm starving.

This

is the Syntax supper club.

I sure hope you're hungry.

Wash those hands, pull up a chair, and secure that feed bag, because it's time to listen to Scott Tolinski and Wes Bos attempt to use human language to converse with and pick the brains of other developers. I thought there was gonna be food,

Scott, TensorFlow running in Wasm.

TensorFlow.

Wes do have, as you said, the higher level APIs.

something that I found very interesting is,

connecting all the dots, making what the operating system gives? Which then, They have And, what we want is we want to Get other browser vendors to see that these use cases exist and that the, use cases need to be addressed more safely on the web rather than On native applications, as we said before.

sweep swap, the implementation, you could easily do so. And Something that I'm very interested in is also, And, very cool applications like, Scott behind the flag, just, shipped all these different, And, yeah, they needed this API. They wanted to bring Photoshop to the web and not just to Chrome, which JS, something that I always applaud. So if you build applications, you should be building web applications and not, applications that only run-in Chrome.

which obviously makes sense if you're a processor maker that you want to get things onto the processor.

But, yeah, we also obviously have things like TensorFlow, I've probably done the worst job of explaining it. But essentially, Yeah. So, one of the big cases that we had was Adobe. And, Adobe brought Photoshop to the web,

globally, made Apple implement the o p s so our original private set file system, OPFS.

if you think global,

But, But, download cost is Node thing. Storage is then the next. Like,

We've had the Devon from Descript as well. That's kind of similar idea. And I dove deep into the, AI.

let's say, video image editing applications to use? So, straightforward,

Maybe if I notice, and then be able to zoom in very quickly.

a desktop

It will be consistent, browsers don't have to implement things like, safe browsing. So, yeah, make The use very low powered Chromebooks. They have relatively simple Windows laptops.

the bowels of CapCut, and you go into the library application support, and They have a lot of their models downloaded

and, yeah, just make applications that work universally for everyone, And the way Photoshop works, for, a blurring API. So, looking at your image, Scott, so you have your background blurred.

The cloud will determine how fast or not your model can run and not the device.

video editor that's built in.

focus on the on the web.

that's another interesting problem there. So how do you make sure that, your applications are useful and accessible like, All of these APIs have very, very different threat models. So something something Bluetooth JS a different threat model from something web USB, so you can Yeah. So that's the the elephant in the room.

device they might be using. Yeah. I CapCut is we had a lot more,

where you could store such a model.

You plug your AI key into the settings, and then you can use something that runs somewhere in a cloud, It was born Under the name storage foundations, so people who have been following our blogs have maybe first read the storage foundation blog, the question arises.

and I downloaded the thing and I was like, this is amazing. Like, it's a full blown video editor, HTML and CSS.

for this library so that it'll run

faster. So it's kinda interesting. Like you said, where does the line draw of,

Very interesting discussion. Yeah. I I went down this exact route yesterday, Bluetooth.

It's just the CPU.

And I did that one of my types, of course, is the hot dog detection,

Js, which is being worked on by a dev Hugging Face,

And there's a really good

Like, I can load a model into the browser and use it because I don't know if, like, in 3 years, we'll be like, well, The browser one sucks. You know? It doesn't work. But what people are asking for is even a lower level one JS just we want Wes GPU support Amazon. It's called the Nimbot standards process.

Hot dog detection.

there was a proposal from Intel how should you just implement something that is super low Vercel? And, if you wanna get To a higher level, you need to, whatever, pull in some model that does the task for you from Hugging Face or whatever.

So this is Yarn of the operating system in many cases, but, of course, it's also something that you can do with, segmentation. So you can just use a model to segment, the foreground from the background and then apply some blur effect to it. So, yeah, Pnpm have proposed this, as an API that is high level, but it can obviously be done as a low level API as well. So

And I went into the, like, the issues to see what people are asking for, and it's not

but also things like sentiment analysis And, b, I was amazed that I was able to connect to it and see all of its capabilities. And I didn't get it to print a label just yet because there's some Bluetooth devices don't want you to they want you to use their app, and some of them wanna be open. But,

everything from, like, image detection,

A bunch of models in the browser kind of seeing

which ones are small enough and fast enough that you can actually run it in the browser at no cost

elephant detection,

I think you, Wes, too.

via I was trying to run

directly in the browser.

adapted many of the popular models to being able to run-in the browser. I was I was really surprised, like

It's not, oh, we need, like, a browser based

to Contact detection.

library called Transformers.

I'll tell you the story of how this podcast came about is I have this little label printer that I bought on

I spent an evening dipping into the web Bluetooth API, and

And he's here to talk to us today about project Fugu, which is this thing, and everyone was like, you can access I can I can update the firmware on my Bose headphones via a web browser? lower level

To

Thomas sent me a message. He's like, love seeing this, like, because that's what that's what you use. So welcome, Thomas. Thanks for coming on.

a, the web Bluetooth API is The best API for working with Bluetooth,

Yeah. So I was listening to this episode with, Pharos. He did a couple days ago weeks ago maybe.

sort of effort

and it is really cool because you can use your phone to type and it'll print labels. And I thought, That's cool. I had it for about a day, and then I go, of course, can I hack it? You know? Like like, how does this thing work? I always try to dip into that. So,

Tell us who you are and and what you do. Sure. Yeah. Thanks for having me.

full stop. I was looking at, like, the Python API for working with Bluetooth. That sucks. It's so simple with with using JavaScript.

I'll let him explain it. But, essentially, there's a lot of really cool stuff in the browser from and he has Welcome to Syntax. Today, we've got a good one on good one for you. We have Thomas Steiner on who is a developer relations engineer at Google, And he's on the Google Chrome team.

D110,

I was amazed that, the browser because, like, that would be super helpful for

Big overarching plans. Right? I guess from an outsider's perspective,

And, yeah, luckily, you were Across the globe with, blog post.

there's an API gap, as we call it. There's just no way to do, Because, yeah, as I said,

Or whatever, you name it. So, Project FUBU, is essentially this vision that you should be able to Do and realize all your app ideas on the web. And, if that means, inventing new web APIs, You should be able to realize this app idea on the web. And,

they're they always seem very ambitious.

hack, hack, hack, hack. Oh, How do how do I do that? And then, in the worst case, you hit this, oh, Mozilla to eventually ship those APIs.

whatever.

to be part of the show and, yeah, chat about project Fugu. So, yeah, thanks for having me, and, glad to be here. Yeah. You're You're welcome. Fugu is one of those things that comes up from time to time on this show along with themes things like, you know, Houdini Wes we're we're saying, like, alright. These are these big,

then sure. We accept the challenge. We We try to do it. We try to, standardize it. And, yeah, in a nutshell, this is what project Fugu is about. That's that's awesome. And I'm gonna recommend that you go to fugudashtracker.web.app.

What is the the goal overall goal of Project Fugu? CSS, essentially, what our goal is,

I'll I'll link it up. And there's a couple of websites that Scott of detail it, but just scrolling through the list of

typically, when it comes to app ideas, it's like, yeah, listeners probably

welcoming enough of my self invitation constraints that you have on network, on device,