MIDI & Music in the Browser

so pretty cool.

Or it could be, like, a Node app that you're sending data over. It's probably the the data is limited. Right? You have three bytes.

So Oh, man. Like a normal keyboard, of what happened. Let's say the user went to click on a keyboard on a website and it played the wrong note. I could figure that out with Sanity. So check it out at sentry.io.

And then in another application, for instance, a digital audio workstation,

So I imagine somebody playing piano. And if you hit it very quickly,

and then you can use like, you could have you could probably have, like, a Node app that listens on, try hook up a a freaking keyboard or a a Roomba

MIDI messages. So if I'm playing, and recording in Ableton

the the d key, also play the g. Right?

or WebMidi. Js, I believe. And, man, what a cool library for that. And the velocity or the attack, as they call it here, that's how

Like, it's giving you the name of the the note itself because this library is taken it's like, we're gonna decode the MIDI messages into things that make sense. So if you're actually doing music with it, you'll wanna use midi. Js,

a chord into Ableton.

This is this is blowing my mind

is a data array.

I'm so inspired right Node, this music.

in Mac,

and records them right here

So if I select a c major scale, for a long time. And and, actually, since getting into web stuff, a lot of that kind of fell off for me for no reason in particular. I don't I just became less interested. But more recently, I've been getting back into some of this stuff, and the Wes Audio APIs that ESLint I had I had a buddy of mine message me on Instagram, and he was like, hey. Did did you get inspired by the Tame Impala Sanity do this? And I was like, oh, shh. You Node? Yes. I I did. That's definitely what I'm doing. I'm recreating it.

way to do this. But in

key or or whatever. I I don't, you know, play it often. My hands aren't used to that. Now I can just be like, hey. Give me the the fourth chord in a a d d major scale MIDI controller that's probably, like, $60. You can get them for super cheap. I got this. It's a simple little MIDI slider off of Etsy for, like, $20.30 bucks. You can get all kinds of cool stuff, out there to control MIDI and use that just for for fun projects. Who cares? Build some art. You can just make art.

Yeah.

the MIDI API But, hey, if you're whether that is Samsung MIDI keyboard,

And now it's playing So, yeah, a Scott of neat stuff going on here. Overall, I I think there's a lot of just opportunity here, and I'm I'm gonna take this a little bit further and try to build something like Truly that I can mess around with. I I even thought about building the arpeggiate arpeggiator directly in the browser, and then I was thinking about it. I was like, man, Ableton does that way better than me trying to write my own arpeggiator because then I gotta write a a metronome and a clock, to keep it all, like, in sync and stuff. Mhmm. I don't wanna get into any of that. So that's WebMidi.

that as data.

You have what type it is. It's a MIDI message.

minor seven, diminished.

up? Oh, I'm sorry. This is not done with the library. This is done with the the web the MIDI API itself. So my understanding even through a library. Yeah. This is just the device itself sending that information to the the browser.

which is, like, custom keyboard events.

a chord within that scale.

I can change which key it is? share my screen here if you're watching on video. And for those of you who aren't watching, we we'll just basically tell you a little bit about what's coming in here. So, basically, what I'm doing in this example here is I'm getting access to my MIDI inputs, and then I'm setting a function to the on MIDI message.

if I play button on this thing I had used the Web Audio API to do audio synthesis. I've done media recorder.

for on, like, a local MIDI that that MIDI source that you just created. So you're sending it from the browser over MIDI, and then you can pipe it into could, obviously, could be a music app like you're doing here, but it could be, like, BetterTouchTool,

You're just messing around with it. Yeah.

Inside of that information,

what if I did use this library called WebMIDI Are you doing stuff with, like, pushing visuals to an LED panel? That's what I'm working on right now. So I've used WLED receives those messages

figuring all this stuff out. It's that's really cool. Yeah. It's it's neat. You Node, I think the coolest thing about it for me is that now I'm able to build, like, a little exploration engine where I can, like, I wanna say I wanna build a a song in d minor, and then I don't have to, like, memorize my hand positions because I I took piano. I I was a music major. I know how to play all the scales. But, like, sometimes I'm just like, oh, what was that scale again? Oh,

which JS,

So let's start the recording. I hit the a key.

things. You know? Like, there's web development, but then there's also music, but then there's also math,

And by doing that, that now shows up in browser as a as a place I can send the MIDI. So I'm sending this MIDI to this bus.

the second row of my keyboard, it's always going to give me, like,

if you wanna see this in action, I'll actually, analog audio and the better it's going to sound. But, also, the the larger this this audio form is gonna be, the more data it is. Right? Because you're sending way more data every single time you have smaller chunks.

Ableton is spreading the notes out.

to me because it it's like one of these overlapping use to program

and you open up MIDI Studio.

is that it doesn't just allow you to receive MIDI because that's the way I always perceived it. Right? You have MIDI devices. The web browser can accept those MIDI messages. You can do something with it. But the browser can also output MIDI messages, which I found to be really super neat. So for outputting MIDI messages, whatever

you know, I I'm in a weird, and that's a seven bit. And by seven bits, it goes zero to one twenty seven.

to get data

But

a way

in and out of the browser. It's it's almost like a WebSocket, isn't it? Yes. Yeah. You you could use MIDI as a way to pipe data out of the browser,

and that's sending three messages,

That's cool. How does it know what the the keyboard is? I I'm assuming the the hardware sends along, like, an identifier byte, and then the library looks that fast you hit the key.

every chord I play automatically

and you can use it here. So for instance, I built this keyboard synth. So if you're watching along, you can see this. Otherwise, you'll just be able to hear it. I built a keyboard synth and allows me to do programmatic music. So what's cool about this this synth here that I built is it's just sending MIDI messages. Right? But these MIDI messages are actually able to do a lot more than just, like, a straight keyboard.

you play the Node.

And let's change this to locked chord mode.

So now

And then I have this I a c driver bus. So you can do this in Windows or Mac. I don't know the Windows, for those of you watching, I have some code on the screen. For those of you listening, I'll describe it. You create a new audio context with window dot audio context,

And you can change major or minor.

and even playing a different audio sample based on that velocity. Like, you have a snare drum, and that velocity, if it's below, you know, point five, it's like a little a recording of a tap of a snare drum. And then you hit it more. It actually plays a different audio file, Yeah. Well, let me tell you. I even did a, theremin, which you don't know. Theremin is is used by, like, radio waves, I believe. And you put your hand into it, and it detects where your hand JS. You go.

This is the hard part because you have the the browser active.

the attack is the velocity,

what what I'm calling a locked chord view.

Also, in the normal chord view, You hit start, and then you can stop it. So I I have this code that you might be looking at. I'm just gonna hit enter on this. And

just by hitting a two key. And then I hit the three key, it makes it a seven, what you can do with MIDI in the browser. So first and foremost, MIDI stands for music instrument digital interface.

Yeah. I lightly Wes it, and it's giving me a zero a point zero zero seven eight. So, yeah, if you're playing lightly or you wanna you you can do all kinds of stuff. And, you know, it's a more advanced digital audio ways of working with MIDI Wes is that they're actually checking that velocity or vice versa.

is going to remain in the c major scale because I'm I'm doing that math wise. And so I added some some keyboard controls of this where I can,

the release, the octave. That's cool. Yeah. So it gives you more information.

a major seven,

and the input message itself comes in as message data.

or generate synth. You're generating synthesizers or something like that. But, typically, you know, if I hit this and I'm doing a drum kit, it's usually just a waveform And that gives you the command, the Node, and velocity inside of the message data.

And then the moment you release the key, it sends another message, which is note off. And, music, generate songs, like, you're sending audio to somebody VST inside of, Ableton or whatever I have is gonna sound way better than anything that I could make myself with, you know, the web audio API. So either way, I'll make that available. We'll have a link in the show notes if you wanna check it out. That code will be open source too if you wanna see exactly how I did it. As always, let's get into sick picks and shameless plugs, Wes. What do you got for sick picks and the shameless plugs? JS is well worth using instead of potentially trying to do it raw yourself because, man, the dealing with the the raw information is kind of tough, especially if you're doing music stuff with this. So with this library, I'm able to do a more standard, like, add event listener. So I'm listening for a note on and note off and control change, these types of events. And this time, when I hit an, key, I get a lot of information with each Node, like,

for instance, while I'm hitting this key down, I I'm saying, oh, play the a key, but also play,

you can make any audio you want in in the browser with the the Web Audio API. And I I went into the

on

play the audio, which is

A lot of neat stuff here.

that would allow you to bypass these, like, phone companies and whatnot. I thought it was kinda interesting. So I, like, I looked into it, and I was like,

is because when you have an m p three or you have a digital audio file, kind of inherently what it's doing is it's sampling the audio.

that's not a lot of overhead at all. I was thinking about doing that same thing. So that's interesting to know that somebody else would want that.

a little bit younger,

a sound using a, in the browser. Maybe turn on my maybe that will help. So, basically, Okay. I can't hear that, so I don't know what it sounds like in my monitor right now. But I'll I'll be definitely show showing you some of this in just a little bit. I'm building something fun with this, so I thought it'd be a good opportunity

so you have to do it. There's a whole maybe we should do a little show on that, like, on what makes you allowed to autoplay, because some some websites can. And there's, like, a algorithm that goes into whether you're allowed to autoplay audio or not. It's honestly kind of a mysterious algorithm.

Safari lets you Yeah. There's lots of projects out there for home automation. There's lots of projects out there for, of you gotta, like, wait for somebody to, like, click something. And the first time they click, you gotta set it up.

Mhmm. And and now

like, load it up, play. Because the thing about the audio API and playing audio in the browser is you have to Actually, this is one that I'm playing with a little bit of these, awful.

to

sometimes it's

And then you you can play it later on. So Why is that mess? Nice. Do you wanna explain to the audience why that is? Well, you can't just autoplay audio, to simply just write, it in, I don't know, maybe

once you're done, once you stop the recording, I'm very interested in this WLED.

a dialer, it actually plays two frequencies at once. And every single number has a different frequency, and that's how that data can be sent over the phone. And I just, like I I reimplemented

you could be, input from MIDI, but you can do a lot with these things. And there's a handful of really great community libraries out there to help you make both music, but also just control things with MIDI messages. So we're gonna be talking in this episode

You know what doesn't need a gesture is the MIDI API. So that means, like, you hit a key, and it would still register the MIDI ESLint. And that what that considers it a gesture, so maybe you could start audio synthesis then. I don't know. I haven't even tried that. Yeah. You have to you have to wait for the first Scott,

Mhmm. So because MIDI is

audio directly in the browser.

From Tame Impala. There's a band, Tame Impala. They re they created their own idea generator synth. I don't even know what they're calling it. Regardless, it's sold out. They had, like, a limited number of them. And it does exactly what mine does, where you can hold one button to do keys and hold another to modulate way that I solve this. I projected color beams of different color onto the wall. So let's say there's, like, six different colors, red, whatever whatever.

then you create a new oscillator with audio context dot create oscillator. So that creates essentially your oscillator. You set what type it is. Is it a sine wave? So you say sine. Then you can set the frequency, and you set a value at a current time. And this is a straight up hertz number. So, like, four forty is a four. It's the first a key or it's like the middle a key. So you go to middle c and you move over to a. That's a four forty.

probably about ten years ago, Firefox launched a operating system that was meant to run on your

And I was just debugging it a second, and it says, like, an an audio context is not allowed to start Yeah.

WebRTC

dev behind Domino, d m n o, doing, like, some, like, secret stuff. Mhmm. I just randomly found a project

Could it be, like, the accelerometer? is going somewhere else. Maybe it's a drum. Maybe it's modulating something. Maybe it's changing.

MIDI in the browser, multiplayer, or something like that could be really super interesting. I'm gonna check this project out. That looks really super neat. I should have actually looked before we recorded this. So that was just like, oh, I wanna make that. No. Sometimes you have an idea, and it's like, I don't even wanna

a record, like a a vinyl record, and you're playing that record. That is actually playing a straight JS, and it's super good. I'm gonna tell you that right now. If you're doing anything with this stuff, the WebMIDI a sine wave directly in your browser with, like, no effort. And this can be used to do a lot of stuff, to create full synthesizers. That was one of my first projects Wes to create a a keyboard that played audio synth, at that I don't know if you could hear that. It was super loud in my headphones. No. Blew my ear out. Either way, it played a sound tone. We'll have this code in the show notes. You can copy and paste it. Throw it in your console. Just hit enter, and you'll get a sine wave. And you can see really quickly what in, like, six, seven lines of Node, you're able to generate tight or wide

that are needed to play. Because when you press a button

Node.

Like like, even another use case, if you're not into music, sometimes in your app, you need to just play a little sound. Yeah. Like, for example, if something finishes

Because it has to be attached to a gesture. Right? We just talked about that you have to click first. But guess what is considered a gesture on

twenty, thirty lines of code,

phone. Yeah. And it was written entirely in JavaScript. And

play your MIDI

Where the difference between analog and digital, and it's some of the reason why people say they like their records better than their m p threes,

and I I turned it off, and then I turned it back on again. And then that was enough to consider it a gesture.

what, seven lines of JavaScript to create that little boop and play it Vercel having to load a whole wave element on websites Yeah. So one one little demo I built a while ago when I was looking into the the Web Audio API was

it was I was at the launch of it, and they gave us they gave us, like, a Android phone that had been flashed with Firefox Bos. And it was really interesting. And what was cool was that you can you still can. You can go into the code Bos for Firefox OS, and

I I couldn't figure out. I'm like, it was working. Now it's not. And so I turned you just I had my my mute button off, implementation. But, clearly, flicking the mute button on the side of an iPhone 13 Pro Max Yeah.

So that's a a big reason why people say digital audio doesn't feel as good as Not just analog audio because analog audio is not losing any information, where digital, you have to chop it up. So, like,

if you zoom in close enough, it's always going to be stepped. It's always gonna be chopped instead of a smooth continuous waveform.

watching a documentary on phone freaking Yeah. Phone freaking. How how, latch onto a click event, before you can actually play audio.

and then,

by him, which was That's this is this is actually fascinating

it's taking just a single chunk of that waveform. So you could imagine, like, a smooth waveform suddenly becomes like stepped blocks. Right? So it tells you, hey. This is coming in from a Samsung

So it turns out that

And I went to look at it. I was like, oh, that sounds neat. I would love to own that. And then it there's like, oh, we're we had a limited amount, and it's sold out. I'm like, you know what? I'm a build in JavaScript. So you can modulate them. So I have a second row where I hit a a chord, and then I can make it minor But you could. Yeah. And what's cool about it is you can augment it. So, I'm impressed with the mouse, but I'm equally impressed with the software that this thing came with. With. So, shout out to Logitech for making something that's you can then create is we're using these to control and play waveforms continuous

Kevin Mitnick. How he But also Steve Wozniak did a bunch of phone freaking. Yeah. You know? Yeah. They basically they blew a whistle they got in Captain Crunch, and that whistle made the same frequency

waveform. Right? That's playing just straight up why why it's called analog audio. Right? It's playing just a straight up waveform.

because it's it's the same with video. Right? Yeah. Auto playing, those can be extremely annoying,

arpeggios inside of a chord or or strum them or do all kinds of stuff.

and then setting the the frequency of them Wes you press down. And then when you let go, you you turn them off. Yeah. So I don't know if you'll be able to hear this because it's

all of those sounds are and I I initially stumbled upon this when I was,

is enough of a gesture to allow it.

but with friends.

I I don't wanna see how they did it because I I love the process of trying to figure out how to do it myself. Yeah. That that's actually, like, a lot of what inspired this keyboard that I made for myself because I got an ad. I got served an ad for a synth created by or a MIDI controller. I don't even know if it has synthesis on it. I think it's a MIDI controller.

Yeah. I know. I was thinking about, like, how you could have a multiplayer Because you'll see here in my my other example Yeah. Ableton

easier

and using webMIDI.j

One more one more neat fact. Sorry. I keep just interjecting on the show. Node. Neat facts. I love it. But this is my app where I built the the dialer.

whether that's, like, hitting enter or I'm curious what the gestures are considered now. Yeah.

or play, It's the way that electronic music instruments MIDI Studio gives you access to, like, an interface. You click on IAC driver. You enable that thing. And then what that does is it essentially sets up a MIDI bus on your computer that you can send audio to. So it creates a MIDI device on your computer.

So every x amount of milliseconds,

and you want it to send less data, what do you do? You lower the sampling rate. The the audio JS gonna sound tinier. It's gonna sound more hollow. It's losing some of that information, about some really interesting audio technical, your device just by giving the browser access to it. I see. I see. The so the browser itself has a list of similar to how you can get all webcams and microphones, you can also get all MIDI devices. Okay. Yeah. So I have all my MIDI devices, all all that stuff. Either way, that's what type of message it's sending. Now in my more advanced Yeah. So you could modulate it. You can do all kinds of stuff with math just by changing the midi notes on the input. And then furthermore, you can use things like Ableton Live to Augmentative for further, where I'm piping in an arpeggiator.

just like WebMIDI JS handles a lot of the nitty gritty stuff. This really just makes it super easy.

just based on whatever, you know, key I'm hitting on the keyboard, similar to what the project I'm showing you in this this, podcast, but, like, for generating synths instead of sending MIDI somewhere else. And that's really great. I think it's really neat. There's actually a really great library that can take care of this as well called Node JS.

Yeah. I I guess that gets us into the last little bit here, which would be the media recorder. Now we're not gonna spend a ton of time, but media recorder, as far as Wes Audio APIs go, is really great for a lot of reasons. Like, potentially, Sanity it two months for free. This software allows you to keep track of all of the errors, bugs, issues, anything that's happening in your site,

s. So being able to

creating a dialer,

more close it is to

look if somebody's done this because I'm so excited about

learning and and doing it JS I don't wanna be, like, someone already did that, or, things,

You can also, if you break it down into actual binary, you go to audio MIDI setup.

MIDI workstation or even you're recording

Mhmm. And instead of sending audio to friends, you're sending Yes. MIDI data, and then you're just reproducing it on the person's machine. So, like, that would be

And the amount of stepped blocks, how Sanity you chop that chop that up we're we're, you know, we're chopping carrots here. How finally you chop that up is what's called the sampling rate. So the higher the sampling rate, the

code Bos. So for those who are, that has Bluetooth and Wi Fi built right into it.

all the, like, native apps. And one of the native apps was the dialer, who was the guy that did Kevin Mitnick.

So yeah. No. Super interesting.

I'm not. The mute button on the side.

but it's going to be much smaller. Right? So that's one of the reasons why, you know, you're when you record or or we do this stuff, we wanna record locally,

Safari, on iOS?

a or a Max MSP, I was able to overlay the two. So I had what the video was showing of the projector plus what it should look like. Mhmm. And I was able to do a if one of these color beams suddenly has a a difference like a snap. You know? Because oftentimes, what we're doing with MIDI

who know what MIDI is and understand MIDI might not understand the waveform is. And if it's a sine wave, it's gonna sound one way. If it's a square wave, it's gonna sound a more more buzzy, sawtooth, those types of things. There's all different types of waveforms you can do for that. But you can generate audio straight up in the browser using an oscillator, and I'll show you an example of that in just a second.

I've done kind of, like, things here and there, but I haven't really dove into it until, your browser, just like the inputs, has a set of default or your your computer has a set of MIDI outputs.

generated unique songs every

Yeah. So what that on Bebop, I believe it's the vertical if you're holding it sideways, the vertical is the pitch or it's the the

physical hardware triggers to trigger music. I I built a a synthesizer out of a projector, actually, Wes. That was, like, one of my coolest projects. What I did is I projected this is actually a really technical, like, interesting in large in size based on how hard I hit the drum. That sounds like a ton of fun to make some of this stuff. So if you have any MIDI devices around, you can experiment with these. Otherwise, the audio synthesis API I have this, like,

the tone or something. One of them is the pitch and one of them the tone. It's kind of like a a theremin in in in an axis of of how you're doing that. Yeah. I have a lot of experience with this stuff. I I did a lot of music programming when I was in college. In fact, I took several years of a music programming class where I was doing, like, complex Yarn off chains to generate, many, many years working in computing on computers, I have finally given in and gotten a mouse. That's right. Alright. I never work with mice ever. I I'm a trackpad guy. You Node? I'm super trackpaddy.

of enough threshold, This message, this function just receives a message. You can see that anytime I hit a key and I know if you're just listening to this, we'll describe it, but you're getting a it's a straight up JavaScript event that gives you some information.

somewhat recently here. So we can start off first and foremost. Let's talk about the WebMIDI API because I think the WebMIDI API

and using, oh, I'm just pushing pixels on a website. No. This is like, I'm creating music. I'm making something that has actual sound. I'm I'm, you know, something that I can show my kids. Hey. Push this key. How does this sound? Or now push this minor note. How does that sound? Like, I could use these as educational tools.

time. We were doing things using,

And I projected that, and I had that image saved of what that should look like. And then I had a video camera pointed

figure out you probably just use the x y coordinates of the person's hand, or you could you could overlay it if something is breaking up

than fire a MIDI note. And that was sent to a controller to play a sound at a certain tone. So you could run through this thing, and it would sound like you're dragging your hand through a keyboard. It was probably one of the cooler projects that I put together. So That would be a fun Canvas project if anyone's looking for something JS take a video and then

dimensions of being able to tweak both the the audio, the quality, the timbre, all that sort of stuff. So I was able to make noises with my webcam just like that. So long story short, I've been I've been experimenting with just tech and music, the intersection, The next three are telling it if it's a note on or a note off. Like, you're either pressing it or you're telling it to stop. Right? Mhmm. And then the next one next four are just simply what channel it's on. So that's the status because you can send many messages on many different channels and receive them on multiple different channels as well. That allows you to What is a channel? And then you connect it to a destination like your speakers.

a specific color.

a million different combinations. Scott a million different combinations, but whatever,

content with that. But the more and more I got into actually,

I hooked up, like, a serial port to it. There's three different, like, motors on it. There's a spinner motor on the side. There's the actual vacuum motor that sucks, and then there's the main brushes underneath that spin. And then also two of those brushes can spin in reverse.

So a channel is exactly kind of like what you think about it. It's kind of like a a pipe. So if I'm sending this message down pipe number Node, in my controller or wherever I'm receiving that message, I can look in pipe one, and I can know that this is coming for this specific thing, I e, like, you have this thing. Right? It has drum pads, and it has a keyboard. I can put this on one channel, and when I send a message here, it could send that in channel one. And then the MIDI receiver can listen and say, anything that's on channel one, that's a keyboard. And then I could send these on channel two. Anything that's on channel two,

It it is funny because I was like, I I mentioned I've been doing quite a bit with this. And at first, I was trying to do everything

So this is the note. So when you're playing and I hit middle c,

where you want the motors to be on, right, because there's five different things, there's

Oh, okay. I see. I see. Yeah. Yeah. Yeah. It's like just putting things in different you go here, you go here, you go here type of way. So the second byte is a data byte,

playing notes and playing scales and wanting to do, like, more real, audio stuff with it, I I realized, like, oh, man. I I really don't wanna be getting into that. I want what I wanna be doing is just worrying more about the audio stuff. So, actually, I'll I'll share some libraries in a little bit that that help with that sort of thing because there is, like, music based MIDI libraries or somebody's already figured it all out. You can say play this Node, and it will just play rather than having to worry about the binary. But, yeah, I I got into it quite a bit. So, basically, what happens, like, you play a keyboard, it sends a note on that note in the velocity.

Can I tell you how hilarious this is that we both got into writing binary,

And in order to send that combination, you send it over in in binary. Right? And I thought I thought it was so hilarious that, like, I have never actually written raw

within the last seven days of our entire career? So I recently did a video on controlling my Roomba with JavaScript.

the permutation of that is.

binary or or bits and bytes like that before. Yeah. And and here we are doing it in the same week.

And in order to send the kind of state of of

Yeah. Totally. Right? So cool. Let me know what you think. If you're building anything like this, let us know. I will make my synth available or my my midi controller my in browser midi controller available. But, just to let you know, it does not generate sound on its own. You need to pipe it into a digital audio workstation, at least right now. I don't feel like I don't feel like doing audio synthesis ESLint is, like, what I want to use it for because it's pretty amazing. Like, I can essentially record audio directly in the browser, save it as a a file, and then download that file directly, entirely in the browser without any additional software just using the media recorder API. So,

dream. I I just have been going nuts with customizing this thing. So I know I'm super late. Stuff. Yeah. Very late to the Sanity. GitHub Vercel. They're, like like, half going after, like, the gamer market, but but, like, as soon as I ordered this thing, get the best sampling rate instead of whatever we have to send over the network for Wes here,

Like, you don't even have to download, like, an IDE or, like, install Python or or any of that. You literally visit a website, user input,

Awesome. That was

these Node. And it it wasn't working earlier.

and get into electronics

a sign that says on air, and I just gutted it and put a ESP 32 and some LED strips in it, and I can control that however I want. But now I'm yeah. Now I'm dipping into these

every single button you could hold and move the mouse in a direction, and it would trigger a different piece of functionality. So I have, like You can do it per app as well. Per app as well. I I have it. So, like, if I hold essentially, the there's the back and forward buttons are typically what they are. I have my forward button. If I hold that and hit to the right, it does paste. If I hit to the left, it does copy. And I've been just copying and pasting that way, and it feels so natural. It's crazy how natural it feels once you get used to all of, like, your own predefined setup. So, yeah, I'm impressed it's a crappy MIDI keyboard.

which is unbelievable.

or just curious about any of this stuff, spend some time just diving into the MDN docs. We'll have a link in the show notes or fire up Node. Js or Wes MIDI. Js We're Costco. My trackpad. Yeah. We're Costco. But you know what? When I broke my finger, the, with the bits and bytes. I was trying to, like, really get into the raw MIDI. I was like, yeah. I'm just gonna use MIDI straight up, and I'm gonna be very, like, But you can do some math on that. So I built this, the trackpad was giving me all kinds of RSI. I'm like, now's the time. I'm gonna get a mouse. I got the Logitech MX Master, And by default, it could be any number one of these things. You might not have any, to be honest, if you don't have any of these things set up. So for instance, I have my audio converter, which is my digital audio interface. That shows up here. I have the keyboard. It actually shows up as a MIDI output as well.

plug it into USB,

I always recommend people to do creative coding and go outside of your your regular button clicking

starred projects in JavaScript in '24.

or any of that cool stuff that you can do there. So a lot of interesting things you can do. If you're a music person

several other ways.

Sweet. It was very good video. Alright. Thanks everybody so much for tuning in, and we will catch you later.

So well worth your time.

every single gamer I I did a TikTok on it, and every single gamer comes out of the woodwork being, like, oh, it's not fast enough refresh rate.

Man. Cool. Man, you do the cool stuff sometimes.

Wow. Yeah. That's interesting, man. I'm fascinated about that stuff. Let's, I got a a midi drum set. I'm gonna go downstairs. I'm gonna hook the midi drum set in, and I'm gonna have and then oh, shoot.

But these things are extremely fun to work on. They're a little microcontroller

There we go. That is

this thing's a tinkerer's

So much cool stuff here. And that's not even getting into, like, WebRTC So I have it set up. Anytime you play the a key, it plays an a chord,

WLED is is one that runs on it. And Oh, what? It's just a it's a really fun thing. Like, if you're trying to, like, tinker

Yeah. You're allowed to just make art.

Three s. Yeah. That's what I have as well.

It's really fascinating because it's not just like, here's a list of the top 15 most starred projects.

I learned a lot from it, so shout out to CJ for putting that together.

arrays right now.

LED, my on air And, like,

APIs are sometimes. Right? It's just one or two little APIs in the browser, and it's just it goes all the way down, and it has such history as well.

from the browser, which is it's using the Wes Serial API. It's unbelievable.

in the description of the show.

Mhmm.

for the mirror in our gym. So when I built the mirror in our gym, I put LEDs behind the entire thing, and then it does it can do, like, cool effects and, like, kinda backlight the mirror. And then with your phone. So, you know, when you when you hit the numbers on a phone, it goes.

in the browser.

we built a website where you could just draw on the website and it would, in real time, paint to the LEDs? Yeah. So that's my idea, at least. There it's a lot of LEDs, though. So, like, the powering them is And it it makes a lot of sense because you can you can store a lot more information No. Play us in. Play us in. Is that

in the last Yarn. And it gives you a lot of context to where we are today

expressive or or creative. Or, you could play And Yeah. That's really cool. I made one of those with my webcam, example, and how long and and how high they should play them. Right? Correct. And there's all kinds of things that people can use MIDI for beyond and the WebMIDI

elements on my page are really fantastic for a number of reasons.

and and, like, highlighting text. And it's it's a totally different

three s.

than they do, like,

to respond to gestures.

awesome. I I can't believe how much, like, how much of a well LED.

You can power them via USB. You can power them via

I have after,

It actually breaks down, like, why. It gets into what they did and why they had the year they had or why people cared about it this specifically bring in sound via WAV files. You can bring in sound via the WebRTC You go to window,

Man, what a a banger of a video. So check that out. We'll link it available Play this note, c four. You know? You Node, play it now. Start out. Like, it's way a nicer API than having to get into some of the nitty gritty of this all, but definitely a a cool thing to mess around with if you're interested in actually generating audio synthesis.

to make fun things rather than, It's all kinds of stuff that you have the array buffer here.

putting buttons on screens and clicking them and updating databases,

and it flashes it

So very interesting.

they care more about ergonomics and productivity literally dialing stuff. So I I went into it, and they have an array of the frequencies

what, play this note, play it at this velocity,

gotta dip into that. Yeah.

my Roomba.

is a bit of a thing I gotta figure out yet because it's it's not just, like it's it's a lot of power that these things take. If you have them all at full Cook, you need, like, like, a massive power supply or or several of them. So I gotta

let me update or give me an update on that once you're you're cooking a little bit more. I'm I'm super interested. Yeah. You should just just order some order a couple of these and, at least, like, get it running so you can kinda get in the headspace of it because it's I bet you can probably get it up and running in, like, twenty three minutes. You flash

in 2025.

sick pick a little microcontroller. This is the one that I

LED arrays That's what I want right there. I'm trying to figure I'm, like, how sick would that be if That's a browser level

it will ignite not to say that I'm burnt out. A lot of people thought that on my video. But,

and experiment with this stuff because this is the type of thing that when I work on it gets me super inspired You can come in here and see exactly, like, what exactly it's coming in from. So for instance, we the device itself JS, this is a Samsung and I did video tracking. So I would hold up a little color square and where the dim where I put it and how big the color square got, I had, like, a three d, it sends a note on, which is zero x 90. And then the second byte is going to be 60 for middle c because it's zero through one twenty seven. And then the last one is the velocity, zero to one twenty seven. And that's also a seven bit byte.

because it makes you so excited about web development in general. Yeah. Imagine this. Right? Here's some, here's some freebie project ideas. Like, we we talked about sending and receiving messages or data or audio or stuff like that or generating. You can use that information in all sorts of cool ways. Like, what if you took the mini message, I hit a specific note, and that note controls the background color on the website. That's a fun little project right there. I hit this key. It changes to this color. And you could just, what, clamp the the notes on a scale to the range of RGB colors and, you know, have some fun really just doing something Tone JS is super good. It handles, like of a drum being triggered in the background. So the MIDI message JS, like, play this note, whatever. It's like, fire this MIDI or fire this waveform. Yeah. That's cool.

Awesome. So my sick pick is going to be just building a simple recorder where it's connecting to your audio source, your microphone, you're talking into it. You can record that. You basically, you create a a new recorder you can check to find all MIDI input and output devices on your in general, when you're working especially with, like, a keyboard, it's gonna be three bytes. The status byte is eight bits, and then the data bits our data bytes are seven bits each. So the first message, the first number that comes in is an eight

That's really fun. And, the microcontroller

extendable and, like, customizable out of the box. If you're a tinkerer, API, what these things are, why they matter, and maybe some interesting things about them that you might not know. Because me, personally, when I think of Wes audio and web MIDI, I'm thinking of maybe playing sounds or, accepting

If you're feeling at the very least burnt out on coding,

Yeah. I'm going to,

actually media stream.

all this, like I I find it lags quite a bit when I'm killing noobs in my game. And it's, like, I'm not gaming with this thing. I'm clicking on buttons

They're $5, Pretty good. Pretty good. Excited to talk about Web Audio. I've

for, like, probably for, like, $10, 5 dollars for the controller, $5 for some wires and whatnot, you can start building some some pretty fun stuff with it.

So

world when you're, like, targeting, like, web developers who don't necessarily

to your browser.

crappy little concepts such as sampling. We're gonna be talking about MIDI in general, the types of messages that are sent, how it's sent, what it means, how we can interact with that in the browser. And then, ultimately, I'll be showing you some really neat demos of stuff we can do. In fact, my desk Play us in, Scott, will you?

You you get the on and off, but you don't get velocity. You know? You don't get how hard you pushed it, which that's a lot of interesting information. Right?

and I use Ableton Live a lot, you can then see the one that you created it would sound different than if it was you pressed it slowly. Right? So, yeah, I just smacked it, and it gave me a one. Now I'm gonna lightly press it. As fast as possible.

unlike Node audio form, like, when you send a waveform, that is Yeah. Audio.

BetterTouchTool, with a description.

Yeah. Because of of JavaScript. It's just creating two oscillators, Man, now now you have me going down and you need to play, like, a little boop or or something is incorrect, like, if somebody tabs away and you need to play an audio alert that something happened,

And those MIDI messages,

bit byte of the status. That status typically means Yes. That's true. Yeah. What's interesting is that the browser also takes care of, like, sampling audio. So those of you who don't know too much about how digital audio works. So let's say you have Shout out. Shameless plugs. Head on over to YouTube. CJ released a a crazy good video on the most

And when you have a MIDI device, it is sending messages.

It's sending messages to whatever the device is and the device, the digital audio workstation, you know, you have Logic, Ableton, those types of things Yeah. Receive those MIDI messages.

can, So if you ever hear sampling rate, that's what that's doing. You can control sampling rate in the browser in an audio buffer. So if you're sending audio over, you know, what, like, over WebRTC, you know, automatically the first digit is

Those are all sent via MIDI. Or,

notes. Like, I have a MIDI keyboard here. And Yeah. You can do, like, pitch bend and modulation, which is which is little touchpads.

So it's just a description

in a

When you send a MIDI message, it's just and you pass in data into chunks. And then the S 3 3 s Yeah. So MIDI, really neat. Right? You can do so much with it. What I did not realize about man, so much cool stuff in this these APIs that I feel like are fairly underutilized,

It's a few messages. And, typically, those messages will look like, is crazy good. Yeah. It's actually pretty good. You think you gotta get into, you know, some third party to do some crazy stuff, but I didn't realize this, Wes. You can set every single the media one, the, audio API.

the notes you can see how they implemented

interface with devices, computers, those types of things.

or, you know, note on this note, telling MIDI that it's a status byte.

data. It and it's not even data of how it sounds.

even, like, being able to change volume with these knobs, that that message is sent via MIDI as well. But MIDI can send you know, it's more than just, like, you think about, like, a key press on your keyboard.

if you're, like, a midi expert out there,

this velocity.

An oscillator is kind of what it sounds like. It creates a waveform. Right? a MIDI payload, I guess it's called. You're gonna go into it's three bytes. Right? Yes. So most MIDI messages in the typical sense you're playing a keyboard are three bytes. It it can also be two or one depending. It can be more than that, I've heard. I don't really,

ringtones

one of the many cool benefits of MIDI. Let's talk about the Wes Audio API though because this is the Web MIDI API. The Web audio API can be used for a ton of interesting stuff too. For instance, you can generate sounds using an oscillator.

MIDI. Right? They they were. Well, yeah. They were. Yeah. Yeah. Twenty years ago. But refresh rate or something like that. Neat stuff. Logitech.

how to play them, and then the phone itself is the one that actually has to turn it into to music, right, which is why it sounded so

are Firefox OS

is how big the waveform is, and the pitch is how,

I always wondered about them. Like, how how come you could just download ringtones back in the day? And it was because it was just a file It's just math. But Yeah. Well, no. It that's you know what? Wes, sometimes math is mysterious.

So you're creating, like, a sine wave, and then you're controlling the Hertz. What is that? The frequency? You're controlling the frequency of the the hertz, and that changes the pitch. So

downloading a ringtone like the keyboard shortcut and everything tool that I use, they have full support for a MIDI controller. And I always thought, like, oh, that's kinda interesting. People are hooking up keyboard shortcuts to their piano.

the velocity

Mhmm. Because

so funny. Yeah. Yeah. It uses the same audio engine that it they use it when you push any of the keys. Like, beep boop boop boop. It's using that same audio engine to to recreate the songs. That's MIDI, man. It's it's so cool because it's small. MIDI is really small where audio and waveforms are really big. So it's very transportable. It's fast to send. It's fast to do anything with. So, is is fairly underrated amongst folks. In fact, even people

So it was like, a, like, a one KB file with TypeScript of

Wes, my man, what's happening today, man? How are you feeling? chords? And I gotta say, the coolest thing about this mouse, besides the fact that it's just nice and comfortable and all that, the software that Logitech made for this

Yeah. Sounds awesome.

to take a time and do an explainer.

Play us in. Okay.

fix performance issues, find out what went wrong. You even get a session replay so you can see a video

dabbled in it, I don't know, three or four times. Yeah. There's this, app called BeBot, which is on the iPhone. I've had it for probably, like, fifteen, Node, fifteen Yarn. And it's generated sound, and it it goes up and down the scales. And I don't really know a whole lot about music and how all of that works, so I I went and built that at one point. Oh, did you? Cool.

Welcome to Syntax. On this episode, we're gonna be talking about the Wes Audio API Beautiful.

Each small CSS and JavaScript, and it was kind of a fun learning experience.

trying out new things in your site, maybe you're pushing it live, and maybe that new thing is broken. Maybe it takes down your site. Maybe you're getting a ton of issues on one specific thing, or perhaps I think that I have covered all the use cases that I thought I should cover in this, and it's not working. So you're gonna want to use software like Sanity at century.i0/syntax,