While my guitar gently speaks — Todd Fisher, Philo Ventures

AI Engineer · 18 min · 260 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12So, I am Todd Fischer.
00:13I love the guitar.
00:14That's one of my passions in life.
00:16Uh, today I want to talk about uh this project I've been working on for a while here, effectively making my guitar speak.
00:23Uh, but of course I want to start out with kind of framing it under this awesome premise.
00:28You know, we've all been to live performances where like your mind was just blown.
00:31It was awesome.
00:32You know, the the first one I remember way back I was in high school.
00:36I went to a Slipnot concert, so a little bit heavier music uh here in the Bay Area.
00:41And I remember at some point the drummer was drumming a cool drum solo, and his drum set started raising up.
00:46And I was like, "Whoa, that's cool."
00:47And everyone got kind of excited, right?
00:49And then halfway through the solo, his drum set started to go like this and tipping.
00:54and his whole drum set was like on the wall while he was upside down playing his drum solo and everyone was just cheering like whoa this is so crazy mind-blowing
01:02like whoa it was so so impactful right I still reme remember it today um fast forward probably about a year ago uh I went to New York to see the Stranger Things on Broadway
01:13the the first shadow and it was really cool because the whole time I was there I was watching it and it felt like it was actually like a a Netflix episode but in real life and it was just like the effects that they did there was so so amazing.
01:26I remember even one time at near the end there was a scene where somebody somebody was falling backwards and it was like a slow motion scene and it probably about a minute long the whole scene but just like the fact that like they they were able to produce that in real life but it looked like it was
01:41a post-prouction slow motion scene just just mind-blowing.
01:45So it's like well that's cool.
01:46So given you everyone here's probably had some experiences with live performances in the past and I think it's really awesome just to like see the creativity uh that people can do with leveraging technology
01:58in some creative way.
02:00And so you know I'm thinking about all the projects that we have as engineers over the years where it's like hey that's a cool idea.
02:06I'm going to write this down.
02:06I have like a big giant list.
02:07Anyone have a list of projects that maybe they'll get to?
02:10Yeah.
02:11So it all happens right?
02:12Uh, and it's really awesome because the last 6 months or so with AI, uh, being able to, you know, push us forward with some of these projects, it's just really cool to see where we can take all these projects.
02:22So, so part of, uh, my goal today is to inspire you guys to find whatever project you're passionate about, go start building it because it's super easy now with AI.
02:31I get easier with AI.
02:32I know sometimes there's caveats, but but in general, uh, I I want to have you guys leave this session being inspired to go build whatever that cool thing is for you personally.
02:42that's gonna, you know, help you to learn stuff or maybe make a difference in the things that you're you're trying to figure out.
02:48Uh, so I want to start off with framing this kind of in the realm of the guitar.
02:52So the guitar acoustic guitar has been around for several years, several thousands or hundreds of years, sorry.
02:58Um, and at some point somebody said, "Hey, I'm going to put a pickup on a guitar.
03:02I'm going to plug it into a speaker."
03:04And now we have rock and roll.
03:05We're going to make it super loud.
03:06That was pretty awesome.
03:08And then of course we have a lot of people getting these stomp boxes, these these effects pedals.
03:12We put them all together.
03:13We have really awesome sounds.
03:15You really famous people have a lot of these.
03:17Uh and then at some point Peter Peter Frampton came along and said, "Hey, what if we get the sound of the guitar, put it through an actual hose, like a physical hose, put it in my mouth, and then basically play the guitar sound in my mouth and inform some sort of words.
03:32So that's pretty awesome that that's known as the talk box.
03:35And then in the last couple decades we have a lot of progress made with software emulation.
03:39Uh you think of you know protools you think of logic of uh fruity loops there's a lot of software even to the point where all the effects are in software now there is the argument made hot take that maybe you don't need all those physical effects pedals anymore.
03:53So so there's that question right and then of course looking forward it's what is that next evolution of the guitar uh with AI in the picture.
04:03So, with that said, I want to take you back several years ago, Halloween time.
04:07I was passing out candy, bunch of trick-or-treaters.
04:09You know, it got kind of boring.
04:11So, I was like, "Hey, what if I go and actually bring my guitar out with my amp and just play?"
04:15Uh, so for the past what, 10 or years or so, I've actually been playing guitar on Halloween, passing out candy, and it's kind of fun.
04:22Uh, and then fast forward probably three years ago, I decided to dress up as Eddie Mson.
04:27Anyone know Eddie Mson?
04:28Stranger Things guitar player guy.
04:31So, pretty awesome guy.
04:32So, I dressed up as him.
04:33I was like, "This is going to be so fun.
04:34I can play the music with him."
04:36Whatever.
04:36And you kind of go the Oh, hold on.
04:42Hold on.
04:42Wait a minute.
04:45Technical difficulties.
04:49All right, there it goes.
04:51[music]
04:56So, a bunch of little Stranger Things awesomeness, right?
04:58Uh, but figured out, hey, what if there was like some more stuff I could put in this whole experience.
05:03And so I decided to go and build a little app that would actually paint the Stranger Things alphabet on my garage door just because it's fun, right?
05:11Uh, and so let me just show you a quick example of how this worked.
05:14So effectively, make sure I have the mic permission.
05:18Whenever I play a note, it would go and communicate whatever lights.
05:24And I even had it to where I could actually set custom messages such as happy Halloween, all that fun stuff.
05:30It was it was overall pretty fun just to kind of mess with that, right?
05:33So once again, you know, whatever I type in there, I could actually spell whatever and and very much a cool nod to Stranger Things.
05:42But it kind of got me thinking like what is the next evolution of this project?
05:46And I settled on this idea of like, hey, how hard would it be to make my guitar speak?
05:50It sounds easy.
05:51Maybe, maybe not.
05:53So, so today I want to share kind of my journey uh in this process and where kind of I'm at today.
05:59Uh so looking around the different tools, there's a framework out there called Juice.
06:03Really awesome.
06:03For anyone building audio software, look into Juice.
06:06It's it's pretty good.
06:07Um and then of course there's a number of plugins uh sorry plug-in formats out there uh for your your digital audio workstation.
06:15And for for those that are not aware, uh your DAW is effectively your IDE but for musicians and and music producers.
06:22And then I I started with some text to speech stuff with Piper and some built-in Apple stuff and then some other really fun uh digital signal processing.
06:30Uh so with that said, my first stab here was I want to get raw text so I could just type in whatever text I want uh push it through the text to speech
06:38uh get the audio clip and then whenever I play a note on the guitar I want to go and play that back.
06:44So let's go see how that works.
06:46So, switching over to my Logic Pro here, and this is the plugin I made.
06:52So, it's just like any other plugin in Logic where you just plop it in there.
06:56It's just, you know, chaining all the effects together.
06:59And so, this is what I came up with.
07:02Developers.
07:04So, it's playing it.
07:06Pretty awesome, right?
07:09Kind of reminds me of something, right?
07:11Start clapping.
07:12Everyone start clapping.
07:13Developers.
07:14Developers.
07:15developers, developers, developers.
07:18Awesome.
07:19Thank you.
07:19That's pretty awesome.
07:20Uh, you guys are great.
07:21Um, so I got it to where it's playing an actual audio file.
07:25That's pretty awesome.
07:26But it turns out, you know, in English or in any languages for that matter, there's more than one word.
07:30So the next kind of evolution, it's a little bit chatty.
07:34The next evolution there is let's actually go and slice it per word now.
07:39So I got to the point where it's playing now.
07:41It's going to slice per word.
07:43Look at me.
07:45I can speak.
07:47Look
07:50me.
07:51I can speak.
07:53So now it's speaking words.
07:54And that's pretty awesome, right?
07:56Uh but it turns out that uh as we speak, there's a number of challenges in how we automatically slice words.
08:03So I looked into this thing called energy gap segmentation.
08:06The general idea here is if you look in any waveform over here, we see that here's a bunch of of words that we're speaking, right?
08:14Uh the idea there is there's typically silence in between words.
08:18So let's just cut it whenever the decibb are very much close to zero, right?
08:22Uh but the issue with that is there's actually sometimes when you know as I'm speaking right now, for example, uh there's actually no silence in between some of my words.
08:30So it gets a little bit challenging to where it's not 100% foolproof, right?
08:34So beyond that, I looked into this thing called sority peak syllab syllabifier.
08:39That's a hard word to say.
08:40Effectively identifying the syllables of the audio signal and identifying that there's vowels in here.
08:46Vowels typically lead to syllables.
08:48That's kind of the idea.
08:49So I said, "Okay, let's take the senority peak, add it to the uh the energy gap, and figure out if we could just make that work automatically."
08:58So with that said, made it kind of work.
09:00So let's just play this one.
09:02Thank you for letting me Nope.
09:06Oh,
09:06here it feels so good.
09:13Get out of my target case once in a while.
09:23So, there you go.
09:23It's working pretty well.
09:25Not quite as good as I want to.
09:27So, long story short, uh I settled on just the ability to go and actually I could drag this and manually edit some of these uh uh segments in here.
09:36So, worked okay, right?
09:37Um but moving on, you know, with the spirit of evolving the thought, evolving the project, right?
09:43It's like, okay, we're we're having the AI say stuff.
09:46That's great, but what if we could actually make it sing?
09:49Let's take it to the next step because this is music.
09:51Why not, right?
09:52Uh so, looked into pitch detection.
09:54And for those that are not aware, uh, as you hear any any noise out there, there's typically multiple frequencies going on at any given time, you know, you think of when you play the the C key on the piano.
10:05Uh, there is definitely a fundamental frequency or the the kind of the the one that we identify as the note, but there's a bunch of other frequencies.
10:12So, we needed a way to go and figure out like how do we detect that fundamental frequency?
10:16So, when I press something on the guitar, how do I translate that into an actual note?
10:21Uh, and then I found this uh, yin pitch algorithm.
10:24Basically what it does is it detects the pitch.
10:27I won't get into all the details, but look it up.
10:29It's kind of a fun uh, really cool way of detecting the pitch.
10:33Uh, but effectively what I do is I play the guitar.
10:36Uh, I detect the the pitch.
10:39I I make what's effectively a pitch saw or aka a synthesized note.
10:44Uh so for those that are not familiar with how audio works on the the computer, you think of all the e electronic music, all that stuff.
10:51Uh that is basically uh a synthesized note.
10:54We have ADSR which effectively are the levers to figure out how to actually make the note sound in different ways.
11:00Uh and then so basically we get the pitch note or the syn synthesized note.
11:04Uh we then push it through uh the the voice clip effectively.
11:08So think of the talk box.
11:09We're kind of filling up the cavity of the talk of the voice that is.
11:13and we push it through a vocoder and it should sing.
11:16So that's kind of the idea, right?
11:18So with that said, let's go ahead and jam out a little bit because I have a guitar and it's fun.
11:23So get back to Logic here.
11:30Uh so I'm going to play some chords.
11:32Uh I originally was gonna uh play a song that is very much related to the talk or the the title of my talk, While My Guitar Gently Speaks.
11:41But because it's going to be posted online, I don't want to muddy up the waters with any copyright things.
11:46So, I will play some chords that may or may not sound similar to a famous song.
11:51So,
11:59[music]
12:19Hey, hey, hey.
12:31So that's the backing track.
12:32So let's go ahead and have some AI speaking on top of it.
12:41I'm here.
12:46No. Hold on.
12:48Technical difficulties.
12:49Let's try that again.
12:53Live demos always the best.
12:54I'm here at this con friends.
13:03[laughter]
13:09All right, technical difficulties again.
13:11So, sorry for that.
13:12But let's just run through it and see what happens.
13:19[music]
13:19Con friends
13:23[music]
13:23that speak.
13:25I'm the guitar that can speak.
13:30So
13:32awesome.
13:32So there you go.
13:33Some bugs to work out, but overall we are able to to say words on top of this.
13:38And for what's worth, I'm trying to mix the uh the the synthesized note with this clarity lever right here.
13:44So balance or mixing the the synthesized note with the actual voice that the AI is giving.
13:49Uh so so that worked pretty good.
13:51Um but there's also this other question, you know, given that uh when you speak it's typically conversational.
13:59What if I could take this to the next level?
14:01And what if we actually had this microphone right here where I could speak into the mic?
14:06It would then respond on the guitar.
14:09So the way that I accomplished this is speaking to the mic used speech to text.
14:13a whisper uh put that into raw text, run a local model on my computer.
14:18It doesn't really matter what LLM, just any local model.
14:21Have a conversation coming and then from that output go and plop it on the guitar.
14:26So, let's see how this works.
14:34Anyone have a question you want to ask my guitar?
14:39Anybody?
14:42What is reality?
14:43Let's do it.
14:46What is reality?
14:50So, it's thinking going through the LLM things and let's see what it says.
14:56That's quite a question to start with.
15:03[laughter]
15:04How does your music help you understand that elusive concept to real be right now?
15:18So, yeah, very existential.
15:19I like that.
15:19So, thank you for that suggestion there.
15:22So, there you go.
15:22We have a choppy version, but but it is working.
15:25So, so that's a win, right?
15:26Uh, awesome.
15:27And thanks for the claps, sir.
15:29Um, but really like it's not quite singing yet, right?
15:33So, moving on in this project, it's like how do I actually make it sing, right?
15:37I found a lot of uh open- source options out there as far as uh uh recordings or or samples if you will.
15:43There's a thing called vocal set out there.
15:45They basically recorded a bunch of singers and I could use those audio files to go and do some fun stuff with them.
15:50And effectively, I took those samples.
15:53I used a project called World, which helps with things like pitch shifting and some other things.
15:58and I'm able to then shift the the pitches of everyone singing on those audio clips and map it to the guitar.
16:05Uh, this is a very heavy process and so I can't really do that live.
16:08I have to effectively pre-bake that and then once it's already pre-baked, I could then go and jam out with it.
16:14So, with that said, uh, let me go ahead and show that example here.
16:24So, uh, because it takes so long, I actually just started with the vowel sounds.
16:27So there's somebody singing all each of the five vowels.
16:31And so we'll see how that sounds, right?
16:41Kind of fun.
16:41Kind of weird, but overall it's working.
16:43It's It's closer to singing, right?
16:46Uh so let's go ahead and throw that on top of all the chords over here, and let's see if we can make it sound decently well.
16:53Oh, so we'll just play these chords again.
16:58[groaning]
17:15So, not quite your opera singer, but getting closer and and so I I think there's something awesome there.
17:20Uh so effectively once again that is going through the whole synthesize process uh putting a sample shifting the pitch shift of the sample and then effectively mapping each fret or each note on the guitar to one of those samples that are pre-baked effectively right.
17:36So with that said you know where do I want to take this next uh very much want to get into more of of the you know AI heavier options if anyone has any other ideas of how to make the guitar sing feel free to come up to me after.
17:49I think it's pretty awesome.
17:50Uh but more importantly, you know, everyone here going back to the the charge to go in and build some awesome, you know, side project, some passion project of yours, like what is that for you?
18:00Go and build awesome things because nowadays with AI, we could build so many really cool things and time is typically not the the big time suck that that it once was, right?
18:09So, with that said, uh be awesome or be good to each other and thank you very much.
18:16[applause]