go
life is like a hurricane
[Music]
[Applause]
[Music]
yeah
when it seems they're headed for that
final contract Shady wording never fails
three main tracks the worst of messes
they call successes Texas
[Music]
[Applause]
trainer
[Music]
that's the reasons
[Music]
they're on a mission hit those targets
every day
[Music]
foreign
[Music]
thank you
[Applause]
[Music]
thank you
[Music]
[Applause]
[Music]
thank you
[Music]
foreign
[Music]
Focus
perfect temperature
[Music]
hmm
[Music]
welcome
to the Copenhagen developers Festival
which I've spent the last three months
trying really hard not to call NDC
Copenhagen because it's not I mean it
kind of is there's a lot of the stuff
here that if you've been to ndc's events
here in Copenhagen before same people
same kind of content but there's a whole
bunch of new and interesting and uh
exciting stuff going on here now
I know some of you have seen me at NDC
events before because I recognize you
and I'm sure some of you have seen me on
ndc's YouTube channel doing uh what my
friends call Dylan talks which are about
the importance of the open web to the
functioning of a digital connected
society and the evolution of email
systems in the face of the Relentless
onslaught of capitalism or I'm not going
to talk about any of that stuff today no
no I'm talking to talk about something
way more important than any of that
stuff
karaoke
I think karaoke is the most important
art form on this planet
because I think that the single most
valuable thing any of us can learn to do
if we want to enjoy the time we have on
this Earth
is to enjoy doing something we are not
good at
because if you suck look at something
but you do it anyway and you have a good
time you're gonna do more of it and you
are going to get better you're going to
discover your strengths and your
weaknesses your limitations you're going
to work out where you should apply
you're going to get better you're going
to end up some things you will end up
being proficient and you will end up
being expert and you will end up being
world-class on other things you might
just suck forever but it doesn't matter
because you are enjoying yourself every
moment that you spend doing it and uh I
think karaoke is one of those things
it's wonderful watching people do it who
are good at singing It's also wonderful
watching people do it who are just
having a great time and they're out with
all their friends and they've had a
couple of drinks and it's one of those
things that kind of when you get too
good at it it's a bit weird you haven't
done like paintball with your friends
from work and someone brings their own
gun
like you got to play Crazy Golf and
someone brings their own golf clubs
Jacob and you're like no no the whole
point about this is you just kind of
turn up and you go with whatever's there
now we've done a lot of awesome karaoke
nights after these kinds of events uh
there is video floating around somewhere
of many people in this crowd packed into
a little room in the sing mayor in Oslo
um frightening the locals with Rage
Against the Machine karaoke I don't
think they knew there was that much
profanity when they put it on the list
but
one of these things we went to one of
these karaoke nights um that was the
usual deal you got the room and you got
the video screen you got the songs and
we had a bunch of these
inflatable guitars because there is
something about having a couple of beers
and putting on Guns and Roses that makes
people want to do this and if you give
them an inflatable guitar it gets better
because now they've got an inflatable
guitar and I'm sitting there and I'm
watching this everyone's having a good
time and I'm thinking I know how to play
this song and I'm sure there are other
people who come to this karaoke Booth
who also know how to play some of these
songs wouldn't it be cool if we could do
karaoke but we could have real guitars
involved with it as well now this idea
kind of burrowed into my brain and
refused to go away
and I thought maybe could it be done and
there's a bunch of different people out
there who've done variations on
instrument karaoke but most of them are
like they play in a band and you can get
up and you can sing with the band and
I've seen I got some friends in London
who've done a version of this where uh
what they do is they get their lead
singer to sit off stage with the
PowerPoint clicker advancing the lyrics
so they don't have to worry about going
too fast or too slow there's a couple of
bats in London where they just give you
like a printed folder with the lyrics in
it and the band play and it's all good
and it's all fun but that wasn't kind of
specifically what I had in mind now when
you said that to reinvent reboot
overhaul to invent something new
you've got to understand where are you
starting from now
how about the history of karaoke which I
know is exactly what all of you came
along here to learn about uh karaoke
started in Japan in the 1960s just as a
way for musicians to perform their own
material without needing to hire a band
and uh it was this guy this is uh
Daisuke in nue and he was the first
person who thought well I could use
these backing machines and just get like
random people in my bar to get up and
sing there's a picture of him with his
original eight-track cartridge Juke
karaoke machine by about 1970 to 1973
karaoke meant people in a bar probably
drinking beer getting up and singing
along to pre-recorded backing tracks and
became established so that's kind of one
of the two fundamental pillars is you
got random people performing and the
accompaniment is pre-recorded now the
other thing which has become my Link's
second pillar of that
is video with the lyrics on it how many
of you know this song uh uh Rolling in
the Deep by Adele
if we started that song now there's a
guitar that goes
ding ding ding and at some point you
have to start singing and you don't know
when you think you do because you've
listened to the song in a car like a
hundred million times but you don't know
how many ding ding dings there are
before you say there's a and so the
video is necessary because on this
particular song If you get it wrong you
can be comfortably wrong for about 90
seconds and then when the drums come in
you're like oh no this is horrible and I
don't want to be here anymore and so I
decided those are the the kind of the
fundamental building blocks of a karaoke
system pre-recorded backing an animated
lyrics on the screen so people know when
to come in and stuff now the history of
karaoke is full of wonderful pieces of
technology that none of you has ever
heard of
did you know about compact disc plus
graphics
yeah
well obviously you did because this is
still how you consume Your Entertainment
lemon but most of us we have no idea
like audio CDs you remember see it's
like MP3s on shiny circles that you
could put in the car and stuff there is
a data sub carrier wave on the CD audio
encoding format which can carry about
112 kilobits of video data now we're not
talking high definition here if any
remembers trying to watch Real Player
over dial up internet it's that quality
of high definition video but this thing
proved incredibly successful as a way of
Distributing karaoke tracks now there
are other things there was a company in
Japan in the 90s who did dial up karaoke
using ASCII and MIDI files and so that
this meant they could have vast
libraries of songs and when you wanted a
song you kind of had to book it like 10
minutes in advance so they could
download it over the dial-up but then
they would play it and if you've ever
listened to MIDI file playback it sounds
like the demos on a Casio Keyboard a
little bit but it's good fun and if you
get a couple of people together and you
pick the songs that's quite a cool
technology but all of these kind of
wonderfully innovative solutions MIDI
files have a download CD Graphics CD
plus video CD
all of them fundamentally existed
because technology was [ __ ] and not good
enough to do what we actually wanted to
do which is just give me as much data as
I can eat 24 hours a day wirelessly
anywhere on the planet I don't care the
problems that this was created to solve
don't exist anymore because we have
broadband and you know a CD-ROM would
hold 650 megabytes of data which was
huge when it came out that is nothing
today like we don't even care you can
stick a thousand of these on your
computer's D drive without even stopping
for breath all of the technical
Innovation that went into karaoke
technology just kind of fell by the
wayside when we got to the point where I
was like yeah just go on YouTube any
song you can imagine it's there it's
this huge Library we got all these songs
in it you click the button and it just
plays the song and that's it and uh I
hope you don't get caught by the lawyers
because it's really not supposed to do
that apparently but it's YouTube cast
now the other really interesting thing
about diving into karaoke technology and
generally there's a whole world out
there of rhythm-based games you know
games like Dance Dance Revolution that
kind of stuff is it is one of the few
times in my life that as an
English-speaking white man I felt like I
don't belong here because the market is
absolutely dominated by Japan and Korea
like I have been in karaoke booths all
over the world we're in one in Amsterdam
a couple of weeks ago where every song I
had ever heard of was buried two levels
deep underneath a menu option called
western music
and you know there is this vast World
other apparently in Japan uh Japan and
Korea it is completely normal that when
a band brings out the new album they
bring out the karaoke album at the same
time which is the same tracks recorded
in the same studio with the lyrics
removed because why would anyone go and
buy an album if you can't do it in a
karaoke Booth that's the kind of you
know the significance that it has
culturally in those countries over here
we're like yeah let's go to a karaoke
booth and oh can we dig through this
menu and see if there's any Bon Jovi in
here um there is don't worry we'll get
to it so
this is what I wanted to do I wanted to
create an experience that was kind of
like a normal karaoke Booth experience
but incorporate guitars into it now if
any of you has played guitar for a while
or anything was learning to play guitar
but in the days before YouTube you have
probably seen things like this total
accuracy guitar backing tracks you'd buy
a book and it would come with a CD and
the CD would have recorded tracks which
had everything but not the guitar and
not the lead singer so you could put the
backing track on you could turn it way
up and you could Jam along and be like
yeah I'm Eddie Van Halen
um and then you'd be like actually he's
much better than me maybe I need AC DC
that's a little bit easier
I thought maybe we could Mash these up
you know maybe there was a way of taking
the the total accuracy guitar backing
track and mixing that with the karaoke
video so you get the music with no
guitar in it you get the words on the
screen so someone can sing along to it
and we could do something on it that way
so I started looking around for places I
could get hold of these tracks now if
any of you have seen uh our conference
band the line Breakers
um so people often say to me who's
officially in line Breakers the answer
is whoever shows up because it's about I
think 10 or 12 different musicians who
live on different continents all over
the world and when we get a certain
number of us together in the right place
at the right time we will do a show and
if you haven't seen us we are going to
be on this stage later on this evening
but the downside of that three weeks ago
four weeks ago we were in Amsterdam and
we had a keyboard player tonight we
don't have a keyboard player because
Vegas couldn't make it so tonight the
laptop has to do all of the keyboard
parts so we can fill in that we have
different combinations sometimes we have
life based sometimes we don't sometimes
we got three guitars sometimes we just
got one
and I needed a way of coming up with
backing tracks that covered all the
different combinations of musicians who
might turn up on the night to play the
show
and I'm going to introduce
your favorite website in the entire
world
karaoke version.co.uk now this is a kind
of side Outlet the main company is
called carafun they do
subscription-based karaoke systems that
Supply most of the karaoke bars in the
UK and the lucky voice franchise all
these kinds of things but they also have
this website where you can go on and if
you want to just get the karaoke video
you can but if you want to download your
own personalized mix you're like I want
all the AC DC but I'm going to play the
cowbell so give me all the AC DC songs
but take out the cowbell part we're
gonna do that bit like you can do that
it has this incredibly cool web audio
interface where you can pick the tracks
you can mix them and then you hit
download now what I thought is could we
pull in individual isolated tracks now
I'm just going to say a word here this
is for you everyone watching this thing
on YouTube later because you will put
this in the comments they're not
technically isolated tracks They are
something called stems an isolated track
if you have a brand with a brass section
the isolated track will be a trumpet
with no effects nothing there will be a
saxophone with no effects nothing
there'll be a trombone with no effects
nothing on that three separate tracks
when you mix all the parts together and
put a little bit of Reverb on it that's
a step and what you can download from
this site is stem so all the brass
instruments will be on one track but
that's actually quite a good sort of
working approximation for the kind of
things that we wanted to do here
so I could go on this site I could buy
these tracks from them for about uh that
two pounds each two pounds to get the
isolated uh stems for a track so you buy
the song you download it and then like
right now I need a way of playing this
stuff that lets me switch the
instruments on and off depending who is
doing the karaoke that night now I do
most of my music production
on my Mac in a Thing Called logic if any
use used uh some of you may have used
logic if you use GarageBand logic is
like GarageBand with the metallic paint
and the alley wheels and all the
upgrades it's an incredibly good
professional grade piece of Studio
software and it will play video there is
a way it's used for people who are
scoring video clips or scoring movie
soundtracks you can pull video into
logic and say we'll play the video over
there so technically you could run a
karaoke night from logic there are two
drawbacks with it
um one logic is fiddly it has things you
want to open a track it takes maybe four
or five seconds to pull in all the
different files and draw the waveforms
you see on the screen and fill all the
bits in which is absolutely fine when
you're working in a studio but when
you're kind of in a bar and people are
watching you want something a little bit
more immediate and the other thing is it
runs on Max and Macs are expensive and I
don't like taking expensive computers to
places where drunk people are waving
guitars around I do it all the time
anyway but I don't like it I wanted to
find something a little bit more
portable and so
I applied a time-honored engineering
problem solving technique do any of you
have a box in your house or maybe a
cupboard or perhaps a garage or an
entire room full of Hardware that you
bought because it looked cool and looked
at it and went yeah and then you put it
in the Box because it might be useful
one day
I had to look in the junk box
and I found in amongst things that I
bought because once upon a time they had
seemed like a good idea that I might
need one day I was right because it
turned out I owned one of these this is
a USB 5.1 audio interface that you can
plug into your laptop and it gives you
six separate audio output channels now
you can in theory put any number of
video tracks into a for any video and
audio tracks into something like a
matroska container file but 5.1 is a
special magic number because it is so
dominant in home Cinema systems if you
can work out a way of getting what you
want into five tracks of audio the 5.1
the 0.1 is weird because lots of
applications don't give you direct
access to lightning we're just going to
put extra bass on that one don't worry
about it you can't mix that separate so
it kind of restricted me to fight but
five was sort of in the thought we
needed because this meant I could take
five separate physical Feeds out of this
thing put them into five channels on a
mixing desk
and have the ability to turn individual
instruments on and off just hit play get
the video going boom that's it oh turn
down the bass bring in life base easy
enough you can do it with the physical
faders now uh
you could do this on screen over the
last couple of weeks I have had the joy
of renting a car and renting a house and
the car I rented had lots of touch
screens in it and the house I rented had
a touchscreen stove which I think is the
dumbest thing human beings have ever
invented black slab of glass very nice
very shiny if you touch it here that bit
gets hot
you know how hard it is to drive a touch
screen when you're trying to kind of do
it with your wrong hand because you're
busy because you have a guitar in this
hand you're talking to someone over
there and you just want to turn a track
down tactile mixing desks where you have
physical knobs and Fighters that you can
find in the dark and just turn it down
turn it up you don't need to look at
what it's doing because your fingers
will tell you which track is on and
which track is off I'm a huge fan of
physical tactile interfaces for this
kind of thing and so I like the fact
that this gave me a way of running it
into a physical mixing desk
so how do you make a karaoke track with
5.1 audio well that's kind of a solve
problem Adobe Premiere which is one of
the kind of Market leading video editing
packages has a thing called the surround
sound mixer so you open a project in
Premiere and what I did is I just said
right give me I got five tracks here so
I'm going to have lead guitar bass
guitar rhythm guitar and then all the
other instruments in a stereo mix back
left and back right and then you just
pick up each track in the surround mixer
here and you're like all right so
backing left let's drag that and lock it
onto that speaker back and right we'll
lock it onto that speaker lead guitar
front left bass guitar bang in the
middle down the front rhythm guitar
there we go rhythm guitar right hand
Channel and now we have five clean
outputs coming out of that video file
all we need now is a player
video player that is not going to fall
over video player that is not going to
stop playing because it got a
notification a video player that has
been used for decades in ill manner of
hostile I.T environments
a player that really whips the llama's
ass
yep
yatarioki runs on Winamp because when
amp does not fall down you give it a 5.1
video and it goes well of course I'm
going to play this what else would I do
um and it's done that since 1997 the
first time I saw Winamp was one of those
what is that and why doesn't it look
like Microsoft Office and can I have a
copy of it please um I still use winam
every week I use it for all kinds of
things it is an astonishingly solid
powerful piece of software
so at this point I kind of ticked all
the boxes I had a format I had a way of
getting hold of these backing tracks I
had a player I had the hard way I needed
to do it it was all kind of held
together with smoke mirrors and gaffer
tape a little bit but
ship it
let's go live now this was around July
last year and my birthday's in August I
just had my birthday last week my
birthday last year I pinged my local
Brewery Tap and I was like hey can I run
a birthday party at your place with life
guitar karaoke and they said yeah cool
so we did
um took all the equipment down set
everything up put up a projector so
people could see the lyrics and
everything folks turned up they had a
good time they played they sang people
played bass people played guitar whole
system worked there was one overwhelming
piece of feedback I had afterwards they
said to me can we have the guitar music
on the video
because the singers yeah they're fine
they got the words they just need to
sing along with that at this particular
one we're kind of relying on the
musicians knowing the songs well enough
to play along with it and there are
people who do that you go and hang out
in any guitar shop for more than a
couple of hours on a Saturday and some
kid will walking off the street now
remember I'm 45 so to me a kid is anyone
born after Reagan became president some
kid walks in off the street and they'll
play Sweet Child of Mine all the way
through and they'll play Led Zeppelin
all this kind of stuff and they're not
in a band or anything but they know the
songs they know how to do it
um I maybe thought that was kind of the
target audience for this it turns out I
was wrong but more on that later but the
overwhelming feedback was can we have
chords on the videos
I was like yes that's technically
possible but this is kind of a one-off
like I did it just to scratch this itch
of could you make guitar karaoke happen
maybe do it again one day a couple of
days later I get an email from the
brewery saying hey that was great do you
want to do it every month
now uh my calendar
I approach calendars the way most people
approach Tetris if there is a gap you
should probably slot something neatly
into it unfortunately one calendars when
you fill up a road doesn't disappear it
just sits there building up your
workload and the other thing is that the
physics of my Tetris calendar actually
looks more like this
[Music]
I somehow managed to find that the third
Saturday of every month for about eight
months I would be able to do this and so
I said yes I sort of exhausted yeah of
course no it's fine I'm sure we'll
figure out ways to make this work which
meant I had two months from August to
October last year to figure out how do I
solve the problem of putting guitar
chords onto a karaoke video so people
Jam along with it now I had absolutely
no idea how to do this you know the
sky's I'm like okay could this be a it
could be a web thing maybe I could do it
with a web maybe I could do web with
animated JavaScript on a canvas in front
of video maybe I need to do it in C
because like there's the VLC player
that's written in C right maybe I need
to learn C2 maybe I can do it in Maui
maybe I could do an iOS application
maybe I could do it real time maybe it
needs to be a processing pipeline
literally it's like I think computer can
do this I just have no idea how so what
I did is I worked backwards I went into
um After Effects Adobe After Effects and
I created manually a couple of different
prototypes of what I thought this thing
needed to look like and I came up with
this
so this is just the regular karaoke
video with the guitar chord shapes just
moving across the bottom and when they
hit that playhead bar there that's where
you hit the chord change and I thought
okay that's it that seems to work I did
this setup for a couple of songs thought
okay that's probably going to work next
question how to help me do that so I
started looking around because I need to
get these quotes from someone and I
discovered there was a research project
it's called Vamp and Vamp is uh Queen
Mary's University in London with a whole
bunch of collaborators all over Europe
and around the world it's a set of
plugins for doing audio analysis so the
idea is that you throw an audiophile at
this thing and it will go uh okay I
found the tempo I can tell you where the
beats are so if you want to go through
like your whole MP3 collection and find
the songs which are the right speed for
you to listen to while you're jogging or
you're on an exercise bike or whatever
in theory it can do that and one of the
plugins that exists for Vamp is a thing
called cordino which claims to do
harmonic analysis of audio files and
it'll tell you what the chords are so I
download this and I run it and that's
when I realize I'm on an M1 Mac
now M1 Max are a nice idea that'll be
really good in about five years when
they finish fixing all the things that
are broken because most of the time M1
Max will run old software built for
Intel because they run it through a
layer called Rosetta which translates
all the instructions from old Elton
Intel instructions to new Apple silicon
instructions and back again and it
mostly Works unless you are trying to do
something like host audio analysis
plugins because then it just goes no and
you're like wow that's like well this
bit We're translating and that but it's
running natively and now that it's not
going to work and I thought well this is
fine this is all open source right I can
just download the source code and I'll
compile the entire tool chain so it runs
on a 64-bit Apple silicon process yeah
no
it's not how open source works
I managed to download the code and it's
like okay I'll build that all right that
oh that's why I need this bill okay I
need that bit I need oh hang on I need
that bit if you're familiar with the
expression Yak shaving
where you're like oh I want to run a
karaoke note with guitars which means I
need chord shapes which means I need to
extract the harmonies which means I need
to run this plugin which means I need to
compile this thing which means I need to
compile a library it uses and then
someone looks over your shoulder and
says hey why are you recompiling cmath.h
and you're like I'm trying to run
karaoke can't you see
yeah so I backtrack a couple of steps
I'm like maybe I don't of course turns
out it all runs on Windows as well
absolutely fine so I grab uh the vamp uh
Library I grab the codino plug-in and I
grabbed this thing it's an app called
Sonic visualizer which gives you a kind
of visual interface to explore these
things now we're going to do a little
bit of a live demonstration now of just
how good and how bad this technology is
uh what I got
is I got a couple of tracks and I've
renamed them so there's no clues in the
file names so I'm going to open this up
and then what I'm going to do is I'm
going to get it to extract what it
thinks the chords are
and see if any of you out there can
recognize the song just from the chord
extraction that has happened here so I'm
going to bring that up so it's going to
drop the chords in on the bottom half of
the screen here I'm going to turn off
the original track and I'm going to play
it
[Music]
thank you
anyone recognize any of it there's a
recognizable progression coming up in a
second
let's bring the original track back in
so you can see what it is
now this one it got pretty good pretty
close let's take a look at another one
so same thing what I'm going to open it
up I'm going to mute the original and
we're just going to get the
automatically extracted chord
progression from it
[Applause]
and
[Music]
if any of you seen a project called Axis
of Awesome
well I I'm afraid I have news for Access
of awesome this
is Don't Stop Believing by Journey it
has five chords it is not a Four Chord
song there are five chords in the intro
to Don't Stop Believing let's kick the
original back in
any second now
and you see that one in there that chord
shouldn't be there most of it it's
picked this up pretty accurately but
there's a couple of chords that just do
not fit they shouldn't be there
foreign
[Music]
I'm going to leave the original music in
because I just want you to see how badly
wrong it gets this
because this was really promising I was
like this is like it's amazing I can
automate the entire pipeline end to end
now you'll recognize the music to this
one I hope pretty much immediately it's
very famous very distinctive and it also
demonstrates how this automated analysis
doesn't work very well when all you've
got is one very distorted guitar
[Applause]
it is
[Music]
but now let's see what we get from the
automated analysis
[Music]
[Applause]
[Music]
what's interesting here is when the bass
guitar comes in it kind of kicks in and
goes oh yeah I can figure out what this
is now he listened for us there's any
second now it goes from this kind of
weird ambient Lodge Church open music
and suddenly it's like oh an E major
chord all right I can work with this so
please
so that was kind of promising but it
wasn't going to solve the problem it's
like it will try it and if it works
that's good and if it doesn't work I
need a better way what it did give me is
it established the fact that there is
this uh there's a command line version
of cordino as well and what that does it
just spits out a text file
[Music]
which has the chord name
and the exact time stamp where that
chord change happens in the music
and so this kind of gave me an interface
between the two halves of this problem
what it meant is okay I still I know I
have this I have this point in the
Middle where I got this file which has
the time and it has the chord name I
don't know how I'm going to get there
and I don't know what I'm going to do
with it once I've got it but that's a
good way of dividing one big problem
into two small problems and then maybe
I'll have a chance of being able to
solve one of them and then problem
number two will output the video
so problem number one codina works on
some tracks for the rest what I did is I
hacked something together uh this is
recorder now this is if you're familiar
with Fred Brooks maxim of when you
design software you should build one
version to throw away this is definitely
the one I should throw away
but I haven't thrown it away yet fifty
percent of it is good solid working code
the other 50 is things I've learned not
to click on because if they do it or
crash but it doesn't matter because the
software is not the product the product
is a bunch of people in a bar playing
guitar and having a good time and all of
these things they are tools that exist
to deliver the next iteration of that
product if I'd spent a lot of time kind
of bulletproofing this and building
features and stuff and what probably
would have happened is I'd never
finished it and none of this would ever
actually have taken place but what this
does it's a net web application running
in the background so it can talk to the
file system it's got some JavaScript on
the front end and it's exploiting the
fact that web browsers can do all kinds
of cool stuff like they can play 5.1
surround video and mix it to stereo
without you needing to do anything
special did you know browsers could do
that I didn't I just put the video in
and it worked so what this lets me do is
basically down the left hand side is a
list of the songs that I haven't got
chords for yet pulled off the file
system I can go in I can click on one of
those
and then I can go on Google and I can
find the guitar chords from one of the
many many websites that exist out there
which will allow anybody to go I think I
worked this song out there is no quality
control here it is completely caveat
Empire so let's have a look at the first
one of Kenny Rogers here now this is
It's kind of right but it's missing some
bits it's a little bit simplified but
that's all right there's another version
on here so I can jump back I can find
one that looks like they've probably got
the intro and the bridges and the
harmonies and the breaks and all those
kinds of things
and then I built a pasa well no what I
built is something that can look at
ASCII text and extract the things that
it thinks are guitar chords calling it a
Pasa is probably disrespectful to people
who actually write pauses
but what I can do is I can copy the text
I can come over here
I can paste it in there now when I said
uh there are things you click on one of
the ways that this application works is
that every time you press a key it
pushes everything back to the server so
you don't need a save button
if you paste and then forget to press a
key it accidentally loses track of your
work and so when you refresh oh there it
is so I'm going to paste that back in
but then what it does when that uh text
area in the middle when that blurs is it
goes through and it runs this chord
recognition filter across it and it
highlights all the chords and then there
is a button here called button because I
didn't know what else to call it but
when you hit button it starts playing
the video and then every time you hit
the space bar it just puts the next
chord out of the transcript and it drops
it into the chord timing chart
and so this way you can grab the ASCII
files off the internet you can drop them
in here and just go through and
relatively quickly now I did have a
prototype of this where I use the midi
interface so I had to play the chords on
a piano I realized I'm not that good a
piano player like not even close like
hitting the space bar I can kind of do
that haven't actually like the black
notes and the white notes that was
beyond what I could do
um do you want to see the regular
expression that we use because chords in
English we have the the major scale
chords A B C D now the only false
positive I got on this was a because
capital letter A occurs in written
English so it crops up in song lyrics it
also crops up in the chords then there
are the accidentals now this is fine
because uh there is nobody in the world
sad enough that they would actually
write a song about C sharp or F sharp
even though those are programming things
so that's not something we need to worry
about there are the flats as well we can
pull those out none of those occurs in
written English
but chords get more complicated than you
think so what I got here I've got the
jazzometer in the bottom corner here so
we've got a B which is a nice simple AC
DC kind of chord then we got B flat and
getting a little bit blue so your B flat
minor B flat minor seven B flat minor
seven add nine
with an F in the bass now we are firmly
into modern Jazz territory now there
aren't a lot of these in the kind of
things people play at karaoke nights
when they do crop up it's normally
because cordino has made a mistake but I
wanted to be able to pick these out now
a lot of the time you have a B flat
minor seven add nine with an F in the
bass what you got to do is get Mark to
play the F get everyone else to play B
flat minor and don't worry about the
seven add nine because you're probably
not going to notice in a bar full of
drunk people this is the regular
expression that will recognize chords in
pieces of ASCII because we love regular
Expressions right I asked chat GPT to
explain this to me and it went I no I
don't know what you did there so it's
all right our jobs are safe humans who
can read regular expressions
so there we go back to our project Gantt
chart here problem number one is now
solved I have the timing charts I can
make them automatically or I can make
them by whacking A spacebar over again
with some ASCII that I found on the
internet and a regular expression
because hey engineering right problem
number two
how do we make the video now uh you do
anything with computers and video sooner
or later you will find ffmpeg because
ffmpeg is amazing for two one it is the
most powerful capable piece of software
there is no video or audio format known
to humankind that ffmpeg cannot read and
write and encode and compress and
decompress
it is also the most hostile piece of
software
like after a couple of weeks with ffmpeg
people go I've missed Lotus Notes I like
Lotus Notes Lotus Notes loved me Lotus
Notes was my friend at its most basic
you say hey ffmpeg minus I input MP3
turn it into output.web and it goes yeah
I can do that
it can do just about anything if you can
find the right Syntax for it you want to
take an input file you want to apply an
audio resampling map it to 44 100 Hertz
two audio channels 320 bits per second
output former MP3 it'll do that you want
to take a 5.1 video file like the guitar
Aoki files we got here and you want to
say take 5.1 extract it but this much
guitar in that channel this much in that
channel this much in that channel back
left back right move them to the front
rescale it to 640 by 360 pixels start at
45 seconds extract a 30 second clip
libx264 encoded 2500 kilobits per second
of video 128k of audio use the AAC
filter and output the result as a flash
video yeah one line easy
now I absolutely believe
that you should be able to go ffmpeg
minus I live in on a Prayer minus guitar
karaoke minus lead guitar rhythm guitar
bass band left band right chord color
this playhead true output
I believe it exists I just don't think
we'll ever find the combination of
keystrokes that makes it happen like it
wouldn't surprise me at all if somebody
went oh yeah ffmpeg can do that because
it probably can
well maybe I don't know
um but ffmpeg does have a lot of really
powerful video mixing overlaying
composition multiplexing capabilities so
that was going to come into the tool
chain somewhere but I needed to do a bit
of processing first now I like C sharp I
like writing C sharp and when you are
doing software development for fun to
run karaoke nights with your friends why
would you do it in a language you hate
or a system that makes you uncomfortable
I'm going to do this in C sharp as much
as I can now all the code for this is on
GitHub if you really want to dive deep
into how it all works and how it hangs
together I just want to show you the
kind of high level approach here so what
I did is I created a chord class
and I broke a chord down into the name
and the extra bit now this is not a
musical uh thing this is a typesetting
thing because when engravers set sheet
music the chord name is nice and big and
then everything else flat minor major
seven diminished that's normally set a
little smaller it's a convention that I
wanted to obey because there's only so
much space on the screen and so when it
goes through this file it pauses each
chord pulls out the name extracts the
extra bit just by doing string split at
the time when does that chord get played
so that represents now there is a read
chords method in there it's going to
pull in the chord times file path select
each line map that onto a chord
um the pausing error by the way any
chord it can't recognize it gives you a
time of -1 so we can just filter those
out any chord that starts before zero
scrub it
not part of the song
then I go through every one of these and
I calculate the distance or the time the
time interval between the previous chord
and this one gives me the duration if
there are lots of very very short chords
it means I need to draw them on two
lines so you can get them all on screen
at the same time if the chord changes
are relatively far apart you can draw
the whole thing on one line
and then this runs through the whole
thing and does that it looks for chords
whose duration is below a threshold and
it says right I'm going to move you on
to line two I'm going to move you on to
line two everything else stays on line
one bring back a collection of chords
from that now this then gets fed into so
we pull out read chords calculate the
duration of the track use that to
calculate how many frames of video we
need to render
and then create a thing called a frame
maker now frame maker is something that
I built for this you say hey how wide's
the video how high is the video how many
frames per second and then you call FM
create frames we'll have a look at that
in a second now there is a thing called
a raw video pipe source which is a
memory mapped interface from.net into
ffmpeg so you can spinup.net you can put
a bunch of video frames frame by frame
by frame in a massive continuous block
of memory and then tell ffmpeg to treat
that as if it was an input stream and
apply all of its encoding stuff to it so
this is the bit which renders all of
those and what it does is it creates a
transparent video
so you got your original song four
minutes of video with lyrics on it now
we have four minutes of transparent
video using a it's a format called webm
and it uses a thing called the vpx9
codec which allows you to have Alpha
channels in video information so now we
have two videos we got the original and
we've got a transparent one full of
animated guitar chords now that frame
maker bit there I'm not going to show
you the code for that because it's
gnarly and it's mostly mathematics but
the only reason that exists is because
of this amazing open source Library it's
a single image shop and uh Jim Bob
SquarePants if you are listening if
guitar Aoki ever makes any money I
promise I will give you some because
this is a wonderful library and none of
the stuff with the chord rendering and
stuff would have been possible without
it
um and uh yeah they've had some uh don't
net drama about their licensing policies
over the last couple years but seriously
this is an awesome piece of Kit and
basically what I've been doing with it
is going through and saying right okay
so we're drawing a frame first of all
are there any chords on this frame look
at the timing figure out when does the
current video frame start and end and
the song If there are then I want you to
go through every chord that belongs on
this Frame calculate physically where to
put it by taking the playhead position
the chord time the playback speed the
frames per second and then there's a
thing here called chord nudgen pixels
now the human brain is weird
if we see things before we hear them our
brain will make them line up because the
speed of light versus speed of sound
means that in the natural world you
often see something before you hear it
if you hear something before you see it
your brain goes no no no this is bad and
wrong and we hate it but you don't have
to get it exactly right you just need to
make sure that you are arrowing on the
side of coming up on screen first so
everything gets nudged off by about 48
pixels which then gives the illusion
that it lands at exactly the right
moment even when it doesn't quite
um spin up a point work out now image
dot mutators image shops mechanism will
say hey I want to take that and I want
to draw the stuff on it and I want to
draw this stuff on it so we're drawing
the chords pretty name which is the nice
name or just the short name and then if
the chord doesn't have an extra bit hey
continue otherwise do all the rectangle
metrics calculations to figure out how
big you draw the flat seven minus 9
diminished with an F in the bass stick
all that on there and then finally draw
the bar over the top and draw the little
white playhead now one of the
optimizations in here is that if the
frame is m it just returns clones of the
same empty frame over and over again so
we only ever draw the empty frame once
in any other scenario we clone that and
use that as the canvas we're drawing
everything else on top of so that gives
us this frames it returns an iron
numerable we wrap those up in a memory
stream and pipe all that into this
ffmpeg processor and then once that's
done
there's just a little bit here now at
this point I'm just using.net to drive
the command line and I'm saying I want
you to grab that video I want you to
draw it on top of that video I want you
to apply these filtering settings to it
I want you to uh there's various options
to render like a 30 second preview and
stuff which you can see in the code if
you want to look at that
and then finally show done Banner now
because this takes four or five minutes
I would often start it running and then
go and do something on the other side of
the room and so the last thing this does
is it draws a massive bright pink ASCII
art done on the screen so I can glance
over and go oh another one's done and I
can go over and I can kick off the next
one so that seems like hey it's the end
of the rendering pipeline process now we
are going to be running live guitar
karaoke right here on this stage later
tonight so this is kind of only the
first half of the talk there will be a
live demo but it's going to be later
when everyone's had a couple of beers
and feels like getting up and singing
and playing some stuff if you want to
see all the code that drove this video
it is on GitHub there is an organization
called guitar Aoki the tools are in
there go look it up poke it around the
music is not you are going to have to
buy your own backing tracks if you want
to play with this because I do not want
to get sued for putting Bon Jovi on
GitHub because that is a terrible reason
to go to prison
we are going to be back on this stage
later on tonight I'm doing a show here
with line Breakers and then guitar
karaoke will be kicking off if any of
you out there have seen some I want to
have a go at that come and talk to me
now and what we'll do is we'll line up a
set so we can get things up and running
figure out who wants to play what and
then we're going to throw it open to the
rest of the festival later but folks
thank you very much for coming thank you
for listening are there any questions
about a talk that went from the history
of karaoke machines to rendering
algorithm pipelines in.net and ffmpeg or
have we achieved The Silence of
enlightenment
I did
so the first version the first one that
ran in August had no chords the second
one there were chords on every song
except one Bohemian Rhapsody was taken
off the list because it's too bloody
difficult
it'll be back one day that's stretch
goals
um
huh
why would I do sound garden
yeah
does anyone know a C flat six
we don't need that like they did that
they did that and they did it for the
rest of us so we don't have to
ha
yes
strumming patterns so the thing about
karaoke is that I believe it assumes a
certain level of familiarity with the
material it's the difference between
singing karaoke and like if you sing in
a choir
um you know I I know classically trained
professional singers so you give them a
piece of music that they've never heard
and you're given the sheet music and
they sing it and they can do that
um karaoke is not like that and so I
sort of assumed that if you know the
song and you can see the chords you can
figure the rest out for yourself and if
you don't get it quite right
um I have had a couple of people go can
we have the guitar tabs on there and so
I've shown them what guitar tabs look
like in real time and being can you read
this and they're like well no I'm like
exactly you know it's uh it's it's
enough that you can kind of figure it
out and get it right but no it doesn't
have strumming patterns if it needs
strumming do you think it needs
strumming patterns I don't know what's
the audience consensus on this
any other questions folks
yes
sorry you're gonna need to yell I'm deaf
it looks like positive grid spark a
product for free you can download
certain future
all right
we're just going to see live on stage if
I've wasted the last year of my life
are there any questions that aren't you
shouldn't have done all that you can get
it for free on the internet
what is it positive grid Spark
uh
oh hang on hang on hang on let's see so
what what is positive grid spark uh
yep
that one that that's a YouTube video
okay I will take a look at that later
um but I I will also bet you a beer that
it does not do get the chords up get the
vocals up mix out the bass guitar bring
in the rhythm guitar and let someone
play the solo for The Final Countdown
there's a lot of people doing similar
things in this kind of space with
various tuition tools um if you've seen
Rocksmith the computer game
um Rocksmith is basically this but like
for a very very sort of advanced user
interface and things
um but now that looks interesting I'll
check it out any other questions yes
is that a vehicle
um
about 15 minutes
um of which a good chunk is waiting for
karaoke version to give you the download
link because it generates the mixes on
request so you'll click the button it
takes about 20 30 seconds for it to
generate each file download the files I
have an automation script that turns
them into a Premiere project so
everything's already mixed in the right
place you render that with the Adobe
pipeline that takes about 90 seconds
then if codino picks it up it just picks
it up you're done if you need to make
the chords manually that uh if you're
familiar with the song you can I've got
it right on one pass through at real
time before so for the four minute song
it takes you four minutes to do it often
I'll screw up the first time and need to
go back and tweak a couple of things and
then the uh the video rendering pipeline
takes on the M1 Mac it takes about 150
percent of the duration of the song So a
four minute song it'll be a six minute
encoding job to spit out the completed
video at the end of it but uh when when
we do this um so Mark Rendell comes
along and we do this in London uh fairly
regularly and we had a thing gonna add
five new songs every time we do it to
keep expanding the repertoire and it got
to a point where putting those five
songs in would be about a two-hour job
so so I knew I needed two hours on
Friday to bash the songs and get the
chords do the rendering drop them in and
then they'd be ready to go for us to run
through on Saturday so uh yeah it's not
quite real time but it's uh I can't
imagine doing it on a 486 that's for
sure
any other questions folks
all right thank you for coming we'll see
you later this evening