what's up what's up what's up what's up
welcome back everybody today we're
working on our video editor and the
topic for today is quality of life
improvements so we've been working on
this video editor and the pitch for the
thing is sorry that's me talking again
in the past but pitch is if we have this
script on the right hand side maybe we
can navigate the video faster right so
the idea is like if I want to say like
hey I want to include this stuff in the
video maybe I'll click on this maybe
maybe maybe maybe and then I'll hear
that stuff and then I'll cck down here
and I'll say like oh then I can
experiment experiment and that comes out
exactly with where I clicked right so
this stuff is autogenerated by um
whisper
CPP uh by just like running through our
audio uh that we've extracted from fmeg
and feeding it in there so that's
working like okay I can go into like the
timeline here and I can like start to
like clip some stuff so if I say I want
this I want this I want this we can kind
of like preview that by questions more
often jumps around
um and if IIT save I can close this and
I can run FFM peg this like this script
basically parses the save file that has
all of the clips that I have in there
and it'll run FFM Peg generate a video
give it a sec he's got to think a little
bit he's thinking and if I play that the
same questions more often that's
something that I've been like thinking
about I don't know if and at the very
least it's I think it's the the same
things that I clipped and previewed in
our editor so we're getting pretty close
close to something that is like usable
um however when I try to use it it's dog
and uh the big reasons for that are
that your using this script is like
quite a struggle like skimming it and
trying to figure out
where like how do I say this with no
line breaks I it's just like word soup
to me right like I I need to be able to
parse this stuff faster the whole point
was that I'm better at reading words
than I am at skipping audio but this
isn't really coming true with with with
the way it's laid out right now so
the first thing I want to do is see if
we can find some way to like split this
stuff up so that it's like more readable
I'm thinking right now uh that we'll
just kind of like look for Dead Space in
the audio and add line breaks there
maybe that'll work okay maybe it won't
um but we'll kind of see how that feels
um another thing that I found when I was
trying to like use this is like I can
kind of like move these clips around uh
but if I like you know if I work by how
do I say do this let's see let's delete
the existing Clips
delete delete delete so if I say I want
like this sentence so a pitch from the
project is so I might do something like
I might come here and I might like click
on the so the pitch for the project is
and then I might click here and I go
okay so it's probably like something
like this for the project it's probably
more like this is but then like say I
want to like remove this section of
audio here that's like kind of dead um I
want the ability to like kind of like do
the inverse of adding a clip which is
like removing a chunk of a clip by kind
of like deleting it here uh and then
maybe I'll have you know tried to remove
some Dead audio and then been like oh
you know what this actually doesn't flow
nicely when I listen to it back so I
want to be able to like merge these two
clips together but right now that's not
really possible um and then the other
thing is like it's hard for me to find
where I am so if I if I am looking at
this stuff over here uh we'll see where
it trying to like find that in the
script is like impossible right now it
might be easier once we add line breaks
but maybe either some form of like text
search so that I can look for what we're
looking at but probably more what's a
better solution is like a way for me to
like basically track where we are in the
video in the script so maybe by default
we kind of scroll along with this thing
um but when I scroll it kind of like
stops doing that for a little bit while
while I'm like interacting maybe I it
only does that if the Last Action wasn't
interacting with the script not really
sure yet um but yeah some way of
basically like tracking one with the
other uh maybe it will be maybe actually
before we do that maybe it'll become a
lot easier if we add some form of
um progress
indicator right cuz that that hasn't
been done yet and maybe that would make
it quick right if you could see maybe in
the scroll bar where where the current
position is and then when you like
quickly scroll through you have this
like visual indicator maybe that's good
enough uh so let's try that let's try
that oh yeah this plan we're just trying
to make it so that like if I were to
actually try to use it it wouldn't uh
suck ass that's that's the plan that's
the plan chat messages hello everybody
hello hello hello um Echo thanks for the
prime big appreciate it uh what else is
going on here hello hello uh L braks in
Dead Space is a good start if you delete
line breaks if you delete adding line
breaks might be good I don't understand
what that means if you delete adding
line braks might be good oh you're
saying like if it adds too many and I
have the ability to like manually go in
and adjust it yeah that's not a bad
idea maybe I want to highlight detect
the same color as clip yeah that's not a
bad idea as well except right now all
Clips are uh the same color so probably
um I'm not really sure I I experimented
with the idea of like taging Clips with
with words and colors and stuff but it
turns out that like in in practice there
are so many clips that like naming each
individual one doesn't really makes
sense but I might experiment with the
idea later of like clip groups so I
might like like like create a label and
say that I'm creating like five Clips
with this label and then have those all
be the same color or something then you
could like at least like chunk up the
video into like chapters or something
with that scheme not really sure we'll
play with it at some point but that's
for a little bit down the line
Captain H thanks for the prime
appreciate it appreciate it okay so I
think let's start with line breaks let's
start with line
breakes um so we are at 0545 and let's
get some line breaks in there what's up
what's up what's up yeah that's
right that is I don't know how that
started but it's just I can't stop now
we have to do it forever we have to do
it forever there's no other option um
word time stamp generator is where this
stuff lives right now
uh
and so what we do is where's our like
main Loop we have this like thread that
starts up when we launch the program and
he basically says hey I'd like to like
pull some samples from
our video so where's that here he like
pulls a sample in from the video by like
basically just like walking through and
looking trying to sample the input at
16,000
HZ um once he he fills up this like
audio buffer which we pre-allocate to be
like 8 seconds long right now just for
whatever reason that's just the number
we picked and then once we have a full
buffer we feed that buffer into
whisper and he generates text for us we
like look at the text and we say okay so
these are the this is the start and end
in both character position and time for
each of the thing each word in the text
and then we just append that text
here
so here's what I'm
thinking here's what I'm thinking
I think we want some sort of like
runtime configurable threshold for where
the new lines are
inserted which kind of means that like
on generation we don't really want to
apply it yet I don't
think there's like kind of two options
here one is that we have this like text
buffer and basically every time we
change the like dead sample size or
something we rejig this so we just like
uh maybe we like what
remove all new lines and then find
segments farther than NS seconds
apart and then add a insert new lines
that seems like kind of stupid
to me that seems stupid to me um because
like the new lines aren't part of the
text if that makes sense they're kind of
like this thing we're tacking on after
and if we're going to be splitting them
uh how do I say this if we're going to
be splitting the text up frequently like
if I'm going to go through the UI and
just like try a bunch of values maybe it
makes more sense to have some sort of
like text
iterator and we'll have like some
struct and he's going to have like a
next
API um and he's just going to return
some
data
okay
and as well as maybe his like time
stamps so output is like outut outut
output is something like
text and then segments okay so this is
just going to look like what like our
actual thing looks like right now right
so he's going to have some like segment
bounds associated with this
text
um
and does that that Mak
sense maybe it doesn't have to be an
iterator CU now I'm thinking about okay
so on the gooey side we have to click on
a character and figure out what time
stamp that's associated with so what
we're going to end up kind of with on
the gooey side is we're going to end up
with an it like a big list of these like
outputs
right um and so if we were going to ask
like
hey back
and hey back end which what what time
stamp is
this how do we know what to
ask oh wait that's actually kind of fine
that kind that kind of actually works
out because um we can like keep we can
write down the lengths of each of these
segments so we can be like this guy's
like five long this guy's 10 long this
guy's you know 11 long or 10 long again
for ease of use and then if we're trying
to find something in this segment we can
do something like we know that it's 5 +
10 + 10 plus
offset and then we don't have to change
any of our API xns thanks for the
prime so I think that makes sense I
think that makes sense so then in that
case like this iterator
method I think works better for uh the
like C ffi does it let me think about
that so right now what we
do right now what we do welcome
Ginger
um right now what we do is we have like
App
State and like on like our Zig app side
he like basically feeds his like
internal State over here every once in a
while and then the rust gooey we're not
going to get into why why the we're
using a zig app on Rus I guess we have
to uh Zig app because I feel like
writing zig rust gooey because I wanted
to use this gooey framework staple them
together baby um anyways so they
basically just both call into this every
once in a while and this app State one
of the many things he has on in him is
just
like text
buffer and so if we were to do some sort
of like iterator based thing
here uh this becomes tricky because now
we need some sort of like function to
call on this thing to like iterate
um we could
maybe we could
maybe you know kind of like make an
array of multiple things like
this and which would require like us to
make do like a new
allocation of however many new lines are
going to be in there which is like
probably fine that's not a lot and then
it's simple like the guey doesn't have
to add any more like function calls back
to the zig side he just like looks at
what's in here
that doesn't seem too awful that doesn't
seem too
awful yeah maybe we try that maybe we
try
that um yeah okay okay so that
means in our like app thing when we say
we like set our snapshot we like set
this like text buffer here that's just
like from the word timestamp map let's
get the current text
and so we just have to adjust that a
little
bit um and I guess in that case it kind
of makes sense for like the word time
stamp map to like
own
the CH like the the
split so maybe we do something like this
uh
split indices we use indices here
instead of like using the buffers
directly because uh this is like an
array list who when you put P data into
him he's going to like uh he might run
out of room and then he goes like I
need more room and he's like hey OS can
I get more room please and then but that
ends up in a different spot so we know
pointers are scary here um but indices
are a little bit nicer to deal
with um and then I guess really in our
snapshot like back to this thing instead
of it being like this we can just have
like split indices
here splits
yeah yeah okay then we don't have to
like allocate a new thing all we have to
do is like copy over this like split
indices weer which is easy which is
easy uh okay so
where where are there reasonable gaps a
we won't com it who needs comments
we'll just forget later and then get
mad uh okay so somebody here I guess is
going to have to
say Pub function
set
something uh set split time we'll call
it split set
split set split
threshold that seems reasonable we'll
say Threshold at which to uh how much
audio
how much time passes between words
before we new
line I forgot to announce My Stream in
Discord did I I don't think so I thought
I did it earlier but maybe I forgot to
press enter I'm pretty sure I did it but
I
guess maybe not maybe
not
um okay so this guy is going to say what
um we're going to look at the ourselves
ourself
and I guess we're going to take in some
sort of threshold in
seconds float 32s probably
fine
um you don't see Discord I like
opened Discord and typed it out I guess
I just didn't send it which is uh oh
well sorry guys
sorry uh okay so this guy's going to
also have to store the split
threshold and does he belong here
or
here like does like the init thread need
this I guess the init thread could use
it um as it's like appending new stuff
so it could be that like init thread is
going in here he's like reading text
reading text reading text and he goes
like I've got new text what do I do with
it and as he's getting new text he incre
he populates the split indices thing
because otherwise we would need to
recalculate it from like the other
thread every once in a while every time
there's new
data where this way he can just like do
it as he goes which makes sense and then
if this changes we like recalculate all
of the existing indices and then he just
keeps going from where he
is yeah that seems reasonable to me that
seems reasonable to me so it should it
going in like the shared data makes
sense
there uh so he's going to say hey I
would like to look at the Shared data
and before I do that though I I always
have to lock my
mutex otherwise I get in big big trouble
uh and I also have to unlock my mutex or
else again I get in big big
trouble uh and then you just says self
shared uh split thresholds equal
threshold
s uh and for now then he's going
to I guess
iterate um
iterate the
segments for these things for each
segments still with the mutex lock
that's fine uh because otherwise the
other the in thread might be adjusting
this thing as we iterate that would be
bad we would get in trouble again so
many things to get in trouble with uh so
this is a segment and he's going to have
like a start and end
time and so I guess we're gonna actually
want to iterate uh maybe by index is
easier here
so we can look at uh self maybe we do
const segments we Alias it just so that
we uh don't have to type as
much so for this and maybe I want even
want items here there we go segments L
minus
one uh I'm doing minus one because I
want to iterate in chunks in like
Windows of
two uh then we say con a is equal to
segment ments i b is equal to
segments I + 1 okay and then we just say
if uh a
do hello why can I not a do why isn't
Auto Computing
anything these should be segment bounds
so I guess we'll just try it
if a start or end I guess it plus
threshold s is less than or equal to B
start we insert a new
split and this split is going to be I +
1 right the new line is associated with
the thing after the new line not the
thing before it I think that makes more
sense that makes more sense to
me at
least h yeah I think that makes more
sense to
me which means that this thing is now
failable
and what's next I guess before we do any
of this we have to clear this
thing
um now let's think about this say we try
to allocate a bunch of new split indices
and this fails at some point do we like
we if we're going to be good citizens of
our program we should probably make sure
that we only ever reassign our indices
if if we
succeed so it kind of indicates that
like we should do this all at the
end
um yeah so if we
succeed split indices we clear the old
thing we say get the out of here I
guess D in it clear and free which one
makes more
sense um I guess they're kind of like
the same right if I were to if I
were to De init this and then reassign
it versus clear and three I think it's
the same thing I guess with clear and
threee I could push I like the allocator
doesn't go
away but that's fine it doesn't really
matter I'm going to dnit
this and then this is equal to split
indices and then I just have to make a
new thing so
uh stood
list segment bounds in it with self Alec
yep okay okay const split indices not
const bar uh if we fail we have to fre
free this
thing uh-huh
yep but otherwise we assign it here and
then we call it a day and then this guy
has to act on the local one
um and I think we're
chilling yeah I think all that makes
sense is there anything else to do
here
no okay so I guess on the thread side
now
uh in it thread this is the wrong wrong
file wrong
file um but on the init
thread does ZIG move in the C++ sense by
default I think that's one of my
favorite things about Zig now that I've
gotten back into like C style languages
like like Zig is the fact that moves
don't exist you don't have to worry
about it because there's no destructors
the concept of a move doesn't need to
happen right because the only reason why
you need St move or like C++ moves is
because uh you want to make sure that
your Destructor only runs on the thing
that matters right so if you have like
unique uh pointer in and you say like x
equals this uh what language is this is
C++ I should write
Auto uh and then uh make stood make
unique I think is how people do this
four or something I don't remember it's
been a second so like if you take this
guy and you say uh you know Auto yals X
um if this guy if there was no such
thing as like a
move which there wasn't for a while
right uh what would happen is these
things would go to scope and then we why
would like free the
pointer and then X would free the same
pointer and then you have like a double
free no no no bad bad bad right um and
so you know you have like this
now and what this does is it essentially
says like hey we have this like crazy
like double Ampersand type shut the
up it doesn't matter who cares but
there's this like new types of
references that basically say like hey
when I call my move when I call like
this assignment we're going to say that
we have like special type that tells us
hey we should like take that everything
that was in X before and get get rid of
it so when he calls this version of the
function he's going to go like uh x.
pointer is equal to null and then when
he tries to free down here they go like
my pointer is null nothing to
free um and so like the only reason that
you need this stuff is because of this
right but in Zig in Zig you don't have
any like hidden control flow they call
it right so there's no destructors that
are going to run at the end of the block
you just wouldn't write this line of
code you just would right like because
the the zig way of doing this is you'd
have like x equals Alec
create uh i32 x =
4 y = x right and then here you would
just you would write like defer x uh
Alec destroy X but you just wouldn't
write that you would just write Alec
destroy
y so all assignments are shallow and Zig
yeah I think that's a good way to think
about it is like you just do shallow
assignments and then it's up to you to
do the right thing uh and then generally
what happens is like you could create
like a clone method that does a deep
copy you could create like a D in it
function that walks the tree of things
and dealcat them nicely um but you just
have that control and so you don't have
to you don't have to worry about like
the language you you just have
to worry about you you and I
kind of like that better right
now oh even because because there's
something about the like uh the
responsibility of like oh I up
versus like C++ me those two
things just like feel very different and
it's one of the things that pisses me
off with so much when I'm running C++ is
just feel like I can't predict what's
going to happen and of course you can't
predict what's going to happen but you
just like forget small details and then
just like weird happens and you
forget and it's just like you have to
like really dig into the language where
in Zig is just like oh I just didn't do
that right thing that thing right you
know is move from C++ sped by rust no no
no no I don't think so I think C++ moves
came in in like uh C++ 11 and that must
have been before rust was at least big
I'm pretty sure that was pre- rust 1.0
when did rust come out rust do you think
I'm going to get press program language
oh I did wow I actually didn't think I
would um
Wikipedia rust playing
Wikipedia when did this thing come out
first appeared to 2015 so surely surely
before I think you are misunderstanding
C++ unique pointers no no no no no I
okay let's clarify here let's clarify
here I'm saying if there was no move
right you stood unique pointers um they
this this version doesn't exist it's
impossible because unique pointers
cannot copy they can only move what I
was saying if you did not have moves you
would you couldn't have a unique pointer
that didn't double
free uh at least without like some sort
of like special
API uh okay what were we doing I got
distracted talking about languages again
hey thanks verifier that's a nice thing
to say nice thing to say I think move
was already a thing when they introduced
uni pointer the problem is with earlier
Auto pointer I'm pretty sure unique
pointer came in in C++ 11 with st move
or like with move semantics I'm like
almost
positive we can look this up unique
pointer CVP
reference um since C++ 11 and I'm almost
positive that like move assignment
this came in in C++ 11 I'm pretty sure
as well does it tell
me uh no not really
huh not really but like I mean I'm I'm
like 95% sure that this was C++ 11 as
well uh okay okay uh we were doing
something and it was in the in it thread
we want to make sure that we respect the
line breaks now so here when we like
append stuff I guess we just have to um
append indices as
well so here we're just going to do this
same
Loop where we like look through segments
and apply indices but we're only going
to do it for the segments that we just
added so it's going to look very similar
but not exactly as similar so I don't
know if I'm going to want to factor out
a function yet we'll see how it looks
like when we finish um but we're
basically going to
say um H we're doing this on a per
segment level
here so so we could either like stash
last segment here oh fifier thanks for
the prime by the way thank you thank you
um and then like on first iteration
check if no like set it and then every
other iteration do a comparison that's
don't like that uh you can use
iterators and pull first segment out
early uh or you could use you could like
do a second walk at the end
uh where you go where you compare in
Windows where what I'm talking about
here is trying to like figure out which
segments that we are pushing right now
have a large gap between them I think I
like this version best so we'll do
something like um
f uh VAR last segment is equal to
samples
next um or
else uh
I don't know what to do here
you can't return because you need to do
this stuff down here uh but you could do
this if you you could get crazy and we
could put this stuff up here before this
and do it in
defer I don't know if I I don't think I
like that that's like really that that's
not easy to follow um what probably
makes more sense is to create a function
that lets us early return without a
problem I think that makes sense that
makes sense so we're going to say like
function
push
samples push whisper
segments
yeah yeah okay and he's going to take in
the
like uh whisper samples right so whisper
next returns something and he's going to
take in that thing so this is segment
it uh segment it is this thing
and we going to take in by reference I
guess
probably variables of snake case and
functions or camel case and Zig wild I
mean they just uh they in by convention
right you there's no there's no
uh you know nothing forcing you to do it
that way um but I think the the main
reason is because it makes it easier to
avoid uh name Collision so they have
like types are some type you have
function which is Su type returns Su
type and then you have variables Su type
is equal to Su
type like this uh sum type is a sum type
which is constructed with this function
sum type and so by having different
naming conventions for the type the name
and the variable you end up with like
lack of collision right otherwise like
if you had this some type then you can't
name this variable some type and that's
like annoying so I think
practically it makes sense to have
different different case naming schemes
for
different different
purposes
um okay so we are going to iterate
these segments and we are going to push
them into shared so I guess we'll just
Tak in shared by refence as well
um
yeah uh and this is
shared is that it is that just what it's
called shared is that the type name it
is okay um and then we just kind of do
everything in here
[Music]
so yeah it's kind of actually we don't
even really need to take the segment
iterator we could take whisper uh the
segment iterator makes more sense
here uh yeah so we will just call here
push whisper
segments and we will pass
in segment it and this should be a SE
called segment it that makes more sense
I don't know why we called it samples
it's not really a sample it's like a a
section of text so calling it a sample
is kind of
weird um what's the other thing
shared yeah
okay uh this is certainly
failable I'll be shocked if it isn't and
then just all of this stuff goes in here
but not the resetting of the
buffer which makes it
simple for us to do this
yeah although you could argue that it's
kind of a bug if this first thing that
we pull out is
null uh but maybe it's fine it's not the
end world I guess like is it a bug or is
it just kind of like a little bit of
surprise who knows who
knows okay so that makes it so that here
we should append that
segment and I guess we have special
handling here where we have to check
um oh no hold on VAR last
segment this like even on the first
sample or the first segment for this
like
run we need to compare with like the
previous run right because we're like
doing like seconds 0 to 8 and then we're
doing 8 to 16 and so to like ignore a
gap between 8 and if this like if the
first thing in this segment comes out at
like 10 then like ignoring that Gap is
silly
um so probably what makes sense
is we only pull out the first sample if
this thing is empty so we're going to
say
shared if shared
segments items
Len is
zero then we push
we do this Con segment is equal to
segment it
next or
else or else else jeez it's hard man
typing is hard and then here we can
initialize the last segment
with shared segments
items shared
segments items land minus one cuz now
it's gued to be greater than
zero uh and then we can just do this and
we can compare with the previous
thing do you have multiple buffers open
inm or is it multiple PS of a terminal
multiplexer multiple buffers in Vim I
use Vim buffers a lot
um yeah I don't really have much to say
about it I just use them uh and then all
we have to do here is we just have to
say instead of doing a and b and this
like Loop
here we just have to say if last segment
end is less less than or equal to
segment
start then we add the
thing that seems really straightforward
actually uh split indices doesn't exist
here
shared yeah okay
and is that it he doesn't like this like
Undeclared segment it so this probably
has to be like whisper Runner segment
iterator probably he doesn't like that
sample text doesn't exist here so we
have to call this
segment text and here we're suppos to
calling this segment iterator
and
um
whisper I see why we named it that now
because we didn't want to get confused
by our like converg converted segment or
segment and that segment but it's okay
we'll do this whisper segment text there
we go we don't have a threshold but I
think that comes from
shared uh split indices append I +1
doesn't make sense anymore um but I
think that we can just say that it's
shared segments items Len minus
one so if we just pushed in the second
element at index one then the new line
is appro is like associated with index
one which I think makes
sense yeah sure it
yeah um and then here on each of these
guys we have to say last segment is
equal to
segment there we
go
um I
wonder I wonder how smart the compiler
is I wonder if in release if it's smart
enough to like Ping Pong between two
different different memory locations so
that it doesn't have to do this copy at
the end I'm sure it doesn't actually
matter but I just L like I don't
know popped in my head I'm curious now
but not enough to look it up you know
any good reason somewh to start start
with the F API uh just just read the
examples man like demo decode see uh I
think that the docs are um but if
you look at this demo decode example at
the very least they're calling the
functions that you would want to call um
and then FF play in the ffmpeg repo is
also going to call the things that you
want to call that's kind of where we
went through it um CU yeah I don't think
you're GNA reading the docs is like
frustrating to some
extent uh oh dude off stream off stream
I hit this bug where I was I was trying
to uh upgrade to FFM 7 and I oh
no no that wasn't it it was
uh I can't remember but I accidentally
called AV format
find stream info I accidentally called
this function twice which you you would
think is not a big
deal right because like what in here
would ever tell you that you can't
do that right you would just
think that it would reinitialize
something that has already been
initialized with the same data it would
just be wasteful um but this resulted in
a seg fault because in with inside your
AV format context not in a variable you
can see here in the like in the
uh C style inheritance you kind
of do something like this where you like
have like a a big thing
a and then a smaller thing B inside and
then if you like hand out references to
B as a pointer you can like figure out
where a is based off of B right so you
can say like oh B is like 10 bytes into
a so if I do like B B minus 10 I now
have a and I can like do stuff with like
the other data in here so what they were
do what they do is in AV format find
stream info which you have to call by
the way to get the stream info uh
they they have some like some sort of
like context pointer in here and they
just like use it they look at all the
stream info and then they free
this pointer and then if you call the
function again you're just like
dereferencing null like null pointers or
like not null pointers you're de
referencing pointers that point to
invalid memory and then your program
just like crashes and you're
like I spent like so long debugging it I
like pull I like built ffmp from Source
ran with GDB set watch points on the
variable to figure out where the it
was ever being modified found it was
freed and I was like why the are
they freeing this and then found it was
in the function that I was calling and I
just accidentally called it a second
time but it's like oh my God man
like what like so all I'm saying
is uh the docs don't say that right and
which is like when I write software I
don't write good docs either but I would
like reading the docs of FFM Peg as the
mechanism for which you learn to
implement it's like you're just not
going to get it everything so like
source code examples FF play like what
you're going to have to go through is my
opinion on
it uh okay distracted distracted
um the doc should don't call it twice
yeah they should they should that would
be great but I mean they don't and
that's not something that is in my
control well maybe it is in my control
but not to the point where I want to
change it right like I'm not going to go
file a PR to fix the docs because I'm a
lazy piece of but
you know it's like you can you can
control what you can control uh you're
on F 7 I think I'm on a six right now
because that's what the what's in the
Nyx repos but I tried with seven briefly
when I was experimenting with self
compil compilation it was okay like
nothing nothing heavily
broke
um okay so here wherever the we
were we push the whisper segments oh
actually here we can we can use that
same function
now which is kind of
cool which is kind of cool
cool uh so we
do segment it and then we just do the
same function
again and then all this goes
away and I think we're
chilling so we need to set the initial
value to something sane
probably
um split threshold so where do we
construct
shared shared from init
data const default split threshold is an
F32
of half a second one second how long are
there like how how often am I living a 1
second Gap in my by talking I guess
we'll just try it and see what happens
um split threshold is equal to default
split threshold I guess maybe it makes
more sense to just actually assign
it
here I can do that so why wouldn't
I why wouldn't
I okay and then I guess it kind of makes
sense as well for us
to uh write down the split threshold
when we save the file right CU like if
we if we pick a threshold and then save
and then close and open the project we
want the same thing back so our like
save data should have a split
threshold that's an F32 and then we just
slap that in here
uh split threshold yep and then when we
recreate this
guy from init
data
um yeah side note uh for those of you
who are watching
yday uh I bit the bullet and wrote a
little parser that does like some kind
of aggressive logging for when things go
wrong so we were talking about yesterday
using like uh the stood Json uh way of
like you you feed it a struct and say
hey can you parse this struct for me um
and I didn't like that because it
doesn't log nicely un fail like when it
fails it just says like Oh missing feels
you're like well what's what's the
missing field so I just kind of like
banged up my own shitty version of this
that throws like has some logs as it's
parsing so he like walks the struct he
goes like oh is this a float thing let
me check if it's a float and if it's
like oh I tried to par the float and it
didn't fail I fail to par the float and
then he's like I failed to par the float
and then he says oh the field failed to
parse this thing for this struct field
in this struct uh which is kind of nice
kind of nice in fact we can probably
demo right here cuz it's going to get
mad that we don't have a field in our
struct from the previous version of this
of the schema I guess we can call it we
have no schema upgrades so get
us
um word time samp generator 368 he's mad
about something he says split indices
doesn't exist but oh here split indices
is equal to split indices and then I
guess we have to Alec that
thing split indices and then the thing
is a uze
here uh and then down here we also need
to do split indices we will split
indices all
right I think we're
chilling uh oh we have to we have to
adjust that thing too split indices is
parsed from split
indices which kind of means we have to
write down the split
indices do we need to does it make sense
to just recalculate them on start or
should we just write them
down like the writing them down is kind
of
useless
um yeah I think I'm not going to write
them down I think I'm going to pull them
from the pars
threshold and then just uh when we load
we'll just rerun our
uh share from init data
we can just rerun our
thing
um yes I think we can just call set
split split thresholds I don't think
there's a problem
there I I don't think we'll find
out uh so we
say VAR R is equal to this thing which
is
whisper then we call R set split
threshold
with uh
shared split
threshold uh and probably we have to do
a little bit we have to get a little
clever
here we have to do like shared mutex
lock shared mutex unlock
is this thing failable no uh cons split
threshold is equal
to
this and then we can call it like this
because he's going to relock the mutex
underneath that's like I guess the
technically correct we do
it let's see what happens error is
ignored at
313 okay fine fine uh type is wrong at
354 what is it complaining about is not
indexing why type pointer to segment
bounds does not support indexing oh am I
just supposed to copy this thing I guess
this is a pointer so I can just do
this
okay uh expected type segment bounds
found usze at split indices do append I
+ 1 at
358 uh cuz this should be uze of course
uh then it expected this thing found
that thing at line
216 so this should be segment
dot push whisper
segments oh we need to construct our
segment out of the thing okay so this is
the whisper
segment
okay and then no name text in segment at
225 so this should be whisper segment
here are we chilling no we're never
chilling we are never chilling ever ever
ever ever
ever um split indices append why doesn't
he like that no field threshold s and
shared oh is it just
called why
not oh it's called split threshold
that's a
weird um I kind of want that to be at
the very least called uh split threshold
seconds at the very
least at the very
least uh this now has to be like this
and then probably in this version we
also want this to be split threshold s
just for
clarity just for clarity
346 this needs an s on it as well cuz
the rename in the language Server
doesn't cover those like I guess like
inferred structs oh so here we're
demoing the nice little errors so he
says field split threshold s is not
present in Json data for word time stamp
generator save data isn't that sick
that's so much better than it just
saying missing field because now I can
just go fix
it uh because I can just open test
Json
and in here I can just say split
threshold
s is 1
.0 yo sick okay uh now the question is
do we see any new lines
no should we see some new lines maybe um
let's check if that code is
running at the very least oops memory
leak that's not
good that's not good at all I shouldn't
write memory leaks we've never written a
memory leak in our
lives and we will never write another
one in our lives this is the only one
ever actually
uh
probably it's that we didn't free this
thing
oopes the pros and cons okay okay so why
are there no splits in here let's check
that the code is running right so on in
it we call set split the should and can
we just at least log like
hi smiley
face this should get spamed we should
see a lot of smiley
faces yo yeah okay so you did do it and
it just must be that nothing was above
the threshold so can we like try
reducing the threshold and see what
happens uh test Json so if we change
this to
like
0.1 out of curiosity just to see oh wait
we're rendering it yo what's up what's
up what's up right we're not
doing anything with it so of course of
course it's not doing anything of course
um let's let's log the split indices
just to see if it looks kind of
sane um stood debug print split
indices indices is
any
of split
indices your
okay so we did get some not a lot right
this is a pretty short list for how much
text there is in here um but it's
something okay okay okay okay so I guess
on the gooey side um we should do
something with these so here we'll say
that we are at 50 minutes in holy
and we're looking
at um show line breaks in
UI I think I went with do 7 Seconds oh
you've done this already nice nice nice
nice uh get add- you get commit DM
smiley face I'll fix it later I'll fix
it later just so you guys don't think I
just need to prove it I do need to I
need to prove it guys it's not like I'm
committing smiley face messages
I want to make it clear that I do not
support bad get commit messages right
these have valid commit messages with
valid descriptions I just don't do that
part on stream okay we don't have to we
don't have to Flame me for writing
smiley face on every commit
okay I just I needed to get that off my
chest or else I would think that you
guys might think that I think it's okay
to write SM face I know it's not okay
I'll just fix it later I'll fix it later
everyone in chat smile smile
smile uh okay uh right so let's look at
the rest
side um we need to basically adjust the
text
rendering oh wait we don't even send
this these things over to the gooey side
yet so that's in app. Zig we say set
snapshot and here we're going to have to
add like a new line breakes
thing this is const U size and if we
have this this comes from uh
line split indices so I guess we'll call
this split
indices uh text split indices I
guess and in here we'll say text split
indices is equal to text split indices
boom boom bo boom
then over here when we set the snapshot
we clone the snapshot that we brought in
and he is going to have to duplicate
these indices so that we don't have like
uh so like word time stamp map has
like
data who is accessed by like it's in it
thread right so he these guys are both
talking to this guy then like when we
snapshot we also talk to this guy so all
of these guys have to like lock a
mutex right which means that like what
we do is while the mutex is locked we
copy this stuff out and then unlock the
mutex um just so that like there isn't
like large mutex contention in general
right so that's why we have to copy this
so this is
uh uh text split
indices uh is cloned from text split
indices and this is also text split
indices and then R text split indices
equal to text split indices boom boom
boom boom and then here we also to say
text split indices
is onu
size
okay uh then we need to we have like
this like internal Zig representation of
these things which have nice little
slices but then when we send them over
to the r side they need to be converted
into like C representation which means
we have to take our slices and we just
have to say uh text split indices is the
pointer but then he also has a
len so we pull that
out here and then when we try to free
our like copied stuff from the WR side
because okay so over here we have like
the gooey thread and he gets a snapshot
from
here
snapshot uh which has this stuff in it
so he copies the data out here and then
he copies them again over here and then
the gooey frees it from over here uh
because again we have like the same
problem here that we had over here where
like two threads are going to be
updating the same stuff and so we have
to like clone the stuff in and clone the
stuff out so that the mute text is only
locked for a short amount of time you
know you
know and so we have to when we de
initialize this thing we have to come
from the C representation into the zig
representation um so that we can free it
nicely so this is
just this oops but this is no longer
text this is uh text
split split indices here and here and
then here we have to free that
thing
okay some people may be watching this
and be they may be saying this is why we
don't like writing Zig and we
want to write rust or C++ and I would
say that's a reasonable thought um but
I'm not you and so I don't have that
same thought but it's a reasonable
thought uh so so here we have text split
indices that has to go into the C API
and this length and this thing is going
to be we don't really have U size I
don't think in standard
in so we actually might want this to be
u64 just to
disambiguate this is mostly the fault of
the ffi no I mean some of it is the the
fact that we have to think about like
cloning and freeing
um through mutex protection and stuff
that problem just kind of like goes away
like this one is from the cffi but this
one is
not so somewhat somewhat because of C
but not
entirely um does standard int have U
size man stood in is that like so oh
hell it isn't there so U
uh you have the doctor hand running in
the like it's illegible dude it's gotten
so much worse it's actually like so bad
um when I write stuff down I I realize
that um I can't read it and what I do is
I use it as a memory trigger for the
things that I remember hearing and it's
just like that's so stupid
that's so broken I should have just
learned to use a pencil like at
some point in my life but just like my
whole life I'm just like I can't
read um
okay okay
why is it easy to understand why I do
something but when I try to implement
something in my own it turns incredibly
difficult that's actually like such high
praise it's like like I shouldn't say
High Praise but it's like very
uh how do I say this that's like what I
want to present like the fact that I'm
taking something that is like difficult
for some people to do and articulating
well articulating it well enough that
you're watching you're just going like
yeah yeah yeah yeah yeah that's like
super sick like unfortunately I think
kind of like the problem with this with
like I think educational content in
general which I'm not claiming that I'm
like educational content I'm not really
sure that that's the bucket I fall under
but I think the problem with like
educational content in General on
YouTube and stuff is that you think you
get something you're kind of like trick
yourself into thinking you get something
um but then as soon as like you're
pushed on it you really don't understand
it at all right so you like watch some
veritasium video about fora transforms
and you go like oh yeah I get 4A
transforms and then you like go and sit
down you're like no no I don't get for a
transforms and so I think that's like a
major problem because people think it's
like you you I think it's important in
general to have like a good grasp of
what you know and what you don't know
and I don't like this like world
where you now think you know something
and you don't and now you're like gauge
of your own ability is like wrong um so
I don't know what to do with that but
I'm saying that because you're like why
is it why is it like this and I'm like
well you just have to do it right you
just have to you just have to try and
eventually by doing you pick up the
skills you need
right I'm not saying that you have skill
issues I'm saying that like things take
practice um and I it's really I think
it's cool that at the very least I'm
articulating things in a way that they
make sense um and but it's like
unfortunately you do just have to like
put in the put in the time as
well um okay so text split indices is in
here we free it everything's happy and I
think we have enough oh we were going to
uh fix the fact that we don't want it to
be uiz here we want this to be U
64s so that we
can use this stuff on this ffi side
okay um which kind of gets a little
tricky now because the split indices
here are U sizes but I guess we can just
use them as u4s here as well and then we
don't have to worry about any like type
conversion
um and well that probably won't compile
so we'll just kind of figure out what
goes
on it's frustrating sometimes I must say
I feel the same way when I watch people
do things and then try to do them myself
feel I watch uh I was watching virtual
play track Mania and then I booted up
track Mania and I was like holy I'm
ass like it makes me so mad I'm like I
should be able to do the best of
anybody in the world because I watch
somebody on the internet who's the best
of world and then like then I suck and
I'm like why am I why am I
and it's like oh it actually makes a lot
of sense uh but it's it is frustrating
it is frustrating because you you TR you
you trick yourself into thinking that
you like you are that guy and you're
just
not not you like you know I'm saying you
is in me not you as in you maybe you are
that
guy it's WS D must be easy that's why
I'm not gaming anymore yeah it's just
more fun to watch YouTube
uh this seems to be a u64 as
well uh a bunch of these are kicking
around H you fix the slow program
closing from latest video Yeah somebody
at the end of stream pointed me in the
right direction and then I was like that
doesn't make sense and then I went off
stream and then I looked I was like oh
that's exactly the right thing to do uh
so yeah we got there we got
there is watching like watching po
playing fortnite makes want play yeah
dude it's
exactly for I also installed
fortnite again the other day I used to
play like so much fortnite I was like
okay not great but like okay um and then
I like hurt my wrist because my edit key
was on Tab and when you're spamming edit
at like rapid Pace with your pinky all
day every day you're just kind of like
inducing RSI at like maximum speed right
you're just like speedr running getting
RSI so I had to stop playing video games
on mouse and keyboard altogether because
my like wrist just kind of hurts too
much oh overall from like if I like code
all day and then game with my pinky it's
just like up now so I like switch
a controller and I spent like two weeks
just grinding edit courses on my
controller and I'm just like so angry
every time I boot the game up because
it's like I'm just not there I'm not
where I used to be and I watch people
play on the internet and they've like
gotten so much better than I ever was
and it's just like oh I'm so mad all the
time all the time man why does
everything I do piss me off
uh this guy needs text split indices
equal to some nothing
thing what else is he bitching
about uh cons 8 found cons u64 so
somewhere I just up the type
again this should be
u64 uh found uze or expected uze found
u64
text
split where is this coming from so this
guy u64 this guy should also be
u64 did I not save this what is he
complaining about oh is it because of oh
it's it's the the length is a u64 and he
doesn't like that so here I have to
incast okay
okay what I up there in cast
are you want to split keyboard nope no
I'm
not uh he still like
that expected integer Vector found
pointer to conent
64 oh I'm I'm just an
idiot text split indices L and then do I
still need the uh incast here
uh no okay expected this found that so
at
199 this guy is supposed to be
items did you check the shutdown Atomic
while in the wall Loop to skip the last
whisper call um
it's it's here we'll just show you we'll
just show
you uh so whisper allows you to feed in
this like abort call back so in the
middle of its full loop um it will call
this function at like every step of the
loop and if your function returns true
then it will go okay I'll stop now I'll
stop now okay uh so that's all we had to
do that's all we had to
do how are you not getting RSI then just
little confused I just stopped playing
fortnite that just like fixed the
problem which is a little sad cuz that
game is so fun man what a fun
game what's up what's up uh okay so
everything
builds uh and we just have to go into
the Go
Side actually not just fortnite I can't
play like I can't play like any first
person shooter most keyboard anymore
just hurts that's because of gaming I
see well I think it's a combination of
like coding all day then gaming all day
after right if I just cut one of them
then it would be
fine
um okay so here when we lay out the
text when we lay out the text
what are we going to do we are going to
chunk
by
section and we probably have to layout
each section
independently cuz that way we don't have
to like copy the data which is like
maybe maybe we could copy it maybe
that's not the end of the world but like
the the early Optimizer in me wants to
just kind of like lay out each section
individ independently and then stagger
them
uh so here we have this Galley which is
this thing we layout our
job and I guess what we need to do here
is we could make a layout job that has a
bunch of sections in it that's probably
the
easiest hey lzq hello
hello so the way that like this like igy
stuff works is they have like an array
of like
jobs jobs job
job and these are I can't remember if
they're applied to the same input string
or if you can like pass multiple strings
to the same job so let's take a look at
that because what happens then is you
have this like layout job that has like
a bunch of texts associated with it with
like different font sizes different
colors Etc and then you can like lay out
that whole job at once in one big ass
Galley so we might be able to leverage
that to just like inject new lines
between sections of text without having
to do any work
um so igui do
RS and we can look at layout
job and so how does this work we have
layout
sections that have
formats um and it doesn't look like we
can like inject stuff in there so it
looks like the layout job itself has one
string so I guess we probably want to
split up the strings also kind of like
interestingly
uh this is Cap string with a capital S
so we are kind of like guaranteed to
just copy all of the text while we do
this it is guaranteed and like maybe we
just don't have to worry about
it break on new line so I guess we could
if this is how it works if he's like
we're going to copy the whole thing
anyways we could just copy it and inject
the new lines ourselves but that'll make
it kind of hard to go backwards uh from
like text position to pixel position cuz
the API to uh the like Zig side is to
say hey I have like character
153
uh could you tell me what time stamp
that is and so if we have character 153
like it'll become 154 if we put a new
line in between before it so it would be
kind of nice to kind of keep that uh
text mapping that would be
nice so
I think it makes
sense I think it makes sense to create
multiple
layouts oh wait what does this do what
happens when you append a job what does
this do what does this
do oh it just like appends it to the
text
okay
um yeah okay we'll just we'll just lay
multiple things
oh you sto stop gaming but for but RSI
was still showing up working that's why
I was wondering how it was working for
you yeah I think just for me I just like
caught it early enough maybe maybe I'm
just lucky maybe it's not RSI I don't
know who knows who knows cuz yeah I hear
that like once it really sets in you're
just kind of like it's really hard to
deal with this is what I've
heard uh okay so maybe we do layouts
plural and we do something like
for uh I in zero to State
text what do we call
them what did we call
them text split indices
L we're going to do something I don't
know what yet but we're going to do
something so we're going to look at the
x uh let idx is equal to text idx is
equal to State text split indices add i
d
reference
and then what then what state text split
indices oh we have the index so then
we're going to look at uh
we have our like text as a string here
so we can do something like s from zero
to text idx we have to we can't just do
this we kind of need to like break out
early we need to break it early or not
break it early so we we need to like
start at wherever the previous text
index was so can you do
like
Windows two is it this is how this
works can you do like a like
this and then
start
end uh that's actually not how this
works
either cuz the first one uh I think that
our thing starts at like 5 10 15 I think
zero isn't in here we don't have a new
line at zero so instead what maybe makes
sense is we say uh let mute
idx is equal to
zero then we do this
stuff then we do this
stuff then we come from last idx to text
idx and say
text text idx equal to last
idx um and then we have to create a
layout job with that thing so we do this
uh but we do it
from
here right so we take this section and
we create our layout job and then we
must have to put this somewhere so we're
going to say that our uh let layout jobs
is like a mutable
[Music]
vector and then we can say uh layout
jobs push
layout or maybe we we we just store the
gyss um and then we just we also
run this as
well and then we have G push this
thing
maybe
okay then we have to iterate GS
here
um so I guess it makes sense maybe for
now we'll just allocate responses over
and over again so here we'll just say
for gley in G I think that's how you do
it in this
language uh and then basically
everything that we were doing here we'll
just do in a
loop and all we have to do now is we
have to maybe stash along with these the
galley
uh we have to stash the like character
position so here we just stash uh last
idx and then uh Galley start
index uh-huh yeah yeah that seems
reasonable and then car pause now it
just gets
uh car pause plus equals start ID start
idx
maybe that might that might be it that
might work that'd be crazy if it worked
but it's
possible uh okay so lots of stuff going
here so we need to cast our U 64s into U
sizes does anybody
understand why the does uze exist I
like understand that on like 64-bit
platforms it's a 64-bit and on 32-bit
platforms it's 32-bit integer but like
is it just because they don't like they
want it to fit in one register so that
when you index a thing oh yeah okay
probably like the assembly instructions
for pointer
operations probably work in the register
unit so if you're trying to say like I
want to Index this array at this
distance like if you had to store that
in two registers you would have to do
some sort of like funky
I guess compatibility but like
I don't really understand like
yeah there was a piece of me in there
that didn't understand why you couldn't
just use 64-bit ins on 32-bit machines
for indexing like it seems like
unrelated to the
the
architecture I guess like for a while 32
bit machines also had a maximum memory
size of 32 bits like two to 32 but
that's not true anymore cuz with like
what do they call it some sort of like
extension I can't remember it starts
with a
P pae
physical address extension
maybe uh but anyways it's fine it's fine
doesn't matter doesn't
matter uh it's funny that somebody in
chat is also talking about being 32
years old and when I'm skimming chat
looking for 32-bit stuff it's it's like
my brain's like oh I can't handle two
numbers that mean different or the same
number meaning two things it's too
hard uh okay so we were at the point
where this thing is E64 and we just have
to do uh
somewhere try
[Music]
into wrap is there anywhere else that
that's a problem of course it is of
course it is it's
here uh let I use size to I try into
unwrap that make things
simpler um uh and then this thing
doesn't have copy
so font ID which is now being used in
many iterations of a loop needs to be
cloned
fine fine and then uh text idx equal
last idx cannot assign I had it the
wrong way around it's last idx equal to
text
idx user space still only sees two to 32
memory bytes on 32 platforms even with
pae oh interesting
interesting uh okay does it look like
we're line breaking oh
maybe I mean it doesn't look like we're
line breaking at anywhere that's
reasonable actually start actually
actually start like actually you're a
dog or
like what is going on here
like the the fastest path to dog fooding
uh using your own stuff or eating kibble
depending on if you're a dog or like a
person yeah it feels like it's off it
feels like it should be like
here okay so we've got something we got
something oh also embedded devices oh
yeah I guess some people really are like
16 bit
huh yeah I'm not really convinced that
this is splitting at the right spots oh
let's just double check that like the
the tech the mapping is still correct
though so do I hear widget when I click
here that audio widget that Audio widget
widget maybe it's a little off but maybe
that's normal
indicate to back to indicate yeah I
think that that's working it could be
that the token like the splits might be
midw um so it be interesting to maybe
look
at look
at the actual
data uh how what's like the simplest way
to do that depending on if you're a dog
top inclusive educational content I
think I was talking about dog
fooding which is just like a stupid
term I used to threaten my boss
and tell him that I would report people
or report him to some labor board
because he made me eat kibble during at
work and he's like well you have to eat
it you have to eat the kibble I'm like
yeah but you can't tell me I have to eat
the kibble I have to do it on my
own what a funny
term
uh okay yeah let's see
let's
see I don't know what I want to see I
want to see something I want to see
maybe in the whisper side word Time s
generator maybe we log the segments that
we calculate here in
uh in set split
threshold so maybe we'll go and we'll
say let
last uh not let in this language VAR
lasts split it index is zero just like
we're doing on the other side and we'll
say for self shared split indices do
items we have split index and then let's
just stood debug
print uh the things at those indices so
then at least we can bisect if it's like
a rust bug like a rust ffi bug or an
actual split calculation bug or maybe
not a bug at all we haven't we won't we
won't be able to rule that out but well
at least Le know what it thinks is
correct at this
point um so this is going to be some
string with a new line and it's just
going to
be self shared text from last split
index to split
index and then last split idx equal
split index
easy of course he doesn't like that
because he's a comp time ant and you
can't modify comp time ant how dare you
how dare
you is there a meaningful way to debug
this visually not really sure not
sure uh okay so here he's splitting at
and Y you no in the middle of the word
you so if this is a bug you if it is a
bug um it is at the source so at the
very least we have confirmation that
this part is working
correctly
um it would be interesting to look at
the tokens that come out we can just
print the
tokens uh and that will give us another
clue another clue for sure so let's
where were we printing here let's just
tack on the same
uh for self shared segments we have
segments that came out of
whisper and we can just say that we want
toint print the segment bounds so we'll
say uh
start
end yeah and then we'll print the thing
so this will be a segment this is a
segment singular that makes more
sense and here we're going to look at uh
self shared text items from
segment car start to segment car end
plus one maybe I think I'm not sure if
the car end is inclusive or exclusive
which is something that I should be
aware of to be honest uh length plus
length seems like it's
exclusive
uh I always for items when working
with arrays or array lists I forget
every goddamn time every time man
okay so
uh let's two and one
less okay
so before we were seeing something like
a big break in midw in
you uh but it would be nice to what's up
what's up what's
up this is in after Pacific time on
Twitch if you want to sing by what do
what swing by oh swing swing and the W I
didn't pronunciate or annunciate it well
enough okay so it's after sing by and
yo
Sing Sing by and you
know so it doesn't look like
this is doing the right is this
splitting at the right spot right like
we're not getting
midw in
here uh let's just double
check Pacific Time twitch if you want to
sing by yeah and so like this is where
we're seeing that split which is just in
like the wrong
spot and what's interesting what's
interesting is that
the we're using the same index indexes
to split here yo what's up what's up
maybe yo what's up what's up what's up
yo what's up what's up shut the up
me I should start it muted
it's too much it's too
much uh okay so it's that the split
indices these come from
where
I oh I'm just an
idiot uh we just need to
write B start here we were splitting at
the segment not the text itself right
because we have we've got like three
levels of inter Direction here I think
maybe right you have
like uh
segments who have indices into
text and then we're iterating these
segments and then we have
split
indices and when we're constructing
these we were iterating these guys and
then storing their indexes here but we
were supposed to be looking at this
thing and then starting this index in
here so I think that's fine I think it's
fine I think that's the bug to be honest
start oh thank God there's
different types because uh I just used
the the time stamp instead of the
character index that would have sucked
as
well okay yo what's up what's up what's
up okay so let's now see
if this correlates with the thing
so thought is is that I can read uh
there's actually just missing words in
here which is kind of funny so like it
just didn't get this word but that is a
big gap okay um these are just too long
right
but it's possible so we have like the
audio widget
rendering why isn't there line break
here why did it
crash whoa whoa whoa what the just
happened aborted Cordon what the
yo what the was
that what the
it's possible so we have like the
audio widget
rendering possible so we have like it's
just missing all of this which is
weird is here is and then why isn't
there a Gap
here right like timestamp Wise It's very
aware that these are 3 seconds
apart cuz you can see you can see it
jumping as I drag the cursor and you can
see the
uh how do I say this what the was
that what the what the just
happened
while indexing this thing I'm just like
I'm I'm go indexing invalid characters
which is
like
um Miss start of a clip I had to add
noise floor to make my reset work
properly makes sense is it always the
same caps could be a timing issue I
think the fact that when I'm hovering
that thing um I'm getting invalid
indices here is like an indicator I'm
doing something really stupid
I just don't know what it
is I don't know what it
is can we see if uh okay
let's let's RM test
Json I was confused earlier but an add I
when I thought you wanted to index into
something uh it's because pointer
arithmetic in Rust is like a pain in the
ass um and so you kind of have to do it
that
way uh there is a bug now so where it
crashes on if there if the save is empty
so let's just look at frame six oh it's
because this is like a p segments Len is
zero yeah so we just have to do
saturating SUB
there um which is fine which is
fine saturating so please there we go
yo what's up shut the fu upad space and
like being able to shut up okay and
we're actually not seeing anything here
anymore oh because the GDB is
just like just spamming thread messages
and so the thread's just taking forever
so we just leave GDB I guess
yo oh no wait error save files in Jason
fail to load save okay that's fine split
indices that that's also fine why the
is our init thread not
running oh is it because we have no
split
thresholds set split threshold like if
we have no
splits uh then we
don't there's something about like if if
we have no segments where there's like a
big gap
um then we will have an empty split
threshold list and if we iterate those
to render then we have a problem so we
kind of need to keep like our zero and
text Max these need to like be injected
manually so if we go into lib
RS uh we just have to do one more of
these
iterations to the text line
at the end of all
this so we just
do text
idx last idx I guess to uh s
l and then we push layo job
yeah I don't know if that's the only
place where we up but that's
definitely at least one place where we
up oh yeah okay that's
better umentation
like a fancy timeline implementation so
and you know we upload those
vide yeah something
okay
okay so the question is do I think it's
doing something reasonable
that that's something that I've been
like thinking
about that's something that I've been
like am I off by
one by and
you know we upload
maybe I'm might be off by one is there
no more new lines what the
want to swing by and you know we
upload those yeah so if I'm one word off
here that would make sense doing it and
it'll be more fun so the thought is
while I'm doing it and it'll be more fun
so there there's no real
excuse there's no real excuse here
uh let's do the can we print them out
again oh I had to sa I have to
hit save so that uh on the next
iteration it will parse these things at
load and then we can uh log them so
let's just wait for it to give some
amount of
maybe let's also visually make it a
bigger line break so that it's easier to
parse because I think that if it happens
to line up towards the edge we won't see
it um so can we do something like uh on
every iteration of this Loop just like
UI allocate
space is that what we can do
IGI
allocate
space uh yeah
okay so here it's just IG V to 0.0 and
like 10 pixels or something I don't give
I just want like something
[Music]
y there we go so here we have that
Gap okay and then
here it doesn't actually look like
there's any gaps in what it
parts so maybe it's just the thresholds
too large
but also this doesn't look like all of
the text am I
high oh because it's
uh honestly seems like you're missing
start the sentence now which is new
Behavior Uh
maybe what say we're working on blah
blah blah
I'm actually going to do it but we have
to try and or sorry blah blah blah blah
words tried opening stream this doesn't
look like it's missing anything but I
wonder if the rendering is this
matching so hard to
read think
BL I think this is the same what's going
on
here while I'm doing it and it'll be
more fun so the thought is yeah just
missing the thought is
here uh okay let's save and then
reload yo
okay yeah I am just missing the thought
is here that I can read faster which is
kind of
odd is it in
here uh
read I can read oh my God oh my
God yeah it is just missing the stuff
I mean it could also be because I
switched models so I'm using kind of
like the shitty model intentionally now
so that could also be a part of
it uh okay
let's try lowering the threshold and
maybe that'll give us another clue so
how do we here if we just say like
0.5
y do the splits oh what the
fatal runtime error begin is less than
end when
slicing 11374 is less than
2818 at
656 what am I doing
wrong what am I doing
wrong this is making it seem like
so here he's saying that the output
index
is small like I'm at like 11374 and it's
then doing
2818 at this line which is means that
this text index that's coming out is not
sorted which shouldn't be
possible I mean clearly it's happening
but it shouldn't happen because when we
construct these split indices we do it
by walking
forwards oh did I up I only maybe I
only fixed the
uh maybe I only fixed the bug where I
was using the segment index I might have
only fixed that in one
spot
um split
indes yeah yeah yeah here we go I didn't
fix it here so here we should
put segment start uh segment car start
yeah
and is that the only other
one oh you don't have to apologize for
trying to use common twitch lingo I'm
just a lazy piece of and haven't
set the set up any commands but you're
totally fine
to expect them to be there because
they're in everybody else's
chat don't worry about
it yo okay is that
okay maybe a chatot project yeah I
should do that at some
point okay so autogenerated from some
sort of like neural doing it and it'll
be more fun so the thought is is that I
can this one makes sense to me this line
break makes sense does this one make
sense generated from some sort of like
neural
network from if since it doesn't have
this from in there it like kind of makes
sense that it would skip that this is 95
to that's like5 seconds we're working
towards that um I think the next again
like it's skipped an um and so that
makes the distance between words a much
higher um I think the next steps here
are
like w for on the bottom we should be
able to like Implement some sort of UI
that lets us like drag and drop like a
left and right where am I I got lost I
wasn't paying attention oh we here of UI
that lets us like drag and drop like so
like why is drop and
like like there should this is a gap
this is a gap that's longer than 0.5 so
why is this not doing anything and you
know what we should do you know what we
should do is um when we click on a word
here let's log the pts that we clicked
that should make this easier to debug I
think that's
important um so we'll just say that on
response
on
response when we do our like pts
conversion we'll just log
this uh print
line
pts this thing F no we don't write
numbers in this one we just write like
this okay so here we have have fun at
7.84 and here we have at 72 so that's
like a reasonable split and if we just
like kind of slowly drag through
these I guess they are kind of
all close to each
other may I suggest that you change the
color of the token being spoken or
similar eventually I will do
that uh but I just talk got there yet
that is the eventual
[Music]
plan so so here the line break is oh 95
so it is like a second here from some
sort of like neural it's just the
problem is that we're missing
data but the splitting itself I think is
kind of working it's just that this is a
bad metric
okay
[Music]
um yeah okay that's
so there's a couple paths to go
down here uh one is that
maybe like the fact that we're missing
words here is like a major
problem you might need to you might need
to pre-roll when you said to do this a
couple days ago I might need to pre-roll
what I don't remember maybe I didn't
understand at the time and then I was
like I didn't maybe I just like ignored
I didn't know what you're saying what do
you mean when you say I might need to
pre-roll so there are few there are a
few options that I'm see I can think of
on here step back a few frames oh right
right right yes yes yes that was the
planner that I never did it but yeah
yeah you're right you're right
so um we are right now we are like
taking some chunk of chunk of audio and
then we're taking like a second chunk of
audio and then we're just kind of like
splitting it in the middle and so like
if there's like a word
here it's just like whisper will just be
like oh what the is
that so there's a chance that there's a
bug there uh there's a chance that we
are up more aggressively than
that right uh there's a chance that the
whisper model that we're using uh I'm
using like the shittiest whisper model
possible and that might be causing like
additional problems on top so one thing
that we're probably going to want to do
here is like kind of overlap a little
bit here and then uh discard like the
last
n% of this
thing
um although I don't really know there's
like some sort of like
intelligent thing that has
to than that I feel like that's it's
more broken than that as you say yeah
yeah I do feel like there's something
else going on here
um I should try like the bigger
model
um yeah we might have like kind of like
jumped the gun here where it might not
make sense to try to do something like
this until we know that the the text
generation is Rock Solid but it's also
hard to validate that the text
generation is good when I can't like dog
food the editor because it's so hard to
read the text um so another thing we
could try is we could just try breaking
on periods that are spit out
from from whisper right we have periods
in the text
somewhere and if we tried splitting on
those maybe those are also naturally
good spots to break and then at least we
can like debug the issues from that but
even even this to be honest even though
it's
wrong um this is still a lot better than
nothing because like at the very least I
have like spots to like latch onto with
my eyes which are kind of
useful are you getting any random texts
yet I had some issues with nonsense
output um I did have a little bit of an
issue earlier today
um caused by floating Point resampling
errors uh so I was doing something
stupid where I was
calculating where in the the thing to go
to based off of a multi I did like
sample rate of the audio divided by
Whisper sample rate and then multiplying
in flo floating Point space and as I got
farther and farther and farther in the
audio it got like dumber and dumber and
dumber like it got to the point where
like an hour into the video it just said
it was like gunshot noises birds
chirping dogs barking and I'm like oh
that's not good so I fix that and it
seems like it's like you know this feels
like okay
into doing the cleanup stuff which like
that those are like correct words um you
got weird stuff with sound is ah okay
I'm getting like blank audio which seems
okay um okay so what let's what are my
options what are my options here it's
like uh
fix uh overlap of
what's it
called uh segments fed to whisper we
have a upgrade model that this one's
easy to try so we might as well just do
this right now we might as well just
upgrade the model right
now uh so we can look
at I think in whisper CPP I've
downloaded the normal model and I was
like we'll just like use the shittier
one for now until this one looks works
but
um it looks like I lost
it so let's download their sane model
again this one is like I think least
likely to be the problem but also the
easiest to
test you
know oopsies I can't copy it
like that I
guess uh I want to copy this now
copy paste okay
give him a sec he's got to think a
little bit fetch from the internet so
now we have
models base Ian bin and we can try at
the very least seeing if that's any
better I doubt it will be but it's the
easiest one to test for sure
so we can just make it
work whisper large is not zillions
faster than medium and small
IRC slower you mean large would be the
slowest one I would think
uh where's my embed file oh there's not
much of a difference in speed oh so it's
just quality or like I guess memory
usage I would have thought that
the I just kind of assumed that they
would be faster because like the the
size of the model is directly related to
how many floating point computations it
does so I would have I would have
surprised that it's not a problem but I
mean if you if you've tried it then you
have more info than me but that's
surprising to me yo what's up what's up
what's up welcome back everybody today
we're working let's pause this so I
don't have to listen to myself and let's
just see
if are there just no breaks happening
here oh because I didn't uh I have to
change the uh the save file to have 0.5
in there I forgot I
forgot uh so it was here 0
.5 yo
today so the thought is yeah we're still
missing the text between fun and here is
that I can read so there's probably we
probably have to fix that buffer
thing um okay so that's kind of this one
out of the way fix the overlap and then
we also want to look at uh maybe trying
split on
punctuation
um and then like
adjusting I guess that's kind of like
the two the two threads I have to pull
on right
now adding more um but before we do that
we need to be able to correlate yeah the
fact that there's just like 10 words in
there that are missed that's got to be
it but then again like that's so much
time um but before we do that we need to
be able 66 um but before we do that we
need to be able to 68 so there's like a
whole second of data in like 8 Seconds
of whisper data being fed in so uh we
also need to look at uh listen to
missing
areas listen listen to sections of audio
that are missing text I think this is
probably where we should start to be
honest uh
so okay I think that Mak sense I think
that makes
sense okay um it is an hour 50 in though
so we're kind of at around the 2hour
mark and so I kind of like to call it
around two hours so I think we I'll try
I'll try to restrain myself from doing
too much off stream um and by too much I
mean I I'll try to restrain myself from
making any progress because I think this
will be interesting um but it's not
something that I think I'm going to get
done in like 15 minutes so I think
tomorrow we'll come back and we will try
like trying to understand why whisper is
missing stuff and uh move on from there
so thanks for watching guys um if you
like what you saw we stream most days at
around 2 o' Pacific time for around 2
hours uh right now we're working on this
video editor but of course there's like
lots of other stuff that we've worked on
as well so if you check out the YouTube
the YouTube link in the twitch
description uh we have all of our
projects organized by by playlist as
well as on GitHub so there's stuff like
operating system neural network uh
terminal emulator QR code detector like
you know there's stuff in there if you
want to go check it out it's around um
if you're watching on YouTube there
should be a twitch Link in the YouTube
description and if you're online between
two and four Pacific time you should
swing by and say hi um and what else
what else uh like subscribe twitch Prime
all that um I think we're going to say
bye to YouTube and then twitch people
let's figure out if we have someone to
raate