well hello friends
welcome back to browser hacking today we
are going to work on some browser
architecture stuff
so um we have this issue right now
that the javascript engine that we
instantiate
when running in the browser for example
here we open a page that has
a little bit of javascript on it and
what we do is we create a javascript
virtual machine
or a js interpreter and then we execute
the script
but the interpreter that we create it um
is owned by the document so this
web page here owns an interpreter so
when we navigate away from it
that interpreter is now gone and
this works more or less okay
for simple pages but as soon as you
throw in
frames or things like window.open
then you need to be able to have um
multiple global objects that exist
within one
javascript virtual machine because like
different frames need to be able to
access each other
if if it's allowed for example
so this is something that our current
architecture
does not allow for so um
for the longest time i've been meaning
to do something about this
and um it's just the kind of thing that
i keep pushing
in front of me instead of dealing with
so today we're just gonna
like hunker down and deal with it um
so basically let me show you
where the um interpreter sits
so this is the um dom document
object in libweb which has an m
interpreter this is the instantiated
javascript interpreter
inside of a given document so whenever
we encounter a script element
such as this one right here inside of a
html document then we will instantiate
this interpreter if not already present
and then use that to execute javascript
um now in the new world
um we're not going to be able to hang
the interpreter off of any specific
document object instead it needs to be
sort of um
i guess like more of a floating thing
that exists
somewhere else and um
a document instead needs to create
um a dom window object
and um whenever you want to execute
scripts in the context of a document
then you take your dom window object
and pass that to sort of our floating
global interpreter
and say hey here is a piece of
javascript code that i want to run
and this is my global slash window
object right
so that's sort of where we want to get
to
um now
i'm not exactly sure how we're gonna
approach
this so i guess we'll just start by um
sort of restructuring the code the way
that um
in a way that would facilitate this kind
of architecture and then
we'll see if we can coerce the code into
complying to the new architecture
so first thing is we need somewhere to
instantiate a single global interpreter
inside of the
um inside of this thing that's what i
was thinking but
let's see the interpreter has
a global object
so um that's currently one of them one
of the flaws here that our interpreter
object
the way that you construct an
interpreter is that you pass it
the type of global object that you want
to use and then you get the interpreter
constructed together with an
with a global object so constructing an
interpreter implicitly constructs a
kind of global object and then you're
stuck with that global object you can't
switch
the global object that runs in an
interpreter so actually maybe that's our
first
architectural issue that we need to
address
so this is the js interpreter class and
if you look here at the bottom we have
a global object pointer
so
really what we would like to do is
instead of
instead of having a global object
permanently here you would
provide a global object when you tell
the interpreter to run some code
and it will run with that global object
until that code exits for whatever
reason
um i think that might be
a decent first step
another approach could be to
separate out so the interpreter would be
sort of a thing that you
almost create like on the stack where
you want to run some code
and it would have it would have more of
a like a virtual machine object
that is persistent in memory and then
um so you would have a javascript vm
object
that sticks around and then when you
want to run some code you construct an
interpreter
and then let's see how would that look
like so it will be something like
like say that you want to run some code
so you have your like um
your like vm my global
pm something like that
and then you would you want to run some
code now so you have
string my javascript code
um and then you would
create an interpreter
uh maybe like passing the vm
and maybe also passing in my
window object so that's like the global
object
we'll say dom window object
um and then interpreter
run my javascript code
that's kind of cool
because then the interpreter would have
the call stack
and exception state things like that and
once execution is finished you can just
throw all of that out
i kind of like that that's a pretty big
architectural change though
but i guess
so what was my other approach my other
approach was essentially to
um
to just have a persistent interpreter
global interpreter saying and then when
you want to
run my javascript code then
um you would do
something like interpreter run my
javascript code
in the dom window object
and then the interpreter will always
stick around
but while it would be running in the
context of different
global objects or window objects
depending on what you pass to run
so in some ways this approach
feels like a smaller step
but this approach feels a lot um
better architecturally because this
approach
allows the interpreter to to be tied
to a given global object
because the the interpreter is transient
whereas here the interpreter sticks
around so it would have to
like constantly retarget itself to new
global objects and
maybe this is easier to reason about so
let's go with this approach
i think i think that'll work
man this this will be a bit of a
bit of a refactor um okay so let's
let's just tolerate it and see if we can
do something incrementally here so we'll
need a vm
object or a virtual machine let's call
it vm
um
okay
run time
time all right
jsvm
hello hello
so what is a vm
a vm vm is this thing that sticks around
always
so this is where we will hang the
javascript heap for example the heap has
to be shared between all interpreters
so that it can so that we can persist
all of these global objects
so it has to have a heap in fact it will
be
responsible for creating the heap so
we'll just make the heap a value member
uh heap okay there's our heap
and we'll provide a friendly
friendly accessories for the heat plus
thing
um
and
i guess we will maybe we should make
this thing reference counted so that if
you want to
let's make it easier to to share
ownership here
wait wait
do that make that private and then we
will create a static
um
thing
that seems fine okay
um okay so we'll start with this
and in vm cpp
hello we will
simply include
um
[Music]
okay so right yeah okay heap needs to be
has some constructor arguments i guess
let's take a look at that
oh right so the heap has a
back pointer to the interpreter so
currently today the architecture is that
the interpreter object holds the
heap by value so this thing is going to
move
so the interpreter will instead
have a non-null ref putter to a
virtual machine
and when you create an interpreter you
will have to provide
a virtual machine okay so this will be a
pretty good
first step i guess for the refactor here
[Music]
runtime vm
um
i don't know i don't know how far far
we'll be able to take this in
in one video like um this
has a little bit of a
large uh time sync feeling to it
but we'll see see what we can do here
so i guess we don't need the const
variant there um
so
when we construct an interpreter we must
now provide
a bm ampersand
let's say
so then everybody who creates these guys
this yeah this will require
messing with a bunch of code because we
instantiate interpreters here and there
and they
now they need to make a vm first um
oh we'll take care of that so
keep go away the m is the new coolness
okay and
do we have anything touching and heap no
we don't
that's really good actually so
let's see so in the javascript
uh rebel program for example like what
breaks now because we changed this
so this is the way we instantiate an
interpreter in the reppl
so it's the js interpreter create and
then the type of global object that
we're using so rebel object in this case
and here we need to provide the vm
so i think what we'll do here is just
something like
um js vm
create
and uh
something like that
it's almost like like we could
i mean i'm kind of getting ahead of
myself i was just thinking that we could
give the vm
a function that produces an interpreter
because then
and then you don't have to um call
js interpreter create you could ask the
vm to make you an interpreter but
let's not get ahead of ourselves so here
we just use star vm again
and we are good
um so obviously this is going to break a
whole bunch of things so
let's just um go through these one by
one
starting with jsbm cpp actually because
here we have to say m keep so we're
going to give the heap a back pointer to
ourselves instead
h so instead of
the interpreter pointer
oh i bet that pointer is used in a
billion places
so that's something we have to deal with
but um we'll get to that so
we'll make a back pointer to the vm
like so and in lib.js forward
where we have all our forward
declarations let's just
declare the vm um
and here um
do we use this in a lot of places heap
interpreter
um there are two calls
not terribly many um
it's when you ask for the interpreter
from a cell
now that is used in a lot of places like
a lot
okay so we currently we need to be able
to go
from um
[Music]
from a heap cell to the interpreter
so i think at least as a stop gap here
we're going to need
some way for the vm to know the sort of
the current interpreter
so we'll move this out of line and
uh maybe we'll add something like this
return
mvm and
and the way you get to the interpreter
is that we just ask the vm for the
current
interpreter something like that
um
yeah i think and then we'll just figure
something out there and then here
uh this is when we start garbage
collection we
do a vm interpreter
um or maybe something like that even
i'm not sure yeah this is going to take
some untangling
but we'll push forward so
jsvm
maybe we'll have a
what if we have a stack of interpreters
so that if you
because i'm thinking like yeah we need
to have a stack of interpreters i think
because you could have a piece of
javascript that runs
that causes a synchronous
event to be delivered in the dom for
example
and then in the callback for that thing
you do something that necessitates the
uh switch to a separate interpreter
and then you don't you don't don't want
to lose track of the first interpreter
that brought you to that callback so
what brought you to that event delivery
thing
so we'll have to like push a new
interpreter on the interpreter stack
i think i i think that's how this is
gonna have to work so
we'll make um interpreter stack
interpreters
i i hope i'm thinking right about this
um
we'll see
we'll see i mean we'll find out pretty
soon if this is all wrong
so interpreter
and then here we'll assert that the
interpreters is not
empty and then we will return
interpreters
last starred
um because you should really only be
calling this from code that
um that needs the current interpreter
for some reason right so it will be a
bug to call it
while js is not being interpreted
and maybe we need an optional variant of
this
um interpreter if exists
interpreter if active let's say
what's this um
and um this is a variant of this
if m interpreters
is empty turn null putter otherwise
return
interpreters last because we needed that
when we do the
when we start to gather roots to do a
garbage collection because we can't
be sure here we might do um
we might start a garbage collection just
because the timer has run out and like
if we have this periodic garbage
collection mechanism right
um and then you might want to do a gc
when there's no interpreter active
so in those cases you might not have one
so here
we'll do something like this
interpreter if exists and if so
we will gather the roots from the
existing interpreter
if one exists
okay um and then
[Music]
how do we get interpreters onto that
stack though
um
i guess we'll have some api for that so
like
push interpreter maybe
pop interpreter
and
what will these do um
well i mean they're kind of simple right
but we just need to get the interpreter
on and off the stack so
in this case
we will simply
append it
to the stack in this case we will
i guess we will assert that the
stack is not empty and
um
we will assert that the
last item in the vector so top of the
stack
is the same interpreter that we are
popping so that we are not confused here
otherwise we might be popping the wrong
thing off the stack which would be
unfortunate and
then we have take last
i guess we can do this popped
take last search
we want to assert that they are the same
actually
there we go
okay okay that looks pretty better um
probably we don't need anybody but
interpreter to call this
so we could probably take a batch here
um yeah i feel pretty good about that
one
okay
so we are in test.js i guess let's go
and clean this up so that it builds
um oops this is the javascript test
runner program i'm just looking over at
the scroll bar here where the red is
so
[Music]
we need to provide a vm um
say like gbm
so the test runner i guess this will be
a good
test vehicle for keeping a persistent vm
um
we'll just make it here
static js
jsvm actually it can't be
like that we'll do it like this
or well we want something that we can
destroy when we exit
so
let's do it this way
okay so star vm
right here okay
and where is main i mean it's here
so we'll construct
jsvm create
and before we exit we will
put her the aforementioned vm
okay very crashy listen to crashalicious
um jsvm create missing of course
we never implemented that
so that's a non-null rav putter to vm
vm priyate
new vm
okay now i'm document
and here
okay so this is where the dom document
constructs its interpreter
if not already present um
so we need to provide it with a global
vm here so i guess we'll do this thing
that i was
saying earlier that we'll just um i
guess for now we can just make a helper
that returns one
like this say that it's static nobody
else needs to know about this little
secret of ours um
global vm
and
i'm thinking what is the reason that
so there will be more than one vm
eventually in
this process if
[Music]
if you want to use web workers
that's one will need more vms
but as long as you're not using workers
everything will go in the
in this global vm so this is like the
main thread vm i guess
so maybe we should call the main thread
yeah
yes and then because um a worker um
a worker is a similar context sort of on
the same
conceptual level as a document because
they are both
um script scripting contexts
sort of so
we'll need something once we get to
workers but we don't have any workers
so that's okay we'll get there someday
and we'll worry about that then but
right now let's just make a main thread
vm and
um if we had something like is main
thread then i would assert that here but
we don't have such a thing
but we also don't have multiple threads
in the web engine so
let's not get ahead of ourselves um
static rep putter two js
bn yeah
if no vm the nvm is
beautiful
so now we are creating a very different
execution environment for javascript
in the browser especially because um
now the javascript heap will persist
so when you go from page to page we're
not like
clearing out the heap we're going to be
reusing the heap
but that will also be significantly
faster than like
constructing a new heap and throwing
away that heap whenever you switch
from page to page
and it's just absolutely necessary to
support a bunch of
basic features that we need so it's not
like we have a choice
but it should remove um
some performance bottlenecks that we
currently have where you
we spend a lot of time allocating and
de-allocating heat blocks
when you navigate anyway um so this is
the
js interpreter in the spreadsheet
application
um so here it has a persistent
interpreter
i mean everybody does at the moment so i
think
for him we'll do exactly what we did in
the
dom document case we'll just like
make one of these poppers and
that's the global vm and then we'll deal
with that at some other time
i don't know exactly know what the
architecture should be here um
we just need to the spreadsheet
application is not a primary
concern right now the architecture of
that because it's
you know it's a new application and it
has
way fewer um architectural
needs i think than the browser
okay so this will likely break yes
because nothing
um we're not putting anything into the
interpreter stack
right i just added apis for it but i
didn't actually use them
so um whenever we want to run
code on the js interpreter
what do we do interpreter
[Music]
run right that's what we do here
interpreter run call run with a global
object
and a program um
at least this is what the spreadsheet
application does
so i'm thinking that the run function
can simply um
push itself
onto the vm so like
vm push interpreter
this and
um
let's use let's make an rei helper for
this because otherwise i'm going to
screw this up somewhere forget to pop
because you can see already we have like
two return paths out of this function
so interpreter scope let's say
something like this
maybe the exception state should move to
the vm
[Music]
nah no let's keep it on the interpreter
um
let's see class
interpreters go pay
um
[Music]
and wait a minute
um
wait how do we make this work we could
also do something like scope is vm
enter vm
enter something like this
and this object that it gives you back
that kind of works
although that's not nearly as expressive
as
[Music]
um
is this um
okay let's let's do it this way i kind
of like the way that
reads so that's that's why we're doing
it that way
private um interpreter
and interpreter
okay
interpreter
okay so interpreter
[Music]
um where is the interpreter here
um
oh now we are not in in the interpreter
okay so that badge actually
ended up not being necessary
and in fact we can maybe even get away
these not even being public
that'd be kind of cool
[Music]
well okay let's not get fancy let's
let's
stay focused so push interpreter
interpreter
and here m interpreter um
pop interpreter
right
so the vm will now transition in and out
of interpreters
i think
um still not good
so what am i doing oh i'm just trying to
construct the interpreter and it already
falls apart
okay so we're not trying to run we're
calling vm interpreter
we don't have an interpreter for the vm
um
why are we doing that so we're in the
construction of
window object constructing global
objects constructing object object
constructor calls interpreter let's take
a look at that
object
and here we are calling interpreter
so
uh it might if we just want to get to
the heap
from um from a cell
we should just call heap
although heap will yeah because this if
to get to the
heap from a cell because all the cells
are somewhere inside of a 16k
aligned block then we can actually
derive the base of the block from the
cell pointer by just masking its bits
and then find a pointer to the heap from
there
because each heap block is 16 kilobytes
it's aligned on 16 kilobyte boundaries
so
it's just a pointer masking strategy to
get
to the heat block and then you can find
the heap
from there so i think the solution here
is just not to go through the
interpreter which we were doing for some
reason
in fact if we look at interpreter we'll
see that cell interpreter
goes to heap and then back to
interpreter so this is just completely
redundant indirection
all right so then we are in
define native function
and what does that do um
i guess let's get a line number so that
we know what we're doing so
adder to line executable
as andy would say and 753
so straight here
so native function create with an
interpreter
now why does native function create
require the interpreter
uh it does not it doesn't even use it
so it seems like we can get rid of that
we're probably gonna find a bunch of
things like this
um like places where we're
calling interpreter unnecessarily but
that's like good solid cleanup
regardless
native function yeah but then like
i'm calling interpreter exception here
hmm this is going to break there
right define native function yay
because
so maybe we do need to move the
exceptions to the vm
[Music]
but maybe that's okay
maybe it's okay
[Music]
let's see i mean we can just try it
so
or
yeah i mean this is not going to work
otherwise because here we are
constructing
the interpreter
or wait what is this even doing let's
let's go take a look at it
so we're calling interpreter which calls
js interpreter create
um interpreter
so we are here and then we've
constructed the interpreter
and we are currently here calling um
initialize on the global object so
interpreter
window object initialize that's our
global object the web
bindings window object
so we have an interpreter
it's just that that interpreter is not
[Music]
it's not on the vms interpreter stack at
the moment
so a way to work around this
um to avoid having to move the exception
state from interpreter to vm
would be to just push ourselves on the
vm's interpreter stack here
um while we're constructing that seems
like a
good compromise otherwise this patch is
going to grow to like 25 000 lines
suddenly
so let's see it's vm interpreter scope
go pay interpreter
this but this architecture is kind of
interesting because
ultimately we don't want to construct
the global object here
um we want the global object to be this
thing that persists
outside of the interpreter so this is
oh this will eventually go away so
um we'll see okay so where are we now
now we are in event dispatch all of a
sudden
um so we are in document parser run
and we have a dom node where we're
dispatching an event
event dispatcher dispatch and from there
we want to call the interpreter so let's
go look at event
dispatcher and the interpreter is
fetched
by calling function dot interpreter
uh huh well now we can't really do that
we need to get the interpreter from
elsewhere because
it used to be that
[Music]
it used to be that any cell could have
i mean every cell still has the
interpreter getter but
if you're not currently executing in an
interpreter then there is no
interpreter i mean that's kind of the
new architecture that we're going for
right so um
here we want to do it perform a call
actually
which also is kind of a questionable api
for
interpreter but
that is an entirely separate concern so
call function this
value and what
the heck is the third argument here
wait call
marked value list is that what we're
passing
i forget how this works oh it's like
this
i see okay okay
right
so it becomes a marked value list
that's fair so
um
what do we want to do here um
well for now we need to get the dom
documents interpreter
how do we get to the bound document well
we have an event target object
and we need to get from him to the
dom document so
let's see
event targets currently they don't have
um
it's not necessarily a dom node like
event target can also be something like
an xml http request
which
as you can see here xml http request is
a dom event target
but it is not um a dom node
so it doesn't have a way to get to the
document
but we should have a way to get to
a script context script execution
context
from a given xml http request
that should probably be a thing that
event target has
so event target
needs to have an accessor
script contact execution context
let's say um
um yeah something like that
and then uh
what is a script execution context well
we'll have to define that i think we'll
put that
in web
bindings maybe
[Music]
let's say that it's done
okay
so
script execution context
um yeah this is going to be a very
um iterative process um
i'm definitely not going to be able to
finish everything in one video but it is
good that we've started
so script exec um
nice
name special web doom
and so this will be the the base or a
base class
that's shared among uh dom document and
um a worker
yeah so once we get web workers they
will also
inherit script execution context and
script execution context i suppose
will be sort of your abstracted link
between arbitrary um
js objects in the dom space and the
javascript virtual machine
some kind of a handle i guess anyway
[Music]
right now we just care about
uh finding
something so maybe we'll for now we can
just cheese this
so or wait what is it that i want to do
i want to oh i want to be able to give
you the interpreter
um
i want to get the interpreter so
for now this will be kind of ugly but
we'll do it this way
and put it fix me
so virtual js interpreter interpreter
this shouldn't really work this way fix
me
[Music]
just so that we um remember to clean
that up
okay so
then dom doc
needs to inherit this thing script
dawn
okay
and then we can say interpreter we can
just make this game
virtual override
and
oh right forgot to actually inherit
um
script execution context
very spicy okay so now that we are a
script execution context
we can also go ahead and say that
something like an
xml http request which inherits event
target
we'll say that event target requires
that you pass a script execution context
i think that makes sense and
when we so we'll have a reference to
this
[Music]
um
how's this going to work and then we
kind of want to learn if
if the execution context disappears
then we would kind of like to know about
it
but um so it should probably be a
weekable
so you can make a weak putter to it
and then that's how we'll store that
here in event target so
we put her to script execution context
so that we don't try to use it if it's
gone
okay and
script execution contacts
make a weak cutter
okay
and then when we construct these things
um
this tpp
so this guy needs to initialize the
event target base class here
so we'll just do this because we can
just pass in our document because
document is a script execution context
event target wait why is that ambiguous
you jerk candidate constructor has been
explicitly deleted
has it now
[Music]
y
candidate constructor 63
39 pretty much
node is an event target yeah
anyway i need um
i need help new xml http request new xml
http
so when we're constructing these guys
we need to pass in some script execution
context
which i guess will be our document
wait what's wrong with this oh it's um
abstract class marked as final
it's an event target that doesn't know
how to get to its execution context
um
maybe this thing doesn't need to be
virtual
over
can do like that and then
here in the constructor for this thing
we just call it then target
window um document
and again the ambiguous call
um
oh we're inheriting from we couple
twice
that's a little nasty
see event target with
this
does that work it's kind of weird
i'm a little bit
um skeptical about the
um class layout here
but we still have that double
inheritance yeah
so script execution context can't
inherit from
weakable like that because
john nodes are already weakable
that's a little bit frustrating
but i think maybe we'll just tolerate it
and
not worry about that and
we will use a rock pointer here for now
and then eventually we'll
replace this with some mechanism fix me
this should not be wrong
that double inheritance is a bit of a
there
okay so oopsy-doopsy then we will just
tolerate that
script execution context destructor
missing
makes sense we didn't implement any of
it
maybe this should be a bindings concept
i don't know why i put it in dom
probably move it later um
but whatever it's not
that's the most important thing right
now
okay so
we're still in the same space we just
need to solve this differently now so in
event dispatcher
right so this is where we were um so
instead of asking the function for the
interpreter what we want to do is ask
target
for script execution context
interpreter okay
script exec context
okay where are we now
now we are in interpreter call internal
okay so
now we're performing the call here
call internal
okay so these guys call call internal
and call internal
is apparently a public api of the
interpreter
much like run um so it's gonna have to
push the interpreter on the scope as
well i think that's okay
it is weird like the js interpreter
um is kind of uh implementation detail
of the engine and also a public api but
it's like
has all these different things on it
it's a very
um overloaded api but
in this new world where interpreters are
supposed to be temporary objects like
transient down the stack it will be nice
because we'll be able to scale back
what the interpreter does and what his
responsibilities are and so on
okay so we loaded
we got some ransom script to put the
user agent here let's open up a page
with some js on it yeah go back
so this is pretty cool so now um
everything should be in the same virtual
machine
what happens when we exit do we crash no
just exit that's good
um let's open something that
runs periodically like the set interval
test
okay there's a crash um
so dom window timer did fire
here we are using well we're asking the
wrapper for an interpreter
and
i mean so in the future the way that
this code should look is different right
like it should be something like js
interpreter
interpreter with rapper global object
something like that and then interpreter
call timer callback
um like this is what it should
ultimately look like but
we're not there now so we have to deal
with the fact that it's trying to get
the interpreter from the wrapper object
the wrapper object is the window down
window wrapper object which means that
it's the global object
actually the window object
so here
we just want to go and get the document
actually like that
so we can probably do this in a couple
more places
here we're getting function interpreter
like that we can just get
document interpreter
uh oh this is a callback oh hold on um
okay that one's a little tricky we'll
have to get back
something like uh this is for request
animation frame which
will um you'll get a call back in a
completely
um different context where you don't
have the only thing you have at that
point in callback is like
the address of the javascript function
that you're supposed to invoke
for the raf callback um
so there we don't have a good way to get
to the interpreter the way that we
currently set things up
but maybe it's okay to break that case
for now
and then just kind of return to it once
we're closer to the architecture that we
do want
at least something like this will work
now and let's try to open a new tab
so we're running the script here that
works and what if we have
um where is that setting tool
so this thing is counting up this guy is
counting up over here
eighteen five and if we go back it
shouldn't say
eighteen still right it should be like
twenty something twenty four
so this is pretty cool um
now we have two separate js interpreters
running in a shared vm so they have a
shared
javascript heap um and the interpreter
still hangs
from the document object but
um two interpreters one heap
basically so this is definitely
progress
um so i think we're gonna we're gonna
try to slow down here and do some
commits and stuff
because um we've already
changed up so many things
let's just do a smoke test load google
that did not work well
uh we crashed
when uh oh it looks like we're crashing
in a
garbage collection maybe or this this is
one of those crashes that you normally
get when you screw up garbage collection
um which is fair
let's test.js run definitely not okay
so we should probably make test.js run
um
let's see test js
sig albert
[Music]
what does test.js actually do
sjs create some damn test runner
run file test and here we create an
interpreter
and then interpreter run interpreter run
and then what
that's so weird though we can't see what
happened before here
um all right so vm
i don't know why the call stack looks
like this
that's weird so
let's see if we can just hack our way
around this to see what's going on
or maybe we can even use the debugger
actually because we have a debugger i
just
i just never use it
program exited well it would be nice if
it would
drop you into the debugger when you when
you get a sega board
okay so if it's empty then we will crash
instead
of asserting
okay awesome volatile
undefined opcode just hit me with
something that crashes
okay that's a crash and we are in
run file test
okay so wait this is not the right place
this is the right place this is test.js
okay so
uh get variable
so we're calling get variable
which
a variable calls object get which
expects interpreter to work
i suppose yeah so
again like this needs to
like anything that uses this thing as a
public api it needs to
push itself on the interpreter stack
scope is kind of an unfortunate term
actually because scope means something
else in javascript as well
um what
okay it's still crashing
um stringify info
so like why do all these things need the
interpreter by the way that's kind of
weird
i mean they call object get
and object gap will okay so it's about
the exception
so if we move the exception to the vm
this will probably um
become a lot nicer so
maybe what we should do here is
instead of doing this for everything we
can also like work around the
architectural issue
by just pushing this interpreter on the
vm stack
jsvm interpreter scope
something like that back
um fix me this is a hack
yeah so we'll just like push the
test runners interpreter on the vm
interpreter stack
for its lifetime
all right so now we're dying somewhere
new
cool
is this thing oh it's handling so you're
bored that's why things are so weird
um handle sig albert
um handle cigarette
so this is what's ruining the
back trace stuff so let's just not
do that
[Music]
i just want to see what goes wrong it's
kind of irritating that you can't get a
proper back trace
it's probably something we should look
into like better back traces when you're
getting a signal
um wait that's not what i wanted to do
test yes
okay so where are we now interpreter
create
initializing the object add constructor
blah blah blah blah blah in strict mode
but interpreted create should have
pushed the interpreter on the stack
didn't we do it that way
so he goes on the stack here so glove
logic initialize add constructor define
property put on property
in strict mode
where the hell are we crashing that's
such a weird
place um
let's see um userland test.js
interpreter158
scopes.oh scope stack last okay so it's
checking um
interesting put on property
wait but what if we're not in the scope
this is weird
why is this code structured this way in
strict mode
um property
we will casually call in strict mode
sure that makes sense
but why um why does in
strict mode check the scope stack
what if you haven't entered a scope
like when you're initializing a new
global object you might not
be in a scope i don't
why did this work before um i'm confused
now
scope stack
do we like always enter a scope
scope stack append we append when we
enter scope
right
and when do we enter scope
when we execute the statement
for statement um variable
formula is a for statement
so again we execute something then we
enter a scope
i am so confused by this right now like
why does this actually
why did this work before in a strict
mode
so i guess i mean i don't know why
this didn't break before there's
probably some reason i'm not going to
track it down now i'm just going to say
that if the scope stack
if if it's empty then we'll just say
that we are
in strict mode i guess
because yeah then we'll just consider
ourselves in strict mode while we are
doing like object initialization
maybe this will bite us in the ass
somehow i don't know that's so weird
though
it's probably some explanation for it i
just don't
okay now we are crashing um
when creating a type error okay so we
have lots of crashes to fix it seems
like
type error creates an error and then
error
to get the prototype
prototype is gone
[Music]
so we're running a few tests
if we collect more often then we die
immediately well that's
that's good then we can use that as a
smoke test tg test
js um okay
so why are we crashing so soon
interpreter create so like
we're crashing very very early because
of
garbage collection garbage collection
ruins everything
okay
[Music]
i mean we should we should definitely
like turn off the garbage collector
while we are constructing a global
object anyway
it's just kind of weird
so let's see um
test runner global object initialize
global objection okay so let's go to
global object
initialize and then here let's just say
that like um
the fur gc
because right now we're like we start
constructing
prototypes and stuff and then
and then we just die um
in the middle somewhere in between
because these things are not like
protected properly while they're being
set up
okay
and undefer gc
[Music]
uh great
fun stuff garbage collection problems
so
wait should we let this thing finish up
completely maybe
[Music]
um
maybe this needs to be here instead
initialize
so can we make a defer gc thing here
maybe let's just keep the fur gc
let's just make one right here um
on the end
so we don't gc across the entire
construction
of an interpreter and uh accompanying
global object
i don't know is there like there's
dragons in there um garbage collection
during global object construction seems
like
it's like a minefield right because
you're setting up all these things that
point to each other
and you don't want to be interrupted in
the middle by the garbage collector
saying like
ha this thing does not point to that
other thing that it normally should
of course we are still failing but we
can at least
see what is failing
scope node execute uh okay
so we have a null pointer
interpreter run scope not execute
wait a moment
[Music]
um so
i'm guessing what happens here is that
we moved a bunch of stuff
so normally when the heap starts the
garbage collection um it will look for
all of the roots
all the root objects or root pointers or
whatever these are um sort of
entry points into the object graph and
previously since we always had a pointer
for the interpreter
we would unconditionally
gather roots from the interpreter and
the interpreter will have
all the things like the global object
and the exception
so i'm guessing that currently
the global object will get garbage
collected
if you are not currently running
um something while you get a gc so if
you get a garbage collection
while you're not running while you don't
have a current interpreter then the
global logic
global object will go poof
that's not very good
we actually kind of need the interpreter
to have a stronger
reference to the global object but that
makes sense so we'll make a handle to it
instead of a
raw pointer
i think that'll be a first
step global object
um
handle to that
can we get this to work
let's see
and then uh
i want to static cast this thing
oh wait what kind of thing is this even
it's a handle object okay so we'll just
static cast it to
object
355.
so the handle is essentially the handle
is a strong pointer to a garbage
up to an object on the gc heap and if
you have a handle of something that you
don't need to
manually mark it during gc mark
um so we don't need to do that anymore
and it will also guarantee that the
object stays alive even if we are not
currently running js
so i'm guessing that was the problem
here that we were
um the global object was getting
collected
because um we only mark the global
object if we have a
currently active interpreter
still very crashy sure
[Music]
um let's see global object let's see if
we're destroying these things
are we debug global object i'm just kind
of curious now
okay so we did destroy a global object
um that is normal though like what
happens if we just run the browser
we go here and then we go away from it
and then we collect garbage
which kills a bunch of these
but everything's still good
so yeah
seems fine oh no there we had a crash
okay so it seems like we have some
issues here
okay so
we're still having these issues that
um
things probably get garbage collected
way too soon and
we don't quite know why but at least we
are able to
reproduce it easily by just running a
single test in the sjs app
so this crashes immediately when we
undefer garbage collection
actually one test passes
and then things go wrong
so what is it that hangs off of
interpreter
um stuff that will now go away
or
let's see just thinking what have we
changed
um structurally about the testjs
application so previously we would have
like we would instantiate a single
interpreter for every single test
which means a single interpreter a
single heap and now we keep the heap
but um throw away the interpreter
but nothing hangs off of the heap itself
um
[Music]
hmm
how do we approach this problem um
i guess we need to figure out what's
being deleted and what's surviving so
let's just turn
on heap debug we can learn a bit more
okay that's very verbose
so
this is all just a single garbage
collection i guess this is like throwing
out everything associated with
the test that we just ran
quite a few objects here
here's a global object okay so this is
like a single garbage collection
it looks like
but let's run that again with
with gc after every allocation
sjs
so it's working fine it seems like um
because now we're doing like multiple
garbage collections
and every time you try to allocate
something
this may take a while um
but what is it that we expect to survive
after
gc
it's really nothing i mean we don't
expect anything to survive if the
interpreter is gone
so like if there's just a heap left
then it should be empty
okay so this is taking too long
[Music]
okay this is kind of
this type of issue is kind of annoying
to track down
um
let's see if we can run this on the host
machine instead
and then we have some additional tools
sjs yes okay come on valgrind
now is your time to shine
um
totally bogus address here
invalid right okay now we are talking
and let's see where the first invalid
axis
is
so oh wait what's going on here
ensure property table
creates a deferred gc
wait a minute
ensure property table makes a difference
you see
there's something really goofy going on
here
so we're doing a visit children so we're
this is we are in marking and then
um during marking we call ensure
property table now that's not right
because this will allocate while we're
marking or it may allocate
um and it ends up doing a defer and
undefer
and then when we undefer this rai object
it says oh well there are no more
deferrals so we have to go into gc
and then we gc while we're already in gc
so this here why is this being called
during marking that's what i want to
know
um ensure property table visit okay so
we should not ensure property table
during marking
why do we do that it's presumably so we
can iterate the property table
of a shape and look at all of its keys
we want to
mark all of the keys in a property table
what is a key
in the property table it is a string or
symbol okay so we are
i guess this is marking symbols
yes visit children if it's a symbol then
we'll mark it
but we don't need to do that
it's only necessary if we have a
property table
so if we have a property table
then then we will mark it
don't need to don't need to reify or
like ensure that
one comes into existence
just to mark it
so can we get a valgrind run again
oh wow now it runs a lot more
significantly more so would this even
complete me
it even completes holy moly okay so
let's just see if it runs here
sjs oh it's too loud
but i think that was probably the issue
um this um
the reason that i wanted to run it on
the host was to get valgrind and that's
because we can't run
test.js in the user space emulator yet
because it requires floating point
instructions which we have not
implemented yet
in the user space emulator so
that's another thing that we need to get
back to implementing fpu stuff
um or if anybody's interested in
implementing fpu
support in the emulator then it's
a very approachable project
we have so many approachable projects in
serenity it's crazy
um if anything is interesting to you
then i encourage you to look into it
um so i just wanted to turn off heat
debug so that it doesn't spam
and then maybe the global object
destructor can stop spamming us as well
but man that gc problem here um
and was it an object or no it was in
shape
yeah test js
oh tg
just works that's very cool so
basically uh the problem here was that
because we were calling ensure property
table
this may allocate so it may do a gc
allocation from the javascript heap
and um an allocation can cause a garbage
collection
and when that happened then we would
initially initiate a new garbage
collection while we were already in the
marking phase of an ongoing garbage
collection
which caused everything to you know
get very surprised and go go boom so
here the reason that this is fine
is because um
if we have a property table then we can
mark
all of the um
then we can mark all of the keys in the
property table because the thing is
property table maps from a property name
to a property value
or or sorry from we're in a shape so in
an object shape
property table maps from a project name
to a
to an offset in the object property
storage
and the property name can be either a
string or a symbol
and in case it is a symbol because
symbols are gc allocated we have to mark
it
but if we haven't actually we don't
actually have a property table
we don't need to mark those symbols
because they don't even exist so we
don't need to bring them into existence
just so we can mark them
and um bringing them into existence is
like
this is the act of lazily constructing a
hashmap because the property table is
really just a hashmap
but because of the shape system tracks
transitions from an empty object and
every time you add a property to an
empty object we create
a shape transition so that you can
follow a chain of
added properties from an empty object to
a populated object
[Music]
then what insure property table would do
here is that it just walks
that chain of at this property add this
property add this property
walks that chain constructs a hash map
that maps from all those names
to storage offsets
but we don't need to do any of this if
we if we don't have the hash map because
the hashmap itself
has no particular importance it's just a
like a
way to do a quick lookup if you wanted
to do a slow lookup you could walk that
chain
as well to find the offset anyway
um so this this fixes that problem it's
really good
and apparently we're still running here
so i ran
in g mode which garbage collects after
every
thing i think i ran into g mode yay
garbage collects after every allocation
i guess technically it's before every
allocation
but it's very often
very busy cpu usage 100
it's pretty cool that we can get these
pop-ups here it's so busy that
the tooltip takes a moment to render
i love this progress bar effect by the
way in the taskbar
it's so neat
i wonder if we should show something in
the toolbar for that as well taskbar is
enough
anyway look at that it completed very
very cool
so commits commits commits commits
um dang there's so many things going on
here
gap let's see what we can add from
you know what we're gonna have to look
through these
um because i did a whole bunch of like
little things that were just
nice teas anyway
so like native function didn't need the
interpreter argument
so let's just um can we just fix that up
native function create
yeah yeah we can just do that one
separately okay so those who are
in lib.js
object
not you yes you
and um native function
yes you and you
okay the js remove unused
uh interpreter argument
constructor
okay um
and then we can do the shape fix
the one that we just did
runtime shepay
um don't allocate
um or
fix don't allocate
property table during
a gz marking phase
shape was um
up to no good uh
triggering allocation or um
shape uh wood shape shape shape shape
shape was um
allocating property tables
during inside visit children
which could
cause a garbage
collection to happen
um it's not very good to
start a new garbage collection
while you are in the middle of one
already yeah
okay that's good
and then
you know what that um actually makes me
think that
we should probably protect against this
so let's just quickly
put something or you know what um
let me let me make a note of that to do
assert in
gc if gc is already
ongoing so we don't
accidentally recursively gc
that will be my little to do
but i don't want to get i'm already like
many patches deep here so let's not
start stacking new changes
the strict mode change is really weird
um
i feel weird about that one but
i'm gonna go ahead and
okay but i guess we can just
commit that and mention that we feel a
bit weird about it
uh that's in interpreter
yes okay so let's just
[Music]
um
in strict mode work
outside of scope
one is a little weird
um
don't know why it's okay for
this function
to assume that there is a current
scope on the scope stack
um when it can be called
during um global object initialization
etc
we for now just
make it say we are in
strict mode when there is no
currently active scope
yeah that one's weird
ah some indentation change here whatever
here we are just removing some
unnecessary interactions so let's do
those separately
yes remove some unnecessary interaction
in object constructors
all right and
[Music]
script execution context stuff
this one we should get rid of
let's move this execution context thing
to bindings i don't know why i added
that to dom
let's see move done
script execution context into
endings
web forward script execution context
come with me you are going to go down
okay dom script x
come with me endings bendings bendings
um script x
and uh bindings
wait
yeah yeah this is much better
this one is weird yeah i don't like the
way that the lifetime is managed here
something that we have to come back to
probably this should be some way to
get notified when a script execution
context goes away
all right we'll figure that out some
other time it's not the most important
thing
right right now maybe in the future will
become the most important thing
let me just nuke these here so that my
ide does not get confused
um document needs to
say bindings here
and event target needs to
not really do anything i guess
this guy
okay so what more changes do we have
now things start to become a little bit
intertwined
and one thing i've been ignoring now is
the test web
test runner i think
probably that's broken yeah it's broken
but i'm gonna fix that separately i'm
not gonna
work on that now because i don't have
i'm running out of time here so we're
just going to wrap this up
and
see
[Music]
the script execution context stuff i
think we can add in a separate commit so
yes no
yes yes
um that thing is not necessary there
actually
[Music]
yeah good web and findings
script execution context
um this um
this will be inherited by
dom by um documents
and workers to provide
a common abstraction point
for uh script execution
extract come on come on destruction yeah
yeah that makes
don't sense brokers yet but
we might as well
make this little um
space for them now to
simplify things down the road
okay and then
this we can get rid of
here you can have your precious
cigarette handler
which is really weird but fine
um
[Music]
so that one we should get rid of
check out
hmm
okay so let's let's bulk this all
together
check the style
oh look js plus
well
let's see
yeah separate
heap from interpreter
or let's see um
add jsvm object separate
heap from interpreter
okay um taking a step towards
a world of multiple global objects
taking a big step
this patch adds
a new jsvm object that
represents or that that um
houses the jc heap
jse
[Music]
what are we saying
this means that the heap moves out of
interpreter
and the same heap can
be used by multiple interpreters
and can also outlive them
um the vm
tracks keeps
let's see it keeps a stack of
interpreters
where's um
and this
stack is pushed popped
we push up on this
stack when um entering
exiting uh execution
when uh let's say you have one entering
things
execution with a given interpreter
this allows us to
to make this change uh without
disturbing too much of the existing
code ultimately
the goal here is to make
an interpreter
more a much more
a transient object
that only needs to execute
or needs to exist while you execute
some code
for that
getting there will take a lot
um yeah
that's that's pretty good
um
note that in the file um the
global jsbn is called main thread
um to
distinguish it from
um future worker vms
yes
we should also mention that um there is
still a one two one mapping
between one two one relationship between
interpreter and um the global
object that stuff
has not been
tangled any further by this change
and indeed may not need to be untangled
we'll just say this because basically
i'm thinking that
if the interpreter becomes transient
it's just the thing you put on the stack
while you run some code
then whatever relation like its
relationship with a global object at
that point
obviously changes to not be one to one
because the interpreter is temporary
so um
this will change in the future
yeah okay so that's pretty good
okay so we're gonna have to stop here
for today but
i'm pretty happy with these changes
so basically we've
added a vm object which
we can begin to move more things to
um that represents sort of um
execution environment in a thread
so a threads javascript engine
and thread global
and we've moved the heap to the vm so
that
the heap can exist
across multiple interpreters which is
pretty cool
and the browser works we can do
many garbage collections we open this
thing
everything still works garbage garbage
garbage very cool
this is indeed a checkbox
i run some scripts okay awesome
test.js works
um yeah and then we also fixed uh fixed
that garbage collection
bug and a couple of small little things
here and there
so all in all very productive session
um and we're just gonna have to continue
on this
uh in the future but for now i gotta go
so
if you made it this far thank you for
hanging out
and for uh for watching i hope you saw
something interesting
and um it was nice to hack on the
javascript engine again uh it's been a
while so
um i guess i'll see you next time
bye