all right scaling 1 million check boxes
to 650 million checks something nobody
ever said they needed but man have I
never needed it more in my lifetime till
now on June uh 26th I launched a website
called 1 million checkboxes omcb it had
1 million Global check boxes on it
checking a box checked it for everybody
on the site immediately I did see this
and there's like people people were just
checking boxes this is by the way this
is crazy in fact this was so popular had
a complete Normie some 40-year-old
50-year-old in Rapid City South Dakota
being like did you know there's a
checkbox website where people apparently
are just checking boxes and then other
people are checking boxes like that's
how popular this got this thing hit
mainstream let's see I built the site in
two days I thought I'd get a few hundred
users Max that is not what happened
instead within hours of launching tens
of thousands of users checked millions
of boxes they piled in from Hacker News
our internet is beautiful Mastadon and
Twitter a I'm sorry that they came in
from Mastadon a a few days later uh 1
million check boxes appeared in the
Washington Post and the New York Times
here's what the activity looked like on
the first day I launched I hope you
threw some ads on that give me that aids
baby I don't have Vlogs to check uh for
check boxes from the first few hours
because I originally kept uh kept the
latest 1 million logs for a given day I
wasn't prepared for this level of
activity the site crashed a lot but by
day two I started to stabilize things
and people checked over 50 million boxes
we pass 650 million before I Sunset the
site 2 weeks later that had to be so
effing costly let's talk about how I
kept the site mostly online this is
super cool imagine launching a website
thinking that a couple people are going
to see it but instead that happened you
hit the New York Times oh man that would
be awful the original uh the original
architecture here's the gist of the
original uh architecture uh bits set
lazily rendered UI yeah box checked
checkbox static content yeah I mean this
is this seems pretty reasonable if I'm
going to make a website for a couple
people i' just throw a little bit of
static content a little web a little web
socket bada bing bada boom right easy
all you have to do is send four bytes up
for uh you literally it's it's it's six
bytes per message to the server six
bytes from the server down unless if you
do batching so that seems pretty easy to
do and by the way for those that don't
understand why I said that in fact since
it's 1 million you don't even need that
you could get away with three bytes uh
for websocket you require a minimum of
two bytes this thing is like flags and
various items this right here is the
length if the length is 125 or less it
means that uh 125 bytes or L that means
the data will be stored directly
afterwards so you could literally store
it in three bytes to be able to check 1
million check boxes because this right
here is going to be approximately 16
million that it could represent since
there's 1 million check boxes you could
do 24-bit integers and send it up little
annoying but whatever you could do that
and so that means you'd have Flags you'd
have your length being uh three and then
you'd have three bytes you could do and
so you'd have five bite messages coming
up and coming down you could pretty much
send them in batches which means you're
going to still probably get the the two
bytes uh for that length and then you'd
probably have three B runs right you
could easily just go data you know data.
length and that would be how many btes
you have right like it'd be crazy to use
anything else bittes are stored in the
balls right next to the microplastics
our checkbox state is just 1 million
bits nice it's just a bit oh damn so
he's storing it in the bits array dang
wait a second how do how do you send up
the state then let's see 125 kilobytes a
one let's see a bit is one oh yeah I
forgot to I forgot to think about just
sending down all the checkboxes how do
you send down all the check boxes it
would only make sense to send them down
in bits yeah that makes perfect sense
you have to do 125k
dang and then you'd have to apply
patches
maybe I guess you you you could apply
bite patches oo that would that totally
change how you'd want to do that and
sorry I know we should read what he's
doing but in my head I my mind's just
running with like you'd have to actually
have like your
offset and then potentially the bite to
patch
in that sounds exciting arle you
couldn't do arle because arle oh you
could do Arley what the hell am I
talking about you could totally do arle
but arle likely would suck and here's
why hear me out on this one is that the
check boxes will probably produce I
would assume a
uniform value is my guess what is rly
run length and coding so if you watch my
Doom video it's a lot easier I have a
doom video on my main channel the prime
Engen but rle is really really simple
let's just pretend we have the following
sequins wwww JJ D what would but what
would Jesus Jesus dud right if I were to
runlength and code this I start off with
the first bite all right so now there's
one W what's the next bite ah it's a w
again all right there's two W's uh let's
The Next Bite ah three W's what's the
next bite ah four
W's all right Next Bite ah it's a j okay
so there's one J what's the next bite ah
it's another J 2J all right what's the
next bite ah it's a d so it's a new one
1 D all right well we are done so just
take the bottom of every one of these so
that means I would encode this as 4 w2j
1D there you go uh bits already zipped
by the transport layer they should be
there's it depends on how they're
requested you know if you send it over
say websockets you you're not you you
might not be getting uh 1D hell yeah
baby but you may not be getting a per
message deflate so I mean there is it
just depends on how the things are sent
down but I would assume it's just
statically put in maybe actually I would
I would assume it's not statically put
in one would actually actually it might
be easiest to statically put it in and
what what I mean by that is that you
could actually have the server running
with the game State effectively always
on and so you just take that state and
send it down because you have to send
down 125k no matter what so you might as
well send it down uh with gzip and all
that it could be good anyways whatever
regardless right all right I wasn't
prepared for this level of activity all
right here we go the original
architecture we did this our checkbox
state is just 1 million bits a bit is
one if the corresponding check box is
checked and zero otherwise clients store
the bits in a bit set an array of bytes
that makes it easy to store access and
flip raw bits really I don't know what a
bitet is is that just an array buffer
what the heck is a
bitet is are they just saying a u8 array
is this fancy talk for u8 array is that
what this is yes oh okay yeah
I is bitset is bitset just must be
convenient operations on top of a u8
array it's like an array of U ones that
doesn't make any sense it can't be that
it there must be there must be it it
it's going to be something like this
where like get
get will take in an I I will literally
be uh what's it called I will be divided
by eight that's the bite you need and
then the bit you need so You' literally
go like this you take your bite array
right you take your u8 array you do this
and then you'd modulo or not modulo
sorry you would uh right shift over um
you'd right shift over uh I modulo
8 and then you'd mask it with a uh an
ampersand uh
0x1
right that's like that's what a git
would be I assume that's what a bit
shift or a get would be imagine not
having binary pattern matching shut up
you're over complicating it how would
you over complicate this wait there's no
other way to do
this can can you tell me how I'm over
complicating it I've never heard of a
bit set like how else would you store
this that would have to be a get I use a
bit set in my yeah yeah I would assume
this is this a bit set must just be
convenient a API over a u8 array it's in
the standard is is it in the standard is
a bit array JavaScript is that is that
really is that real yeah see they just
have typed arrays right you just have
uint at array right yeah you just have a
uint eight array and and and array Buffs
so you couldn't really do that you'd
have to you'd have to you'd actually
yeah in C++ but this ain't C++ bro this
is Ja this is Javascript okay this this
is not C++ uh definitely not in the
standard yeah needs a library what's
JavaScript why not JavaScript am I okay
so that's all bitset is is just I I'm
pretty sure I have exactly created the
implementation of a bit set right a set
is literally the same thing except for
you would just take a uh you would take
a one and shift it over by uh by I
modulo 8 minus one right that's all it
is and then you you would you would just
you would just or that
into uh u8 uh u8 I divided 8 right
that's all that all that is that's like
an an or equals I think I think I I'm
like 99% sure I'm correct on this one
this would be I right you just set I I
guess you could unset you could have
unset just to make the API easy I guess
you could have like set true or false
still over complicating it how how how
am I over you my soul Theory you you
must State how I'm over complicating it
because I don't know how else you would
act I don't know how else you would set
a
single bit in an array of
data it's a VCA bits no there's no such
thing as a VCA bits bro there is no such
thing as a vector of bits that does not
exist when you create a Boolean you
actually take up 64 bits like in Rust a
struct a single booing is 64 bits either
trolling or never program yeah like you
may not you may not realize this but a
struct of three booing is 192 bits in in
Rust why is a bull 64 bits because
computer reads are 64 bits right like if
you're on a 64-bit operating system
obviously if you're on like a 32 or some
realtime 16bit or some nonsense like
that then it's different but a computer
can't read a bite or a bit like that
doesn't exist a c computer doesn't read
a bit it reads a line which is your
architecture which is going to be 64
bits right and then if you have a u8
it's going to just mask out the first
bite and then hand that to you there is
no such thing as a bit array you make a
bit array my gosh okay I I'm feeling
okay we're moving on anyways all right
clients store the bits in a bits I I do
want to understand how I will have to
look into I'll have to understand how
bits work bit shifts work that it can do
a constant it can do a constant time
regardless of how much you shift it
because it just feels like there's some
use for modulo there modulo and barel uh
modulo and and divisions by two could be
happening in
in in constant time which just feels
interesting right it just feels like I I
don't I'm confused as as how that works
anyways whatever doesn't matter uh
clients store the bits in a bit set
array of bytes that make it easy to
store access and flip raw bits and
reference that bit set when rendering
checkboxes yep clients tell the server
when they check a box the server flips
the relevant bit and broadcast the fact
to all connected clients to avoid
throwing up uh throwing a million
elements into the Dom client only
renders the check boxes in view plus a
small buffer ooh look at that small
buffer react window Manny built it and
react using react window crazy crazy
that you did that dude carousels
remember we dude I I swear once a stream
we talk about a carousel uh bits shifted
uh pants shed [ __ ] wait I don't know
what you're saying but thank you for
that I appreciate it I could have done
this with a single process I wanted an
architecture I could scale and an excuse
to use reddits for the first time in
years so the actual server uh setup
looked like this reverse proxy plus
static con uh content checkbox date
updates nice and then this one nice okay
yes this is what you'd kind of need this
is in fact this is getting really close
to the architecture I've chosen for the
Real Time games we're building the Real
Time games we have the game engine and
then we have a bunch of relay engines in
which the relay all all it does is it
just takes in bits and sends out bits to
anyone that's connected like that's it
uh checks uh hit uh engine X for static
content and then make a get for the bits
at State and a websocket connection for
updates engine X acts like a reverse
proxy forwards those requests to one of
the two flask servers are run via uh
gunny cor gunny corn gunicorn gunny corn
can I say Gorn or is a g unicorn uh a
state of stort and reddis which is a
good primitive for flipping individual
bits clients tell flask when they check
a box flask updates the bits and Reddit
and writes the event to a pub sub
message Q both flask servers read from
that uh Pub sub and notify connected
clients you know for me I think it'd be
more fun to build this all yourself
except obviously the reverse proxy with
engine X seems great right gooni corn oh
goon Caven we goon Caven like because
all you need is a server that is 128k
and rece it l literally you have bit set
as a service I don't know it kind of
seemed fun uh the do-it-yourself seems
way more fun but I mean I guess reddis
is kind of neat that you just have you
get Pub sub for free right you get Pub
sub for free anyways uh the stateus
stord reddis which is good primitiv for
flipping individual bits I didn't
realize that Reedus had a good primitive
for flipping individual bits I didn't
realize that redis would had this
available bitet is not the bottleneck
it's probably websockets oh yeah it's
definitely websockets for sure for sure
that's where you're going to be spending
all of your time you're literally going
to be in system land for most of your
time red is bit Maps has bit uh it has
bitwise oh it did okay hey that's kind
of fun that's kind of fun let's see
finally the flask servers uh do simple
rate limited on request per session new
sessions per IP foolishly stored in
reddis redis nice let's go nice job nice
job and regularly send full State
snapshots to Connected clients in case
they miss an update because say the tab
in the back was backgrounded if you have
a websocket running you can miss updates
it can well yeah well disconnecting is
different because then you have to
reconnect I would strongly recommend I
think rle actually would do really well
here exor rle would do really well here
and so would uh just like like just the
real- time engine of Huffman and coding
and and all that would just do actually
do really well here because most of this
you don't like if you just did like if
you did um cuz you could technically
send down updates as as like o trees or
I mean sorry quad trees it's not 3D
space you could quad tree it out and
send down like a list of boxes that have
been updated in in in small amounts and
very very exciting stuff you could do
right there's all sorts of fun stuff
that could happen uh here's my let's see
here here's an abbreviated
implementation of the flask side here's
a bunch of stuff get this get bit get
the bit set index get this one set the
bit index let's go get full State get
the whole bit set basic 64 really base
64 and utf8 it oh damn I don't get that
part websockets support binary
websockets support the
binaries couldn't you just send it
couldn't you just raw dog the couldn't
you just raw dog the output why base 64
encoded I don't I don't understand the
get full State um index new value okay
okay nice so it does it does it does an
index with the new value okay that's
pretty cool does a little index with the
new value in it okay I wonder can you
when you publish can you publish
non-json strings can you just publish
something that you could parse because
jsing creates a lot of garbage uh
petition to rename dumps to dumpy so you
have a json. dumpy yeah I think that's
that's pretty reasonable uh just because
you know when you're if you're massively
doing Json it can it can actually become
kind of an annoying thing whereas if
you're just like you know you could
imagine just all you're really doing is
sending a lot less stuff you know what I
mean parenting yeah we're just parenting
even a formatted string you could just
format this string this this is even a
hard thing to have you just literally
just have a
format
right serializing data between processes
yes it's a it's a real problem people
don't once you hit a certain scale Json
becomes
difficult yeah yeah it's it's an open
TCP yeah it's an open TCP channel in
which comes the payload it has its own
framing built into TCP it uses HTTP to
hold open the connection right that's it
uh all right get the full State get the
count all right we have a snapshot then
we have this little initial State
jsonify damn you're J oh that's why
you're doing an API call wait I don't
get it you're API calling to base 64
undoing it that's what's happening
you're sending you're sending 128k of
jeson well it's not even 128k I don't
know how big that would be I assume I
forget what what's what's the what's the
b64 penalization is it like
133% my colleague hit me with the fact I
haven't checked servers on average are
doing 60% yeah this was from an Intel
paper in like 2007 or 2008 or something
like that uh I don't know how true it it
still is but it's it's like 33% right
yeah so that that would be
um a full snapshots a lot of a lot of
ass data a lot of a lot of a whole lot
of ass data crazy to do that especially
especially if you're doing joning it's
just a lot of work your server just
doesn't really need to do but I guess
long as you're not sending those too
often you don't have too many you could
probably get it done pretty easily I my
my assumption is probably didn't it It
ultimately did not matter because they
probably needed like what three servers
to serve
everything I heard ass data yeah yeah
yeah yeah okay cool principles for
scaling before I talk about uh what
changed let's look at the principles see
this is me I'm pretty much doing the
same thing my co-workers who did uh C++
did to me I would be like who cares it's
just like 1K of extra data I'm
allocating and they're like 1K of extra
data right whereas we're I like it's the
same thing right it's the same thing
it's the same thing where're you know
arguing over like at the end the
principles that really matter is if you
can scale on three machines it probably
doesn't really matter right three versus
One machines who cares like if you want
something to be kind of you know
resilient you probably need a minimum of
two two machines so tip from pythonistas
in chat default Jason liba is slow for
large data got a 50% improvement with or
Jason Library oh
interesting uh are the pack struck we
need Str we need struck packed right
principles for scaling before I talk
about what changed let's look at the
princi I had in mind when scaling bound
my cost I needed to be able to uh let
see I need to be able to math out and
upper bound on my cost I aim to let
things break when they broke my
expectations instead of going serverless
and scaling into bankruptcy yeah by the
way this is a good this is a very good
idea the the man is
smart this is very very
smart embrace the short term I assume
the site's popularity was fleeting I
took on technical debt and aimed for
okay solutions that I could hack out in
hours over great solutions that would
take me days weeks beautiful I love this
this is such a great philosophy give the
guy give the guy some claps give the guy
some claps let's go uh use Simple self
hosted Tech I Ed my running my own
servers like a log into boxes and run
commands I tried only to add
dependencies that I could run into bug
on my own again very very smart dude
just knowing you caught lightning in the
bottle and you need to scale it take the
shortcuts have fun I optimize for fun
not money scaling my site was uh my way
was fun so uh so saying no to
advertisers
I like it I like it it would I will say
that it would have been a cool
opportunity to go to something to
whoever you're hosting on be like hey I
will give you a hosted bu this for
$5,000 you could probably do that with
how much with how much traffic he was
getting for like a week probably could
have probably could have paid for
everything and just had a nice little
nice little paycheck for such a great
idea uh keep it Global the magic of the
site was jumping anywhere seeing
immediate changes so I didn't want to
scale by for example sending clients
view of only the check boxes they are
looking at very very smart by the
way because it is true you could you
could like theoretically you could load
it in chunks like if you loaded
everybody up at the top you
theoretically could have sent it down in
chunks so that way you can see it as
quickly as
possible
right true day one the pop within 30
minutes of Lun the site activity looked
like this all right this is 30
minutes that's so cool
that's so cool the site was still up but
I knew I wouldn't tolerate uh the load
much longer uh the most obvious
Improvement for my Ser was more servers
fortunately this was easy engine X could
easily reverse proxy flask instances
into another uh VM and my state was
already in reddis I was already spinning
up more boxes okay so I I'm I'm curious
how many boxes did he get up to
originally I assumed another server or
two would be sufficient and said traffic
grew as I scaled uh I hit number one on
Hacker News activity on my tweet
skyrocketed I looked for bigger
optimizations my Flash servers were
struggling my assumption is it's the
Json it's the Json and Bas 64 in CU if
you're Bas 64 in 128k and then on top of
that then Json doing that like I mean
you're going to be spending a huge
amount of time just in coding which is a
lot of which is just like a lot reddis
was running out of connections H did you
notice I wasn't using a connection pool
my best idea was to batch updates nice
that's a good idea great idea I hacked
something that looked like this oo great
by the
way yeah you get a bunch of messages and
then you batch them okay cool just do a
little basic little socket IO I I'm
pretty sure is not very efficient by the
way uh I'm surprised even going with
socket.io who cares just just raw dog
that websocket and send down you know
you're just sending down Json and you
get framing for free do you need headers
do you need all the stuff that socket.io
even offers I'm curious why socket.io uh
I didn't bother with backwards
compatibility nice job I figured folks
were used uh used to the site breaking
and would just refresh nice job I added
a connection pool that's definitely uh
did not play nicely with the gunicorn
and flask uh but it did seem to reduce
the number of connections to redus let's
go I also beefed up my redus box uh easy
to do since I was using digital oceans
managed redus smart smart right there
that's some good ass python code yeah
this is smart I really like where he's
going with this one this is great
vertical scaling by the way amazing it
is such people reach for horizontal
scaling so fast as opposed to just like
ah I'll just make it
faster then things got trickier uh it
was not a good night to to have plans
around 4:30 p.m. I accepted it I had
plans I had spent June at a camp at ITP
uh let's see at a let's see a school in
NYU the night of the 26th was our final
show I had signed up to display a face
controlled Pac-Man game and I invited
some friends I had to go I brought an
iPad and put omcb on it I spun up
servers while my friend Yuri and my
girlfriend Emma kindly stepped in to
explained what I was doing to strangers
when they came by my booth I had no
automation for spinning up servers oops
so my conventions
evolved by the way good hack like why
why spend days trying to make it auto
scale when you can just manually scale
like it's really not that hard cursed
sver this just if this just cursed sver
1 mil 3 1 mil 3 1 mil four 1 mil 5 m six
you can see you could look at this you
can see what's that what's what's the
like the W what's the wack uh cursed is
it is it cursed meme yeah you you could
just watch this happening to his nameing
inventions you can just watch it slowly
happening oh my gosh I bet you there
there's a pretty funny there has to be a
pretty funny uh what's it called meme
right here where it starts off with like
wojack smiling and then slowly just
getting more and more cursed as the time
went on I got home uh from the show
around midnight I was tired but there
was still more work to do reducing the
number of flas processes on each box I
originally had more workers than the
number of cores on the box this didn't
go well especially when especially if I
bet you the time spent was was
serializing so this makes perfect sense
if your time is spent waiting you
theoretically could get away with more
more processes than uh CPUs but since
that's not it yeah doubling the batching
Windows substantially reduced load I
tried doubling again this appears to
help even more I don't know how this
principle worked uh or how to I don't
know how to pick a principal the number
here yeah keep going until it stops
working
wow bandwidth I pushed the updates I was
feeling good and then I got a text from
Greg uh technology I'm so proud you're
going to hit uh W let's see you're going
to hit w a server bill for bandwidth or
is the bandwidth not cray cray cray cray
cray
gulp yeah definitely going to get a w
there definitely going to get a top I
realized I hadn't thought hard enough
about bandwidth digital oceans bandwidth
pricing is pretty sane
one to put it into perspective my Doom
game that I sent down with my own
handrolled compression that was beating
gzip by a good
portion was cos me $150 an hour to
run under less
load just to like put that into
perspective so one could imagine that
since this guy is dunking
128k just because maybe they got out of
sync this has to be pretty good uh a
generat per server compounding free
allowance I had a terabyte of free data
from past work in a pre-launch and I
didn't think omcb would make a dent I
did the back of the envelope math and I
let's see see I send snapshot States 1
million bits one megabit every 30
seconds let's go oh my gosh with a
thousand CS that's 200 gigabytes a
minute oh my gosh I I love the fact man
like the entire point of my stream has
been to talk about this exact problem
for the last like three months of
programming and here it is oh my gosh
this is so good this is so good and is
just in you know that's just in state
snapshots right and that's with a
thousand
clients how much do you want to bet at
Peak he was sitting way above a thousand
way above a TH or 120 gigabytes an hour
and we're probably going to have more
clients than that and we haven't even
started to think about updates yeah
updates think about how many updates in
batching you're getting that has to add
another huge amount it was 2 A.M I was
very tired I did some bad math maybe
confuse gigabyte hours with gigabyte
minutes and I freaked out I thought I
was already on the hook for thousands of
so I did a couple things frantically I
texted Greg who helped me realize that
my math was way off good I ran I I IPS
link show Dev e zero on my engine box to
see how many btes had sent confirming my
math was way off nice job I started to
think about how to reduce bandwidth and
how to cap my cost I immediately reduced
the frequency of my state snapshots easy
win by the way so what this is is
effectively what he's doing is you can
think of this in the terms of video as
opposed to
checkboxes because what you have is you
have your you have what is called an
iframe an iframe is like the State
really this is like a this is RTC right
uh or rtmp you could rtmp this so hard
right you can think about uh your
checkboxes as effectively your
resolution right so this is your
resolution you're sending down you have
about 1 million
bits uh so all you need to do now is to
actually just send down this thing so
iframes would be effectively something
in which you can base all frames
afterwards on so afterwards you just
need something small to send down to
patch this thing that's effectively how
like MP4 works or h265 works is that you
have something that's like a big a big
boy and then you have uh I believe
they're called P frames afterwards if
I'm not mistaken P frames and B frames P
frames are future forward patches and B
frames are past future frames right
which is B frames are crazy to me I
don't understand B frames at all okay I
don't know I don't know what the hell is
going on here uh I don't know how that
works but you can imagine that you're
doing video that's all you're doing is
video so sending an iframe every 30
seconds is a lot right you're sending
these big you're sending these big
frames quite a bit uh obviously the the
true answers don't send those and just
anybody who's out of
sync here here's what here's what I
would personally do is that at the
beginning of every message I would have
a number right I'd have a one bite
number maybe two bytes if you feel like
that would be better depending on how
many updates you get get and this would
be my uh this would be my frame right so
it's like it's just it's just running
loops around 65,000 and if this number
isn't plus one the number that I
currently
have then I would say a big message you
must refresh to keep going and then I
would disconnect my websocket and just
wait because maybe they don't go back
maybe they don't go back to that tab for
a long time so I'm just sitting there
with zero cost right like some simple
things adding a little frame number is
always such a good idea this is actually
how I caught
a bug in the web well it's not a bug
it's a feature of how websockets work in
JavaScript if you do binary websockets
without setting a very magical flag
there's a magical flag that actually
exists you will get uh you will get
promised payloads of the data which is
crazy by the way because remember
websockets are in order data what the
hell am I why the hell do I have a
promise to read the payload what the
hell's wrong with you why are you giving
me a blob okay this isn't a file store
you know the size of it and you have to
read the whole thing how am I getting
out of order data well you do and when
you get out of order data by simply
adding this little frame number to the
out of order data I was able to goof it
up you get a blob instead of an array
buffer it turns out there's like a super
hidden flag in websockets it's not
hidden it's just who the hell would ever
think about using that there is a a flag
that you can set to not get blobs but
instead you get array buffers and then
you can do that that's right why not
constru binary strings you wouldn't want
to use binary strings you want to use a
ray buffers because strings in
JavaScript I believe cannot break utf8
if I'm not mistaken maybe they can maybe
I'm completely wrong on that but if you
have a string you have to you have to
like the reading out of a string is
really inconvenient versus a blob uh and
second a string in JavaScript is a rope
which a rope is probably pretty
inefficient is my assumption right
strings are utf16 right I want to say
they're UTF 16 but I might be wrong on
that altogether it's in the first two
bites of a socket you get out of order
because the messages are split into two
packets no no no no that's not it that's
not it that's that's not um that's not
the problem that's not the problem with
websockets uh it that's not how this
works they don't come in out of order
there is no such thing as out of order
uh bytes in a websocket there's no
interleaving of messages and in
websockets when you have this if your
first bite's your flags right your
second bite if it's if it's 120 uh if
it's 126 then the next uh two bytes will
be the actual true length right so that
mean you have a four bite if it's 127
the next four bytes would be the length
then after that it's in order data there
is no interleaving there's no
multiplexing right there's no
multiplexing in websockets it's a it's
that you're you might be confusing H2 uh
http2 has
multiplexing this thing doesn't all
right anyways let's go I immedately
reduced the frequency of uh State
snapshots and then with some help from
Greg pair down the size of incremental
updates uh sent to the client yeah just
don't yeah yeah there you go perfect
yeah you don't even need to use you
don't even need to use arrays like if
you really think about it just use a
string or even better just use binary
right you could just do commas right but
if if you want to make it really easy
just use binary and it'd be really small
then right you could you could send this
down and like this this this right here
especially if you have these smaller
numbers could be sent down so tiny right
string mle you could just do like an
array an array do join with a with a
comma binary and python is crazy oh okay
maybe that's fair I moved stuff from a
bunch of dicks into a list of sending
two arrays of uh indices with true and
false implied this was five times uh
shorter than my original implementation
nice job hey that's a bonus though can
we get a bonus that's he that is a good
thing right there good job and then I
used Linux TC utility to slam a hard cap
on the amount of data I could send per
second TC is famously hard to use so I
wrote a configuration script with uh
claud's help I've never used TC
beautiful I don't even know what this
means awesome I've never used uh TC uh
this limits the traffic over eth zero my
public interface to 250 megabits per
second that's a lot of bandwidth two
gigs a minute uh or just uh under three
terabytes a
day that's pretty funny and at least
that's $30 a day right uh but at least
the re let's see but it let me reason
about my cost and one gigabyte uh and a
cent a gigabyte I knew I wouldn't go
bankrupt overnight at around 3:30 I got
in bed my server was pegged at 250 uh
megabits a second limiting for much of
the night I originally thought I was
lucky to add this limits when I did I I
now realized somebody probably saw my
tweet about reducing bandwidth and tried
to give me a huge bill dang look at that
it would have got real expensive imagine
if you got up into this one dude dang
blue traffic is uh for my workers to
engine X purple is the engine X Out To
The World the timing is suspicious
timing is a bit suspicious there's a
definitely a lot of sus right here right
yeah okay yeah you meant just yeah yeah
I'm on your team pegged by that dude
imagine getting pegged day two it's
still growing I woke up a few hours
later the site was down uh I hadn't been
validating input
properly dude I hate the internet why
can't you guys just have nice things
okay hey you're not a hacker you're just
an [ __ ] there's a difference between
being a hacker and an [ __ ] an [ __ ]
is somebody who takes somebody's
personal project and ruins it for
everybody else a hacker is actually
going and finding real vulnerabilities
in real websites that are meant to be
real okay you're just an [ __ ] that's
it you're just [ __ ] totally
different I broke this guy's personal
project I'm a winner is there any
private information that I can steal no
is there anything other than hurting
people's feelings no it's just me being
an [ __ ] what do you mean drop table
Prime yeah I know [ __ ] hat hacker it's a
[ __ ] hat hacker it's a [ __ ] hat hacker
right the thing is it's not like this
thing with storing credit cards or
storing information you could possibly
steal it's it's storing 128k of binary
okay so there's nothing to it it's
stupid there's nothing to do at all with
it you can't you can't win anything the
brown hat hacker people are going to be
like what the [ __ ] a brown hat hacker
but it's so true Brown hat hackers can
we start that can we start that this
chat started Brown hat hackers okay
they're brown hat hackers they're ass
hats if if you if you try to ruin
someone's personal project for your own
fun you're an [ __ ] if you ruin
someone's personal project because
you're learning how to hack you're still
an [ __ ] if Defcon was uh if Devcon
was still there you'd be banned yeah I
was here yeah there's a huge difference
between it now if you go to if you go to
Defcon and show your project you're
going to get hacked like that's fine
you're going into it oh that reminds me
are you going to Defcon with th I am
like I think there's a difference
there's obviously a difference here I
think we can all agree that the target
of the hack changes its meaning but
that's how you show right yeah yeah I
ruined my own project it's just fine
yeah let let the person ruin their own
project right big time anyways the site
didn't prevent folks from checking boxes
above 1 million someone had checked
boxes in 100 million
[Laughter]
range imagine them uh this let them push
the coure the count of uh checked
boxes this let them push the count of
checked boxes to 1 million tricking them
to sight and think uh thinking things
were over redis had also uh added 1
million zeros between uh bit 1 million
and bit 100 million which 100x the data
I was sending to the clients this was
embarrassing I let's say I'm new to
building uh for the web but like I know
you should validate your inputs it was a
quick fix I stopped engine X copied the
first million bits of my old bits set
and truncated a new bit set I wanted to
keep the old one for debugging I taught
my code let's see taught my code to
reference the new bit set and added
proper validation not too bad brought
the site back up let's go oh my gosh
look at this
20,000 20,000 this uh makes the story uh
better than it does make the story look
this hourly checked
boxes in the thousands three million
checked boxes in an hour oh my gosh the
site was slow the number of boxes uh per
hour quickly exceeded the uh day one
Peak the biggest problem was the initial
page load this made sense we had hit uh
we had to hit reddis which was under a
lot of load and we were making too many
connections uh to it due to bugs and
connection pooling I was tired and I
didn't feel like uh feel equipped to
debug my connection pool issue so I
embraced this uh shortterm and spun up a
rdus replica to take load off the
primary and spread the connections out
but there was a problem after spinning
up the replica I couldn't find its
private
IP I made a new one but I don't know
where it
went that's so good to connect to my
primary I used a DNS record there was a
records for its public and private IPS
digital ocean told me to prepend replica
to those records to get my replica IP
this worked for the public one but
didn't exist for the private DNS record
I really wanted the the private IP I
thought sending traffic to a public IP
would risk uh traversing the public
internet which would mean being build
for way more bandwidth I still couldn't
figure out how to find the rep replica's
private IP in an official way I'm sure
you can tell me how I took a different
approach and started making connections
to private IPS close to the IPS of my
Reds primary and other servers this
worked on the third or fourth
dry let's go well done I hardcoded that
IP as my replica okay I mean you know
what this is one way to solve it and I
think we should be very happy about that
that's just gigachad well played GG
stabilizing my flask process kept
crashing recording me to babysit the
site the crashes seem to be running out
of reddest connections I'm wincing as I
type this now but I still didn't want to
debug what was going on there it was
late and the problem was Fuzzy so I
wrote a script that looked for the
number of running flas processes and
bounced my system D unit if too many
were down beautiful I threw that into a
cron tab on my boxes updated my engine X
config to briefly take servers out of
rotation if they were down I should have
done this sooner this appeared to work
pretty well the site stabilized uh
around 12:30 a.m. I posted some stats to
Twitter and got ready to go to bed and
then a user reported an issue I think
replicas make the total check strange
it's 800k than 300K at this moment it's
100 plusk I just saw a few K briefly oh
God I understand the bug it's hard to
fix I love this one again again we got
ourselves uh well what's it called Mr
Incredible meme going on here uh By the
way when I say build your own projects
do you see some of the great parts of
building your own projects like the this
person right here
Nolan just got more experience than 10
years of you on the job it did because
you normally don't like that's the
that's the reality of working at a
company these problems are solved slowly
and over time with papers and and all
this stuff and everyone you know and
you're just one Cog in a wheel
especially as the company gets
disproportionately bigger you know I've
written a classic bug to keep client
checkbox State synchronized I did two
things sent the clients incremental
updates when check boxes were checked or
unchecked clients sent occasional full
State snapshots in case they missed an
update these updates didn't have
timestamps a client could receive a new
Full State snapshot and then apply an
old incremental update resulting in them
having a totally wrong view of the world
yep if only if only if only if only if
only there was some sort of indicator
I'm telling you this is why I frames and
P frames and all these things exist you
got to know the ordering of what you're
getting because these could come in and
out of order right I was embarrassed by
this I've written a whole lot of State
machine code and I know better it was at
almost 1:00 a.m. and I barely slept the
night before it really it's not that you
don't know better it's just that you
haven't gone through it right whenever
you're doing this like he doesn't
realize he's playing with video and
there you go oopsy daisies oopsy daisies
didn't get your monotonically incre
increasing value it was a struggle to
write code that I ironically thought I
could write my sleep but I timestamped
each full State snapshot you could just
throw a you could literally just throw
an incrementing counter running around
something right it's pretty easy I I
timestamped each update to my reddest
Pub sub added a Max time stamps of each
incremental update and batches I sent to
client taught the clients to drop update
batches in their uh time stamp was
behind the time stamp of the last full
SN full State snapshot and really should
also potentially consider dropping
anything that was behind other updates
uh also I love the I love that he says I
taught I didn't real it's beautiful I
taught the client I sat the client down
and said hey
you don't do that and then it didn't do
it it taught he's a
teacher this this isn't perfect clients
can apply batch of mostly stale updates
as long as one update is new but it's
substantially better I don't think we
should add that line but I'm really not
sure think carefully I'm very tired but
if we get the
Le oh he's oh he's using a jipp me to
Claud oh you're using jippy or Cloud to
do this sounds difficult I ran my
changes by cloud for shipping to prad
Claude suggestions weren't actually
super helpful uh but uh talking through
why they were wrong gave me more
confidence rewrite and
go the natural outcome at web filings we
rewrote our kerning engine from python
to go it just happens it's just the way
honestly drop it just it just makes
perfect sense I woke up the next morning
and the site was still up hackly
restarting your servers is great this uh
was great timing the site was attracting
more mainstream media attention I woke
up to an email from The Washington Post
I moved my attention from keeping the
site up to thinking about how to wind it
down I was still confident folks
wouldn't be interested in the site
forever I wanted to provide uh not and
rust [ __ ] article I wanted to provide
the thing is it's not that he wasn't
going to do it in Rust because of that
it's just that he didn't have four
months of planning plus six months of
implementation to get the site up in
Russ so you know it just he wasn't there
yet and plus the white paper don't
forget the white paper and he's also
probably not a furry the the furry thing
probably really hurt him there uh I
still moved my attention from keeping
the site up to thinking about how to
wind it down I was still confident folks
wouldn't be interested in the site
forever and I wanted to provide a real
ending before everyone moved on I came
up with the plan i' let's see I'd make
check boxes freeze if they were
unchecked quickly if they weren't
unchecked quickly I wasn't sure that my
current setup could handle this it might
result in a spike of activity and I'd
asked my servers to do more work after
taking a break for the day I got brunch
with my friend Elliot a super talented
performance engineer and I asked uh if
he was down to give me a hand he was and
from around 2: p.m. to 2: a.m. on Sunday
we discussed implement ations of my
sunsetting plan and then rewrote the
whole back in a go the go rewrite was
straightforward we ported without many
changes Lots by the way that's because
go and
python when you look at the code
sometimes they feel very
similar right they they just do feel
very similar by the way 12 hours is a
quick
rewrite they they feel very similar go
is python done right go can become very
verbose where python isn't but generally
speaking go was very straightforward
lots of sticking uh lot lots of sticking
points were things like finding a go
socket iio library that supports the
latest version of the protocol or you
could just raw dog your websockets right
the speed up was staggering yeah that's
because you're using python for
generally its worst possible case right
like this this is this is the you you
were for looping you for looped
bro uh things were actually so much
faster that we ended up uh needing to
add better rate limiting originally uh
we scaled too well and bots on the site
were able to push absurd amounts of
traffic through the site the site was
dosed on Sunday night but but addressing
this was pretty simple I just threw the
site behind Cloud flare and updated my
engine X config bit a
bit
again again Brown hat hacking such Brown
hat hacking going on there right like
who does
that dude again what's the purpose of
that why would you why would you just
try to ruin somebody's day like it's not
like you can make money off of dosing it
dude this is just Brown hat hacking
Brown hat hackers okay yeah by the way I
wonder if yeah I wonder if Cloud Flair
contacted them about Enterprise account
uh sunsetting of the site the site was
Rock Solid after the go rewrite I spent
the next week doing interviews enjoying
the attention and trying to relax good
job by the way very this great article
too uh and then I got to work on
sunsetting checkboxes with freeze if
they weren't unchecked quickly H which
would eventually leave the site totally
frozen the architecture ended up being
pretty simple oh that's kind of funny
sunsetting it by hey this thing if if
it's not checked within the next 10
minutes it cannot be oh that's kind of
fun keep the oh that's that's such a fun
idea oh I love the sunsetting plan uh I
added a hash table that tracked the last
time the box was checked this would be
too much state to pass to the clients
but it was fine to keep in R yep along
with the time to freeze value when
trying to uncheck a box we'd first check
whether now and last unchecked it's time
to freeze if we uh if it let's see if it
is we don't uncheck the Box and in
instead update Frozen bitset to note
that the relevant checkbox is now Frozen
I distributed Frozen bitset state to the
clients in the same way that I
distributed checkboxes which were
checked and taught the clients to
disable a checkbox if it was in a Frozen
bitset and I added a job to periodically
search the bits that should be frozen
but where yet nobody had tried to check
uncheck them and freeze those redus made
it so easy to avoid race conditions with
this implementation I put all the
relevant logic into a Luis
script I didn't realize you could script
with
Lua I love Lua by the way dude the wait
Bros using Lua now how did Lua get in
here by the way how how did Lua get in
here I thought we were in go I know Lu's
a scripting language I'm really good at
Lua I literally use neovim by the way by
the way I use neovim and I have a wife
and four kids so I'm just saying all
right I'm a sex you wouldn't understand
you can do it directly in reddis I
didn't realize you could directly do it
in reddis
oh you can embed super easy that's why L
is so great I didn't realize that reddis
support uh reddis supports reddis or Lua
powered
oh that's super cool by the way this I
did not know redis did a Lua powered
Atomic updates hey you know what w
that's a w so what I had learned well a
lot this was the second time I'd put a
server with a real backend on a public
internet and the last one barely counted
learning in a high intensity but low
stakes environment is
great side project people building a
site in two days with little regard uh
for a scale was a good choice I can't
even read regard anymore correctly I
can't I'm way too Lithuanian to
understand that I just can't every time
I'm just like ah just a little regard
and it's just like it's just it's just
it just hurts right it's so hard to know
what I will do well on the internet
nobody explained the site to seem uh
that excited about it and I doubt I
would have launched at all if I spent
weeks thinking about scale having a
bunch of eyes on the site energized to
keep it up and helped me focus on what
mattered um I'm happy with uh the tech I
used reddis and engine X are incredible
running things myself made debugging and
fixing things so much easier it was a
little painful to not have full control
of my reddis instance the site cost me
something like 850 bucks to run
donations came in pretty close to that
so I'm not uh so I'm not in the whole
too much hey this we can buy some coffee
maybe I need to buy a little bit of
coffee huh in the future I might spend
some time trying to uh decode the p
pricing Pages for convex or durable
objects but I'm proud of scaling it on
my own terms and I think it was worth
noting that things worked out okay this
also validated my belief that people are
hungry for constrained let's see
Anonymous interactions with
strangers I love building sites like
this and I was going to build uh more
regardless but again regardless see it's
just killing me it's just killing me but
I'm more confident than I've ever uh
than ever it was a good idea wrapping up
this was an absolute blast I've got one
more story to tell about the site it's
about teens doing cool things this blog
is too long to tell right now so I'll
let's see so stay tuned I'll update my
newsletter Twitter and various platforms
when it's live in the meantime uh build
more stupid websites the internet can
still be fun it can be and it should be
I was aiming to strike a balance between
letting someone trivially spam my whole
site and letting a bunch of students
behind an IP play at all which is why I
which is why I did this instead of just
limiting per IP uh I later gave up this
approach although I had more time I
would tried to keep it in with
sufficient time oh this is just like the
little bullet points right or is this oh
this is just these must be little ones
that were yeah it's these things okay
yeah we're not going to read those
because it wouldn't make any sense I
thought they're footnotes I just wanted
to make sure it was a footnote I know I
was making sure it was a footnote I was
like wait a second wait a second where
can I read the story the one check one
the the the million checkbox
one by the way this guy is going to get
hired hope you know anyways that was
awesome that was what a great great read
and what a great experient uh experiment
well done by the way well done