yo I'm gonna delete the postgres
directory on db2 should be safe yeah
yeah
classics sounds to me
is this actual
at 6 PM UTC a beautiful day in Silicon
Valley work was going on as usual at
gitlab HQ so do we know the person the
person that's making this uh video Kevin
Fang hit subscribe by the way hit
subscribe everybody and press like on
the video do we know like does this
person work does Kev does Kev work at
gitlab
it was a real event a few years back I
know that's why I'm questioning because
it looks like there's a lot of like good
stuff I'm very curious for uh poor Kevin
sounds a bit like fireship yeah just
kidding kitten lab is fully remote yeah
is a code sharing platform just like
GitHub except open source the main
character of the story team number one
was about to sign off stop footage not
actually aware of the disaster that was
ahead he was working in UTC time so one
of these countries unfortunately isn't
UTC time
all the places really when you think
about it it's supposed to be universal
time
this evening was interrupted at 6 PM
when he was paged in for high database
load after scrolling through the
dashboards he saw that the total number
of Snippets was Rising dramatically and
assumed the root cause was spam this was
a fair assumption as gitlab had been
experiencing similar spam issues in the
past week just less severe over the
hours the team attempted to block
certain monkeys this is a terrible
problem to have especially like right on
the weekend this is just a terrible
problem to have oh my goodness but that
sucks like this guy I can't believe we
live in a world
where people spend time and money
to hurt somebody else on the interwebs I
don't know I still have just a hard time
in general with that entire notion
it's just it's just wild clueless I'm
clueless
let's find this delete spam users
okay 99 p.m they were alerted to the
database having an elevated number of
locks I mean classic right transactions
are made to a record it will enforce and
lock forcing further rise to the record
to wait for the first one to finish this
ensures that the variets do not
interfere with each other more spam more
rights more locks more latency the
engineers continue to search for other
sources of stock footage again not
actually him everybody just in case
you're wondering
um
all right the engineer okay so here we
go this is I like this team number one
said he was gonna sign off as it was
getting late suddenly he didn't wake up
early this time for database replication
lag gitlab had two databases a primary
one and a secondary replica users will
write to the primary database
db1.cluster.getlab.com which will
forward the same right to the secondary
db2 the second database cluster was in
standby and only used for failover
purposes
it honestly just sounds like a cluster
this
I feel like these are like the worst
possible issues when on a Friday when
people don't even want to be there later
in the day
and like everything starts breaking
right like everything starts going wrong
I feel like I've I've had a similar
experience to this where everything goes
wrong and it's just like failure bleeds
into failure bleeds into failure and it
just feels like man
this is this is rough lesson they've
learned in the past where DB1 was a
single point of failure the process of
forwarding The Identical rights to db2
is called replication yeah and over four
gigabytes of data in DB1 failed to
replicate in db2 this was a novel issue
without proper documentation so team
number one stayed online to support the
team the database they used was
postgresql which had a command PG based
backup to create a backup from a live
database the plan was to remove the
existing incomplete data on db2 and to
write PG based backup to copy the
current DB1 dated there we go not fixing
any problems but
you know or fixing any sources of
problems but most certainly fixing some
problems feels good okay they're making
progress here I like db2 I probably
would have done something similar maybe
from there according to the plan team
number one SSH Dante BB2
removed the existing data and attempted
to run the command at first the command
would just hang seemingly doing nothing
but after retrying the command
complained that DB1 did not have enough
replication clients no problem team
number one proceeded to SSH onto DB1 and
increased this value in the config upon
attempting to reload postgres it
complained that there were too many open
connections which happens when Max
connections is set too high he lowered
this value and this time the settings
applied without issue going back to db2
he ran
have you can just raw dog access
the the fact that he is literally going
between terminals right now just like my
cackles are up and you're just feeling
bad because you know that
the CH like one time I deleted my home
directory okay and it's not
I would like to say I'm a semi-talented
engineer it happens okay like sometimes
you make mistakes it just happens and to
be bobbling around between two separate
terminals one where it's like
totally don't oopsie Daisy one terminal
and totally you can it's okay other oh
man
I also comp I have I've never heard the
story so this is very very very exciting
PG based back up again and once more it
appeared to just hang and do nothing at
this point frustration began to kick in
the engineers thought perhaps the prior
attempts to run bass backup before the
configuration changes had created some
buggy files in the data directory
interfering with the current run the fix
would be to remove these files again oh
no well might as well give it a shot
thought Team Member One a hard reset to
start on a clean slate so to say he
prepared the command to RM RF the
directory and ran it in his show
[Music]
gosh I feel so bad for this guy oh my
goodness oh no
no not RMR oh no
I should immediately after pressing
Enter he noticed the shell in which he
ran the command was the one connected to
the live production DB1 he slams control
C harder than he ever had before but it
was too late of the over 300 gigabytes
of data only 4.5 was left
if you recall a db2 was previously wiped
up data before writing the backup
command gitlab now officially had zero
data in any of their production database
servers
guys I may have just accidentally
deleted DB1
hello Good Golly what a way to end a
Friday evening
you know what I just rmrf to DB1 out of
existence please tell me this is real
looks like we need to call for some
backup the team scrambled to find a
backup of the production data they
checked for the database backups that
were supposed to be uploaded to S3 but
there was nothing there then they
checked for disk snapshots but they
found they didn't actually take these
snapshots for their database servers as
they assumed the other backup procedures
were sufficiently they checked for
logical volume snapshots or backups of
the file system gitlab had a staging
database for testing which periodically
captured snapshots from DB1 to remain up
to date with prod these snapshots were
normally captured once every 24 hours
but luckily team member one had taken a
snapshot six hours before that okay now
there were two choices they could copy
either the entire lvm snapshot or just
the postgres data directory from staging
to prod the amount of data was similar
in both options but they opted to copy
the data directory yes that would be
easier to restore from problem was their
staging environment used as your classic
in a different region without premium
storage which they cannot retroactively
upgrade therefore limiting data transfer
of rates to around 60 megabits per
second copying the data to production
took a solid 18 hours and nearly 24
hours later gitlab was back up to normal
operation the only caveat was that all
database data such as projects issues
and Snippets created in the six hours
between the lvm snapshot and the outage
were pertinently lost
[Laughter]
[Music]
who uses azure
[Music]
oh my goodness imagine being a billion
dollar company
and you can't go faster than 60 megabits
a second like imagine being in that
position be like we'll just I'll give
you a hundred thousand dollars right now
and they're like
Sorry rules rules are rules can't
upgrade you
nothing can do about it sorry
this affected around 5000 projects 5 000
comments and 700 users during the backup
restore process progress was tracked in
a publicly visible Google doc and they
even had a recovery stream on YouTube
it was discovered that the replication
lag was actually caused by a background
job trying to delete a gitlab employees
account due to it being reported for
abuse by a troll in combination with the
other spam postgres maintains right it
was it was even part of the original
problem oh my goodness that is so
awesome oh that is so good head logs on
disk every operation is first written to
these logs before being applied to the
database this way if the database
crashes and restarts in the middle of a
database edit the logs can be used to
restore the database to a consistent
State more pertinent to the issue at
hand right ahead logs are also sent to
db2 for replication there is a maximum
disk usage configured for these so old
logs will be deleted when the limit is
reached the large employee removal
operation plus the spammers caused so
many operations that logs which hadn't
been sent to db2 yet were deleted
thereby causing db2 to become
permanently out of sync they also
discovered that the database backups
weren't uploaded to S3 because the
server which was taking the database
backups was using the wrong version of
this is so funny
oh my goodness who even really needs
type so we even need types in
programming languages when there's these
kind of Errors okay this is where all
the real errors happen anyways come on
distress these failures should have sent
warning emails but they never received
any of them because they forgot to
enable dmarc authentication on the
backup server
logging onto production servers changing
configs and running random untested
commands is obviously not the best
situation to be in but it's good
practice to let someone else review
exactly what command you're running and
where you're running it before you've
run it for real how many I do think that
that is really really important which if
you're gonna rmrf on a production server
especially one involving data
never hurts to double check like it just
it just doesn't it just never hurts to
double check
ever
how many times have you asked someone a
question only to immediately realize
what the answer was or submit a pull
request and immediately find 300 bugs
that you overlooked yeah you're in the
zone it's easy to tunnel vision and make
mistakes a mental race the second one
actually that second one is such a good
observation do you look at your own pull
requests
there I find the most amount of problems
with my code when I look at my pull
request not in my editor
right I always look at the diff first
because it just helps me see it in a
whole new like I don't know I don't know
what it is but when I make a PR looking
at it in the pr view like enables a
whole new brain mode right it's like
find the mode whereas like the
other ones like make it this one's like
find it's like a completely different
brain mode and if for whatever reason I
see things I could never see looking at
the code
I highly advise against it never never
look at your own code I overlooked when
you're in the zone it's easy to tunnel
vision and make mistakes a mental reset
or a second pair of eyes is always
helpful yep but the only reason they had
to go through all these Shenanigans is
because this replication lag scenario
was never tested or well documented no
one on the call was familiar with how PG
based backup worked and how it was
actually normal behavior for it to hang
for a bit so in fact all they had to do
was wait and everything would have been
fine
thorough load testing could have also
exposed this replication like issue
before it occurred in prod but it's
possible
they didn't have the same primary
secondary setup as production nothing's
worse than it's successful you you aired
successfully like
oh oh
such a noof which is a problem in and of
itself next after DB1 was deleted they
found that many of their backups were
broken these backup steps or more
generally all rarely run or use
procedures should be manually or
automatically tested on a somewhat
frequent basis months or years can go by
without any need to restore from backup
but when disaster strikes and someone
accidentally deletes the database the
documented procedure better actually
work on paper their recovery story
looked fine but if someone had tested
all of them they would have found that
one doesn't actually work two would take
18 hours to restore from three didn't
exist and four would only work if one of
the databases was still alive lastly the
straw which blew the databases back the
deletion of the gitlab employees account
so during and before that is just
that what is this what is this deletion
what is this deletion business I want to
I wonder what that employee was doing or
the incident when a gitlab user is
removed for spam the software would
perform a synchronous delete on all the
user data so if the user had 50 million
projects all of them will be deleted
immediately the change they proposed was
to instead soft delete by marking the
user as deleted and then the real delete
can be done asynchronously and in a
controlled manner
yeah what is that so guys for those that
haven't seen this
what is the Fate you think is going to
happen is Team Member One getting
rmrf'd himself
remain promoted
died which one
type one here one in the chat for fire
two in the chat for remain three in the
chat for promoted four in the chat for
died
died of natural heart problems from this
okay most people think just remained
remained or promoted all right and then
some died
some people die okay okay this is
looking good I like this I like where
this is going a lot of deaths in
remaining remaining dead in the end team
number one was obviously not fired there
were many factors which is just maybe an
annual reward
in the end please don't fire team number
one but maybe create an annual award in
his name
team member one was obviously not fired
there were many factors which led to
that rmrf moment and many more factors
which led to it taking 18 hours to
recover from there none of which were a
single person's fault gitlab's CEO
personally apologized for the outage and
gitlab never accidentally deleted their
production database ever again that's it
that's pretty good don't accidentally
delete the database
wait what
what
that was pretty good I like that I like
that
very good all right I gotta go pee
I'll be right back