uh welcome to have been pwned and how to
serve billions of requests and terabytes
of data without going
broke uh I'm going to start this off by
telling you a little bit about myself my
name is stepan y I'm Icelandic and for
this whole reason of being Icelandic you
probably have a measurable portion of
the entire country here in the room with
you I'm a principal software engineer at
Lucin in re also Microsoft
MVP and I may have been pwned conf and
maintainer and have been for the for a
while that's a link to my blog want to
take a look at
it but to start this off uh have I been
pwned to answer all your questions then
yes you
have it was launched on December 4th
2013 by Troy hunt Troy Hunt's an
Australian he used to frequent NDC but
unfortunately he's not here with us
today it allows like the gist of it is
it allows internet users to check out if
their data have or has been leaked in a
data
breach uh some
statistics currently it contains
breaches for 6 674
websites over 12 billion email addresses
which don't have to think about very
long to realize that there more email
addresses than are humans in the world
and over 850 million passwords or their
hashes to be more
correct have been pwn has a notification
service you can register and verify your
email and how in pwn will let you know
if that email is found in uh data bries
as they come
in it also has uh domain search
functionality for example if you have
your own domain or if you are a system
administrator you can verify your domain
with have been poned and it will then
let you know if any email addresses
under that domain are found in a data
bridge is this is very
handy then
it also has an ingestion pipeline uh
which is used by law enforcement uh and
government so for example agencies like
the FBI uh and the NCA NCA for those
that don't know is pretty much the
equivalent of the FBI in the UK they
actually feed data or passwords into
have been pwned uh through an API
pipeline that we
have and just over the last year they
ingested millions of
passwords so it's good to get these
things these are these are passwords
that they find for example in their
investigations uh SE teams from
different governments also have access
to have been poned they get they get
free access to it so they can like if
any any government email addresses are
found in data breaches the SE teams are
notified and as far as I know and the
last time I checked it 34 governments
signed up it's USA UK Germany Iceland I
think pretty much all the Nordic
countries for
example
uh but
going to ask you very quickly if you
recognize this number I'm not I'm not
expecting you to you probably don't and
you're not supposed
to but be clear a little bit later
on but many of you might recognize this
number for the sole reason of like we
all know how to
count but this number is actually uh
according to data breaches the most
popular password in the
world so I'm really really hoping that
none of you actually have this as a
password Suare uh if you do please go
ahead and change
it but if you worked with all computers
you would probably recognize that this
is a hexadecimal
string but to be more correct this is
actually sh one hash of 1 2 3 4 5 6 and
why is that
important because we can actually use
the Shan hash to see if 1 2 3 456 has
been a data
Bridge simply take the five first five
characters go to this URL get a text
file back and you look for the rest of
the hash in the file and there you see
the number
behind this is how often 1 2 3 4 5 6 has
been seen in a data
Bridge yes that's
37.6 million times and it's probably
been in a data breach a lot more often
these are just the data breaches that we
can actually see that the password has
been in
there so have a been pwn has an API the
P password API and that's sort of
becomes the focus of this talk because
that's where most of the traffic goes
like I said in there there are 850
million password hashes in that
API All the known breach passwords that
we have found in the data breaches we
both have sha one and nlm hashes uh ntlm
hashes are good because like I said or
like it says you can download the hashes
for offline use like in a big
file so you can you can check off line
you don't have to have to use the API
for everything um and for example in the
case of ntlm if you're if you're a
system administrator for an active
directory domain you can actually fetch
that file and see if any of your users
have well what's considered uh unfit
passwords for use without knowing what
the password
is it can be integrated in a number of
ways and it has been so just just to
name uh three cases one password uses
have been pwned for its watch to Fe
feature uh they have a
feature where they actually like go over
your passwords that you have in your
password manager and can actually tell
you no this one is not too complex and
this one is actually vulnerable Mozilla
uses have a been pone for Firefox
monitor so they can tell you if the
email address you're signing up for has
uh or using in any of your logins has
been in a data breach and the email line
where I used to work uh actually can
tell users when they're logging in if
their password is not safe anymore
so these are just a just a few examples
for example the1 line one this is what
happens when you log in with a bad
password you get notified it says Hey
because you know the only place where
they actually have your password when
you're logging in so they say hey like
this password you're using is not safe
you should go to account management and
change it and enable two Factor
authentication if you haven't and then
it like gives you links to like good
password hygien uh in the middle you see
the one password integration it has
Watch Tower and the big Red Square there
is actually checking they have a been
toned API to see if the passwords are
any of your passwords in your one
password Vault are
insecure and Firefox monitor tells you
that like your email address has been
exposed in several data breaches yes
that's my email address I'm actually
happy that it's just 11 probably could
be a lot more um and then it tells you
like which data bees you've been in and
what data was actually leaked in that
data
breach so you can EAS look up and see if
a password is unfit for use uh just
consider any password that's in there to
be unfit for use and why is that
because these password lists or pretty
much since we have the passwords that
means the bad guys have the passwords as
well and they regularly go through this
list and just like go rampant on
services with partnet like checking
every or usually the most popular
passwords and basically see what what
sticks and unfortunately a lot of them
stick
now the password itself is never sent to
the AL Pon API and neither is the full
hash of it uh and that's because of K
anonymity and I'm going to explain that
in a little bit but that's used to
prevent uh for example if your network
traffic was going through a proxy
whoever controls the proxy can't
actually see what password you're
looking up because that would be
bad so pretty easy to use calculate the
sh one hash of the password issue a
request look up the rest of it in the
text file and see if you can find the
rest of the hash in
there but to make the example like even
more clear uh let's start with Hunter
tube this is a this is a story from IRC
I don't remember how many of you
remember IRC
uh see if you can spot the joke in
there so basically this is a chat going
on between two people I want him saying
hey if you type in your password it just
shows up Stars so the guy types in his
password hun2 uh do look like stars to
me so the other guy pays it back saying
no it looks like stars to me oh so you
can go Hunter 2 my Hunter 2 and Hunter 2
right does that look funny to you and
the guy said yeah I just see a bunch of
stars all I see is like when you type
Hunter two it shows up with stars wait
so you know my password uh no because
like I copy pasted your stars so it like
shows correctly up with
you oh
okay so let's use Hunter 2 as an example
this is the sh one hash from Hunter 2
and
what we're interested in is these
parts the green part is actually what we
will be appending to the
URL so we do a get request to the API
the green
part we get back a text file and we will
be looking for a line starting with red
part and in the case of 100 two this is
what you'll find and yes someone or some
people have used2 23,000 times as a
password probably because they thought
it was a good idea it isn't
um so that's that's how easy it is to
actually use the API just to sort of go
over the hosting
environment uh hosting environment is
that simple really uh the hashes are
just text files they're stored in Azure
blob
storage email addresses are stored in
Azure table storage where we actually
use the domain as a partitioning key and
the uh everything before the ad sign is
a row key
both the email and domain search is run
as Azure functions uh the hash files are
also served by Azure functions like all
they do is like just an HTP trigger
fetches the taex file and returns it
back and this is actually all these as
functions are open sourced at least the
pan post P passwords one are so you can
actually look them up there if you
want uh so traffic is growing
fast this is the request per month that
the P passwords API receives
started out pretty small uh then it is
gradually increasing and it currently
seems to be almost increasing in an
exponential rate um I hope it doesn't
continue doing that because we're going
to have to figure something out
then but just to look at the numbers as
of this month like all of the Shan D nlm
hashes are 60 GB that's all the text
files now considering like for both ntlm
and ch1 you really just have over 1
million tax files because there's a
limited key range in there the average
tax file is 30
Koby and we are currently serving over 5
billion HTTP requests per month so
putting that down is it's over 2,000
requests per second every second every
minute every hour every day all the time
on
average so that's a
lot uh when I was actually gathering
this data I figured out like when when
did I actually put in a proposal for
this talk and in The Proposal I actually
said I was mentioning 1.6 billion
requests per month that was uh because
the numbers I had were from March
222 so I'm sorry that uh this was a lie
because currently this is up to five
like I said 5 billion requests as of
this
month and to sort of try to put this
into perspective
uh of how much data this
is this is is 126.80
that so that sort of sort of puts it
into perspective how much data is
actually being
transferred so if you do some maths
serving this like straight out of Asher
uh with nothing in between would be
expensive if you think about us fetching
tax files from blob storage it would be
5 billion read
operations uh I actually had to update
that number should be 126 terab not 125
but it's not going to make all
difference and putting this into the
Azure calculator comes out to many
dollars and not such wow to be more
exact it would be more than
$12,600 per month just to just to serve
this data and this like we're just
reading data and putting it back that's
because Asher actually builds you for
the amount of data that you send through
the network and for of course for the uh
read operations from the
blobs but as like probably will be all
of you figured out this is completely
static data so there's no reason for us
to do this all the time every time so of
course we cash all the things in there
but caching is not enough like we could
cash like just in the Azure function and
save on the read operations but we'
still be serving those terabytes of data
so we had to figure out uh something
else so the obvious reason would be to
put a CDN in front of it and we went
with Cloud
flare they have a CDN uh their CDN is
fast it's actually very very fast uh
Global reach they have 250 Edge nodes
all over the world like 10,000 networks
connect pretty much directly to them
that's isps um like Cloud providers and
a lot of other people they have a free
tier which does a lot of the things that
I'll be talking about without you having
to pay anything actually so this seemed
like a very perfect fit and when I say
it's faster than I think I actually mean
that literally because they say that 95%
of the world's population is within 50
milliseconds of a cloud for data center
it takes you 100 milliseconds to
Blink just to put that into
perspective so cloudfare caching uh it
cloudfare is very simple you basically
just route your domain through them and
they they do a lot of things just out of
the box and a lot of things is just
pretty much just like flipping on
switches uh there are some things you
have to think of though it depends on
the content type you're serving it's
going to be easy or not but we figured
okay we can just set the cash TTL or the
time to live to a very high value and
that is currently set to 30
days so we're basically telling all the
cloud for Edge nodes to Cache The Blob
files for 30 days on the edge nodes but
you get a lot of other perks uh as well
from cloud fa like they they give you
https they can give you HTTP 2 and 3
like all the latest protocols brought
compression they just do all of these
things for you and you don't have to
worry about like flipping those switches
on your own web
servers the initial implementation that
we actually did
was pretty much just create a cloud for
worker that's it's pretty much just a
snippet of of JavaScript that you can
run on the cloud for
nodes so what that does is it gets a
request goes to the origin which is the
aure
servers uh fetches the data sets some
course headers so people can like do
this directly from their browsers
validates the request like is it a get
request there the like hash prefix valid
is five characters Etc and then we do
URL normalization because one of the
things that is sneaky is that URLs are
case sensitive well technically not all
of it because the scheme HTTP or htps
isn't and the host name is not but the
rest of the URL is Cas
sensitive so to for example if we would
fetch
uh if we would fetch the hash file for
fffff all in lower case web servers
would look at it differently as in if it
was all uppercase so what the cloud
flare worker does is basically just gets
the URL converts the range to uppercase
so we're always fetching the same thing
otherwise we would be could be taking up
multiple cash slots for the same
request the cloud worker is open source
as well if you want to take a look at
it so that's that's there
the initial results of that
implementation was that the cash it was
99.5% which is a
lot so with our current numbers like
that would bring the bill down from
12,000 how was it $12,600 down to 63
which is good or actually it's great and
like I mentioned you can download all
the hashes for offline use and clouds
were even cached the big single fall
downloads which were like almost 15 GB
they of course are compressed we're not
like giving a 60 GB text file so it's
the uh single file downloads are
compressed
already but this this what they do and
like I said great success so we had all
static data cashed for a long time or
until we
purched not a 100% hit ratio because
cloudfare has other customers than us so
they might they might need some space on
their Edge no so they might like boot
some of our stuff off their off their
cash to get make room for something else
but
like almost every request was going
through them and not to the origin so
everything was awesome
right uh is it
though
so in December 2021 that's when we
actually open sourced the pawn passwords
API and at the same time we got a lot of
new passwords from uh the FBI and the
NCA like 225 millions of them I think
the I think the number was that we we
went up from like 400 or something
million or 500 something million up to
like 6 or 700 so there was a very
sizable increase to the amount of data
that was in the uh that was in The Blob
files and like getting a lot of new data
like for this kind of service would be a
good thing
right well turns out there are limits to
how big a single item in the cach can be
and why do I say a single
item because I mentioned the big files
that you can download like The like I
want to get a zip file of all the Shan
hashes like one big file 15 gab like
almost 15 GB that was before they added
225 million new passwords in there so
the limit for Pawn password was 15
GB and how do we know this well we can
ask tooy
wallet what I mean by asking tooy wallet
is this
this is this is the expenditure report
Troy got that month see everything is
fine up until December 20th when it's
Bill skyrocketed like absolutely
skyrocketed
11,448 Australian doll which is like I
think it's around the exchange rate at
the time I think it's around like $8,000
us so we had to take a look at this like
what's going on we got to
investigate so we started looking at the
cloud for
dashboards and this is that uh
that that day so I'm going to point a
few things out this was in 24 hours all
of a sudden the origin had served 3
terab of
data and it was due to cash misses now
you can see like I don't can't really
tell how clear it is like the little
green bar there is the text files they
were all being cached so that those
weren't a problem but what it says been
there the binary files the big zip text
files they were hardly ever being
cached
so that was a something that was an
oversight but when you have a service
with this amount of traffic little
oversight can result in a pretty massive
Bill uh we solved the issue by removing
the file downloads s ofile
downloads put in uh like Azure
expenditure like monitors and alerting
so we now get alerts if the expenditure
starts starts Rising too quickly and we
to bring back the syn of all downloads
We decided to create a downloader uh we
also thought use this opportunity
because as we've added the API for like
the FBI and others to put stuff in there
we realized that the data would be
changing a lot more often than it did so
we decided okay we'll just create a
downloader we'll open source it and I
can just download all the files because
you have a limited range of like file LS
that you can get so it's pretty easy to
just go through them it's pretty much
just a for
loop everything was good for a
while but then we added the nlm hashes
that was February this
year so adding the ntlm hashes uh was
like wasn't hard like we already had all
the data so we decided okay we'll just
like this was a this was like a big
feature request so we decided okay we'll
just put all the Anum hashes to like new
text files put that up there and update
the Azure
function and we thought well
everything's
good until Troy slides into my DMs uh
saying like wondering why the erress
data is still quite high and we it took
a while to figure this out uh but in the
end it just pretty much just came down
to numbers
um what happened when we actually
introduced the anlm hashes was that we
added 1 million5 files again so we went
from 1 million files to 2 million
files
so that the amount of files that were
being increased uh basically caused
Cloud to start like we were taking up
instead of taking like a million slots
for the cash items we started taking 2
million and like I said they have other
customers so they might have to boot
things off the cash um so items were
getting like booted off their Edge cach
more
frequently and unfortunately that means
that the sh one hashes which were
primarily ones being fetched because
this was a new feature it meant that
they started get get booting up more
often and like we saw earlier when like
the cash hit rate drops the bills go
up so we started talking with cloudfare
okay how can we get the cashit rate back
up and Cloud for has a thing called
Cloud for cash
Reserve now cash Reserve is I think it's
still in beta but what it is is a cash
for the
cash so we spend less
cash uh but how does it
work uh it's not it's not very complex
sits between the origin and the edge
nodes and it's just magic but it's not
really magic uh it's built on what they
call
R2 uh which is is a bit of a naming like
they're making a little bit of fun of
Amazon there because they have S3 does
the same thing it's S3 compatible so
they call it R2 it's basically just
another like storage bucket so what they
actually do is they sit between the edge
nodes and the
origin and as the like request hit the
origin they start saving them to uh to
their cash
Reserve but to do this they have some
requirements uh CU these are things that
are being stored like this is this is
not just
this is no longer just living in the ate
this is living in a bucket somewhere so
they have some requirements like of
course the data needs to be cashable um
B it is we're just like serving a bunch
of text files should have time to live
at least 10 hours like check um and then
needs to return a Content link header
like we already know how big the files
are so not a
problem where is
it we thought this wasn't a problem
because like we tested this of course
locally and we always got a Content
length header
but when we pushed it to Azure we
stopped getting the cont length header
and we were trying to figure out what's
going
on now Azure which in this case like has
I serving this data they saw that the
requests were coming in for text files
and text files are not jpegs or zip
files so they're usually uncompressed
data so I in Azure thought hey I'm just
going to be I'm just going to be helpful
this is static data I'm just going to
dynamically compress it on the way out
as I'm
responding so trying to be helpful I
guess uh but the way Dynamic compression
works is it gets the actual data and it
as it's streaming it out it just
compresses it that is if the client
actually can handle
it but I'm wondering if any of you can
spot the problem
here so like I said it's dynamically
compressing things so it doesn't know
how big the response is going to be
until it's finished
responding so is it ringing any
bells what's the first thing that gets
as like the first part of the response
from HP
server it's the headers
right the headers include content length
which should tell you how big the
response is going to be but as as is
responding it doesn't know how big the
response is going to be because it's not
done compressing it it doesn't compress
it all in memory because if you would
serving a 15 GB file like You' be taking
up all the memory on the
server so as the response heads are sent
before the actual response is
sent well
obviously uh it removes the content
length
header and this took a while to figure
out but we thought all right they are
compressing the data like because they
they're trying to be helpful and makes
sense for like most use cases but we'll
just go ahead and disable the
compression natur Dynamic
compression but we
can't and I can tell you I really really
tried I think I tried every possible
setting I could find even some settings
I probably shouldn't have found um try
to mess with I tried to mess with like
configuration files and everything else
to turn this off until I finally just
basically got a response saying nope
sorry you
can't and upon re receiving the response
this was sort of my
reaction uh I was not happy
because this was a requirement for us to
be able to put the cash Reserve in there
and while we couldn't fix this like the
bills are just going
up so I started thinking a lot about it
but I actually did find the fix in the
end though so this is me hitting the uh
development like Azure function directly
not going through Cloud
FL as you can see I'm like the content
encoding V that means broadly that means
it's compressed and I'm getting a
Content length header
back now
this was not this was not the cleanest
solution but I figured hey like if I
just compress the response before I gets
it there's no reason for is to compress
it so in the aure function because I can
assume or like I can assume I know that
the files are not that big I can allow
myself the luxury of just reading them
all into memory compressing them there
and then just handing is back the
compressed response so that's pretty
much what I'm doing I'm just taking over
the responsibility of of doing the
compression depending on if the client
supports compression or
not
so went through the requirements so now
we fulfill all the requirements and upon
seeing this work I was quite
happy
uh so this is the current
situation uh with the functions this is
a single day of requests going into um
into Cloud for the for the hash
ranges and as you can see like we get
16731 million request and 16731 million
requests are served by Cloud 184 went to
the
origin which if you put this into
numbers it's a
99.9999% hit ratio which is literally
one in a million so like you can
actually see when you're doing Quest 2
Cloud flare you can see the response
headers if it was a c hit or not so if
you ever look at the cash headers uh
coming from the P password API and you
see a cash miss you can be pretty happy
because you're literally one in a
million uh right
there so this was this was pretty
sucessful and this is actually something
that we did just like 3 months
ago so okay we thought okay let's wrap
things up without the cloud FL caching
uh the as reg fees and read operations
like I said would exceed $12,000 month
and that is not really sustainable if
you're trying to run a free service uh
especially if there's just like one
person paying for all
this the a network Eis fees are free
because they say like the first 100 gbt
are free and we like since we put in
Cloud for cash Reserve we never exceed
that so we're not paying any e fees
anymore at least from last year the
storage BL read operations are free as
well because they give you 10,000 read
operations per month and we never
actually hit those unless we actually
purchas the cash which is
rare so the entire Azure storage bill
right now is just like
$1.25 for the 60 GB we store in the blob
storage cler does not build EAS fees and
this is what actually makes this
possible but for cash Reserve they
actually do bill you for the amount of
data stored and how many operations they
have to read from the storage similar to
The aure Blob storage they have to store
a lot of data and it's not free
so they bill you for reads rights uh and
usually rights aren't that many they're
like we only write to the cash Reserve
like I said when we purchas the cash but
otherwise we're just reading from it and
usually the edge nodes are taking like
uh a bunch of the hits so that's not big
either so I think the total amounts to
like $16.5 us per month like I said
earlier we before we actually hit cash
reserve and before we actually started
getting all the cashes evicted we
calculated would be around $67 per month
but like after adding the nlm hashes
that would have been rising pretty
quickly so 16.5 us per month instead of
12,000 is pretty much a 99%
saving so recapping all of this like
sort of what what we've learned from all
this is well the obvious the obvious
learning from this is Cash static data
there's no reason for you to spend CPU
Cycles or memory or whatever on
constantly serving data that's never
going to change like you can you can
just cach it and return it like that
directly if you have static data with
like long lifetimes for example if
you're serving let's say zip files that
rarely change or you're serving images
that rarely change or like you rarely
update a CSS or whatever should think
about serving it over a CDN because that
will actually bring that data a lot
closer to the users so it'll be a lot
faster to fetch it and it's never going
to hit your servers at least rarely
uh only the scheme and the host in a URL
are case insensitive it's what I was
explaining with the URL normalization so
these two URLs are not the same and not
not according to web servers a lot of
web servers they do automatically like
if they would get one of these and they
were like working against the case
senstive File system and not finding
they would just try okay can I find
another file that
fits but you should never assume that uh
the URLs are or whatever is serving the
URLs is is uh case
insensitive you need to make sure that
you can easily Purge or invalidate the
cach when you have like either on your
side or the CDN uh thankfully like
cloudfare uh
Asher like Asher CDN Amazon cloudfront
they all have apis that that do this for
you or dashboards where you can like
click a button and do this uh to make it
easier but a good thing to also think
about is for example in the case of CSS
or JavaScript uh it would be good for
you to just not have like my site. Js
and when it changes then users are not
going to get the new version because all
of them have cached it it would be good
for example to append something to the
file name like the hash of the file or
something else because then if the file
changes you'll be getting a different
file
name so that's uh that's a good thing to
do as
well be aware of caching limits in our
case the the 15 GB one was like it was
not really well documented it's not sort
of an extreme case but usually the CD
intercast providers will actually tell
you like you know you need to do this
and this and this for everything to work
one of them is for example the headers
make sure that they comply with whatever
the CDN requirements you have because
otherwise you're putting a CD in front
it's not going to do anything because
you're not telling it the correct things
to
do and that sort
of that sort of brings it down to the um
the gist of it of of actually how this
is done I mean like none of this is
really complex for ex especially for
static data but as soon as you start
getting Dynamic data there are all the
things you have to think
about
now I decided to uh leave a lot of room
for questions if you have any
um this is it's not a topic that like a
lot of people think about but probably
should if you're serving any kind of
data so if you have any feel free to
bring them
on
AF might change
so the question is are we afraid that
cloudfl might change his pricing policy
uh no not at the moment at least uh
they they like make themselves out to be
like uh say they make themselves out to
be fast and have these pricing policies
usually for these big companies when
they're changing their pricing policies
they usually get a very big hats up or
like at least some some uh a very good
time in advance for you to actually
respond to it so it's not really
something we're worried about at the
moment at
least anyone
else
yeah so the question is like because
we're storing the fouls based on the
first five digits of the hash like would
it make a difference if we put it down
to six uh the difference would be that
well the obvious difference would be we
go from 1 million files to 16 million
files
um so then now this is sort of a
convenience thing but like the more
digits we actually put in the hash the
more chances are of someone actually
identifying what password you're looking
up because we'd be giving like uh we'd
be giving servers or everyone in the
middle proxy servers like more
information even if it's just six digits
but you've you've cut the you've got the
like possibilities of the passwords down
a lot they will be would be smaller
files that be returning but we found
that five digits uh pretty much The
Sweet Spot
there anyone
else all right so thank you very much
then uh you can find my blog if you want
find me on Twitter send me an email if
you have more questions I have a bunch
of iin P stickers here in front which
you can feel free to grab if you want um
don't forget the feedback on your way
out and enjoy the rest of NC thank you
very much