please remember there are additional
resources and things such as code
samples at elithecomputerguy outcom so
if you're watching one of these classes
and you need to know what code looks
like or if you need links to the
resources that we're using please go to
elithecomputerguy
docomo please remember that free to the
end user classes are not actually free
it cost me a lot of money to be able to
provide this type of content so if you
could click on the Donate button and
throw in a couple of dollars every month
this will help me be able to continue to
provide you this type of material
welcome back as you know I am Eli the
computer guy and today's class is
sanitize strings in Python so what do we
mean by sanitizing strings basically
what we mean by sanitizing strings is
that we're going to clean up the string
of variable value so that we can use it
within our code so one of the problems
that I've talked about before is this
whole idea with program progamming or
with databases with garbage in equals
garbage out right if you put bad data in
or poorly formatted data into your data
store when you go go to pull the
information out or when you go to do
queries it is going to be a bit of a
disaster for you so simple thing so if
basically if you have users inputting
information uh can you clean up the
information that they're they're putting
into your system so that ites doesn't
create vulnerabilities so one of the
things I'm going to show you how to do
today is it's called an HTML Escape so
if a user basically is trying to input
uh values into your system to try to
hack it some well some way basically be
able to do an injection attack what'll
happen is this little Escape method will
go through and it will change out all
the HTML tags into something that looks
like HTML but isn't actually HTML uh so
that you will not add vulnerability to
your PL platform if it's displayed uh to
other users other things such as
different systems deal with
capitalization differently right uh so a
lot of modern systems if you have a name
that has an uppercase first letter and
lowercase other letters and then you
have the exact same name where it's all
lowercase many modern systems will see
that as the same variable
value many will many not all so one of
the things that you might want to do is
simply whenever users are inputting data
into your system simply make everything
lowercase or make everything uppercase
so that the case with the case for all
those variable values in your database
or your data store is the same and you
don't run into any kind of weird
problems or basically uh if you're
grabbing uh data uh possibly something
from something like an API call being
able to go in and be able to delete
parts of the entire amount of data that
comes in so you just get the variable
value that you're looking for so that's
the type of thing that we're talking
about when we talk about sanitizing
strings I'm going to show you some ways
to sanitize strings today uh but there
are many ways uh to do this and what you
need to be thinking about is your
particular project and your particular
platform what vulnerabilities will your
platform have what are the concerns that
you have with the data coming in and
then you design your sanitization
process based off of those issues now
here's an example of sanitization using
the lower method the upper method and
the title method so let's say we have
some kind of form that users are
plugging information into and so they
plug in their name of Bob uppercase B
lowercase o uppercase B right now again
with most modern database systems that
type of thing uh this would not be an
issue but with some systems this may be
different than uppercase B lowercase o
lowercase b or all lower cases or all
upper cases or whatever else and so
basically what these methods allow us to
do lower upper and title is that they
will modify uh the variable uh so that
everything will be in lower case
everything will be in uppercase or the
first letter will be capitalized and
everything else will be in lowercase so
if we go and we run this basically we
can see the original name is Bob b o b
so uppercase right then we go name.
lower now everything is lowercase then
we do name. uper everything is an
uppercase then we do title first B is
uppercase everything else is lowercase
and so this is one of those things that
you can do to sanitize the the values
that are coming in right if you were
getting things uh like street names
email addresses people's names whatever
else simply making all of those
lowercase or uppercase but most likely
lowercase and inputting that into your
database that would just create a
consistency of the variable values uh
how they're they're styled or how
they're formatted so that you might not
run into the issues going into the
future now the next method that you can
use is something called the replace
method so what the replace method does
is wherever it sees the character that
you designate it will place it with a
different character uh so up here right
so we have the single quotation marks
and then within the single quotation
marks we have Bob within double
quotation marks so let's say we're
pulling this value again maybe out some
kind of CSV file a comma separated file
F value file uh so we had some kind of
program we exported the CSV file and
then now everything within that CSV file
not only is it separated by commas but
it's also has double quotation marks
around found all of the values and so if
you want to put that data into your
database you just care about the name
you don't want those additional
quotation marks so basically what we can
do here is we say name. replace we put
single quotation marks and we say we
want to get rid of the double quotation
mark and we want to replace it with
nothing so single quotation mark no
space single quotation mark right so if
we do that basically we do the replace
we can see that the original Bob so Bob
within the double quation marks is here
and then Bob without the double
quotation marks is here so we replaced
the double quotation marks with nothing
right just to show you that we're
actually replacing it with things so I
could do this I could do let's say three
three hashes for some reason or three uh
yeah three uh lines for some reason now
I do this we can see we have Bob within
double quotation marks and now we have
Bob here uh with three uh three hyphens
on either side so basically what the
replace allows you to do is it allow
allows you to go in there and then
basically be able to replace specific
text uh with whatever it is that you
want to replace with if you don't want
to replace with anything you can simply
replace it with single quotation mark
single quotation mark nothing in the
middle and so that will get rid of it
and so that can be very useful again uh
for getting rid of punctuation marks
again A lot of times when you're
importing data you have a lot of uh
punctuation marks you might have a lot
of special um you know special
characters based off of the system that
importing from and so this is a way to
Simply get rid of those types of special
characters uh to get you to the value
that you're looking for the next method
that we're going to look at is the strip
method so if we use the strip method
without any additional arguments what
this is going to do is is going to get
rid of the white space either to to both
the front and the back of the value that
you're bringing in so again this can be
an issue let's say here so Bob we have
three single spaces and then we have
three uh to begin with with and three
single spaces at the end right this is
going to be different than Bob with no
single spaces so that's a problem that
you can run into right somebody somebody
might not be trying to be nefarious they
simply put a space right so they're
looking at that web form for some reason
they just put a space in the beginning
because you know they're being users and
if you try to do space with Bob that is
going to be different than Bob without
any spaces and so what uh the strip does
if you have no additional arguments is
it will simply strip out white space uh
so we have print we're doing an F string
I'm putting these hyphens here just so
you can see where the spaces are or
aren't doing the same thing down here
after we've stripped uh the white space
to either side so when we run this we'll
see hyphen hyen hyphen then we have
three spaces Bob then three spaces
hyphen hyphen hyphen right so this is
the entire variable value and again if
you put that into the database you can
have problems so here what we're doing
is we're stripping out the white space
around the variable value so they we
only have Bob and so now you can see
that there's no white space around that
variable value Bob uh for the name and
so that's one of things that you should
really do again whenever you're getting
users uh input from users sometimes they
they do a couple of spaces they do weird
things maybe they type their name and
then they do a space right after they
type their name for some strange reason
and that can be a problem because the
name with the space is the entire
variable value and depending on what
you're doing that might run you into
issues later uh then we can come down
here and again when we start looking at
um uh things like strip uh we can strip
uh from both the front and back of the
string uh whatever characters right uh
so let's say we're just trying to Hash
up like we're just trying to make it so
if somebody tries to put in like HTML
information we're just going to screw it
up for him for whatever reason so this
person plugs into username H1 Bob close
H1 right so they're trying to make Bob
all big they're trying to add some kind
of HTML tags into this so what we're
going to say here is uh from either the
front or back we want to eliminate these
little uh the little angle brackets here
right we want to get rid of those and so
what we're going to do is when we run
this we're going to see this is the
original Bob so Bob within the H1 tags
and if you look at Bob down here you'll
know notice the angle brackets at the
end have been eliminated so we simply
got rid of those two uh again basically
to screw it up and also just to kind of
show you how that works uh beyond that
here let me just uh comment out that uh
we can keep going down so we print
username uh so we can left strip so what
left strip does so strip will strip both
to the front and back of that variable
value L strip will strip to the left and
R strip will strip from the right and so
if we do this then right so basically
we're doing here username equals
username DOL strip so left strip H1 so
if we come down here we see we had uh H1
Bob close H1 then we killed the angle
brackets on either side and now you'll
notice with with Bob there is no longer
an H1 to begin with because we l
stripped we left stripped off the H1 uh
from that left side if we go down here
again we can do right strip so R strip
for H1 if we want to get rid of this H1
over here we can strip that out and now
we have just Bob within these particular
angle brackets so this can be very
useful like we're going to do um oh
we're going to be doing more projects uh
with using um OS application so we Ed
the OS module before so you can use
things like inmap so inmap will scan a
network for you and then present you
with with all the values that it found
so many times if you want to take those
values and dump it into a database you
have to clean those values up so one of
the things that you can do is you can
use strip and you can use l strip you
know to remove if especially if you know
exactly how the formatting is going to
look okay you remove remove some from
the left you move some from the right
and then you just get the variable value
that you're looking for uh that's one of
the things that that can be very useful
for you with with output that
standardized output that you know that
you're going to get something for uh so
anyways so the big thing with strip so
if you don't do anything else if if you
never do anything else just make sure
you know the strip method if you just
use it without arguments will strip
white space to the front and back of the
variable you need to do that anytime
you're getting values from the user
basically almost anytime you're getting
values from anything you need to strip
the white space from the front to the
back because that can cause you problems
and then from here again you can use a
strip to strip from the front and back
specific characters and you can use l
strip and R strip to strip SCP specific
characters again from from the the front
or from the back of the variable value
now we're going to look at the HTML
Escape method so this is a type of
method where you might need to do a
little bit of research to figure out
exactly you know what vulnerabilities
you're looking for if you're going to be
inputting into a SQL database if you're
going to be inputting into some type of
other data store you might need to look
for specific uh uh functions or modules
that allow you to clean up that
particular type of data so with this is
for is this is basically to prevent
attacks that you may have seen in the
old days you go back about five years
ago on a lot of web forums so remember
not that long ago people would would
throw up their web forum and then all of
a sudden within the the discussion of
the web Forum you'd have iframes that
would pop up and JavaScript that would
pump up and there' be links there'd be a
whole bunch of vulnerability garbage
there uh that would cause problems for
the people that were actually trying to
use the board and the reason was is
because uh before we had good
sanitization procedures basically
wherever you had text somebody could
actually input HTML JavaScript CSS that
whole thing into the into the message
into the input and then the web browser
would simply display all of that
regardless of what anybody wanted and so
that was a big vulnerability back in the
day so you could actually add like an
iframe or you'd even add JavaScript
directly into the the message into the
discussion on these these bulletin
boards and then you can have things like
uh oh you know we used to have um
registry edits actually get dumped
directly into Windows machines simply
from that type of thing and so what
we're going to be using today is we're
going to use the html. Escape so we're
going to be using the Escape method and
what this is going to do is it's going
to change uh anything that looks like an
H1 tag into something in text that looks
like HTML tags but are not actual HTML
tags you may have heard this in the
hacking world right now things like with
uh oh fishing attacks where you'll get
an email and it'll say like click on
this thing and it will look it'll the
character will look almost exactly like
the normal character looks like but it's
slightly different that's the kind of
same thing that we're doing here is that
when this gets printed onto a page that
has HTML you'll see what looks like
these tags but they're not actually the
tag themselves let me actually run the
script so you can see what I'm talking
about so user message equals hello world
with an H1 and then we have a herf for
clickme going to cnn.com the most
dangerous website in the world uh
basically we are going to write that
we're going to write that to our
sanitized. HTML uh page then we're going
to sanitize a string we're going to
sanitize a string and then print the
exact same uh the same string after it's
been sanitized to the same page uh so
I'm going to run this we don't get any
response because we don't need it we
have the sanitize HTML got created I
double click on this and so this is what
you can see
right so the nons sanitized the nons
sanitized text that went in says hello
world and then click me right the
sanitized text we have things here that
look like HTML tags but they're not
actually HTML tags and so therefore the
web browser uh doesn't present them as
HTML tags like for this clickme right I
can click on
that no and it takes me directly to
cnn.com and my brain will be melted uh
if I go and I take a look at source so I
can do show page Source what I can see
here all right let me uh can I expand
this what I can see here is see hello
world is within H1 AHF CNN.com click me
right that's all within the arfs and
then have an HR that's just for a line
but then what you'll notice down here in
the Santa Iz string instead of having
the H1 instead of having the HTML tags
we have these asky codes right and these
asky codes will provide us something
that looks like the uh the HTML tags but
are not actually functional HTML tags
and so this is one of those things that
you can use like uh you'll see this a
lot like if you use WordPress and you
create a block that allows you to add
code so people can see it right so if
you're doing uh so if you go to Eli
thecomputer guy.com and uh you go to it
and you look at our code blocks one of
the ways that we can show things like
HTML code or CS code and it doesn't
screw things up is is basically it
formats it using these asky codes
instead of using the actual text that
it's supposed to use and so that's how
the sanitization procedure protects your
uh your system uh from from being
vulnerable the other nice thing about
this is that you can actually instead of
stripping this out so if you strip this
out um then nobody realizes that
somebody's trying to hack them they
don't real realiz that this might be a
bad actor by leaving all of this in
simply escaping out what they did it
kind of proves that this is probably not
somebody that you want to be dealing
with because they're trying to hack your
system but if we go over take a look at
it again it's very very simple script so
we're going to import the HTML module
and again this is where I say depending
on what you're trying to escape out of
you may need a specific module this is
the HTML module user message equals
again the H1 with the a ORF with open so
we're going to open the sanitized. HTML
we're going to overwrite it so w is
overwrite as file we're going to do
file. write the user message and we're
also going to do file. right HR that's
going to give us our line so this is
going to give us the user message with
the HTML formatting intact then we're
going to go here sanitized message
equals html. Escape user message
variable value and then we're simply
with open sanitize HTML a for append add
to that file as file file right
sanitized message and that's going to
give us the sanitized message and that's
really all there is to it and again
that's one thing to think about with
like the sanitization uh of these
variable values is many times it can be
very very very simple uh and you know if
you add this all into one particular
function that you create for all
variable values that are coming into
your system you can just kind of Stack
these on top of each other uh to really
clean things out uh pretty well but
anyways that's the HTML Escape and
something that you should be looking at
again if you have if you have anything
where people input information and then
it will be displayed to other users
simply doing a basic thing like HTML
Escape can be very very valuable from a
from a security standpoint for your
platform so there you go there is a
basic overview of how to sanitize
strings in Python uh so in the previous
class we learned how to filter uh
variable values so basically if people
are trying to give you data and it's not
formatt it properly right you want an
email address and they give you a phone
number you want a phone number they give
you an email address I showed you how to
filter that so basically uh your code
will see what they presented say no
that's not what I'm looking for and then
Reas for it so that doesn't go into the
system so filtering is a very important
uh component of actually making sure
that your data is clean and then
sanitizing the data once you know okay
they actually plugged in their name then
actually cleaning that data up getting
rid of things like white spaces that's
the next uh thing that you're going to
need to do to be able to clean up your
data again having clean data is the most
important thing for your system uh when
you're when you're going and you're
going to be adding up data again you
want to know all the sales for a day or
the average of sales for a day or if you
need to know all of your different users
making sure that your data is clean and
appropriately formatted is going to be
one of the most important tasks for you
so that it doesn't really mess up users
uh going down the pike again if you have
if you have a user account um and it's
uh you know and it's uppercase b o and
then you have a different user account
that's all lowercase Bo o and you have a
different user account with a mix right
that's going to be issues if you have
white spaces around it that's going to
be an issue um again one of the problems
that I had realistically um again
probably 15 years ago most of this stuff
has been cleaned up by now most
programmers know to do this but again I
had uh it was called mind spr there's an
internet service called Mind spring
again 15 15 plus years ago anyways one
of the interesting things that I found
with it is it didn't do this lower or
upper thing uh with with your username
value so when I would plug in my my
email address to log into the system if
I didn't use the exact capitalization
that I used when I set up my account it
would not see my username because it
wasn't the same username and it wouldn't
allow me to log in to actually be able
to access my my my email again you want
to talk about frustration you're sitting
there trying to figure out what the
hell's wrong with your password your
password is fine how you plugged in your
email address that's different than what
you originally did and that can be a
problem right you're sitting there
you're not thinking or your user is not
really thinking about it they they type
in their email address or they type in a
display name however they want to type
it they're not thinking about the
capitalization because they just assume
the data will get cleaned up you don't
clean up the data properly and now all
of a sudden they have to remember okay
the the first the first letter is
uppercase and the third letter is
uppercase all right that can be a bit of
a disaster for for trying to figure out
so anyways that's how to sanitize
strings again what you will need to do
for your particular projects might be
pretty specific so just use these basic
methods and tools that I showed you and
go from here to figure out how you need
to clean up your data for your
particular project uh as always I enjoy
teaching this class and look forward to
see you the next one