hello again as you know I am Eli the
computer guy and today's class is
Introduction to the Deep
Web okay that was a little bit
melodramatic there but I do feel that
the melodrama is necessary because so
many of you folks think that the Deep
Web is something very very very
impressive and scary and cool they
thought that there is a a worldwide web
that we all know about and then that
there is a deep web that is hidden away
in the dark recesses of the internet
seems really cool and really amazing and
really sexy and awesome doesn't it and
so I keep getting questions on my daily
blob show about what is the Deep Web and
I get so tired of answering the same
question Time and Time and Time and time
again I decided to do this class for you
guys on introduction to the Deep Web so
that you understand what the Deep Web is
and why really truly it's not nearly as
cool as all you folks think it is I I
don't know why you know in technology um
there are things that come and go you
know in popular culture there there are
Concepts that get to be really sexy and
cool for a while and everybody is
worried about it and apparently right
now the Deep web is one of those things
everybody thinks wow there's a deep web
there's a worldwide web and then
underneath is the Deep Web well the
reality is the true honest to golly
honest to Spaghetti Monster reality is
is all the Deep Web is is its web pages
that are not indexed by search engines
so when you go to Google and you type in
a search query all the normal websites
pop up uh or have the ability to pop up
in that organic search so you know if
you're searching for whatever it is a
website might pop up there what the Deep
Web is is the Deep Web is all of the web
pages that are out there on the internet
that for whatever reason are not indexed
by the search engines so if you did a
search they would not show up in Google
they would not show up in bing they
would not show up in yahoo or altav
Vista or whatever else that is all the
Deep Web is the Deep Web are the web
pages that are not indexed by search
engines now the question is the thing
that you may be thinking you may be
might be thinking yeah but Eli but Eli
why are they not indexed by search
engines right all web pages are indexed
by search engines so what are they
trying to hide that they're not indexed
by search engines well in the modern era
you know 2013 and so forth many web
pages are not indexed by uh search engin
such as Google and Yahoo because they
are behind login walls so if you go to a
site let's say you go to a WordPress
site or you go to some kind of Community
website and you have to log in the only
way to be able to access the content on
that website is if you are in fact able
to log in now all the search engines
they have these little algorithms that
run around and they're called either
spiders or robots so these spiders or
robots go out and they navigate the web
so basically they kind of just let them
loose and what happens is these spiders
and robots they go out they catalog all
the web pages that they can find they
catalog all the web pages that are
linked to by other web pages and then
they just kind of Follow That and go
along and go and grab information and
basically index all of these different
web pages that are out there well their
issue is Imagine they're going along and
they're indexing indexing indexing
indexing and they're about to index a
website but when they get to the website
it asks for a username and password well
that robot that spider can't get P
because obviously it's not a real user
so it doesn't have a username and
password so websites much of uh websites
such as Facebook are considered Deep Web
simply because they are not indexed by
Google there's a huge huge massive
argument going on right now where Google
says that they should be able to index
more of Facebook than Facebook allows
them to index that's a whole big thing
but but what you should understand is
that there are simply a lot of pages
within Facebook that cannot be indexed
by Google or Bing or anybody else
therefore that would be considered the
Deep Web if you set up a community site
and let's say you close it off so it's
it's private only private members can
access the information so you set up a
bulletin board but then you don't want
this to be public you only want you know
your fellow users to be able to access
it the same thing would be true when one
of these spiders or robots from Google
or Yahoo go to uh to scan your web pages
they will be blocked by the username and
password so that's the first reason uh
why there is a lot of deep web web pages
out there just simply you need login
credentials to get there that's it the
other reason uh that there are deep web
web pages is you can actually
ask through coding for the search
engines to not index your web page for
whatever reason so there's something
called the robots.txt file if you put
that into the root of your website so
basically the same place where your
index.htm or HTML or whatever is if you
put in what's called a robots.txt file
within that file you can ask the search
engines not to index your site now
you're thinking well there you go Eli
there you go you're asking the sech
transs not to index your site that means
that means you're doing something pretty
nifty neat or cool aren't you not
necessarily so you know a lot of us a
lot of us are
not overly worried about uh putting
information out onto the internet but we
don't want to overdo it right so let's
say you have a club or you have an
organization and you want to be able to
put on to the web uh let's say the board
members names so so if other members
need to get in contact with somebody or
somebody else needs to see a listing of
who the board members are you want to be
able to put that on the web and it's one
of those things that you know you don't
really care if the public sees it on the
other hand you don't necessarily want
this thing showing up in a Google search
either you know if somebody goes to the
web page if somebody knows what the web
page is that's fine if they see it if
they don't know what the we Pages you
don't want them to know about it well
then that's what you might put into
robots.txt file to state that you don't
want the search engines to index the
pages because again you don't want it to
show up in Google or Yahoo or Bing so on
and so forth again this is not nefarious
this is not cool this is not sexy this
may be the membership roster for your
local Elks club or rotary club or
something like that again essentially um
you don't really care really care um if
it's public you just would kind of
prefer it's just not completely and
utterly out there so robots.txt files
were used for a long time you know when
you go back to websites that were built
like 10 years ago lots of people were
putting up web pages and they simply
didn't want them to be indexed by the
likes of Google and so they use this
robots.txt file now the important thing
to understand about the robots.txt file
is it's only a suggestion for the web
crawlers so the spiders and the robots
that Google and Yahoo and all that send
out there when they go to your website
they will look for robots.txt file if
they read the W robots.txt
file it tells them what you would prefer
that you prefer that the site not be
indexed but they can go ahead and index
it regardless this this isn't like when
you use a robots.txt file this isn't
like a mandatory federal law or
something it's just kind of like like a
nicity so those are the main reasons
that we get What's called the Deep Web
so these are the websites that are not
indexed by Google Yahoo any of these
search engines and again it's generally
because either there's some kind of
login credential or there's the
robots.txt file so let's go over to the
computer for a second just so you can
see a couple of things and maybe maybe
this will get a little bit clearer for
you so I just brought up this web page
here if you go to robots tx.org you can
get a lot more information about the
robots.txt file and basically again all
this file is is you can just create this
a notepad it's not HTML it's not
anything like that it's just robots.txt
all you do is you put in this user agent
colon star disallow colon slash and
basically what that says is it says to
all those crawlers and robots please
please do not Index this website and
that's all you have to do and then you
too have a website on on the Deep Web
not only that but even on things like
WordPress so elithecomputerguy
docomond WordPress if you go in here you
will see if you go down to settings
reading there's actually a little check
box search engine visibility you can
actually discourage search engines from
indexing this site again it is up to the
search engines to honor this request but
generally they do so if I check off this
box it will tell the search engines
please do not Index this site so even
though it's a full site even though it's
a big site all of that basically it will
drop off of Google and Yahoo and Bing so
on and so forth that's all you do uh to
create the Deep
Web that's really all there is to it
right um I don't know why people think
it's amazing and cool and fancy again I
think it's just I think it's simply one
of those we have gotten so used to being
able to use search engines to find
anything on the internet
it seems that if you can't use a search
engine to find information then it must
be again cool or nefarious or something
like that the reality is you know a lot
of times again there there's login
credentials that block robots from
searching there's a robots.txt files
that block uh or that try to block
search engines from indexing and there's
also really really really crappy code
again when you're thinking about the
internet a lot of you folks a lot of you
folks actually weren't even born when
the internet you know popped out in 1992
what you you may not realize is that
there are a lot of websites still up and
running that got coded a long long long
time ago when we were all still trying
to figure out what the hell we were
supposed to be doing so some of these
Deep Web websites are just websites that
were coded so horribly that even the
robots don't know what to do with them
you know what I'm saying but that is all
the Deep Web is the Deep Web is simply
simply web pages that are not indexed by
Google or any of the other search
engines for whatever
reason that's
it so as you know I am Eli the computer
guy today's class was introduction to
the Deep
Web as always I enjoy teaching this
class and look forward to see you at the
next
one