The Deep Web you don’t know about
NEW YORK (CNNMoney)
What we commonly call the Web is really just the surface. Beneath that is a vast, mostly uncharted ocean called the Deep Web.
By its very nature, the size of the Deep Web is difficult to calculate. But top university researchers say the Web you know — Facebook (FB, Fortune 500), Wikipedia, news — makes up less than 1% of the entire World Wide Web.
When you surf the Web, you really are just floating at the surface. Dive below and there are tens of trillions of pages — an unfathomable number — that most people have never seen. They include everything from boring statistics to human body parts for sale (illegally).
Related story: Shodan, the scariest search engine on the Internet
Though the Deep Web is little understood, the concept is quite simple. Think about it in terms of search engines. To give you results, Google (GOOG, Fortune 500), Yahoo (YHOO, Fortune 500) and Microsoft’s (MSFT, Fortune 500) Bing constantly index pages. They do that by following the links between sites, crawling the Web’s threads like a spider. But that only lets them gather static pages, like the one you’re on right now.
What they don’t capture are dynamic pages, like the ones that get generated when you ask an online database a question. Consider the results from a query on the Census Bureau site.