Underbelly of the web

It's called the Deep Web:

Mike Bergman, founder of BrightPlanet and credited with coining the phrase, said that searching on the Internet today can be compared to dragging a net across the surface of the ocean: a great deal may be caught in the net, but there is a wealth of information that is deep and therefore missed. Most of the Web's information is buried far down on dynamically generated sites, and standard search engines do not find it. Traditional search engines cannot "see" or retrieve content in the deep Web—those pages do not exist until they are created dynamically as the result of a specific search. As of 2001, the deep Web was several orders of magnitude larger than the surface Web.

Time Magazine recently reported on it in a cover story titled "The Secret Web: Where Drugs, Porn and Murder Live Online."  The white paper, "The Deep Web: Surfacing Hidden Value (Bergman, Michael K., Journal of Electronic Publishing)," offers some unbiased information about the Deep Web.

To access the Deep Web you need Tor --

...free software and an open network that helps you defend against traffic analysis, a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and state security.

Developed to ensure privacy online,

Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location.

Tor's about page offers some history and the site offers a wealth of information on how to set up the browser. Here's an example of a Tor network:

***

Rezonansowy, Tor network map, Wikimedia Commons

***

There are a number of reasons why the Deep Web is invisible; BrightPlanet offers a primer.  One of the reasons is dynamic web pages.  To get an approximate idea of how a page is dynamically generated, try Hypergurl's HTML generator; but keep in mind that a page on the Deep Web would be generated by code using data from a database, when the destination address is touched.

Be careful out there -- you may see more than you want to.

-- Marge