DeepWeb and SurfaceWeb: What lies beneath ?

There is a popular misconception that Google and the other mainstream search engines index all the content that is out there on the web (if you are prepared to dig enough). In fact, Google / Bing etc only access the surface web in much the same way that a trawler with a drag net only picks up the fish fairly close to the surface.

Perhaps, like experienced fishermen, they don't want to go too deep for fear of what might end up in the net.

Estimates vary but a very popular statistic is that the DeepWeb is about 500 times larger than the SurfaceWeb. This might seem absurd but lots of government data is stored out there in a way that is not visible to the search engines.

A part of the DeepWeb that has attracted a fair bit of media attention is the DarkWeb which can only be viewed as part of the Tor network. The websites on this section of the web have a .onion title albeit that this is not a recognised domain like a .com.

SilkRoad was the most famous site on the Tor network / dark web and was involved in almost all the illegal activities that spring to mind. The FBI and Europol have closed down a number of versions of the site (proving that nothing is 100% anonymous) but tribute sites continue to spring up.

Both the DeepWeb and DarkWeb present big challenges to both Governments and Law Enforcement all over the world.

Both the Deep and DarkWeb's have proved invaluable to individuals fighting against oppressive regimes which stifle free speech. The value is precisely because these networks are free from censorship. Anne Frank would have been posting to the DeepWeb rather than scratching away at her diaries in Hitler's Germany.

On the other hand it seems pretty absurd that a specific search engine exists on the DeepWeb to search for and supply illegal drugs.

The statement that one man's terrorist is another man's freedom fighter neatly captures the problem here. The internet is global and the standards across the globe are not consistent. Large corporations have exploited these different standards in respect of tax and regulation and the criminal fraternity are doing so in respect of criminal and civil (or common) law.

The Budapest Convention on CyberCrime seeks to address this issue and start to enforce common standards but at this point it only has 50 signatories worldwide and Russia, China and India are notably absent.

A search engine now exists called Onion City which allow access to the Dark Web without using the Tor Network. This is probably a good thing as increased visibility should facilitate regulation.

Looking forward it will be interesting to see if if the first attempts at global regulation of the internet (which is obviously required) will go for the highest common factor in terms of standards or the lowest common denominator.