The big picture
In a previous blog I made passing reference to The Deep Web (also called the Deepnet, the Invisible Web, the Hidden Web, etc.). Much larger than the Surface Web (BrightPlanet puts it at 400 to 500 times bigger), the Deep Web is invisible to standard search engines.
Most of the Deep Web is comprised of straightforward private and public sector password-protected information. It includes things like unpublished/unlisted file directories (See our profile on DesignRush), archived newspaper articles, private financial information, medical data, and engineering databases. Bottom line: the Deep Web is not the exclusive domain of nefarious activity, as is generally depicted in the popular media. In fact, it’s mostly comprised of indigestible data that is of possible interest to the rest of us only through the filter of Big Data analytic tools.
The mysterious, constantly evolving, “Dark Web”
Nonetheless, there are ‘hidden sites’ (often with temporary addresses) that are sanctuaries for serious criminal activity. Often referred to as the Dark Web, such forums purposefully hide things like illicit drug transactions, child pornography, stolen credit card numbers, human trafficking and arms sales.
Access to the Deep Web and Dark Web sites is gained through an anonymizing browser like The Onion Router (TOR) that transmits hidden information via globally interlinked servers. Interestingly, TOR was created by the U.S. Naval Research Laboratory so that political whistleblowers could communicate information anonymously without fear of reprisal. Unfortunately, TOR was so effective at creating anonymity that criminals started using it.
Like layers of an onion?
A Google search of ‘The Deep Web’ reveals claims that it has many layers, ranging from easy-to-access university portals to the so-called super secret “Marianas Web,” the latter supposedly requiring a quantum computational key. Others counter that this is all speculation, especially conjecture surrounding the Marianas Web.
Silk Road and Bitcoin
The most infamous TOR site was the Silk Road, which was first taken down in late 2013, only to reappear this year, when it was hit by the theft of Bitcoin currency valued at $3.6M. Pending a critical court case next month, it appears to be down again. It’s hard to keep up with Silk Road’s competitor sites, like Agora, and a plethora of other portals where users can buy hard drugs and other illegal goods.
Bitcoin, paired with TOR, is an easy way to buy and sell anonymously on the Web. However, because Bitcoin is not backed by any government, its value has fluctuated wildly, falling 20% in value since the beginning of this month (October, 2014).
The downside of TOR anonymity
With alarming reports of major internet security breaches growing month to month, many folks are opting for the apparent anonymity of TOR and similar browsers. They have no interest in doing anything illegal. However, those who make this choice are raising a red flag to the NSA and other Homeland Security agencies. Understand that if your communications are hidden, you look suspicious. And keep in mind that government and other entities are highly adept at plumbing the breadth and depth of the Deep Net/Dark Net.
The Deep Net/Dark Web is a double-edged sword, capable of unleashing the worst (sometimes the best) in human nature, e.g., as with sharing valuable medical research across firewall-protected silos for the benefit of mankind.
Deep Web and how it differs from the Dark Web:
- Most of the Deep Web is packed with routine private and public sector password-protected archival data. When a webmaster tells a search engine not to look at something, it’s because including it (e.g., programming data) would hurt SEO rankings and be of no interest to readers.
- The Deep Web includes things like YouTube vids (most of which are not archived for standard search), online books and libraries, e.g., via university/college portals, and Google Docs.
- The Dark Web, by contrast, is ultimately a neutral domain, with sanctuaries both for serious criminal activity and socially beneficial, even vital, communication. More about that in a minute…
The Deep Web is vital to big data analysis
Because the Surface and Deep Webs are growing exponentially, it’s hard for organizations to make sense of all the critically important available information. That’s a problem because companies need accurate analysis of Big Data to be competitive and to manage their internal operations. Bottom line: businesses that rely only on the Surface Web make less-informed decisions and are at risk of failure.
Another growing benefit of the Deep Web information is broadened interconnectivity between professionals who need complete, easily available updates on university and other research results. The Deep Web, for example, contains comprehensive, hard-to-access information on diseases and epidemics that is of critical importance to public health.
The redeeming potential of the Dark Web
So, the Deep Web is far from mysterious. Nor is the Dark Web limited to terrorist communication, illegal drug transactions and other illicit activities. In fact, it also facilitates services and communication that serve the social good.
On the positive side, the Dark Web—
- Is home to various e-mail services, file storage and sharing, alternate search engines, human rights social media, chat sites and news outlets;
- Contains sites that provide anonymous communication for whistleblowers;
- Provides a relatively secure way for political dissidents and human rights activists to communicate with like-minded individuals, potentially saving lives in countries with oppressive leaders. Facebook or Twitter, by comparison, are easily monitored by authorities; and
- It’s free of advertisements!
