If you want to search the deep web yourself, here’s my advice: don’t. If you want to learn more about the deep web, you can find plenty of information about the deep web using Google (how ironic). The best deep web search engines function in various ways, whether it be crowd-sourcing URLs and page descriptions or continuously collecting them, but they certainly do not function in similar ways to traditional search engines such as Google. While the “deep web search engines” mentioned above are capable of indexing a good part of the deep web, the vast majority of it remains unindexed, and no search engine is capable of finding everything contained in it. Additionally, the Uncensored Hidden Wiki has its name for a reason, as the content of that page is certainly uncensored. The negatives of this can be mitigated by site admins to ensure that the links are usually accurate, but there are no guarantees when using the links on this page. On the other hand, anyone can change the links to wherever they want, or alter the descriptions of the links.
onion domain names change extremely often).
On the bright side, crowd-sourcing the links is one of the best ways to collect a large number of useful URLs, and keep them up to date (especially since. The search engine operates by searching the provided descriptions of the pages at these links. Anyone can register on the Uncensored Hidden Wiki, and after that, anyone can edit the links contained in the database. The Uncensored Hidden Wiki operates a little differently. That being said, it still comes nowhere near to scratching the surface of the whole deep web, but it indexes a good portion of the content that most people would want to look for. onion URLs, Ahmia has created one of the largest indexes of the deep web. Additionally, Ahmia allows onion service operators to register their own URLs, enabling them to be found. onion URLs from the Tor network, then feeds these pages to their index provided that they don’t contain a robots.txt file saying not to index them (4). There are many “deep web search engines”, but I’ll focus on two: Ahmia, and the Uncensored Hidden Wiki.Īhmia was developed by Juha Nurmi as part of the Tor Project, and it is one of the closest things to a deep web search engine (3). If the crawlers index the page anyways, then legal action can be taken against the creator of the crawlers, and the search engine can end up on a bot reporting site like (2).
Finally, if the creator of a page doesn’t want it to be indexed by popular search engines, they can include a suitable robots.txt file, which tells the crawlers not to index the page. Also, if a page contains illegal content, Google will likely not want that content appearing in search results, so they won’t index it. Additionally, this page might require some sort of authentication such as filling out a search form and clicking submit, or having a certain certificate. For one, Google’s crawlers might never come across this page simply because no other previously crawled page links to it. Google’s search engine could be unable to find this page because of several reasons. Now, consider a single page in the deep web. (1) These crawlers start from a list of known web addresses, visit those pages, then follow the links contained on those pages, and continue following links found on the new pages, collecting information about each page as they go. Google’s search engine functions by using “crawlers”. Note: the deep web shouldn’t be confused with the “dark web”, which pertains strictly to pages containing illegal content such as child pornography, terrorist forums, and illegal auctions/transactions. It is estimated that search engines like Google index only 4% of the entire world wide web, meaning that the deep web is nearly 25 times larger than the internet you and I have used our whole lives. If search engines like Google, Yahoo, and Bing are unable to index the deep web, then how do deep web search engines work? We’ll try to answer this question by first what I mean by “deep web”, then explaining why Google can’t crawl the deep web, and finally looking at how some popular search engines like Ahmia and the Uncensored Hidden Wiki “search” the deep web.Īccording to Google’s online dictionary, the deep web is “ the part of the World Wide Web that is not discoverable by means of standard search engines, including password-protected or dynamic pages and encrypted networks”. Google Can’t Search the Deep Web, So How Do Deep Web Search Engines Work?