Opting out of search engines
I decided to remove (unindex) my website from search engines.
Why?
Most search engines have become more like LLMs in how they process and surface information to users. Most search engines now have a AI-generated summary as the most prominent response to user queries. The most noticeable impact of this is a significant reduction in the number of referrals to websites.
In a recent post, I outlined how my website is approaching Google Zero, referrals from Google to my website has dropped 91% since 2022. This was a reminder for me to re-assess my relationship with search engines.
It got me thinking about the flow of internet traffic. Really, this shift is akin to Route 66 being supplanted by the modern inter-state highway system in the USA. Route 66 was developed in the 1930s and became the first paved highway in the USA. Initially the majority of the traffic was people from the east travelling west looking for jobs in the emerging industries in California. After World War 2, thanks to the boom in manufacturing and unionization, more people could afford to buy cars. It became popular to go on vacation with your car. Since Route 66 meanders directly through countless small towns, many local businesses sprung up to serve the people passing through.
The push for a faster way to traverse the country eventually led to the decline of Route 66. The objective of the national inter-state highway system was to enable travelling long distances in the shortest amount of time possible. This led to highways being constructed away from towns, with ramps giving access to and from towns. Once the various inter-state highways opened, traffic along Route 66 dried up and many local businesses along it started to fold. These businesses were largely replaced by typical franchise businesses along the inter-state highways.
Route 66 is now used lightly and is seen more as a nostalgic route to travel for pleasure. That sounds like an apt analogy for the current evolution of the digital highway system AKA the world-wide web!
AI-powered search is double-edged. It looks like a net win for users, users get more succinct answers that can resolve their query faster, albeit with questionable accuracy. On the other side, it looks like a net loss for society. The life-blood of AI search is the content of the internet. It uses a costly process to digest that content into a form that can produce summaries. The side effects are less referals to websites in general, search traffic skewing towards larger websites (businesses), and a big increase in energy consumption.
I don’t want my content to be part of that enterprise. You could categorise this as a minuscule boycott or sour grapes (bitter from a loss in traffic). I am not bitter. I am used to technological change and accept it as part of life. It is a case of re-evaluating my relationship with search engines.
I see the application of AI in Search in a similar vain to the usage of palm oil. Palm oil has many applications. It is contained in nearly 50% of the packaged products that we purchase. Palm oil production is cited as a big contributor to environmental destruction. If you buy products that contain palm oil, you are increasing demand for it. Increasing demand is a driver for irresponsible production. You are a minor contributor to a big problem when you purchase products containing palm oil.
It takes a big shift in consumer sentiment and opposition against the sourcing palm oil from producers that participate in deforestation to motivate manufacturers to switch to a friendlier alternative. Sometimes, public policy from goverments is required to force the issue.
The surge in AI usage requires massive data centers, more and more are being built at breakneck speed. Is this being done in a responsible manner? Why are we opted into AI search summaries by default?
Technological improvements herald new opportunities but also have side effects. We should consider the whole package.
How do you exclude your content from search engines?
For now, I added a meta tag to the head of all of the pages on my website with the noindex rule. The following tells all crawlers not to index your page:
<meta name="robots" content="noindex">You can be specific and limit it to a particular crawler, such as Google’s web crawler with the following:
<meta name="googlebot" content="noindex">For the noindex rule to be effective, the page must not be blocked by your robots.txt file. It has to be accessible to crawlers. You can think of the “noindex” rule as a slightly stronger action than using a robots.txt file.
Of course, you are relying on crawlers to be well-behaved. There may be crawlers who ignore this rule and index your content anyway. To be forceful, you would need to configure the server to not serve pages to crawlers. That is a more involved process.
Final thoughts
Excluding my website from search engines means that less people will visit it. I would prefer not to take this action. However, I think AI search summaries are taking the internet in a harmful direction. The side effects are significantly less referrals to websites, skewing search traffic towards larger websites, and a big increase in energy consumption. I do not want this website to be a part of that web, even if its contribution is indirect and small.
You might think that it is inevitable that AI summaries will be used in all search engines, in fact all of the big boys already have AI summaries as their primary response to search queries (Google, Bing, DuckDuckGo, Baidu, Yandex, Naver). Well, as a producer of content, you have a say. You can take your content away from search engines. As a user, you can choose a search engine that does not use AI summaries such as Ecosia. Your actions count. Act consciously.