Is your RSS reader getting a 403 error for a feed?

by Rob O'Leary on
The RSS feed icon has a prohibited mark in front of it - a red circle with a diagonal line through it. It is set against an orange background similar to the colour of the RSS feed icon

A reader informed me yesterday that they were receiving a 403 error (permission denied) for my RSS feed in their RSS reader (FreshRSS). They thought that maybe they were blacklisted for polling my feed too often, every 6 hours initially. That is acceptable rate and should not led to being blocked.

The reason is that probably my webhost Cloudflare is blocking the request as it has classified the source as a bot. Cloudflare has tools designed to block malicious bot traffic such as Block AI bots and Bot Fight Mode. When enabled, these features will mitigate unwanted bot traffic, but they can end up blocking some good guys too like RSS readers! 😥

screenshot of bot tools available in

OpenRSS complained about being blocked by Cloudflare last year. They were able to get Cloudflare to unblock them eventually.

Unfortunately bots are a growing problem. An inauspicious milestone was reached last year, traffic from bots has surpassed that of humans. The troubling aspect is that now about one third of all internet traffic is malicious. [1]

Even for my humble personal website, I think there is merit in thwarting bots in some fashion. It is a shame that it is hampering legitimate syndication of content and the indie web. We need to ensure that the weeds don’t crowd out the flowers.

How does Cloudflare decide who to block?

Cloudflare keeps a directory of verified bots and signed agents. When you enable the Block AI bots, it will add a security rule to block requests for one of the following reasons:

If you’re not on the good list or naughty list, Cloudflare will run an additional check to see if a visitor is a bot based on some heuristics that will generate a score. If the score hints at the source being suspicious, Cloudflare presents the requester with a number of challenges, challenges that something like a RSS reader would not be able to fulfill. If Cloudflare is more convinced that the requester is automated and malicious, it will simply block the request.

For a partial list of verified bots and signed agents, you can refer to Cloudflare Radar.

What should I do?

There are a few options:

  1. Turn off the bot blocking features. That should let all requests through including the RSS readers. Not ideal.
  2. Maintain your own whitelist of good bots and add FreshRSS to it. In Cloudflare, you can create a custom rule that explicitly allows all traffic by an IP address or by a user agent string.
  3. Make a request to Cloudflare to add FreshRSS to its verified bots list.

It is preferable to go with option 3 and help Cloudflare recognize good bots. Maintaining my own whitelist for bots and possibly user instances is not practical in the long-run. In the case of FreshRSS, since it is self-hosted, it may pose a problem to have it added as a verified bot since the IP addresses are going to differ user by user for their hosted instances. I don’t know how verification works for that scenario.

How do I stop Cloudflare from blocking a RSS reader?

You can request that a RSS reader is added to Cloudflare’s verified bots list by filling out a Google Form, or through an application form in the Cloudflare dashboard.

OpenRSS outlined their experience in becoming a verified bot in a blog post, it doesn’t appear to be a well-defined process. They needed to submit the Google Form twice. No notification was given that Cloudflare was working on it as an issue. It took approximately 5 months for Cloudflare to add them.

  1. The Thales Bad Bot Report, an annual survey of the state of the internet, reported that bot activity accounted for 51 per cent of internet traffic in 2024. It found that malicious bots now account for 37% of all internet traffic. ↩︎

Tags