- cross-posted to:
- technology@lemmy.world
- cross-posted to:
- technology@lemmy.world
DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.
DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.
Actually currently it contains this:
User-agent: * Disallow: /
Well, that actually is a blanket ban for everyone, so something else must be at play here.
https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy
Reddit is serving different file to google
That’s real rich, coming from Reddit.