Sam_uk
I’m Cyborg, part human part robot
- 3 Posts
- 5 Comments
Joined 3 years ago
Cake day: June 6th, 2023
You are not logged in. If you use a Fediverse account that is able to follow users, you can follow this user.
I think it would just be
User-agent: * Disallow: / User-agent: Googlebot Allow: /
It’s any day now I think, EU pension funds are moving out https://www.removepaywall.com/search?url=https%3A%2F%2Fwww.ft.com%2Fcontent%2F9d90d557-48e5-4f4b-a927-88071cef8ea9
Would you be up for re-enabling Google indexing? It is crappy, but still…
I wonder if it was a short term fix that got forgotten about…
In the bottom
User-agent: * Disallow: /Blocks all scrapers and superseeds the text above AFAIK
Slightly different emphasis, but some similarity with https://karrot.world/