paperbay.org Mastodon - https://paperbay.org/users/a/statuses/113160099393210726

Did you know? The Facebook crawler doesn’t consider most HTTP return codes except for 403!

If a page returns 403, the Meta crawler won’t access it. Other status codes, like 404 or 451, won’t stop it from attempting to scrape your page’s content. The crawler will continue and won´t apply any delay.

#crawler #meta