Crawl-delay is not working in robots.txt
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user-agent in a few minutes.
How can I stop this user-agent?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user-agent in a few minutes.
How can I stop this user-agent?
No comments:
Post a Comment