Page MenuHomeSoftware Heritage

Review / extend web app robots.txt
Open, HighPublic

Description is currently being crawled by a random bot.

While this bot is quite slow and therefore doesn't incur much load, I don't think being indexed is generally desirable, at least for now; We should look at what other endpoints we should add to the current robots.txt.

Event Timeline

olasd triaged this task as High priority.Sep 22 2018, 2:50 PM
olasd created this task.


being crawled along million-revision VCS histories doesn't sound particularly appealing :-)