Add robots.txt to archive.softwareheritage.org to avoid crawlers
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	olasd
	Jan 31 2017, 3:09 PM

Description

We don't want robots crawling archive.softwareheritage.org using API endpoints.

We need to add a robots.txt to that effect.

rSPSITE puppet-swh-site
	rSPSITE4ebdcae735cb swh::deploy::webapp: add robots.txt alias
rSPPROF puppet-swh-profile
	rSPPROF4ebdcae735cb swh::deploy::webapp: add robots.txt alias

Mentioned In: R65:0a84869862ba: add static/robots.txt, disabling crawling of /api/
rDWAPPS0a84869862ba: add static/robots.txt, disabling crawling of /api/

zack moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Feb 1 2017, 12:52 PM

zack moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.

I've written the trivial robots.txt, but as is it will be deployed as /static/robots.txt.
We need HTTP trickery to that end.