Consider changes to redirects

[Mr0grog]

We should probably consider changes to redirects in the priority calculation, or at least call them out one of the columns of the sheet.

For example, a couple weeks ago, https://www.epa.gov/aboutepa/about-national-health-and-environmental-effects-research-laboratory-nheerl went from being a 404 page to suddenly having content and a 200 status code again, but what really happened was that the 404 page started redirecting to an archived copy of the page at https://archive.epa.gov/epa/aboutepa/about-national-health-and-environmental-effects-research-laboratory-nheerl.html

See the above example in the API:

• Page: https://api.monitoring.envirodatagov.org/api/v0/pages/6ff9a595-1e50-4811-8449-87a5a701b90d
• Version before: https://api.monitoring.envirodatagov.org/api/v0/versions/576b5c05-f1e2-4afa-9a17-26692ff881b2
• Version after: https://api.monitoring.envirodatagov.org/api/v0/versions/839c4bdd-17b6-48ed-8dc0-27263fb86d71

One reason this is important is that it can be hard to tell that pages at archive.epa.gov are in an archive. It looks obvious when you’re there in a browser with modern CSS:

But that big banner is a background image on a pseudo element, and there’s no representative text about the archive anywhere in the page. (Honestly, this seems designed to be inaccessible, but I don’t want to speculate too much about what led to this very problematic technical setup.) In any case, without a lot better rendering support (for part of the solution to that, see edgi-govdata-archiving/web-monitoring-ui#600), there’s just nothing on the page to indicate that it’s an archive. Besides, there are probably a number of other different situations where this is especially relevant and worth calling out.

See also: edgi-govdata-archiving/web-monitoring-ui#509