reivilibre
/
quickpeep
1
0
Fork 0
Code Issues 11 Pull Requests Projects Releases 11 Wiki Activity

Accept forbidden robots.txt — if they forbid us from knowing about things, we will be cheeky
ci/woodpecker/push/check Pipeline was successful Details
ci/woodpecker/push/release Pipeline was successful Details
ci/woodpecker/push/manual Pipeline is pending Details

This commit is contained in:
Olivier 'reivilibre' 2022-06-04 23:54:26 +01:00
parent d18d0635d7
commit fb3eae9226
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
quickpeep_raker/src/raking.rs
View File

@ -555,8 +555,10 @@ pub async fn get_robots_txt_for(url: &Url, client: &Client) -> anyhow::Result<Op
if !resp.status().is_success() { if !resp.status().is_success() {
let code = resp.status().as_u16(); let code = resp.status().as_u16();
if code == 404 || code == 410 { if code == 403 || code == 404 || code == 410 {
// not found or gone? Assume there is intentionally no robots.txt file. // not found or gone? Assume there is intentionally no robots.txt file.
// If they deny us access to the robots file, then they deserve whatever they get and
// we proceed.
return Ok(None); return Ok(None);
} }