如何解决404 HTTP 错误，尽管能够在浏览器中看到该页面？

我正在尝试映射此网站，但在尝试对其进行完全抓取时遇到问题。即使 URL 存在，我也收到错误 404。

这是我的代码：

import csv
from urllib.request import urlopen
from bs4 import BeautifulSoup
import re

csvFile = open("C:/Users/Pichau/codigo/govbr/brasil/govfederal/govbr/arquivos/teste.txt",''wt'')
paginas = set()
def getLinks(pageUrl):
    global paginas
    html = urlopen("https://www.gov.br/pt-br/"+pageUrl)
    bsObj = BeautifulSoup(html,"html.parser")
    writer = csv.writer(csvFile)
    for link in bsObj.findAll("a"):
      if ''href'' in link.attrs:
       if link.attrs[''href''] not in paginas:
             #nova página enconTrada
                newPage = link.attrs[''href'']
                print(newPage)
                paginas.add(newPage)
                getLinks(newPage)
                csvRow = []
                csvRow.append(newPage)
                writer.writerow(csvRow)

   
getLinks("")
csvFile.close()  

这是我在尝试运行该代码后收到的错误消息：

#wrapper
/
#main-navigation
#nolivesearchGadget
#tile-busca-input
#portal-footer
http://brasil.gov.br
Traceback (most recent call last):
  File "c:\Users\Pichau\codigo\govbr\brasil\govfederal\govbr\teste2.py",line 26,in <module>
    getLinks("")
  File "c:\Users\Pichau\codigo\govbr\brasil\govfederal\govbr\teste2.py",line 20,in getLinks
    getLinks(newPage)
  File "c:\Users\Pichau\codigo\govbr\brasil\govfederal\govbr\teste2.py",in getLinks
    getLinks(newPage)
  [PrevIoUs line repeated 4 more times]
  File "c:\Users\Pichau\codigo\govbr\brasil\govfederal\govbr\teste2.py",line 10,in getLinks
    html = urlopen("https://www.gov.br/pt-br/"+pageUrl)
  File "C:\Users\Pichau\AppData\Local\Programs\Python\python39\lib\urllib\request.py",line 214,in urlopen
    return opener.open(url,data,timeout)
  File "C:\Users\Pichau\AppData\Local\Programs\Python\python39\lib\urllib\request.py",line 523,in open
    response = meth(req,response)
  File "C:\Users\Pichau\AppData\Local\Programs\Python\python39\lib\urllib\request.py",line 632,in http_response
    response = self.parent.error(
  File "C:\Users\Pichau\AppData\Local\Programs\Python\python39\lib\urllib\request.py",line 561,in error
    return self._call_chain(*args)
  File "C:\Users\Pichau\AppData\Local\Programs\Python\python39\lib\urllib\request.py",line 494,in _call_chain
    result = func(*args)
  File "C:\Users\Pichau\AppData\Local\Programs\Python\python39\lib\urllib\request.py",line 641,in http_error_default
    raise HTTPError(req.full_url,code,msg,hdrs,fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
PS C:\Users\Pichau\codigo\govbr>

我尝试仅使用主链接执行此操作，并且效果很好，但是一旦我将 pageurl 变量添加到 url，它就会给我这个错误。如何修复此错误？

解决方法

'responseError': function (rejection) {
 console.log(rejection);
 return $q.reject();
}

var method = 'POST';
var url404 = '/tnseiartneiasrt/arsntiearsntiasrntiarnstsie';

var xhr = new window.XMLHttpRequest();
xhr.open(method,url404,true);
xhr.onreadystatechange = function() {
  if (xhr && xhr.readyState == 4) {
    console.log('should have logged an error already :(');
  }
};
xhr.send(null);

accessibility.src.js:431 Uncaught TypeError: a.hasAttribute is not a function
    at accessibility.src.js:431
    at NodeList.forEach (<anonymous>)
    at l (accessibility.src.js:430)
    at l (accessibility.src.js:436)
    at l (accessibility.src.js:436)
    at l (accessibility.src.js:436)
    at describeSeries (accessibility.src.js:3494)
    at accessibility.src.js:4096
    at Array.forEach (<anonymous>)
    at b.onChartRender (accessibility.src.js:4091)