爬蟲錯誤總結錄

1、目標網站編碼utf-8,經過requests請求返回text方法,用到xpath上報錯html

Traceback (most recent call last):
  File "/python_project/app_data/parse_async/setup_parse.py", line 49, in <module>
    get_urls(link)
  File "/python_project/app_data/parse_async/setup_parse.py", line 44, in get_urls
    tree = etree.HTML(html)
  File "src/lxml/etree.pyx", line 3170, in lxml.etree.HTML
  File "src/lxml/parser.pxi", line 1872, in lxml.etree._parseMemoryDocument
ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
值錯誤:不支持帶編碼聲明的Unicode字符串。請使用不帶聲明的字節輸入或XML片斷。

解決:python

將`html = requests.get(url=base_url, headers=headers).text` 改成`html = requests.get(url=base_url, headers=headers).content` 便可app