Question

这是有问题的代码

启动会话处理程序

session = requests.Session()

以编程方式获取SAML断言，打开初始IdP URL并遵循所有HTTP302重定向，并获取生成的登录页面

formresponse = session.get(idpentryurl, verify=sslverification)

捕获idpauthformsubmiturl，这是所有302之后的最终网址

idpauthformsubmiturl = formresponse.url

解析响应并提取所有必要的值，以便构建IdP期望的所有表单值的字典

formsoup = BeautifulSoup(formresponse.text.decode('utf8'))
payload = {}

调试输出：

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): myapps.microsoft.com
DEBUG:urllib3.connectionpool:https: //myapps.microsoft.com:443 "GET /signin/AWS%20CMD%20(Audit)/18216d2a-eef8-4fde-962c-50cf615f3f5b HTTP/1.1" 302 244
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): account.activedirectory.windowsazure.com
DEBUG:urllib3.connectionpool:https://account.activedirectory.windowsazure.com:443 "GET /applications/signin/AWS%20CMD%20(Audit)/18216d2a-eef8-4fde-962c-50cf615f3f5b HTTP/1.1" 302 94
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): login.microsoftonline.com
DEBUG:urllib3.connectionpool:https://login.microsoftonline.com:443 "GET /common/oauth2/authorize?client_id=0000000c-0000-0000-c000-000000000000&redirect_uri=https%3A%2F%2Faccount.activedirectory.windowsazure.com%2F&response_mode=form_post&response_type=code%20id_token&scope=openid%20profile&state=OpenIdConnect.AuthenticationProperties%3DmIDzRLZskQlxxtgB9rjxiHrNVmQJpcUVaK8wuZ3A2PMIyBE8fzxkXDcroNhC4wyof9OK9OlhqH0J_stoYSEIhKiEzx4O3XDW4rS4xyFTitGmztuV3ozOJhX5uafmQm_XmKnXEjEt9CNwFbp2Kju3rRGLAXRViD3byQ7XpwdXkeXoDFLwmy5OIXQgzvPjSsc7Jx7xEXMHckDwElhBOBFXmJVYCkHYx6cB-3yjwGJHX6RQ2lfx6CUg7x2PqPkbo4WsUxbZDAJZsMqYXyVRZGSDqAgU3gSezlHNgZGh-nblkxj7Dw6rdMVKmpNWZWkjp3zI3OjWa91FTrVc0mC9gIQC-BC4zaF-FrwQ4rHPbQlisQoS6-S1qM8ca_cEi6CfFaHh2lrtB-xdNEVum97Mzmlg9g&nonce=1507770263.sCv6L2a21eQuLNKaXL3zog&nux=1 HTTP/1.1" 200 15838
Traceback (most recent call last):
  File "formauth.py", line 62, in <module>

formsoup = BeautifulSoup(formresponse.text.decode('utf8'))

  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)

**UnicodeEncodeError:** 'ascii' codec can't encode character u'\u2019' in position 30342: ordinal not in range(128)

尝试下面的技巧没有帮助：在响应中用空格替换非utf8字符

formresponse.encoding = formresponse.apparent_encoding
formsoupba = bytearray(formresponse.text, 'utf8')
for i, val in enumerate(formsoupba):
    if val > 128:
        formsoupba[i] = 32
        formsoup = BeautifulSoup(formsoupba.decode('utf8'), "html.parser")

将产生以下错误：

返回codecs.utf_8_decode（输入，错误，True） UnicodeDecodeError：＆＃39; utf8＆＃39;编解码器不能解码位置30334中的字节0x80：无效的起始字节

任何帮助将不胜感激

Answer 1

您正在尝试将unicode字符（\u2019，引号）解码为utf-8，这应该可以正常工作。然后有什么东西试图将其编码回ascii - 也许某些bs4解析器？

没关系 - 如果你愿意失去奇怪的’字符，这就是霰弹枪方法：

clean_text = formresponse.text.encode('utf8').decode('ascii', 'ignore')
formsoup = BeautifulSoup(clean_text, "html.parser")

这只是ignore的任何编码错误，这意味着你失去了角色。请查看此处的文档，了解除了忽略之外的其他选项：https://docs.python.org/2/library/codecs.html

更深入的方法是找到页面的实际编码 - https://login.microsoftonline.com:443声明为<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">，但显然不是它包含那些类型的字符。我想这可能会甩掉BeautifulSoup。尝试为bs4提供一些不同的编码，例如cp1252和latin-1。

UnicodeEncodeError：＆＃39; ascii＆＃39;编解码器不能编码字符u＆＃39; \ u2019＆＃39;位置30339：序数不在范围内（128）

1 个答案: