带有cat的.txt文件的UnicodeDecodeError

Question

我正在使用cat test.txt |并将该文件的行传递到我的python程序中。但是，我得到一个UnicodeDecodeError：

回溯

Traceback (most recent call last):
  File "/home/sys_bio_ctgdq/sthe-admin/python3.5/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/sys_bio_ctgdq/sthe-admin/python3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nfsdata/DSCluster/home/bli1/qe-trinity-functional/qe/trinity/hdfs2datacase.py", line 102, in <module>
    sys.exit(main(sys.argv))
  File "/nfsdata/DSCluster/home/bli1/qe-trinity-functional/qe/trinity/hdfs2datacase.py", line 72, in main
    for rawline in fileinput.input(args.targets):
  File "/home/sys_bio_ctgdq/sthe-admin/python3.5/lib/python3.5/fileinput.py", line 265, in __next__
    line = self.readline()
  File "/home/sys_bio_ctgdq/sthe-admin/python3.5/lib/python3.5/fileinput.py", line 370, in readline
    self._buffer = self._file.readlines(self._bufsize)
  File "/tmp/python3_bli1/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 1971: invalid start byte

第72行是for loop行

以下是我处理输入流的方法：

       for rawline in fileinput.input(args.targets, openhook=fileinput.hook_encoded("utf-8")):
            try:
                jobj = json.loads(rawline)
            except ValueError as e:
                log.warn("invalid json on input line %d", line_count)
            except UnicodeDecodeError as e:
                log.warn("Funky characters", line_count)
            else:
                if case_obj is not None:
                    case_str = json.dumps(case_obj, sort_keys=True, indent=4)
                    if first_case:
                        print("[")
                        first_case = False
                    else:
                        print(",")
                    print(case_str)
            finally:
                pass