Question

我需要从字符串的起始处提取UTF-16代码单元的数量，在该字符串处，位置名称从Python sting开始。我正在使用Polyglot NER标记Python字符串中的位置。例如，“奥巴马出生在美国。但我出生在阿拉巴马州”，标志着“美国”和“阿拉巴马州”。 Python Polyglot提取器只返回标记的位置，以及它们从前面开始的单词数。如何从单词出现的字符串开头算出UTF-16代码单元的数量？

需要信息https://github.com/Berico-Technologies/CLAVIN/blob/master/src/main/java/com/bericotech/clavin/extractor/LocationOccurrence.java

的Java界面

Answer 1

只是为了澄清一些@Ignacio Vazquez-Abrams＆＃39;评论。处理或分析文本时，您不必担心给定字符占用的字节数。这就是你采用＆＃39;编码的原因。通过先解码＆＃39;编码文本到单独的text / str表示。

>>> encoded_text = 'hello world'.encode('utf16')
>>> encoded_text
b'\xff\xfeh\x00e\x00l\x00l\x00o\x00 \x00w\x00o\x00r\x00l\x00d\x00'
>>> type(encoded_text)
<class 'bytes'>
>>> len(encoded_text)
24


>>> decoded_text = encoded_text.decode('utf16')
>>> decoded_text
'hello world'
>>> type(decoded_text)
<class 'str'>
>>>
>>> len(decoded_text)
11

我确实在你发布的java代码中看到了UTF-16 code units ...

您可以执行以下操作以从头开始获取字节数：

sentence = "Obama was born in the United States. But I was born in Alabama".encode('UTF-16LE')
word = 'United States'.encode('UTF-16LE')

bytes_from_start = None
for start_byte_position in range(len(sentence)):
    candidate = sentence[start_byte_position: start_byte_position + len(word)]
    if word == candidate:
        bytes_from_start = len(sentence[:start_byte_position])
        print('bytes from start: ', bytes_from_start)
        print('len(sentence[:start_byte_position]): ', len(sentence[:start_byte_position]))
        print('Preceding text: "{}"'.format(sentence[:start_byte_position].decode('UTF-16LE')))
        break

但是，如果 UTF-16代码单元 == bytes ，它仍然不清楚。我有一种感觉，它真的只是从一开始就想要字符数。如果这就是你所需要的，你可以使用str对象的.index（）方法：

sentence = "Obama was born in the United States. But I was born in Alabama"
word = 'United States'
characters_from_start = sentence.index(word)

Python Polyglot中的UTF-16代码单元

1 个答案: