是否有所有国际标志符号的字符集？

时间：2012-02-29 21:25:44

标签： parsing unicode character-encoding string-parsing punctuation

我正在尝试将utf-8字符串解析为“一口大小”的段。例如，我想将文本分解为“句子”。

是否有与所有语言的句子结尾相对应的全面字符集（或正则表达式）？我正在寻找可以捕捉拉丁时期，感叹号和审讯标记，中国和日本句号等的东西。

像上面这样的东西，但相当于一个逗号也会很棒。

3 个答案:

答案 0 :(得分：6)

您需要查看具有\p{Sentence_Break=STerm}属性的\p{Sentence_Break=ATerm}或\p{Terminal_Punctuation}属性的代码点。针对Unicode v6.1运行the unichars script，我们了解到这些代码点符合所有这些标准：

$ unichars -gas '[\p{Sentence_Break=STerm}\p{Sentence_Break=ATerm}]' '\p{Terminal_Punctuation}'
U+00021 ‭ !  GC=Po SC=Common       EXCLAMATION MARK
U+0002E ‭ .  GC=Po SC=Common       FULL STOP
U+0003F ‭ ?  GC=Po SC=Common       QUESTION MARK
U+00589 ‭ ։  GC=Po SC=Common       ARMENIAN FULL STOP
U+0061F ‭ ؟  GC=Po SC=Common       ARABIC QUESTION MARK
U+006D4 ‭ ۔  GC=Po SC=Arabic       ARABIC FULL STOP
U+00700 ‭ ܀  GC=Po SC=Syriac       SYRIAC END OF PARAGRAPH
U+00701 ‭ ܁  GC=Po SC=Syriac       SYRIAC SUPRALINEAR FULL STOP
U+00702 ‭ ܂  GC=Po SC=Syriac       SYRIAC SUBLINEAR FULL STOP
U+007F9 ‭ ߹  GC=Po SC=Nko          NKO EXCLAMATION MARK
U+00964 ‭ ।  GC=Po SC=Common       DEVANAGARI DANDA
U+00965 ‭ ॥  GC=Po SC=Common       DEVANAGARI DOUBLE DANDA
U+0104A ‭ ၊  GC=Po SC=Myanmar      MYANMAR SIGN LITTLE SECTION
U+0104B ‭ ။  GC=Po SC=Myanmar      MYANMAR SIGN SECTION
U+01362 ‭ ።  GC=Po SC=Ethiopic     ETHIOPIC FULL STOP
U+01367 ‭ ፧  GC=Po SC=Ethiopic     ETHIOPIC QUESTION MARK
U+01368 ‭ ፨  GC=Po SC=Ethiopic     ETHIOPIC PARAGRAPH SEPARATOR
U+0166E ‭ ᙮  GC=Po SC=Canadian_Aboriginal CANADIAN SYLLABICS FULL STOP
U+01803 ‭ ᠃  GC=Po SC=Common       MONGOLIAN FULL STOP
U+01809 ‭ ᠉  GC=Po SC=Mongolian    MONGOLIAN MANCHU FULL STOP
U+01944 ‭ ᥄  GC=Po SC=Limbu        LIMBU EXCLAMATION MARK
U+01945 ‭ ᥅  GC=Po SC=Limbu        LIMBU QUESTION MARK
U+01AA8 ‭ ᪨  GC=Po SC=Tai_Tham     TAI THAM SIGN KAAN
U+01AA9 ‭ ᪩  GC=Po SC=Tai_Tham     TAI THAM SIGN KAANKUU
U+01AAA ‭ ᪪  GC=Po SC=Tai_Tham     TAI THAM SIGN SATKAAN
U+01AAB ‭ ᪫  GC=Po SC=Tai_Tham     TAI THAM SIGN SATKAANKUU
U+01B5A ‭ ᭚  GC=Po SC=Balinese     BALINESE PANTI
U+01B5B ‭ ᭛  GC=Po SC=Balinese     BALINESE PAMADA
U+01B5E ‭ ᭞  GC=Po SC=Balinese     BALINESE CARIK SIKI
U+01B5F ‭ ᭟  GC=Po SC=Balinese     BALINESE CARIK PAREREN
U+01C3B ‭ ᰻  GC=Po SC=Lepcha       LEPCHA PUNCTUATION TA-ROL
U+01C3C ‭ ᰼  GC=Po SC=Lepcha       LEPCHA PUNCTUATION NYET THYOOM TA-ROL
U+01C7E ‭ ᱾  GC=Po SC=Ol_Chiki     OL CHIKI PUNCTUATION MUCAAD
U+01C7F ‭ ᱿  GC=Po SC=Ol_Chiki     OL CHIKI PUNCTUATION DOUBLE MUCAAD
U+0203C ‭ ‼  GC=Po SC=Common       DOUBLE EXCLAMATION MARK
U+0203D ‭ ‽  GC=Po SC=Common       INTERROBANG
U+02047 ‭ ⁇  GC=Po SC=Common       DOUBLE QUESTION MARK
U+02048 ‭ ⁈  GC=Po SC=Common       QUESTION EXCLAMATION MARK
U+02049 ‭ ⁉  GC=Po SC=Common       EXCLAMATION QUESTION MARK
U+02E2E ‭ ⸮  GC=Po SC=Common       REVERSED QUESTION MARK
U+03002 ‭ 。 GC=Po SC=Common       IDEOGRAPHIC FULL STOP
U+0A4FF ‭ ꓿  GC=Po SC=Lisu         LISU PUNCTUATION FULL STOP
U+0A60E ‭ ꘎  GC=Po SC=Vai          VAI FULL STOP
U+0A60F ‭ ꘏  GC=Po SC=Vai          VAI QUESTION MARK
U+0A6F3 ‭ ꛳  GC=Po SC=Bamum        BAMUM FULL STOP
U+0A6F7 ‭ ꛷  GC=Po SC=Bamum        BAMUM QUESTION MARK
U+0A876 ‭ ꡶  GC=Po SC=Phags_Pa     PHAGS-PA MARK SHAD
U+0A877 ‭ ꡷  GC=Po SC=Phags_Pa     PHAGS-PA MARK DOUBLE SHAD
U+0A8CE ‭ ꣎  GC=Po SC=Saurashtra   SAURASHTRA DANDA
U+0A8CF ‭ ꣏  GC=Po SC=Saurashtra   SAURASHTRA DOUBLE DANDA
U+0A92F ‭ ꤯  GC=Po SC=Kayah_Li     KAYAH LI SIGN SHYA
U+0A9C8 ‭ ꧈  GC=Po SC=Javanese     JAVANESE PADA LINGSA
U+0A9C9 ‭ ꧉  GC=Po SC=Javanese     JAVANESE PADA LUNGSI
U+0AA5D ‭ ꩝  GC=Po SC=Cham         CHAM PUNCTUATION DANDA
U+0AA5E ‭ ꩞  GC=Po SC=Cham         CHAM PUNCTUATION DOUBLE DANDA
U+0AA5F ‭ ꩟  GC=Po SC=Cham         CHAM PUNCTUATION TRIPLE DANDA
U+0AAF0 ‭ ꫰  GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHAN
U+0AAF1 ‭ ꫱  GC=Po SC=Meetei_Mayek MEETEI MAYEK AHANG KHUDAM
U+0ABEB ‭ ꯫  GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHEI
U+0FE52 ‭ ﹒ GC=Po SC=Common       SMALL FULL STOP
U+0FE56 ‭ ﹖ GC=Po SC=Common       SMALL QUESTION MARK
U+0FE57 ‭ ﹗ GC=Po SC=Common       SMALL EXCLAMATION MARK
U+0FF01 ‭ ！ GC=Po SC=Common       FULLWIDTH EXCLAMATION MARK
U+0FF0E ‭ ． GC=Po SC=Common       FULLWIDTH FULL STOP
U+0FF1F ‭ ？ GC=Po SC=Common       FULLWIDTH QUESTION MARK
U+0FF61 ‭ ｡  GC=Po SC=Common       HALFWIDTH IDEOGRAPHIC FULL STOP
U+11047 ‭   GC=Po SC=Brahmi       BRAHMI DANDA
U+11048 ‭   GC=Po SC=Brahmi       BRAHMI DOUBLE DANDA
U+110BE ‭   GC=Po SC=Kaithi       KAITHI SECTION MARK
U+110BF ‭   GC=Po SC=Kaithi       KAITHI DOUBLE SECTION MARK
U+110C0 ‭   GC=Po SC=Kaithi       KAITHI DANDA
U+110C1 ‭   GC=Po SC=Kaithi       KAITHI DOUBLE DANDA
U+11141 ‭   GC=Po SC=Chakma       CHAKMA DANDA
U+11142 ‭   GC=Po SC=Chakma       CHAKMA DOUBLE DANDA
U+11143 ‭   GC=Po SC=Chakma       CHAKMA QUESTION MARK
U+111C5 ‭   GC=Po SC=Sharada      SHARADA DANDA
U+111C6 ‭   GC=Po SC=Sharada      SHARADA DOUBLE DANDA

反过来说 - 也就是说，给定一个代码点而不是在给定一组属性的情况下找到代码点的属性 - 使用the companion uniprops script，它会提取给定代码点的所有属性： 

$ uniprops -a . \? \!
U+002E ‹.› \N{FULL STOP}
    \pP \p{Po}
    All Any ASCII Assigned Basic_Latin Case_Ignorable CI Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn
       Pattern_Syntax PatSyn POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print
       X_POSIX_Punct
    Age=1.1 Block=Basic_Latin Bidi_Class=Common_Separator BC=CS Bidi_Class=CS Block=ASCII BLK=ASCII Canonical_Combining_Class=0
       Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
       East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
       Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
       Line_Break=Infix_Numeric LB=IS Line_Break=IS Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
       Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
       IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=AT Sentence_Break=ATerm SB=AT
       Word_Break=MB Word_Break=MidNumLet WB=MB _Case_Ignorable _X_Begin
U+003F ‹?› \N{QUESTION MARK}
    \pP \p{Po}
    All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn
       POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct
    Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0
       Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
       East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
       Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
       Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
       Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
       IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST
       Word_Break=Other WB=XX Word_Break=XX _X_Begin
U+0021 ‹!› \N{EXCLAMATION MARK}
    \pP \p{Po}
    All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn
       POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct
    Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0
       Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
       East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
       Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
       Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
       Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
       IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST
       Word_Break=Other WB=XX Word_Break=XX _X_Begin

我怀疑你应该检查整个句子中断属性。

还有a 3rd script in the suite, uninames，其中包含以下内容：

$ uninames sentence
 ;  037E        GREEK QUESTION MARK
        = erotimatiko
        * sentence-final punctuation
        * 003B is the preferred character
        x (question mark - 003F)
        : 003B semicolon
 ⁚  205A        TWO DOT PUNCTUATION
        * historically used to indicate the end of a sentence or change of speaker
        * extends from baseline to cap height
        x (presentation form for vertical two dot leader - FE30)
        x (greek acrophonic epidaurean two - 1015B)
   110BE       KAITHI SECTION MARK
        * marks end of sentence

我发现这三个程序对于探索Unicode属性是不可或缺的。您可以使用the CPAN Unicode::Tussle suite安装它们，也可以单独检查它们here。

答案 1 :(得分：3)

我没有遇到任何有关此类信息的汇编，我希望这是收集它的主要工作。对于一些广泛使用的语言，您可以从芝加哥风格手册中获取信息。有一些关于标点符号的信息在http://unicode.org/repos/cldr-tmp/trunk/diff/by_type/misc.exemplarCharacters-other.html处通常用于不同语言，但仅涵盖一小部分语言，并且不区分句子终止字符。

仅使用字符是不够的，因为例如在英语中，句号“。”出现在许多不终止句子的情境中，如“eg”或“1.5”。

答案 2 :(得分：3)

中文，日文和韩文使用。。泰国人使用空间。请参阅此Unicode full stop equivalents列表。

相关问题
最新问题