在PHP中包装多字节字符以用于控制台输出

时间:2014-08-18 07:49:08

标签: php encoding utf-8 multibyte

我正在尝试将文本片段与固定数量的列对齐。该文本用于记录目的,可能包含一些用户数据,因此我不能假设任何有关输入的内容。为了便于查看,我想确保在标准LINUX控制台中查看时,此文本只有一定数量的固定字符。

PHP没有与其wordwrap - 函数等效的多字节。有几个框架有自己的版本。我一直在尝试几个多字节函数,其中包括this question中的答案,以便为unicode显示编码一些UTF-8文本。大多数答案似乎使用'mb_strlen'来计算子串长度。

为了确保我想要显示的文件被识别为UTF-8,我在文本前面添加了一个字节顺序标记。据我所知,这'应该'导致linux正确识别和格式化。

然后,我尝试编码一些长字符串。在指定的N个字符位置之后,诸如“自上车......”之类的字符串将被正确切断。但是,诸如“₩箕式紫甘氨檀紫红色₩......”之类的字符串则不会。它包含一些在UNIX控制台中具有双倍长度的字符,并且会比预期更晚切断。事实上,在这种情况下,它似乎只是预期显示的字符数的两倍,因此看起来部分或全部双宽字符被读取为宽度1.常量'mb_internal_encoding'正确设置为UTF-8,使用:

mb_internal_encoding("UTF-8");

请注意,当我实际将mb_internal_encoding设置为其他值时,我得到其他奇怪的结果,然后字符串将被截止50%太快,因此在位置26处切断70个字符的字符串。 -offs发生在角色的中间,导致一些mojibake。

我执行工作的类中的换行符代码如下所示,mb_wordwrap重定向到其中一个示例函数:

private function BreakIntoLines($text) {                
    $dtxt = $this->mb_wordwrap($text, self::LINE_LENGTH, PHP_EOL, true);
    return explode(PHP_EOL, $dtxt);
}

然后使用经典的c风格文件编写函数进行文件编写,下面的代码片段会编写其中一行:

fwrite($this->file, $line);
fwrite($this->file, PHP_EOL);

我使用一系列UTF字符验证了输出,它们似乎都正确显示。我似乎完全迷失了问题,这里发生了什么?

注意:尝试自己编写算法似乎是一项过度的工作。似乎也有组合字符。例如,序列\ u68 \ u0300实际上只占用终端中的一个字符。试验,代码:

$str = json_decode("\"\\u0068\\u0300\"");
var_dump($str); 
echo mb_strlen($str);

打印出:string(3)“h”2

1 个答案:

答案 0 :(得分:2)

使用字形函数

对于这个问题的一部分,有一个很好的和干净的解决方案。使用链接问题中的mb_wordwrap函数,但将mb_*函数替换为grapheme_*函数。字素是' intl'的一部分。 php的扩展名,实际上可以正确计算字符长度。有关详细信息,另请参阅this page

因此,我们使用grapheme_strlen代替mb_strlengrapheme_substr代替mb_substr,依此类推。然后在自动换行函数中,我们可以将字符串视为字形集合,这些字形是内部C风格的php字符串,每个字符串都带有N个ASCII符号。这实际上是mb_ *函数运行的方式,因此代码保留了基本相同的结构。

例如:

$str = json_decode("\"\\u0068\\u0300\"");
var_dump($str);         
echo grapheme_strlen($str);

将输出:string(3) "h̀" 1

它仍然无法一次性正确计算更宽的字符。因此,为了实际完全解决问题,我们需要一个实际的实现。有一个C实现使用了双宽字符的范围列表和一个带有二进制遍历的更长列表,用于所有0宽字符,这在php中重新实现。

下面的代码理论上应该正确地包装正常文本,同时还支持所有奇怪的UTF-8字符。它不支持带有其他包装规则的语言(包装在非空格字符中),也可能不完全支持奇怪的空格(ASCII范围之外的空格)。但是,此代码应保证$width个字符的行宽。

/**
 * Word-wrap a multi-byte character (UTF) string
 * @param string $string Initial string. 
 * @param int $width Maximum width of a line
 * @param string $break Which character(s) to line break with
 * @param bool $cut Whether to force-chop long words. 
 * @return string The chopped string. 
 */
private function mb_wordwrap($string, $width = 75, $break = "\n", $cut = false) {
    $string = (string) $string;
    if ($string === '') {
      return '';
    }
    $break = (string) $break;
    if ($break === '') {
      trigger_error('Break string cannot be empty', E_USER_ERROR);
    }
    $width = (int) $width;
    if ($width === 0 && $cut) {
      trigger_error('Cannot force cut when width is zero', E_USER_ERROR);
    }
    if (mb_check_encoding($string, 'ASCII')) {
      return wordwrap($string, $width, $break, $cut);
    }
    $result = '';        
    // Width on display
    // note: stringLength != stringWidth!
    $breakWidth = $this->trueStringWidth($string);
    $lastStartWidth = $lastSpaceWidth = 0;        
    // These all measure 'length'. 
    // Length in characters       
    $breakLength = strlen($break);
    $lastStartLength = $lastSpaceLength = 0;        
    $G_sz = grapheme_strlen($string);        
    $wPos = 0;
    $lPos = 0;
    // Iterate over graphemes
    // Measure using TrueWidth
    // Cut using ASCII (for speed)
    for($i = 0; $i < $G_sz; ++$i) {        
        $char = grapheme_substr($string, $i, 1);
        $charLength = strlen($char);
        $charWidth = $this->trueStringWidth($char);  
        $lookahead_wPos = $wPos + $charWidth;                 
        // If we already have a line break, preserve it and start anew
        if($breakLength !== 1) {
            $possibleBreak = substr($string, $lPos, $breakLength);
        } else {
            $possibleBreak = substr($string, $lPos, $breakLength);
        }
        if ($possibleBreak === $break) {
            $result .= substr($string, $lastStartLength, $lPos - $lastStartLength + $breakLength);
            $lPos += $breakLength - $charLength;
            $wPos += $breakWidth - $charWidth; 
            $lastStartLength = $lastSpaceLength = $breakLength;
            $lastStartWidth = $lastSpaceWidth = $breakWidth;
            continue;
        }            
        // if we match any 'whitespace' character,
        if(preg_match("/\\h/u", $char)) {
            // Exclude the space itself, do not use the lookahead
            if($wPos - $lastStartWidth >= $width) {
                $result .= substr($string, $lastStartLength, $lPos - $lastStartLength) . $break;
                $lastStartLength = $lPos + $charLength;
                $lastStartWidth = $wPos + $charWidth;
            }
            $lastSpaceWidth = $wPos;
            $lastSpaceLength = $lPos;
            continue;
        }
        // look-ahead one character
        $nextChar = grapheme_substr($string, $i+1, 1);      
        // If we are about to overflow, AND the last space was too far back, 
        if($cut && $lookahead_wPos - $lastStartWidth > $width && $lastStartWidth >= $lastSpaceWidth) {
            $result .= substr($string, $lastStartLength, $lPos - $lastStartLength) . $break;
            $lastStartLength = $lPos;
            $lastStartWidth = $wPos;
            continue;                    
        }  
        if ($lookahead_wPos - $lastStartWidth > $width && $lastStartWidth < $lastSpaceWidth) {
            $result .= substr($string, $lastStartLength, $lastSpaceLength - $lastStartLength) . $break;
            $lastStartLength = $lastSpaceLength = $lastSpaceLength + $charLength;
            $lastStartWidth = $lastSpaceWidth = $lastSpaceWidth + $charWidth;
            continue;
        }            
        $wPos += $charWidth;   
        $lPos += $charLength;
    }
    if($lastStartLength !== $lPos) {
        $result .= substr($string, $lastStartLength, $lPos - $lastStartLength);
    }
    return $result;
}

private function trueStringWidth($str) {
    $w = 0;
    for($i = 0; $i < mb_strlen($str); ++$i) {
        $char = mb_substr($str, $i, 1);
        $w += $this->trueCharWidth($char);
    }
    return $w;
}

private function trueCharWidth($char) {
    $ucs = $this->uniord($char);
    // For non-unicode characters, return 1. 
    // Consoles replace them with 'replacement characters' which have width 1!
    if($ucs === FALSE) {return 1;}

    // Do some bit math...

    $combi = [
        [ 0x0300, 0x036F ], [ 0x0483, 0x0486 ], [ 0x0488, 0x0489 ],
        [ 0x0591, 0x05BD ], [ 0x05BF, 0x05BF ], [ 0x05C1, 0x05C2 ],
        [ 0x05C4, 0x05C5 ], [ 0x05C7, 0x05C7 ], [ 0x0600, 0x0603 ],
        [ 0x0610, 0x0615 ], [ 0x064B, 0x065E ], [ 0x0670, 0x0670 ],
        [ 0x06D6, 0x06E4 ], [ 0x06E7, 0x06E8 ], [ 0x06EA, 0x06ED ],
        [ 0x070F, 0x070F ], [ 0x0711, 0x0711 ], [ 0x0730, 0x074A ],
        [ 0x07A6, 0x07B0 ], [ 0x07EB, 0x07F3 ], [ 0x0901, 0x0902 ],
        [ 0x093C, 0x093C ], [ 0x0941, 0x0948 ], [ 0x094D, 0x094D ],
        [ 0x0951, 0x0954 ], [ 0x0962, 0x0963 ], [ 0x0981, 0x0981 ],
        [ 0x09BC, 0x09BC ], [ 0x09C1, 0x09C4 ], [ 0x09CD, 0x09CD ],
        [ 0x09E2, 0x09E3 ], [ 0x0A01, 0x0A02 ], [ 0x0A3C, 0x0A3C ],
        [ 0x0A41, 0x0A42 ], [ 0x0A47, 0x0A48 ], [ 0x0A4B, 0x0A4D ],
        [ 0x0A70, 0x0A71 ], [ 0x0A81, 0x0A82 ], [ 0x0ABC, 0x0ABC ],
        [ 0x0AC1, 0x0AC5 ], [ 0x0AC7, 0x0AC8 ], [ 0x0ACD, 0x0ACD ],
        [ 0x0AE2, 0x0AE3 ], [ 0x0B01, 0x0B01 ], [ 0x0B3C, 0x0B3C ],
        [ 0x0B3F, 0x0B3F ], [ 0x0B41, 0x0B43 ], [ 0x0B4D, 0x0B4D ],
        [ 0x0B56, 0x0B56 ], [ 0x0B82, 0x0B82 ], [ 0x0BC0, 0x0BC0 ],
        [ 0x0BCD, 0x0BCD ], [ 0x0C3E, 0x0C40 ], [ 0x0C46, 0x0C48 ],
        [ 0x0C4A, 0x0C4D ], [ 0x0C55, 0x0C56 ], [ 0x0CBC, 0x0CBC ],
        [ 0x0CBF, 0x0CBF ], [ 0x0CC6, 0x0CC6 ], [ 0x0CCC, 0x0CCD ],
        [ 0x0CE2, 0x0CE3 ], [ 0x0D41, 0x0D43 ], [ 0x0D4D, 0x0D4D ],
        [ 0x0DCA, 0x0DCA ], [ 0x0DD2, 0x0DD4 ], [ 0x0DD6, 0x0DD6 ],
        [ 0x0E31, 0x0E31 ], [ 0x0E34, 0x0E3A ], [ 0x0E47, 0x0E4E ],
        [ 0x0EB1, 0x0EB1 ], [ 0x0EB4, 0x0EB9 ], [ 0x0EBB, 0x0EBC ],
        [ 0x0EC8, 0x0ECD ], [ 0x0F18, 0x0F19 ], [ 0x0F35, 0x0F35 ],
        [ 0x0F37, 0x0F37 ], [ 0x0F39, 0x0F39 ], [ 0x0F71, 0x0F7E ],
        [ 0x0F80, 0x0F84 ], [ 0x0F86, 0x0F87 ], [ 0x0F90, 0x0F97 ],
        [ 0x0F99, 0x0FBC ], [ 0x0FC6, 0x0FC6 ], [ 0x102D, 0x1030 ],
        [ 0x1032, 0x1032 ], [ 0x1036, 0x1037 ], [ 0x1039, 0x1039 ],
        [ 0x1058, 0x1059 ], [ 0x1160, 0x11FF ], [ 0x135F, 0x135F ],
        [ 0x1712, 0x1714 ], [ 0x1732, 0x1734 ], [ 0x1752, 0x1753 ],
        [ 0x1772, 0x1773 ], [ 0x17B4, 0x17B5 ], [ 0x17B7, 0x17BD ],
        [ 0x17C6, 0x17C6 ], [ 0x17C9, 0x17D3 ], [ 0x17DD, 0x17DD ],
        [ 0x180B, 0x180D ], [ 0x18A9, 0x18A9 ], [ 0x1920, 0x1922 ],
        [ 0x1927, 0x1928 ], [ 0x1932, 0x1932 ], [ 0x1939, 0x193B ],
        [ 0x1A17, 0x1A18 ], [ 0x1B00, 0x1B03 ], [ 0x1B34, 0x1B34 ],
        [ 0x1B36, 0x1B3A ], [ 0x1B3C, 0x1B3C ], [ 0x1B42, 0x1B42 ],
        [ 0x1B6B, 0x1B73 ], [ 0x1DC0, 0x1DCA ], [ 0x1DFE, 0x1DFF ],
        [ 0x200B, 0x200F ], [ 0x202A, 0x202E ], [ 0x2060, 0x2063 ],            
        [ 0x206A, 0x206F ], [ 0x20D0, 0x20EF ], [ 0x302A, 0x302F ],
        [ 0x3099, 0x309A ], [ 0xA806, 0xA806 ], [ 0xA80B, 0xA80B ],
        [ 0xA825, 0xA826 ], [ 0xFB1E, 0xFB1E ], [ 0xFE00, 0xFE0F ],
        [ 0xFE20, 0xFE23 ], [ 0xFEFF, 0xFEFF ], [ 0xFFF9, 0xFFFB ],
        [ 0x10A01, 0x10A03 ], [ 0x10A05, 0x10A06 ], [ 0x10A0C, 0x10A0F ],
        [ 0x10A38, 0x10A3A ], [ 0x10A3F, 0x10A3F ], [ 0x1D167, 0x1D169 ],
        [ 0x1D173, 0x1D182 ], [ 0x1D185, 0x1D18B ], [ 0x1D1AA, 0x1D1AD ],
        [ 0x1D242, 0x1D244 ], [ 0xE0001, 0xE0001 ], [ 0xE0020, 0xE007F ],
        [ 0xE0100, 0xE01EF ]
      ];

/* test for 8-bit control characters */
if ($ucs === 0)
  return 0;
if ($ucs < 32 || ($ucs >= 0x7f && $ucs < 0xa0))
  return 0;

/* binary search in table of non-spacing characters */
if ($this->binaryIntervalSearch($combi, $ucs))
  return 0;

/* if we arrive here, ucs is not a combining or C0/C1 control character */

return 1 + 
  ($ucs >= 0x1100 &&
   ($ucs <= 0x115f ||                    /* Hangul Jamo init. consonants */
   $ucs == 0x2329 || $ucs == 0x232a ||
  ($ucs >= 0x2e80 && $ucs <= 0xa4cf &&
   $ucs != 0x303f) ||                  /* CJK ... Yi */
  ($ucs >= 0xac00 && $ucs <= 0xd7a3) || /* Hangul Syllables */
  ($ucs >= 0xf900 && $ucs <= 0xfaff) || /* CJK Compatibility Ideographs */
  ($ucs >= 0xfe10 && $ucs <= 0xfe19) || /* Vertical forms */
  ($ucs >= 0xfe30 && $ucs <= 0xfe6f) || /* CJK Compatibility Forms */
  ($ucs >= 0xff00 && $ucs <= 0xff60) || /* Fullwidth Forms */
  ($ucs >= 0xffe0 && $ucs <= 0xffe6) ||
  ($ucs >= 0x20000 && $ucs <= 0x2fffd) ||
  ($ucs >= 0x30000 && $ucs <= 0x3fffd)));
}

private function uniord($c) {
    if (ord($c{0}) >=0 && ord($c{0}) <= 127) {
        return ord($c{0});             
    }
    if (ord($c{0}) >= 192 && ord($c{0}) <= 223) {
        return (ord($c{0})-192)*64 + (ord($c{1})-128);
    }
    if (ord($c{0}) >= 224 && ord($c{0}) <= 239) {
        return (ord($c{0})-224)*4096 + (ord($c{1})-128)*64 + (ord($c{2})-128);
    }
    if (ord($c{0}) >= 240 && ord($c{0}) <= 247) {
        return (ord($c{0})-240)*262144 + (ord($c{1})-128)*4096 + (ord($c{2})-128)*64 
                + (ord($c{3})-128);
    }
    if (ord($c{0}) >= 248 && ord($c{0}) <= 251) {
        return (ord($c{0})-248)*16777216 + (ord($c{1})-128)*262144 + (ord($c{2})-
                128)*4096 + (ord($c{3})-128)*64 + (ord($c{4})-128);
    }
    if (ord($c{0}) >= 252 && ord($c{0}) <= 253) {
        return (ord($c{0})-252)*1073741824 + (ord($c{1})-128)*16777216 + (ord($c{2})-
               128)*262144 + (ord($c{3})-128)*4096 + (ord($c{4})-128)*64 + 
               (ord($c{5})-128);
    }
    if (ord($c{0}) >= 254 && ord($c{0}) <= 255) {    //  error 
        return FALSE;
    }
    return 0;
}   //  function _uniord()



// It is assumed the interval array is sorted!
// It is assumed we have a SIMPLE array (indexed 0, 1, 2, ...). 
private function binaryIntervalSearch($array, $element) {
    if(count($array) === 1) {
        if($array[0][0] <= $element && $element <= $array[0][1]) {
            return true;
        } else {
            return false;
        }
    } else if(count($array) === 0) {
        return false;
    }        
    // split the array into two halves and a central element. 
    $tC = count($array) >> 1;
    // rightmost left element
    if($array[$tC-1][1] >= $element) {
        return $this->binaryIntervalSearch(array_slice($array, 0, $tC), $element);
    } else if($array[$tC][0] <= $element) {
        return $this->binaryIntervalSearch(array_slice($array, $tC), $element);
    }
    return false;   
}