如何从C stdio.h getline（）替换/忽略无效的Unicode / UTF8字符？

时间：2019-06-14 20:24:15

标签： c++ c c++11 unicode utf-8

在Python上，open Python函数具有以下选项errors='ignore'：

open( '/filepath.txt', 'r', encoding='UTF-8', errors='ignore' )

通过此操作，读取具有无效UTF8字符的文件将不会用任何内容替换它们，即它们将被忽略。例如，带有字符Føö»BÃ¥r的文件将被读取为FøöBår。

如果从Føö»BÃ¥r中用getline()读为stdio.h的行，它将被读为Føö�Bår：

FILE* cfilestream = fopen( "/filepath.txt", "r" );
int linebuffersize = 131072;
char* readline = (char*) malloc( linebuffersize );

while( true )
{
    if( getline( &readline, &linebuffersize, cfilestream ) != -1 ) {
        std::cerr << "readline=" readline << std::endl;
    }
    else {
        break;
    }
}

如何使stdio.h getline()而不是FøöBår读为Føö�Bår，即忽略无效的UTF8字符？

我可以想到一个压倒性的解决方案，它会在读取的每一行上遍历所有字符，并构建一个不包含这些字符的新readline。例如：

FILE* cfilestream = fopen( "/filepath.txt", "r" );
int linebuffersize = 131072;
char* readline = (char*) malloc( linebuffersize );
char* fixedreadline = (char*) malloc( linebuffersize );

int index;
int charsread;
int invalidcharsoffset;

while( true )
{
    if( ( charsread = getline( &readline, &linebuffersize, cfilestream ) ) != -1 )
    {
        invalidcharsoffset = 0;
        for( index = 0; index < charsread; ++index )
        {
            if( readline[index] != '�' ) {
                fixedreadline[index-invalidcharsoffset] = readline[index];
            } 
            else {
                ++invalidcharsoffset;
            }
        }
        std::cerr << "fixedreadline=" << fixedreadline << std::endl;
    }
    else {
        break;
    }
}

相关问题：

  1. Fixing invalid UTF8 characters
  2. Replacing non UTF8 characters
  3. python replace unicode characters
  4. Python unicode: how to replace character that cannot be decoded using utf8 with whitespace?

3 个答案:

答案 0 :(得分：3)

您将看到的东西与实际发生的事情混淆了。 getline函数不执行任何字符替换。 [注1]

您看到一个替换字符（U + FFFD），因为当要求控制台呈现无效的UTF-8代码时，您的控制台会输出该字符。如果大多数控制台都处于UTF-8模式，则将执行此操作。也就是说，当前语言环境为UTF-8。

此外，说一个文件包含“字符Føö»BÃ¥r”最多是不精确的。文件实际上不包含字符。它包含一些字节序列，根据某种编码，这些字节序列可以解释为字符，例如，通过控制台或其他用户演示软件，它们会将它们呈现为字形。不同的编码产生不同的结果。在这种特殊情况下，您有一个文件，该文件是由软件使用Windows-1252编码（或大致等同于ISO 8859-15）创建的，并且正在使用UTF-8在控制台上进行渲染。

这意味着getline读取的数据包含无效的UTF-8序列，但是（可能）不包含替换字符代码。根据您显示的字符串，它包含十六进制字符\xbb，在Windows代码页1252中是十六进制（»）。

要在getline（或任何其他读取文件的C库函数）读取的字符串中查找所有无效的UTF-8序列，则需要扫描该字符串，而不需要扫描特定的代码序列。相反，您需要一次解码一个UTF-8序列，以查找无效的序列。这不是一个简单的任务，但是mbtowc函数可以提供帮助（如果您启用了UTF-8语言环境）。如您在链接的联机帮助页中所见，mbtowc返回有效的“多字节序列”（在UTF-8语言环境中为UTF-8）中包含的字节数，或-1表示无效或序列不完整。在扫描中，您应按有效顺序遍历字节，或删除/忽略开始无效序列的单个字节，然后继续扫描直到到达字符串末尾。

下面是一些经过简单测试的示例代码（用C语言编写）：

#include <stdlib.h>
#include <string.h>

/* Removes in place any invalid UTF-8 sequences from at most 'len' characters of the
 * string pointed to by 's'. (If a NUL byte is encountered, conversion stops.)
 * If the length of the converted string is less than 'len', a NUL byte is
 * inserted.
 * Returns the length of the possibly modified string (with a maximum of 'len'),
 * not including the NUL terminator (if any).
 * Requires that a UTF-8 locale be active; since there is no way to test for
 * this condition, no attempt is made to do so. If the current locale is not UTF-8,
 * behaviour is undefined.
 */
size_t remove_bad_utf8(char* s, size_t len) {
  char* in = s;
  /* Skip over the initial correct sequence. Avoid relying on mbtowc returning
   * zero if n is 0, since Posix is not clear whether mbtowc returns 0 or -1.
   */
  int seqlen;
  while (len && (seqlen = mbtowc(NULL, in, len)) > 0) { len -= seqlen; in += seqlen; }
  char* out = in;

  if (len && seqlen < 0) {
    ++in;
    --len;
    /* If we find an invalid sequence, we need to start shifting correct sequences.  */
    for (; len; in += seqlen, len -= seqlen) {
      seqlen = mbtowc(NULL, in, len);
      if (seqlen > 0) {
        /* Shift the valid sequence (if one was found) */
        memmove(out, in, seqlen);
        out += seqlen;
      }
      else if (seqlen < 0) seqlen = 1;
      else /* (seqlen == 0) */ break;
    }
    *out++ = 0;
  }
  return out - s;
}

注释

  1. 除了底层I / O库的可能的行尾转换外，它还将在Windows之类的使用两个字符CR-LF序列作为一行的系统上用单个\n替换CR-LF -结束指示。

答案 1 :(得分：3)

正如@rici在他的回答中很好地解释的那样，字节序列中可能有几个无效的UTF-8序列。

iconv（3）可能值得一看，例如参见https://linux.die.net/man/3/iconv_open。

  

当将 IGNORE”字符串附加到 tocode 时，目标字符集中无法表示的字符将被静默丢弃。

示例

如果解释为UTF-8，则此字节序列包含一些无效的UTF-8：

"some invalid\xFE\xFE\xFF\xFF stuff"

如果显示此内容，将会看到类似

some invalid���� stuff

当此字符串通过以下C程序中的remove_invalid_utf8函数传递时，无效的UTF-8字节将使用上述的iconv函数删除。

那么结果就是：

some invalid stuff

C程序 

#include <stdio.h>
#include <iconv.h>
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>
#include <errno.h>

char *remove_invalid_utf8(char *utf8, size_t len) {
    size_t inbytes_len = len;
    char *inbuf = utf8;

    size_t outbytes_len = len;
    char *result = calloc(outbytes_len + 1, sizeof(char));
    char *outbuf = result;

    iconv_t cd = iconv_open("UTF-8//IGNORE", "UTF-8");
    if(cd == (iconv_t)-1) {
        perror("iconv_open");
    }
    if(iconv(cd, &inbuf, &inbytes_len, &outbuf, &outbytes_len)) {
        perror("iconv");
    }
    iconv_close(cd);
    return result;
}

int main() {
    char *utf8 = "some invalid\xFE\xFE\xFF\xFF stuff";
    char *converted = remove_invalid_utf8(utf8, strlen(utf8));
    printf("converted: %s to %s\n", utf8, converted);
    free(converted);
    return 0;
}

答案 2 :(得分：1)

我还设法通过尾随/减少所有非ASCII字符来修复它。

这大约需要2.6秒来解析319MB：

#include <stdlib.h>
#include <iostream>

int main(int argc, char const *argv[])
{
    FILE* cfilestream = fopen( "./test.txt", "r" );
    size_t linebuffersize = 131072;

    if( cfilestream == NULL ) {
        perror( "fopen cfilestream" );
        return -1;
    }

    char* readline = (char*) malloc( linebuffersize );
    char* fixedreadline = (char*) malloc( linebuffersize );

    if( readline == NULL ) {
        perror( "malloc readline" );
        return -1;
    }

    if( fixedreadline == NULL ) {
        perror( "malloc fixedreadline" );
        return -1;
    }

    char* source;
    if( ( source = std::setlocale( LC_ALL, "en_US.utf8" ) ) == NULL ) {
        perror( "setlocale" );
    }
    else {
        std::cerr << "locale='" << source << "'" << std::endl;
    }

    int index;
    int charsread;
    int invalidcharsoffset;
    unsigned int fixedchar;

    while( true )
    {
        if( ( charsread = getline( &readline, &linebuffersize, cfilestream ) ) != -1 )
        {
            invalidcharsoffset = 0;
            for( index = 0; index < charsread; ++index )
            {
                fixedchar = static_cast<unsigned int>( readline[index] );
                // std::cerr << "index " << std::setw(3) << index
                //         << " readline " << std::setw(10) << fixedchar
                //         << " -> '" << readline[index] << "'" << std::endl;

                if( 31 < fixedchar && fixedchar < 128 ) {
                    fixedreadline[index-invalidcharsoffset] = readline[index];
                }
                else {
                    ++invalidcharsoffset;
                }
            }

            fixedreadline[index-invalidcharsoffset] = '\0';
            // std::cerr << "fixedreadline=" << fixedreadline << std::endl;
        }
        else {
            break;
        }
    }
    std::cerr << "fixedreadline=" << fixedreadline << std::endl;

    free( readline );
    free( fixedreadline );

    fclose( cfilestream );
    return 0;
}

使用memcpy的替代版本和较慢版本

使用menmove并不会提高速度，因此您可以选择一个。

这大约需要3.1秒来解析319MB：

#include <stdlib.h>
#include <iostream>
#include <cstring>
#include <iomanip>

int main(int argc, char const *argv[])
{
    FILE* cfilestream = fopen( "./test.txt", "r" );
    size_t linebuffersize = 131072;

    if( cfilestream == NULL ) {
        perror( "fopen cfilestream" );
        return -1;
    }

    char* readline = (char*) malloc( linebuffersize );
    char* fixedreadline = (char*) malloc( linebuffersize );

    if( readline == NULL ) {
        perror( "malloc readline" );
        return -1;
    }

    if( fixedreadline == NULL ) {
        perror( "malloc fixedreadline" );
        return -1;
    }

    char* source;
    char* destination;
    char* finalresult;

    int index;
    int lastcopy;
    int charsread;
    int charstocopy;
    int invalidcharsoffset;

    bool hasignoredbytes;
    unsigned int fixedchar;

    if( ( source = std::setlocale( LC_ALL, "en_US.utf8" ) ) == NULL ) {
        perror( "setlocale" );
    }
    else {
        std::cerr << "locale='" << source << "'" << std::endl;
    }

    while( true )
    {
        if( ( charsread = getline( &readline, &linebuffersize, cfilestream ) ) != -1 )
        {
            hasignoredbytes = false;
            source = readline;
            destination = fixedreadline;
            lastcopy = 0;
            invalidcharsoffset = 0;

            for( index = 0; index < charsread; ++index )
            {
                fixedchar = static_cast<unsigned int>( readline[index] );
                // std::cerr << "fixedchar " << std::setw(10)
                //           << fixedchar << " -> '"
                //           << readline[index] << "'" << std::endl;

                if( 31 < fixedchar && fixedchar < 128 ) {
                    if( hasignoredbytes ) {
                        charstocopy = index - lastcopy - invalidcharsoffset;
                        memcpy( destination, source, charstocopy );

                        source += index - lastcopy;
                        lastcopy = index;
                        destination += charstocopy;

                        invalidcharsoffset = 0;
                        hasignoredbytes = false;
                    }
                }
                else {
                    ++invalidcharsoffset;
                    hasignoredbytes = true;
                }
            }

            if( destination != fixedreadline ) {
                charstocopy = charsread - static_cast<int>( source - readline )
                               - invalidcharsoffset;

                memcpy( destination, source, charstocopy );
                destination += charstocopy - 1;

                if( *destination == '\n' ) {
                    *destination = '\0';
                }
                else {
                    *++destination = '\0';
                }
                finalresult = fixedreadline;
            }
            else {
                finalresult = readline;
            }

            // std::cerr << "finalresult=" << finalresult << std::endl;
        }
        else {
            break;
        }
    }
    std::cerr << "finalresult=" << finalresult << std::endl;

    free( readline );
    free( fixedreadline );

    fclose( cfilestream );
    return 0;
}

使用iconv

的优化解决方案

这大约需要4.6秒来解析319MB的文本。

#include <iconv.h>
#include <string.h>
#include <stdlib.h>
#include <iostream>

// Compile it with:
//     g++ -o main test.cpp -O3 -liconv
int main(int argc, char const *argv[])
{
    FILE* cfilestream = fopen( "./test.txt", "r" );
    size_t linebuffersize = 131072;

    if( cfilestream == NULL ) {
        perror( "fopen cfilestream" );
        return -1;
    }

    char* readline = (char*) malloc( linebuffersize );
    char* fixedreadline = (char*) malloc( linebuffersize );

    if( readline == NULL ) {
        perror( "malloc readline" );
        return -1;
    }

    if( fixedreadline == NULL ) {
        perror( "malloc fixedreadline" );
        return -1;
    }

    char* source;
    char* destination;

    int charsread;
    size_t inchars;
    size_t outchars;

    if( ( source = std::setlocale( LC_ALL, "en_US.utf8" ) ) == NULL ) {
        perror( "setlocale" );
    }
    else {
        std::cerr << "locale='" << source << "'" << std::endl;
    }

    iconv_t conversiondescriptor = iconv_open("UTF-8//IGNORE", "UTF-8");
    if( conversiondescriptor == (iconv_t)-1 ) {
        perror( "iconv_open conversiondescriptor" );
    }

    while( true )
    {
        if( ( charsread = getline( &readline, &linebuffersize, cfilestream ) ) != -1 )
        {
            source = readline;
            inchars = charsread;

            destination = fixedreadline;
            outchars = charsread;

            if( iconv( conversiondescriptor, &source, &inchars, &destination, &outchars ) )
            {
                perror( "iconv" );
            }

            // Trim out the new line character
            if( *--destination == '\n' ) {
                *--destination = '\0';
            }
            else {
                *destination = '\0';
            }

            // std::cerr << "fixedreadline='" << fixedreadline << "'" << std::endl;
        }
        else {
            break;
        }
    }
    std::cerr << "fixedreadline='" << fixedreadline << "'" << std::endl;

    free( readline );
    free( fixedreadline );

    if( fclose( cfilestream ) ) {
        perror( "fclose cfilestream" );
    }

    if( iconv_close( conversiondescriptor ) ) {
        perror( "iconv_close conversiondescriptor" );
    }

    return 0;
}

使用mbtowc的最慢的解决方案

这大约需要24.2秒来解析319MB的文本。

如果您注释掉行fixedchar = mbtowc(NULL, source, charsread);并取消注释行charsread -= fixedchar;（破坏无效字符删除），则将需要1.9秒而不是24.2秒（也已编译） -O3优化级别）。

#include <stdlib.h>
#include <string.h>

#include <iostream>
#include <cstring>
#include <iomanip>

int main(int argc, char const *argv[])
{
    FILE* cfilestream = fopen( "./test.txt", "r" );
    size_t linebuffersize = 131072;

    if( cfilestream == NULL ) {
        perror( "fopen cfilestream" );
        return -1;
    }

    char* readline = (char*) malloc( linebuffersize );
    if( readline == NULL ) {
        perror( "malloc readline" );
        return -1;
    }

    char* source;
    char* lineend;
    char* destination;
    int charsread;
    int fixedchar;

    if( ( source = std::setlocale( LC_ALL, "en_US.utf8" ) ) == NULL ) {
        perror( "setlocale" );
    }
    else {
        std::cerr << "locale='" << source << "'" << std::endl;
    }

    while( true )
    {
        if( ( charsread = getline( &readline, &linebuffersize, cfilestream ) ) != -1 )
        {
            lineend = readline + charsread;
            destination = readline;
            for( source = readline; source != lineend; )
            {
                // fixedchar = 1;
                fixedchar = mbtowc(NULL, source, charsread);
                charsread -= fixedchar;

                // std::ostringstream contents;
                // for( int index = 0; index < fixedchar; ++index )
                //         contents << source[index];

                // std::cerr << "fixedchar=" << std::setw(10)
                //         << fixedchar << " -> '"
                //         << contents.str().c_str() << "'" << std::endl;

                if( fixedchar > 0 ) {
                    memmove( destination, source, fixedchar );
                    source += fixedchar;
                    destination += fixedchar;
                }
                else if( fixedchar < 0 ) {
                    source += 1;
                    // std::cerr << "errno=" << strerror( errno ) << std::endl;
                }
                else {
                    break;
                }
            }

            // Trim out the new line character
            if( *--destination == '\n' ) {
                *--destination = '\0';
            }
            else {
                *destination = '\0';
            }

            // std::cerr << "readline='" << readline << "'" << std::endl;
        }
        else {
            break;
        }
    }
    std::cerr << "readline='" << readline << "'" << std::endl;

    if( fclose( cfilestream ) ) {
        perror( "fclose cfilestream" );
    }

    free( readline );
    return 0;
}

使用memmove

以上所有其他版本的最快版本

您不能在这里使用memcpy，因为存储区域重叠！

这大约需要2.4秒来解析319MB。

如果注释掉*destination = *source和memmove( destination, source, 1 )行（破坏了无效字符的删除），其性能仍然与调用memmove时的性能几乎相同。在这里，调用memmove( destination, source, 1 )比直接进行*destination = *source;

慢一点 
#include <stdlib.h>
#include <iostream>
#include <cstring>
#include <iomanip>

int main(int argc, char const *argv[])
{
    FILE* cfilestream = fopen( "./test.txt", "r" );
    size_t linebuffersize = 131072;

    if( cfilestream == NULL ) {
        perror( "fopen cfilestream" );
        return -1;
    }

    char* readline = (char*) malloc( linebuffersize );
    if( readline == NULL ) {
        perror( "malloc readline" );
        return -1;
    }

    char* source;
    char* lineend;
    char* destination;

    int charsread;
    unsigned int fixedchar;

    if( ( source = std::setlocale( LC_ALL, "en_US.utf8" ) ) == NULL ) {
        perror( "setlocale" );
    }
    else {
        std::cerr << "locale='" << source << "'" << std::endl;
    }


    while( true )
    {
        if( ( charsread = getline( &readline, &linebuffersize, cfilestream ) ) != -1 )
        {
            lineend = readline + charsread;
            destination = readline;
            for( source = readline; source != lineend; ++source )
            {
                fixedchar = static_cast<unsigned int>( *source );
                // std::cerr << "fixedchar=" << std::setw(10)
                //         << fixedchar << " -> '" << *source << "'" << std::endl;

                if( 31 < fixedchar && fixedchar < 128 ) {
                    *destination = *source;
                    ++destination;
                }
            }

            // Trim out the new line character
            if( *source == '\n' ) {
                *--destination = '\0';
            }
            else {
                *destination = '\0';
            }

            // std::cerr << "readline='" << readline << "'" << std::endl;
        }
        else {
            break;
        }
    }
    std::cerr << "readline='" << readline << "'" << std::endl;

    if( fclose( cfilestream ) ) {
        perror( "fclose cfilestream" );
    }

    free( readline );
    return 0;
}

奖金

您还可以使用Python C扩展（API）。

解析319MB而不将其转换为缓存版本2.3大约需要UTF-8 char*秒

大约需要3.2秒来解析319MB，将其转换为UTF-8 char *。 并且还需要大约3.2秒来解析319MB，将它们转换为缓存的ASCII char *。

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <iostream>

typedef struct
{
    PyObject_HEAD
}
PyFastFile;

static PyModuleDef fastfilepackagemodule =
{
    // https://docs.python.org/3/c-api/module.html#c.PyModuleDef
    PyModuleDef_HEAD_INIT,
    "fastfilepackage", /* name of module */
    "Example module that wrapped a C++ object", /* module documentation, may be NULL */
    -1, /* size of per-interpreter state of the module, or 
                -1 if the module keeps state in global variables. */

    NULL, /* PyMethodDef* m_methods */
    NULL, /* inquiry m_reload */
    NULL, /* traverseproc m_traverse */
    NULL, /* inquiry m_clear */
    NULL, /* freefunc m_free */
};

// initialize PyFastFile Object
static int PyFastFile_init(PyFastFile* self, PyObject* args, PyObject* kwargs) {
    char* filepath;

    if( !PyArg_ParseTuple( args, "s", &filepath ) ) {
        return -1;
    }

    int linecount = 0;
    PyObject* iomodule;
    PyObject* openfile;
    PyObject* fileiterator;

    iomodule = PyImport_ImportModule( "builtins" );
    if( iomodule == NULL ) {
        std::cerr << "ERROR: FastFile failed to import the io module '"
                "(and open the file " << filepath << "')!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }
    PyObject* openfunction = PyObject_GetAttrString( iomodule, "open" );

    if( openfunction == NULL ) {
        std::cerr << "ERROR: FastFile failed get the io module open "
                << "function (and open the file '" << filepath << "')!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }
    openfile = PyObject_CallFunction( 
            openfunction, "ssiss", filepath, "r", -1, "ASCII", "ignore" );

    if( openfile == NULL ) {
        std::cerr << "ERROR: FastFile failed to open the file'"
                << filepath << "'!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }
    PyObject* iterfunction = PyObject_GetAttrString( openfile, "__iter__" );
    Py_DECREF( openfunction );

    if( iterfunction == NULL ) {
        std::cerr << "ERROR: FastFile failed get the io module iterator" 
                << "function (and open the file '" << filepath << "')!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }
    PyObject* openiteratorobject = PyObject_CallObject( iterfunction, NULL );
    Py_DECREF( iterfunction );

    if( openiteratorobject == NULL ) {
        std::cerr << "ERROR: FastFile failed get the io module iterator object"
                << " (and open the file '" << filepath << "')!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }
    fileiterator = PyObject_GetAttrString( openfile, "__next__" );
    Py_DECREF( openiteratorobject );

    if( fileiterator == NULL ) {
        std::cerr << "ERROR: FastFile failed get the io module iterator "
                << "object (and open the file '" << filepath << "')!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }

    PyObject* readline;
    while( ( readline = PyObject_CallObject( fileiterator, NULL ) ) != NULL ) {
        linecount += 1;
        PyUnicode_AsUTF8( readline );
        Py_DECREF( readline );
        // std::cerr << "linecount " << linecount << " readline '" << readline
        //         << "' '" << PyUnicode_AsUTF8( readline ) << "'" << std::endl;
    }
    std::cerr << "linecount " << linecount << std::endl;

    // PyErr_PrintEx(100);
    PyErr_Clear();
    PyObject* closefunction = PyObject_GetAttrString( openfile, "close" );

    if( closefunction == NULL ) {
        std::cerr << "ERROR: FastFile failed get the close file function for '"
                << filepath << "')!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }

    PyObject* closefileresult = PyObject_CallObject( closefunction, NULL );
    Py_DECREF( closefunction );

    if( closefileresult == NULL ) {
        std::cerr << "ERROR: FastFile failed close open file '"
                << filepath << "')!" << std::endl;
        PyErr_PrintEx(100);
        return -1;
    }
    Py_DECREF( closefileresult );

    Py_XDECREF( iomodule );
    Py_XDECREF( openfile );
    Py_XDECREF( fileiterator );

    return 0;
}

// destruct the object
static void PyFastFile_dealloc(PyFastFile* self) {
    Py_TYPE(self)->tp_free( (PyObject*) self );
}

static PyTypeObject PyFastFileType =
{
    PyVarObject_HEAD_INIT( NULL, 0 )
    "fastfilepackage.FastFile" /* tp_name */
};

// create the module
PyMODINIT_FUNC PyInit_fastfilepackage(void)
{
    PyObject* thismodule;

    // https://docs.python.org/3/c-api/typeobj.html
    PyFastFileType.tp_new = PyType_GenericNew;
    PyFastFileType.tp_basicsize = sizeof(PyFastFile);
    PyFastFileType.tp_dealloc = (destructor) PyFastFile_dealloc;
    PyFastFileType.tp_flags = Py_TPFLAGS_DEFAULT;
    PyFastFileType.tp_doc = "FastFile objects";
    PyFastFileType.tp_init = (initproc) PyFastFile_init;

    if( PyType_Ready( &PyFastFileType) < 0 ) {
        return NULL;
    }

    thismodule = PyModule_Create(&fastfilepackagemodule);
    if( thismodule == NULL ) {
        return NULL;
    }

    // Add FastFile class to thismodule allowing the use to create objects
    Py_INCREF( &PyFastFileType );
    PyModule_AddObject( thismodule, "FastFile", (PyObject*) &PyFastFileType );
    return thismodule;
}

要构建它，请创建具有以上文件内容的文件source/fastfilewrappar.cpp和具有以下内容的setup.py：

#! /usr/bin/env python
# -*- coding: utf-8 -*-
from setuptools import setup, Extension

myextension = Extension(
    language = "c++",
    extra_link_args = ["-std=c++11"],
    extra_compile_args = ["-std=c++11"],
    name = 'fastfilepackage',
    sources = [
        'source/fastfilewrapper.cpp'
    ],
    include_dirs = [ 'source' ],
)

setup(
        name = 'fastfilepackage',
        ext_modules= [ myextension ],
    )

要运行示例，请使用以下Python脚本：

import time
import datetime
import fastfilepackage

testfile = './test.txt'
timenow = time.time()
iterable = fastfilepackage.FastFile( testfile )

fastfile_time = time.time() - timenow
timedifference = datetime.timedelta( seconds=fastfile_time )
print( 'FastFile timedifference', timedifference, flush=True )

示例：

user@user-pc$ /usr/bin/pip3.6 install .
Processing /fastfilepackage
Building wheels for collected packages: fastfilepackage
  Building wheel for fastfilepackage (setup.py) ... done
  Stored in directory: /pip-ephem-wheel-cache-j313cpzc/wheels/e5/5f/bc/52c820
Successfully built fastfilepackage
Installing collected packages: fastfilepackage
  Found existing installation: fastfilepackage 0.0.0
    Uninstalling fastfilepackage-0.0.0:
      Successfully uninstalled fastfilepackage-0.0.0
Successfully installed fastfilepackage-0.0.0

user@user-pc$ /usr/bin/python3.6 fastfileperformance.py
linecount 820800
FastFile timedifference 0:00:03.204614

使用std :: getline

这大约需要4.7秒来解析319MB。

如果您使用UTF-8删除了从最快的基准中借用的stdlib.h getline()删除算法，则需要运行1.7秒。

#include <stdlib.h>
#include <iostream>
#include <locale>
#include <fstream>
#include <iomanip>

int main(int argc, char const *argv[])
{
    unsigned int fixedchar;
    int linecount = -1;

    char* source;
    char* lineend;
    char* destination;

    if( ( source = setlocale( LC_ALL, "en_US.ascii" ) ) == NULL ) {
        perror( "setlocale" );
        return -1;
    }
    else {
        std::cerr << "locale='" << source << "'" << std::endl;
    }

    std::ifstream fileifstream{ "./test.txt" };
    if( fileifstream.fail() ) {
        std::cerr << "ERROR: FastFile failed to open the file!" << std::endl;
        return -1;
    }

    size_t linebuffersize = 131072;
    char* readline = (char*) malloc( linebuffersize );

    if( readline == NULL ) {
        perror( "malloc readline" );
        return -1;
    }

    while( true )
    {
        if( !fileifstream.eof() )
        {
            linecount += 1;
            fileifstream.getline( readline, linebuffersize );
            lineend = readline + fileifstream.gcount();
            destination = readline;

            for( source = readline; source != lineend; ++source )
            {
                fixedchar = static_cast<unsigned int>( *source );
                // std::cerr << "fixedchar=" << std::setw(10)
                //         << fixedchar << " -> '" << *source << "'" << std::endl;

                if( 31 < fixedchar && fixedchar < 128 ) {
                    *destination = *source;
                    ++destination;
                }
            }

            // Trim out the new line character
            if( *source == '\n' ) {
                *--destination = '\0';
            }
            else {
                *destination = '\0';
            }

            // std::cerr << "readline='" << readline << "'" << std::endl;
        }
        else {
            break;
        }
    }
    std::cerr << "linecount='" << linecount << "'" << std::endl;

    if( fileifstream.is_open() ) {
        fileifstream.close();
    }

    free( readline );
    return 0;
}

恢复

  1. 2.6秒使用两个带索引的缓冲区修整UTF-8
  2. 3.1秒使用带有memcpy的两个缓冲区修整UTF-8
  3. 4.6秒用iconv删除无效的UTF-8
  4. 24.2秒用mbtowc删除无效的UTF-8
  5. 2.4秒使用一个指针直接分配的缓冲区修整UTF-8

奖金

  1. 2.3秒内删除无效的UTF-8，而不将其转换为已缓存的UTF-8 char*
  2. 3.2秒删除无效的UTF-8，将其转换为缓存的UTF-8 char*
  3. 3.2秒修剪UTF-8并缓存为ASCII char*
  4. 4.7秒使用一个指针直接分配的缓冲区用std::getline()修剪UTF-8

使用的文件./text.txt包含820.800行，每行等于：

id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char&id-é-char\r\n

以及所有使用编译的版本

  1. g++ (GCC) 7.4.0
  2. iconv (GNU libiconv 1.14)
  3. g++ -o main test.cpp -O3 -liconv && time ./main
相关问题
最新问题