Question

我想将两个单词列表合并到一个文件中。必须删除所有重复项。每个单词都用换行符分隔。我已经搜索过这种程序，但我找不到任何东西。我在寻找合适的东西吗？是否有这个c / c ++实现？

Answer 1

// read input
std::ifstream in( file_path );
typedef std::set< std::string > wordlist_type;
wordlist_type wordlist;
std::string word;

while ( in >> word ) {
    wordlist.insert( word );
}

// repeat with other files to merge more wordlists

// now output to a new file
std::ofstream out( output_path );
for ( wordlist_type::iterator it = wordlist.begin(); it != wordlist.end(); ++ it ) {
    out << * it << '\n';
}

Answer 2

文件有多大。如果你可以将它们都放在内存中，使用STL这是相对简单的：

std::vector<std::string> v(
        (std::istream_iterator<std::string>( ifile1 )),
        (std::istream_iterator<std::string>()));
v.insert(v.end(),
         std::istream_iterator<std::string>( ifile2 ),
         std::istream_iterator<std::string>());
std::sort( v.begin(), v.end() );
std::copy( v.begin(), std::unique( v.begin(), v.end() ),
           std::ostream_iterator<std::string>( ofile, "\n" ) );

或

std::vector<std::string> v1(
        (std::istream_iterator<std::string>( ifile1 )),
        (std::istream_iterator<std::string>()) );
std::sort( v1.begin(), v1.end() );
v1.erase( std::unique( v1.begin(), v1.end() ), v1.end() );
std::vector<std::string> v2(
        (std::istream_iterator<std::string>( ifile2 )),
        (std::istream_iterator<std::string>()) );
std::sort( v2.begin(), v2.end() );
v2.erase( std::unique( v2.begin(), v2.end() ), v2.end() );
std::set_intersection( v1.begin(), v1.end(),
                       v2.begin(), v2.end(),
                       std::ostream_iterator<std::string>( ofile, "\n" ) );

如果他们不适合记忆，你可能需要对每个人进行排序文件（使用system调用本地实用程序），然后执行手动合并：

class FilterDuplicates
{
    std::ostream& myDest;
    std::string myLastOutput;
public:
    Outputter( std::ostream& dest ) : myDest( dest ) {}
    void write( std::string const& word ) const
    {
        if ( word != myLastOutput ) {
            myDest << word;
            myLastOutput = word;
        }
    }
};

ifile1 >> s1;
ifile2 >> s2;
FilterDuplicates out( ofile )
while ( ifile1 && ifile2 ) {
    if ( s1 < s2 ) {
        out.write( s1 );
        ifile1 >> s1;
    } else {
        out.write( s2 );
        ifile2 >> s2;
    }
}
while ( ifile1 ) {
    out.write( s1 );
    ifile1 >> s1;
}
while ( ifile2 ) {
    out.write( s2 );
    ifile2 >> s2;
}

Answer 3

#include <string>
#include <set>
#include <iostream>

int main()
{
    std::set<std::string> s;
    std::string word;
    while (std::cin >> word)
        s.insert(s);
    for (std::set<std::string>::const_iterator i = s.begin(); i != s.end(); ++i)
        std::cout << s << '\n';
}

用法：

cat input1 input2 | program > output

Answer 4

如果您有权访问unix

cat file1 file2 | sort | uniq > file3

Answer 5

像这样......

std::set<std::string> words;    
std::string word;

while(cin >> word)
  if (words.insert(word).second)
    cout << word;

编辑：哎呀，太急于简化......

合并两个单词列表

5 个答案: