我对如何设置输出文件的编码感到困惑。
我有一个包含内容的测试文件＆＃34; qwe＆＃34; （每行一个字符）。我测试了一些ISO-x编码。我读取文件并生成输出文件。但输出文件始终以UTF-8编码。这本身就令人困惑，因为我从未明确地编写代码来使输出文件UTF-8编码。更令人困惑的是，在一个不同的程序中，我输入了UTF-8并获得了一些ISO编码的输出......再次，没有我告诉要改变他的编码。
这是我的测试代码：
#include <iostream>
#include <fstream>
using namespace std;
int main(){
string in_file = "in.txt"; // some ISO encoding (e.g.)
ifstream in(in_file.c_str());
ofstream out;
out.open("out.txt");
while (in.good()) {
std::string line;
getline(in, line);
out << line << endl;
}
out.close(); // output file is in UTF-8
}
生成一些带有UTF-8输入的ISO的其他程序的代码很长，我找不到测试程序和实际程序之间的差异。但也许理解为什么测试程序的行为方式，已经使我能够找到另一个问题。
所以，基本上我的问题是，为什么输出文件设置为UTF-8，或者是什么决定了ofstream对象的编码。
好的，所以我让我的实际代码更方便一些，所以我现在可以更容易地向你显示。
所以，我在表面层运行两个函数，从输入列表构造一个trie，它还包含为graphviz生成DOT代码的代码。
/*
*
* name: make_trie
* @param trie Trie to build
* @param type Type of tokens to build trie for
* @param gen_gv_code Determines wheter to generate graphviz-code
* (for debug and maintanance purposes)
* @return
*
*/
bool make_trie(Trie* trie, std::string type, bool gen_gv_code=false){
if (gen_gv_code){
gv_file
<< "digraph {\n\t"
<< "rankdir=LR;\n\t"
<< "node [shape = circle];\n\t"
<< "1 [label=1]\n\t"
<< "node [shape = point ]; Start\n\t"
<< "Start -> 1\n\t\t";
}
Element* current = wp_list->iterate();
state_symbol_tuple* sst;
std::string token = "<UNDEF>"; // token to add to trie
// once the last entry in the input list is encountered, make_trie()
// needs to run for as many times as that entry has letters +1 - the
// number of letters of taht stringa lready encoded into the trie to
// fully encode it into it.
bool last_token = false;
bool incr = false;
while (true){
if (type == "tag") { token = current->get_WPTuple_tag(); }
else if (type == "word") { token = current->get_WPTuple_word(); }
else {
cerr
<< "Error (trainer.h):"
<< "Unkown type '"
<< type
<< "'. Token has not been assigned."
<< endl;
abort();
}
// last_state is pointer to state the last transition in the trie
// that matched the string led to
sst = trie->find_state(token);
incr = trie->add(current, sst, gv_file, gen_gv_code);
// as soon as the last token has been encoded into the trie, break
if (last_token && sst->existing) { break; }
// go to the next list item only once the current one is represented
// in the trie
if (incr) {
// Once a word has been coded into the trie, go to the next word.
// Only iterate if you are not at the last elememt, otherwise
// you start at the front of the list again.
if (current->next != 0){
current = wp_list->iterate(); incr = false;
}
}
// enable the condition for the last token, as this is a boundary
// case
if (current->next == 0) { last_token = true;}
// free up memory allocated for current sst
delete sst;
}
if (gen_gv_code){
gv_file << "}";
gv_file.close();
}
return true;
}
/*
*
* name: Trie::add
* @details Encodes a given string into the trie. If the string is not
* in the trie yet, it needs to be passed to this function as many
* times as it has letters +1.
* @param current list element
* @param sst state_symbol_tuple containing information on the last
* state that represents the string to be encoded up to some point.
* Also contains the string itself.
* @return returns boolean, true if token is already represented
* in trie, false else
*
*/
bool Trie::add(Element* current, state_symbol_tuple* sst, \
std::ofstream &gv_file_local, bool gen_gv_code){
if (current != 0){
// if the word is represented in the trie, increment its counter
// and go to the next word in the list
if (sst->existing){
(((sst->state)->find_transition(sst->symbol))->get_successor())->increment_occurance();
if (gen_gv_code){
gv_file_local
<< (((sst->state)->find_transition(sst->symbol))->get_successor())->get_id()
<< "[shape = ellipse label = \""
<< (((sst->state)->find_transition(sst->symbol))->get_successor())->get_id()
<< "\nocc: "
<< (((sst->state)->find_transition(sst->symbol))->get_successor())->get_occurance()
//~ << "\naddr: "
//~ << ((sst->state)->find_transition(sst->symbol))->get_successor()
<< "\" peripheries=2]\n\t\t";
}
return true;
}
// if the current string is a substring of one already enoced into
// the trie, make the substring an accepted one too
else if (sst->is_substring){
(((sst->state)->find_transition(sst->symbol))->get_successor()) \
->make_accepting();
}
// if the word isn't represented in the trie, make a transition
// for the first character of the word that wasn't represented
// and then look for the word anew, until it *is* represented.
else {
(sst->state)->append_back(sst->symbol);
// as the new transition has been appended at the back
// "last" is that new transition
// make an empty successor state that the new transition
// points to
((sst->state)->get_last())->make_successor();
// increment total state count
increment_states_total();
// give the newly added state a unique ID, by making its ID
// the current number of states
(((sst->state)->get_last())->get_successor())->set_id(get_states_total());
if (gen_gv_code){
gv_file_local << (sst->state)->get_id() << " -> " << get_states_total()
<< "[label=\"";
if (sst->symbol == '"') {
gv_file_local << "#";
}
else{
gv_file_local << sst->symbol;
}
gv_file_local << "\"]\n\t\t";
}
get_states_total();
// if the length of the input string -1 is equal to the
// index of the last symbol, that was processed, then that
// was the last symbol of the string and the new state needs
// to become an accepting one
if (sst->index == (sst->str_len-1)){
// access the newly created successor state
// define it as an accepting state
(((sst->state)->get_last())->get_successor())->make_accepting();
}
else if (gen_gv_code){
gv_file_local
<< get_states_total()
<< "[shape = circle label = \""
<< (((sst->state)->find_transition(sst->symbol))->get_successor())->get_id()
//~ << "\naddr: "
//~ << ((sst->state)->find_transition(sst->symbol))->get_successor()
<< "\"]\n\t\t";
}
}
} else { cerr << "list to build trie from is empty" << endl; abort();}
return false;
}
输出文件打开如下：
gv_file.open("gv_file");
传递的就像这样：
make_trie(trie_words, "word", true);
由于这是关于编码的问题，实现的细节并不重要，只有将DOT代码写入输出文件的位。
我的测试输入是这样的（以UTF-8表示）：
ascii-range
ütf-8-ränge
我的输出是这个（在ISO-8859中）
digraph {
rankdir=LR;
node [shape = circle];
1 [label=1]
node [shape = point ]; Start
Start -> 1
1 -> 2[label="a"]
2[shape = circle label = "2"]
2 -> 3[label="s"]
3[shape = circle label = "3"]
3 -> 4[label="c"]
4[shape = circle label = "4"]
4 -> 5[label="i"]
5[shape = circle label = "5"]
5 -> 6[label="i"]
6[shape = circle label = "6"]
6 -> 7[label="-"]
7[shape = circle label = "7"]
7 -> 8[label="r"]
8[shape = circle label = "8"]
8 -> 9[label="a"]
9[shape = circle label = "9"]
9 -> 10[label="n"]
10[shape = circle label = "10"]
10 -> 11[label="g"]
11[shape = circle label = "11"]
11 -> 12[label="e"]
12[shape = ellipse label = "12
occ: 1" peripheries=2]
1 -> 13[label="Ã"]
13[shape = circle label = "13"]
13 -> 14[label="Œ"]
14[shape = circle label = "14"]
14 -> 15[label="t"]
15[shape = circle label = "15"]
15 -> 16[label="f"]
16[shape = circle label = "16"]
16 -> 17[label="-"]
17[shape = circle label = "17"]
17 -> 18[label="8"]
18[shape = circle label = "18"]
18 -> 19[label="-"]
19[shape = circle label = "19"]
19 -> 20[label="r"]
20[shape = circle label = "20"]
20 -> 21[label="Ã"]
21[shape = circle label = "21"]
21 -> 22[label="€"]
22[shape = circle label = "22"]
22 -> 23[label="n"]
23[shape = circle label = "23"]
23 -> 24[label="g"]
24[shape = circle label = "24"]
24 -> 25[label="e"]
25[shape = ellipse label = "25
occ: 1" peripheries=2]
}
所以...我怎么能确保我的输出也用utf8编码？
答案 0 :(得分：1)
在UTF-8中，某些字符被编码为多个字节。例如，ä需要两个字节进行编码。您读取字符串的代码完全忽略了这一点，并假设每个字符有一个字节。然后，您将分别输出字节;这不是合法的UTF-8，所以无论你用什么来计算字符集，推断它必须是ISO-8859。
具体而言，ISO-8859中编码的两个字符Ã然后€与UTF-8中编码ä的2个字节完全相同。
如果，正如我前段时间建议的那样，你看一下原始字节，这将更加明显。