所以，我正在制作一个单词过滤器，用星号替换坏词，但是如果使用像ąężźć等特殊字符那么可能有很多单词组合。
如何使boost :: ireplace_all将这些视为基本字符aezzc？
所以
boost::ireplace_all("żąć", "a", "*");和boost::ireplace_all("zac", "a", "*");
会分别导致ż*ć和z*c吗？
编辑/扩展示例：
const std::set<std::string> badwords =
{
"<not nice word>",
"<another not nice word>"
};
void FilterBadWords(std::string& s)
{
for (auto &badword : badwords)
boost::ireplace_all(s, badword, "*");
}
int main()
{
std::string a("hello you <not nice word> person");
std::string b("hęlló you <nót Nićę wórd> person");
FilterBadWords(a);
FilterBadWords(b);
//a equals "hello you * person"
//b equals "hęlló you * person"
//or as many * as the replaced string lenght, both are fine
}
答案 0 :(得分：3)
Boost Locale通过ICU支持主要校对：
让它发挥作用非常棘手。基本上，使用char字符串进行烘烤，因为Boost字符串算法对代码点一无所知，只是逐字节地迭代（并比较）输入序列（嗯，{{ 1}} char，但这里有点令人困惑。）
因此，解决方案在于转换为utf32字符串（使用char可以使用GCC，因为std::wstring是32位）。 Utf16通常也应该“正常工作”，但它仍然存在我刚才概述的遍历问题，只是很少见。
现在，我已经创建了一个快速而又脏的自定义Finder谓词：
wchar_t 非常低效因为它会在每次调用时构造单字符字符串。这是因为template <typename CharT>
struct is_primcoll_equal
{
is_primcoll_equal(const std::locale& Loc=std::locale()) :
m_Loc(Loc), comp(Loc, boost::locale::collator_base::primary) {}
template< typename T1, typename T2 >
bool operator()(const T1& Arg1, const T2& Arg2) const {
// TODO use `do_compare` methods on the collation itself that
// don't construct basic_string<> instances
return 0 == comp(std::basic_string<CharT>(1, Arg1), std::basic_string<CharT>(1, Arg2));
}
private:
std::locale m_Loc;
boost::locale::comparator<CharT> comp;
};
方法不是do_compare的公共API的一部分。我离开派生自定义collator<>并将其用作读者的练习。
接下来，我们通过包装collator<>来模仿replace_all界面：
find_format_all现在我们只需要字符串扩展转换，我们很高兴：
template<typename SequenceT, typename Range1T, typename Range2T>
inline void collate_replace_all(
SequenceT& Input,
const Range1T& Search,
const Range2T& Format,
const std::locale& Loc=std::locale() )
{
::boost::algorithm::find_format_all(
Input,
::boost::algorithm::first_finder(Search, is_primcoll_equal<typename SequenceT::value_type>(Loc)),
::boost::algorithm::const_formatter(Format) );
}
void FilterBadWords(std::string& s) {
using namespace boost::locale::conv;
std::wstring widened = utf_to_utf<wchar_t>(s, stop);
for (auto& badword : badwords) {
detail::collate_replace_all(widened, badword, L"*"/*, loc*/);
}
s = utf_to_utf<char>(widened, stop);
}
在我的系统上：
#include <boost/algorithm/string/replace.hpp>
#include <boost/locale.hpp>
#include <iostream>
#include <locale>
#include <set>
#include <string>
const std::set<std::string> badwords =
{
"<not nice word>",
"<another not nice word>"
};
namespace detail {
template <typename CharT>
struct is_primcoll_equal
{
is_primcoll_equal(const std::locale& Loc=std::locale()) :
m_Loc(Loc), comp(Loc, boost::locale::collator_base::primary) {}
template< typename T1, typename T2 >
bool operator()(const T1& Arg1, const T2& Arg2) const {
// assert(0 == comp(L"<not nice word>", L"<nót Nićę wórd>"));
// TODO use `do_compare` methods on the collation itself that
// don't construct basic_string<> instances
return 0 == comp(std::basic_string<CharT>(1, Arg1), std::basic_string<CharT>(1, Arg2));
}
private:
std::locale m_Loc;
boost::locale::comparator<CharT> comp;
};
template<typename SequenceT, typename Range1T, typename Range2T>
inline void collate_replace_all(
SequenceT& Input,
const Range1T& Search,
const Range2T& Format,
const std::locale& Loc=std::locale() )
{
::boost::algorithm::find_format_all(
Input,
::boost::algorithm::first_finder(Search, is_primcoll_equal<typename SequenceT::value_type>(Loc)),
::boost::algorithm::const_formatter(Format) );
}
}
void FilterBadWords(std::string& s) {
using namespace boost::locale::conv;
std::wstring widened = utf_to_utf<wchar_t>(s, stop);
for (auto& badword : badwords) {
detail::collate_replace_all(widened, badword, L"*"/*, loc*/);
}
s = utf_to_utf<char>(widened, stop);
}
static_assert(sizeof(wchar_t) == sizeof(uint32_t), "Required for robustness (surrogate pairs, anyone?)");
int main()
{
auto loc = boost::locale::generator().generate("");
std::locale::global(loc);
std::string a("hello you <not nice word> person");
std::string b("hęlló you <nót Nićę wórd> person");
FilterBadWords(a);
FilterBadWords(b);
std::cout << a << "\n";
std::cout << b << "\n";
}
¹显然，Coliru执行环境中的语言环境支持不完整
答案 1 :(得分：0)
作为一个额外的解决方案，使用少一点提升（好吧，你可以编辑它以完全删除提升..）：
const std::vector<std::string> badwords =
{
"badword1",
"badword2",
"badword3",
"badword4"
};
char PolishReplacement[0xFF];
const std::map<std::string, std::string> PolishReplacementMap =
{
{ "ł","l" },
{ "ą","a" },
{ "ę","e" },
{ "ć","c" },
{ "ż","z" },
{ "ź","z" },
{ "ó","o" },
{ "ś","s" },
{ "ń","n" },
{ "Ł","L" },
{ "Ą","A" },
{ "Ę","E" },
{ "Ć","C" },
{ "Ż","Z" },
{ "Ź","Z" },
{ "Ó","O" },
{ "Ś","S" },
{ "Ń","N" }
};
//preconstruct our array, we love speed gain by paying startup time
struct CPolishReplacementInitHack
{
CPolishReplacementInitHack()
{
for (unsigned char c = 0; c < 0xFF; ++c)
{
char tmpstr[2] = { c, 0 };
std::string tmpstdstr(tmpstr);
auto replacement = PolishReplacementMap.find(tmpstdstr);
if (replacement == PolishReplacementMap.end())
PolishReplacement[c] = boost::to_lower_copy(tmpstdstr)[0];
else
PolishReplacement[c] = boost::to_lower_copy(replacement->second)[0];
}
}
} _CPolishReplacementInitHack;
//actual filtering
void FilterBadWords(std::string& s)
{
std::string sc(s);
for (auto& character : sc)
character = PolishReplacement[(unsigned char)character];
for (auto &badword : badwords)
{
size_t pos = sc.find(badword);
size_t size = badword.size();
size_t possize;
while (pos != std::string::npos)
{
possize = pos + size;
s.replace ( s.begin() + pos, s.begin() + possize, "*");
sc.replace(sc.begin() + pos, sc.begin() + possize, "*");
pos = sc.find(badword);
}
}
}
这可能是不可移植的（Windows + Locale +编码依赖？），但是非常快（200 ms / 25000随机句子，i7，调试，没有优化）。