使用Boost Spirit / Fusion轻松解析带有枚举字段和STL容器的结构

时间:2019-06-28 11:16:25

标签: c++ parsing boost-spirit boost-fusion

要增强功能,我实际上需要增强精神来编写一个简单的解析器来填充某些数据结构。

大致如下:

struct Task
{
    const string dataname;
    const Level level;
    const string aggregator;
    const set<string> groupby;
    void operator();
};


struct Schedule
{
    map<Level, ComputeTask> tasks;
    // I have left just to make it seems that 
    // the struct wrapping over the map is not
    // useless (this is not the full code)
    void operator()(const InstancePtr &node); 
};

关于Task,我不知道如何使用employee example中提到的BOOST_FUSION_ADAPT_STRUCT或变体来使其与枚举和STL容器字段一起使用。

Schedule类似的问题,但是这次我也使用了一种用户类型(也许已经注册为Fusion,它是否是递归的?)。

我正在设计文件格式,结构定义和文件格式可能会更改,因此我更喜欢使用boost而不是手工制作但难以维护的代码。我也这样做是出于学习目的。

文件如下所示:

level: level operation name on(data1, data2, data3)
level: level operation name on()
level: level operation name on(data1, data2)

一行{是mapSchedule的一个条目,在:之前是键,然后其余的行定义了Task。 在level被与enum Level相对应的某些级别的关键字代替的情况下,operation的类似情况,name是允许的名称之一(在一组关键字中),{ {1}是一个关键字,括号内是用户提供的零个或多个字符串,这些字符串应填充on()中的set<string> groupby字段。

我希望它具有可读性,甚至可以添加不会增加可读性的英语关键字,这是使用某些解析库而不是手工代码的另一个原因。

如果您认为我的问题不够清楚,请随时询问更多细节。

谢谢。

2 个答案:

答案 0 :(得分:1)

我建议用户@sehe提供解决方案。这是非常灵活的。

但是我也想分享纯C ++解决方案。正如我在上面的评论中所写的那样,您的输入语言相当简单。您甚至可以使用标准提取器运算符阅读第一个元素。其余的可以使用std :: istream:iterator循环读取。

您还可以使用C ++ std :: regex来验证输入。因为您的语言是Chomsky-Type-3常规语言,所以这很容易实现。如果输入字符串有效,则可以使用std :: regex元素和std :: regex_token_iterator来获取数据。

我为您创建了一个示例。数据打包在一个结构中。对于此结构,我已经覆盖了inserter和extractor运算符。因此,使用std :: iostream函数可以轻松进行输入和输出。

总的来说,我有一个内胆,可以读取完整的输入文件并将数据放入向量中。因此,我用构造函数参数定义了变量。就是这样。所有数据将根据需要提供。为了调试,我将结果打印在屏幕上。

作为练习,我将数据放在地图中。

#include <iostream>
#include <string>
#include <vector>
#include <map>
#include <iterator>
#include <regex>
#include <sstream>


std::istringstream testData(
R"#(level1: levelA operation0 name0 on(data10, data12, data13)
level2: levelB operation1 name1 on(  data1  )
level3: levelC operation2 name2 on()
level4: levelD operation3 name3 on(data2, data3)
level5: levelE operation4 name4 on(data4, data5, data6, data7)
level6: levelF operation5 name5 on(data8, data9)
)#");


const std::regex InputFileRegEx(R"#((\w+)(?:[\:\s]+)(\w+)(?:\s+)(\w+)(?:\s+)(\w+)(?:\s+)(?:on\s*\()(.*)(?:\)))#");

struct Data
{   // Our Data
    std::string levelLeft{};            // Left Element for Map
    struct Right{                       // Right element for Map. Sub Struct
        std::string levelRight{};
        std::string operation{};
        std::string name{};
        std::vector<std::string> data;  // The data in the on( section
    } r;

    // Overload the extractor operator. With that someting like "Data d;std::cin >> d; " is easiliy possible
    friend std::istream& operator >> (std::istream& is, Data& d) {
        std::string line; getline(is, line);                // Read a complete line
        std::smatch sm{};                                   // Prepare match result values
        if (std::regex_match(line, sm, InputFileRegEx)) {   // CHeck, if the input string is valid
            // Copy all data
            d.levelLeft = sm[1]; d.r.levelRight = sm[2]; d.r.operation = sm[3]; d.r.name = sm[4]; std::string str(sm[5]);
            str.erase(remove_if(str.begin(), str.end(), isspace), str.end()); std::regex comma(","); d.r.data.clear();
            if (str.size()) std::copy(std::sregex_token_iterator(str.begin(), str.end(), comma, -1), std::sregex_token_iterator(), std::back_inserter(d.r.data));
        }
        else is.setstate(std::ios::failbit);
        return is;
    }
    // Overload inserter operator. Only for debug purposes and for illustration
    friend std::ostream& operator << (std::ostream& os, const Data& d) {
        // Print normal data members
        std::cout << d.levelLeft << " :: " << d.r.levelRight << ' ' << d.r.operation << ' ' << d.r.name << " --> ";
        // Print the mebers of the vector
        std::copy(d.r.data.begin(), d.r.data.end(), std::ostream_iterator<std::string>(os, " "));std::cout << '\n';
        return os;
    }
};

using MyMap = std::map<std::string, Data::Right>;

int main()
{
    // Read all test data in an array of test data. The one-Liner  :-)
    std::vector<Data> dataAll{std::istream_iterator<Data>(testData), std::istream_iterator<Data>() };

    // For debug purposes. Print to console
    std::copy(dataAll.begin(), dataAll.end(), std::ostream_iterator<Data>(std::cout, "\n"));

    MyMap myMap{};  // Put all Data in map
    for (const Data& d : dataAll) myMap[d.levelLeft] = d.r;

    return 0;
}

因此,main函数很小,其余函数也不是真正的大代码。相当简单。

希望这能提供一些见识。

答案 1 :(得分:0)

因此,作为示例进行一些假设并不能使含义很清楚。但是这里:

带有随机枚举:

enum class Level { One, Two, Three, LEVEL };
  

旁注:std::set<>可能需要是一个顺序容器,因为通常groupby操作不是可交换的(顺序很重要)。我不知道您的域名,

适应:

BOOST_FUSION_ADAPT_STRUCT(ComputeTask, level, aggregator, dataname, groupby)
BOOST_FUSION_ADAPT_STRUCT(Schedule, tasks)

请注意,我巧妙地将适应字段按语法顺序放置。这对将来有很大帮助。

想到的最简单的语法:

template <typename It>
struct Parser : qi::grammar<It, Schedule()> {
    Parser() : Parser::base_type(_start) {
        using namespace qi;

        _any_word    = lexeme [ +char_("a-zA-Z0-9-_./") ];
        _operation   = _any_word; // TODO
        _group_field = _any_word; // TODO
        _dataname    = _any_word; // TODO

        _level       = no_case [ _level_sym ];
        _groupby     = '(' >> -(_group_field % ',') >> ')';
        _task        = _level >> _operation >> _dataname >> "on" >> _groupby;
        _entry       = _level >> ':' >> _task;
        _schedule    = _entry % eol;
        _start       = skip(blank) [ _schedule ];

        BOOST_SPIRIT_DEBUG_NODES((_start)(_schedule)(_task)(_groupby)(_level)(_operation)(_dataname)(_group_field))
    }
  private:
    struct level_sym : qi::symbols<char, Level> {
        level_sym() { this->add
            ("one", Level::One)
            ("two", Level::Two)
            ("three", Level::Three)
            ("level", Level::LEVEL);
        }
    } _level_sym;

    // lexemes
    qi::rule<It, std::string()> _any_word;
    qi::rule<It, std::string()> _operation, _dataname, _group_field; // TODO
    qi::rule<It, Level()> _level;

    using Skipper = qi::blank_type;
    using Table = decltype(Schedule::tasks);
    using Entry = std::pair<Level, ComputeTask>;

    qi::rule<It, std::set<std::string>(), Skipper> _groupby;
    qi::rule<It, ComputeTask(), Skipper> _task;
    qi::rule<It, Entry(), Skipper> _entry;
    qi::rule<It, Table(), Skipper> _schedule;
    qi::rule<It, Schedule()> _start;
};

我将输入更改为时间表中Level的唯一键,否则实际上只会产生一个条目。

int main() {
    Parser<std::string::const_iterator> const parser;

    for (std::string const input : { R"(ONE: level operation name on(data1, data2, data3)
TWO: level operation name on()
THREE: level operation name on(data1, data2))" })
    {
        auto f = begin(input), l = end(input);
        Schedule s;
        if (parse(f, l, parser, s)) {
            std::cout << "Parsed\n";
            for (auto& [level, task] : s.tasks) {
                std::cout << level << ": " << task << "\n";
            }
        } else {
            std::cout << "Failed\n";
        }

        if (f != l) {
            std::cout << "Remaining unparsed input: " << std::quoted(std::string(f,l)) << "\n";
        }
    }
}

打印

Parsed
One: LEVEL operation name on (data1, data2, data3)
Two: LEVEL operation name on ()
Three: LEVEL operation name on (data1, data2)

此外,还定义了BOOST_SPIRIT_DEBUG

<_start>
  <try>ONE: level operation</try>
  <_schedule>
    <try>ONE: level operation</try>
    <_level>
      <try>ONE: level operation</try>
      <success>: level operation na</success>
      <attributes>[One]</attributes>
    </_level>
    <_task>
      <try> level operation nam</try>
      <_level>
        <try>level operation name</try>
        <success> operation name on(d</success>
        <attributes>[LEVEL]</attributes>
      </_level>
      <_operation>
        <try>operation name on(da</try>
        <success> name on(data1, data</success>
        <attributes>[[o, p, e, r, a, t, i, o, n]]</attributes>
      </_operation>
      <_dataname>
        <try>name on(data1, data2</try>
        <success> on(data1, data2, da</success>
        <attributes>[[n, a, m, e]]</attributes>
      </_dataname>
      <_groupby>
        <try>(data1, data2, data3</try>
        <_group_field>
          <try>data1, data2, data3)</try>
          <success>, data2, data3)\nTWO:</success>
          <attributes>[[d, a, t, a, 1]]</attributes>
        </_group_field>
        <_group_field>
          <try>data2, data3)\nTWO: l</try>
          <success>, data3)\nTWO: level </success>
          <attributes>[[d, a, t, a, 2]]</attributes>
        </_group_field>
        <_group_field>
          <try>data3)\nTWO: level op</try>
          <success>)\nTWO: level operati</success>
          <attributes>[[d, a, t, a, 3]]</attributes>
        </_group_field>
        <success>\nTWO: level operatio</success>
        <attributes>[[[d, a, t, a, 1], [d, a, t, a, 2], [d, a, t, a, 3]]]</attributes>
      </_groupby>
      <success>\nTWO: level operatio</success>
      <attributes>[[LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], [[d, a, t, a, 1], [d, a, t, a, 2], [d, a, t, a, 3]]]]</attributes>
    </_task>
    <_level>
      <try>TWO: level operation</try>
      <success>: level operation na</success>
      <attributes>[Two]</attributes>
    </_level>
    <_task>
      <try> level operation nam</try>
      <_level>
        <try>level operation name</try>
        <success> operation name on()</success>
        <attributes>[LEVEL]</attributes>
      </_level>
      <_operation>
        <try>operation name on()\n</try>
        <success> name on()\nTHREE: le</success>
        <attributes>[[o, p, e, r, a, t, i, o, n]]</attributes>
      </_operation>
      <_dataname>
        <try>name on()\nTHREE: lev</try>
        <success> on()\nTHREE: level o</success>
        <attributes>[[n, a, m, e]]</attributes>
      </_dataname>
      <_groupby>
        <try>()\nTHREE: level oper</try>
        <_group_field>
          <try>)\nTHREE: level opera</try>
          <fail/>
        </_group_field>
        <success>\nTHREE: level operat</success>
        <attributes>[[]]</attributes>
      </_groupby>
      <success>\nTHREE: level operat</success>
      <attributes>[[LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], []]]</attributes>
    </_task>
    <_level>
      <try>THREE: level operati</try>
      <success>: level operation na</success>
      <attributes>[Three]</attributes>
    </_level>
    <_task>
      <try> level operation nam</try>
      <_level>
        <try>level operation name</try>
        <success> operation name on(d</success>
        <attributes>[LEVEL]</attributes>
      </_level>
      <_operation>
        <try>operation name on(da</try>
        <success> name on(data1, data</success>
        <attributes>[[o, p, e, r, a, t, i, o, n]]</attributes>
      </_operation>
      <_dataname>
        <try>name on(data1, data2</try>
        <success> on(data1, data2)</success>
        <attributes>[[n, a, m, e]]</attributes>
      </_dataname>
      <_groupby>
        <try>(data1, data2)</try>
        <_group_field>
          <try>data1, data2)</try>
          <success>, data2)</success>
          <attributes>[[d, a, t, a, 1]]</attributes>
        </_group_field>
        <_group_field>
          <try>data2)</try>
          <success>)</success>
          <attributes>[[d, a, t, a, 2]]</attributes>
        </_group_field>
        <success></success>
        <attributes>[[[d, a, t, a, 1], [d, a, t, a, 2]]]</attributes>
      </_groupby>
      <success></success>
      <attributes>[[LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], [[d, a, t, a, 1], [d, a, t, a, 2]]]]</attributes>
    </_task>
    <success></success>
    <attributes>[[[One, [LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], [[d, a, t, a, 1], [d, a, t, a, 2], [d, a, t, a, 3]]]], [Two, [LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], []]], [Three, [LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], [[d, a, t, a, 1], [d, a, t, a, 2]]]]]]</attributes>
  </_schedule>
  <success></success>
  <attributes>[[[[One, [LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], [[d, a, t, a, 1], [d, a, t, a, 2], [d, a, t, a, 3]]]], [Two, [LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], []]], [Three, [LEVEL, [o, p, e, r, a, t, i, o, n], [n, a, m, e], [[d, a, t, a, 1], [d, a, t, a, 2]]]]]]]</attributes>
</_start>

完整列表

Live On Coliru

//#define BOOST_SPIRIT_DEBUG
#include <boost/spirit/include/qi.hpp>
#include <boost/fusion/adapted.hpp>
#include <vector>
#include <map>
#include <set>
#include <iostream>
#include <iomanip>
#include <experimental/iterator>

enum class Level { One, Two, Three, LEVEL };

struct ComputeTask {
    std::string dataname;
    Level level;
    std::string aggregator;
    std::set<std::string> groupby;
};

struct Schedule {
    std::map<Level, ComputeTask> tasks;
};

//////////////////////
// FOR DEBUG DEMO ONLY
static inline std::ostream& operator<<(std::ostream& os, Level l) {
    switch(l) {
        case Level::One: return os << "One";
        case Level::Two: return os << "Two";
        case Level::Three: return os << "Three";
        case Level::LEVEL: return os << "LEVEL";
    }
    return os << "?";
}

static inline std::ostream& operator<<(std::ostream& os, ComputeTask const& task) {
    os << task.level << ' ' << task.aggregator << ' ' << task.dataname << " on (";
    copy(begin(task.groupby), end(task.groupby), std::experimental::make_ostream_joiner(os, ", "));
    return os << ')';
}

/////////////
// FOR PARSER
BOOST_FUSION_ADAPT_STRUCT(ComputeTask, level, aggregator, dataname, groupby)
BOOST_FUSION_ADAPT_STRUCT(Schedule, tasks)

namespace qi = boost::spirit::qi;

template <typename It>
struct Parser : qi::grammar<It, Schedule()> {
    Parser() : Parser::base_type(_start) {
        using namespace qi;

        _any_word    = lexeme [ +char_("a-zA-Z0-9-_./") ];
        _operation   = _any_word; // TODO
        _group_field = _any_word; // TODO
        _dataname    = _any_word; // TODO

        _level       = no_case [ _level_sym ];
        _groupby     = '(' >> -(_group_field % ',') >> ')';
        _task        = _level >> _operation >> _dataname >> "on" >> _groupby;
        _entry       = _level >> ':' >> _task;
        _schedule    = _entry % eol;
        _start       = skip(blank) [ _schedule ];

        BOOST_SPIRIT_DEBUG_NODES((_start)(_schedule)(_task)(_groupby)(_level)(_operation)(_dataname)(_group_field))
    }
  private:
    struct level_sym : qi::symbols<char, Level> {
        level_sym() { this->add
            ("one", Level::One)
            ("two", Level::Two)
            ("three", Level::Three)
            ("level", Level::LEVEL);
        }
    } _level_sym;

    // lexemes
    qi::rule<It, std::string()> _any_word;
    qi::rule<It, std::string()> _operation, _dataname, _group_field; // TODO
    qi::rule<It, Level()> _level;

    using Skipper = qi::blank_type;
    using Table = decltype(Schedule::tasks);
    using Entry = std::pair<Level, ComputeTask>;

    qi::rule<It, std::set<std::string>(), Skipper> _groupby;
    qi::rule<It, ComputeTask(), Skipper> _task;
    qi::rule<It, Entry(), Skipper> _entry;
    qi::rule<It, Table(), Skipper> _schedule;
    qi::rule<It, Schedule()> _start;
};

int main() {
    Parser<std::string::const_iterator> const parser;

    for (std::string const input : { R"(ONE: level operation name on(data1, data2, data3)
TWO: level operation name on()
THREE: level operation name on(data1, data2))" })
    {
        auto f = begin(input), l = end(input);
        Schedule s;
        if (parse(f, l, parser, s)) {
            std::cout << "Parsed\n";
            for (auto& [level, task] : s.tasks) {
                std::cout << level << ": " << task << "\n";
            }
        } else {
            std::cout << "Failed\n";
        }

        if (f != l) {
            std::cout << "Remaining unparsed input: " << std::quoted(std::string(f,l)) << "\n";
        }
    }
}