Question

我试图将扭曲的Protocol的力量与声明性二进制数据解析器construct的延展性混淆。

到目前为止，我的MessageReceiver协议以下列方式累积来自tcp通道的数据：

def rawDataReceived(self, data):
    '''
    This method bufferizes the data coming from the TCP channel in the following way:
        - Initially, discard the stream until a reserved character is detected
        - add data to the buffer up to the expected message length unless the reserved character is met again. In that case discard the message and start again
        - if the expected message length is reached, attempt to parse the message and clear the buffer
    '''
    if self._buffer:
        index = data.find(self.reserved_character)

        if index > -1:
            if len(self._buffer) + index >= self._fixed_size:
                self.on_message(self._buffer + data[:data.index(self._reserved_character)])

            self._buffer = b''
            data = data[data.index(self.reserved_character):]
            [self.on_message(chunks[:self._fixed_size]) for chunks in  [self.reserved_character + msg for msg in data.split(self._reserved_character) if msg]]

        elif len(self._buffer) + len(data) < self._expected_size:
            self._buffer = self._buffer + data
        else:
            self._buffer = b''
    else:  
        try:
            data = data[data.index(self._reserved_character):]
            [self.on_message(chunks[:self._fixed_size]) for chunks in  [self._reserved_character + msg for msg in data.split(self._reserved_character) if msg]]
        except Exception, exc:
            log.msg("Warning: Maybe there is no delimiter {delim} for the new message. Error: {err}".format(delim=self._reserved_character, err=str(exc)))

现在我需要改进协议，以考虑到消息可能带有或不带有可选字段的事实（因此不再有固定的消息长度）。我以下列方式使用construct对消息解析器建模（有意义的一部分）：

def on_message(self, msg):
    return Struct(HEADER,
        Bytes(HEADER_RAW, 3),
        BitStruct(OPTIONAL_HEADER_STRUCT, 
            Nibble(APPLICATION_SELECTOR),
            Flag(OPTIONAL_HEADER_FLAG), 
            Padding(3)
        ),
        If(lambda ctx: ctx.optional_header_struct[OPTIONAL_HEADER_FLAG],
            Embed(Struct(None,
                   Byte(BATTERY_CHARGE),
                   Bytes(OPTIONAL_HEADER, 3)
                   )
            )
        )
    ).parse(msg)

所以现在我需要更改缓冲逻辑以将正确的块大小传递给Struct。我想避免在Struct方法中调整要传递给rawDataReceived的数据，考虑到construct对象中已知可能的消息候选者的规则

有没有办法将缓冲逻辑推送到construct对象？

修改

通过简单地使用Macro和Adapter s，我能够部分实现将缓冲逻辑推入内部的目标：

MY_PROTOCOL = Struct("whatever",
    Anchor("begin"),
    RepeatUntil(lambda obj, ctx:obj==RESERVED_CHAR, Field("garbage", 1)),
    NoneOf(Embed(HEADER_SECTION), [RESERVED_CHAR]),
    Anchor("end"),
    Value("size", lambda ctx:ctx.end - ctx.begin)
)

这极大地简化了调用者代码（由于Glyph的建议而不再在rawDataReceived中）：

def dataReceived(self, data):
    log.msg('Received data: {}'.format(bytes_to_hex(data)))
    self._buffer += data
    try:
        container = My_PROTOCOL.parse(self._buffer)
        self._buffer = self._buffer[container.size:]
        d, self.d = self.d, self._create_new_transmission_deferred()
        d.callback(container)
    except ValidationError, err:
        self._cb_error("A validation error occurred. Discarding the rest of the message. {}".format(err))
        self._buffer = b''    
    except FieldError, err:     #Incomplete message. We simply keep on buffering and retry
        if len(self._buffer) >= MyMessageReceiver.MAX_GARBAGE_SIZE:
            self._cb_error("Buffer overflown. No delimiter found in the stream")

不幸的是，这个解决方案仅部分涵盖了这些要求，因为我找不到让construct告诉我产生错误的流的索引的方法，因此我不得不放弃整个缓冲区，这是不理想。

Answer 1

要获取发生错误的流位置，您需要使用Anchor并编写自己的NoneOf版本。假设HEADER_SECTION是另一个Construct，请替换NoneOf，如下所示：

SpecialNoneOf(Struct('example', Anchor('position'), HEADER_SECTION), [RESERVED_CHAR]))

SpecialNoneOf需要从Adapter继承，并将NoneOf中的 init 和_validate与来自Validator的_encode和_decode结合起来。在_decode中，替换

raise ValidationError("invalid object", obj)

与

raise ValidationError("invalid object", obj.header_section + " at " + obj.position)

将header_section替换为HEADER_SECTION构造的名称。您将不得不更改生成的容器的结构或找出使用Embed的不同方法来使此方法有效。

用于协议解析的python构造

修改

1 个答案: