Question

我正在使用风暴0.10 我有一个列表，我迭代，然后发出元组

List<List<Objects>>

我想发出一个单独的元组，而不是发出多个元组，这个元组是一个嵌套的List，就像这样，

我的主要问题是序列化。阅读风暴文档表明，Java序列化是昂贵的，而Storm使用Kryo序列化。另外，常规的Java POJO类不应该在没有kryo的情况下通过线路发送。所以我想发送List<List<Object>> valueList = new ArrayList<List<Object>>(); for (CustomObject o: List<CustomObject>) { v.add(new ArrayList<Object>{ { add(o.getFirst); add( o.getName); add(o.getAddress); } }); } collector.emit(STREAM_NAME, new Values(valueList)); 如下，

DateSerial(Left([DateColumn],4),Mid([DateColumn],5,2),Right([DateColumn],2))

所以问题是 - 这是用风暴使用Kryo完成的吗？

Answer 1

您只需发出List，

即可

collector.emit(new Values(listCustomObject));

然后当您在bolt中阅读时，

List<CustomObject> listCustomObject = (List<CustomObject>) tuple.getValue(0);

希望它有所帮助。

编辑：确实我忘了提到对象必须是Serializable！为了处理清单，我做了类似的事情：

import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;


public class CustomObjectList implements Serializable {

    private static final long serialVersionUID = 6877020084704724252L;

    public class CustomObject {
        ...
    }

    private final List<CustomObject> list = new ArrayList<>();

    ...
}

EDIT2 ：由于我做了太多事情，我通过修改storm starter ExaclamationTopology <中的ExclamationBolt来完成/测试了一个快速的玩具示例/ p>

看起来像这样。

package storm.starter;

import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.base.BaseRichBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;


public class ExclamationBolt extends BaseRichBolt{

    public static Logger LOG = LoggerFactory.getLogger(ExclamationBolt.class);    


    public class CustomObject {
        private String word;

        public CustomObject(String word) {
            setWord(word);
        }

        public String getWord() {
            return word;
        }

        public void setWord(String word) {
            this.word = word;
        }

    }
    public class CustomObjectList implements Serializable {

        private static final long serialVersionUID = 6877020084704724252L;

        private final List<CustomObject> list = new ArrayList<>();

        public CustomObjectList() {
        }

        public List<CustomObject> getObjects() {
            return list;
        }

        public void addObject(CustomObject o) {
            list.add(o);
        }
    }

    OutputCollector _collector;

    public ExclamationBolt() {
    }

    @Override
    public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
        _collector = collector;
    }

    @Override
    public void execute(Tuple tuple) {
        String id = tuple.getSourceComponent();
        String word;
        if ("word".equals(id)) {
            word = tuple.getString(0);
        } else {
            CustomObjectList list = (CustomObjectList) tuple.getValue(0);
            word = list.getObjects().get(0).getWord();
        }
        word = word + "!!!";
        LOG.debug(word);
        CustomObjectList list = new CustomObjectList();
        list.addObject(new CustomObject(word));
        _collector.emit(tuple, new Values(list));
        _collector.ack(tuple);
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("word"));
    }


}

希望它有所帮助。

Answer 2

使用KryoSerialization。

实施CustomObjectSerializer：

class CustomObjectSerializer extends Serializer<CustomObject> {  
  @Override
  public void write(final Kryo kryo,
                    final Output output,
                    final CustomObject cObject) {
    // do serialization
  }

  @Override
  public CustomObject read(final Kryo kryo,
                           final Input input,
                           final Class<CustomObject> aClass) {
    // do deserialization
  }
}

将CustomObjectSerializer注册到Storm配置： conf.registerSerialization(CustomObject.class,CustomObjectSerializer.class);

在风暴中，如何发出原始列表

2 个答案: