Question

我在Apache Flink项目中遇到以下情况。

包含不同对象的3个流，例如

人 - ＆gt; String id，String firstName，String lastName（即101，John，Doe）

PersonDetail - ＆gt; String id，String address，String city，String phoneNumber，long personId（即99，Stefansplatz 1，+ 43066012345678,101）

PersonAddDetail - ＆gt; String id，String AddDetailType，object AddDetailValue，long personId（即77,1，Hansi或78,2,1234或80,3，true）

我想将这些流中的对象（不确定这是否是正确的措辞）聚合到我放入新流的新对象中。聚合应该基于Person id，作为额外的catch我需要使用特定的AddDetailType过滤掉PersonAddDetail（假设我只对类型为1和2的对象感兴趣）。

聚合对象应该看起来像

PersonReport - ＆gt; long id，String firstName，String lastName，String address，String city，String phoneNumber，ArrayList＆lt; PersonAddDetail＆gt;细节

现在的问题是，如果这是可能的，如果是，我该如何实现它。每一个欢迎的投入。

Answer 1

您的问题听起来像是join操作。你可以这样做：

personDataStream.join(personDetailDataStream).where(new KeySelector<Person, Long>() {
    ...
}).equalTo(new KeySelector<PersonDetail, Long>() {
    ...
}).window(TumblingEventTimeWindows.of(Time.seconds(2))).apply(new JoinFunction<Person, PersonDetail, PersonWithDetail>() {
   ...
});

请注意，通常在无界（无限）集合上无法进行连接操作，因此需要将其绑定到窗口中。

Answer 2

感谢@Jeremy Grand评论我自己提出了一个解决方案，我想分享我的想法和代码。我介绍了一个名为PersonContainer的新类

public class PersonContainer {

private String id;

private Person person;
private PersonDetail personDetail;
private List<PersonAddDetail> personAddDetailList = new ArrayList<>();

public PersonContainer(Person person) {
  this.id = person.getID();
  this.person = person;
}



public PersonContainer(PersonDetail personDetail) {
    this.id = personDetail.getOTTRID();
    this.personDetail = personDetail;
  }

  public PersonContainer(PersonAddDetail personAddDetail) {
    this.id = personAddDetail.getOTTRID();
    this.timeStamp = ttrDetailAddEvent.getDATECREATED();
    this.personAddDetailList.add(personAddDetail);
  }

  public PersonContainer merge(PersonContainer other) {
    if (other.person != null) {
      this.person = other.person;
      return this;
    }
    if (other.personDetail != null) {
      this.personDetail = other.personDetail;
      return this;
    }
    if (other.personAddDetailList.size() > 0) {
      this.personAddDetailList.addAll(other.personAddDetailList);
      return this;
    }
    return null;
  }

  public String getId() {
    return id;
  }

  public Person getPerson() {
    return person;
  }

  public PersonDetail getPersonDetail() {
    return personDetail;
  }

  public List<PersonAddDetail> getPersonAddDetailList() {
    return PersonAddDetailList;
  }

  public boolean isComplete() {
    return person != null && personDetail != null && personAddDetailList.size() > 1;
  }
}

这是重要的一部分，因为我要先将三个输入流的对象映射到这个公共对象，然后再将这些流联合起来。

所以这就是我所做的，我描述了评论中的单个步骤。简而言之，我将三个输入流映射到新引入的容器的新流。然后我在三个流上进行联合并使用迭代模式来键入这些对象并使用我的自定义合并方法合并它们。最后，我定义了一个自定义完整方法来区分完全合并的容器，这些容器最终映射到输出，而不是已经反馈到合并过程的容器。

//Filter PersonAddDetail to have just the types needed
DataStream<PersonContainer> filteredPersonAddDetail = unfilteredPersonAddDetail.filter(new FilterFunction<OboTtrDetailAddEvent>() {
      @Override
      public boolean filter(PersonAddDetail personAddDetail) throws Exception {
        return personAddDetail.getAddDetailType().matches("1|2");
      }
    });

//map Person stream to common object
DataStream<PersonContainer> mappedPersonStream = personInputStream.map(new MapFunction<Person, PersonContainer>() {
  @Override
  public PersonContainer map(Person Person) throws Exception {
    return new PersonContainer(Person);
  }
});

//map PersonDetail stream to common object
DataStream<PersonContainer> mappedPersonDetailStream = personDetailInputStream.map(new MapFunction<PersonDetail, PersonContainer>() {
  @Override
  public PersonContainer map(PersonDetail PersonDetail) throws Exception {
    return new PersonContainer(PersonDetail);
  }
});

//map PersonAddDetail stream to common object
DataStream<PersonContainer> mappedPersonAddDetailStream = filteredPersonAddDetail.map(new MapFunction<PersonAddDetail, PersonContainer>() {
  @Override
  public PersonContainer map(PersonAddDetail PersonAddDetail) throws Exception {
    return new PersonContainer(PersonAddDetail);
  }
});

//union the three input streams to one single stream
DataStream<PersonContainer> combinedInput = mappedPersonStream.union(mappedPersonDetailStream, mappedPersonAddDetailStream);

// Iteration pattern is in place here and I'm going to recursively try to merge corresponding objects togehter
IterativeStream<PersonContainer> iteration = combinedInput.iterate();

// Group objects by there shared ID and then use reduce to merge them
DataStream<PersonContainer> iterationBody = iteration.keyBy(new KeySelector<PersonContainer, String>() {
  @Override
  public String getKey(PersonContainer personContainer) throws Exception {
    return personContainer.getId();
  }
})
    .reduce(new ReduceFunction<PersonContainer>() {
      @Override
      public PersonContainer reduce(PersonContainer personContainer, PersonContainer other) throws Exception {
        return personContainer.merge(other);
      }
    });

// use the containers complete method to check whether the merge is finished or we need to wait for further objects in the stream   
DataStream<PersonContainer> containersNotCompleteYet = iterationBody.filter(new FilterFunction<PersonContainer>() {
  @Override
  public boolean filter(PersonContainer PersonContainer) throws Exception {
    return !personContainer.isComplete();
  }
});

// partially merged or not merged at all containers are put back on the stream
iteration.closeWith(containersNotCompleteYet);

// fully merged containers are processed further
DataStream<PersonContainer> completeContainers = iterationBody.filter(new FilterFunction<PersonContainer>() {
  @Override
  public boolean filter(PersonContainer PersonContainer) throws Exception {
    return personContainer.isComplete();
  }
});

// finally the container is mapped to the correct output object
DataStream<PersonReport> personReport = completeContainers.map(new MapFunction<PersonContainer, PersonReport>() {
  @Override
  public PersonReport map(PersonContainer personContainer) throws Exception {

    // map personContainer to final PersonReport

    return personContainer;
  }
});

这种方法对我有用，好处是我可以处理在流的后期到达的对象（假设在其他对象之后几分钟就会出现PersonAddDetail）并且我不需要定义某种类型的窗口。无论如何，感谢您的输入

如何将3个流中的数据聚合到一个组合对象？

2 个答案: