Question

我是hazelcast-jet的新手，我的用例是在hazelcastIMDG中检查其值之后，从Kafka源读取并过滤。

即使在创建管道之前，我也正在获取并加载IMDG地图。见下文

 IMap<String, Policy> policyMap =jet.getHazelcastInstance().getMap(POLICY_MAP_NAME);
            Utility.populatePoliciesMap(policyMap);

在buildPipeline方法中将policyMap作为参数传递。

我创建了如下所示的管道

StreamStage<TimestampedEntry<String, Long>> streamStage = pipeline.drawFrom(KafkaSources.kafka(brokerConsumerProperties(), projectionFn, getIngestTopic()))
                .addTimestamps()
                .flatMap(ingestData -> traverseArray(ingestData.getMapRequestParameterTree().toArray(new String[ingestData.getMapRequestParameterTree().size()])))
                .filter(hash -> policyMap.get(hash)!=null)
                .window(sliding(MINUTES.toMillis(1), SECONDS.toMillis(10)))
                .groupingKey(wholeItem())
                .aggregate(counting())
               .map((TimestampedEntry<String, Long> e) -> entry(e.getKey(), createBlacklistObjectEvent(Utility.fetchPolicy(e.getKey()), e.getTimestamp(), e.getValue())));
        timestampedEntryStreamStage.drainTo(Sinks.map(BL_MAP_NAME));

但与此同时，我正在获得例外

线程“主”中的异常java.lang.IllegalArgumentException： “ filterFn”必须在以下位置可序列化 com.hazelcast.jet.impl.util.Util.checkSerializable（Util.java:301）在 com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.attachFilter（ComputeStageImplBase.java:129）在 com.hazelcast.jet.impl.pipeline.StreamStageImpl.filter（StreamStageImpl.java:71）在 com.visa.rls.handler.HazelcastJetIngetstResultHandler.buildPipeline（HazelcastJetIngetstResultHandler.java:120）在 com.visa.rls.handler.HazelcastJetIngetstResultHandler.run（HazelcastJetIngetstResultHandler.java:84）在 com.visa.rls.handler.HazelcastJetIngetstResultHandler.main（HazelcastJetIngetstResultHandler.java:58）引起原因：java.io.NotSerializableException： com.hazelcast.map.impl.proxy.MapProxyImpl位于 java.io.ObjectOutputStream.writeObject0（ObjectOutputStream.java:1184）在 java.io.ObjectOutputStream.writeArray（ObjectOutputStream.java:1378）在 java.io.ObjectOutputStream.writeObject0（ObjectOutputStream.java:1174）在 java.io.ObjectOutputStream.defaultWriteFields（ObjectOutputStream.java:1548）在 java.io.ObjectOutputStream.writeSerialData（ObjectOutputStream.java:1509）在 java.io.ObjectOutputStream.writeOrdinaryObject（ObjectOutputStream.java:1432）在 java.io.ObjectOutputStream.writeObject0（ObjectOutputStream.java:1178）在 java.io.ObjectOutputStream.writeObject（ObjectOutputStream.java:348）在com.hazelcast.jet.impl.util.Util.checkSerializable（Util.java:299） ...还有5个

Answer 1

您在过滤器函数内使用const date = message.author.createdAt; const newDate = date.toLocaleDateString();，但是policyMap不可序列化。它被lambda表达式捕获。您必须在每个远程成员上获得一个IMap实例，您可以为其使用IMap而不是filterUsingContext：

filter

获取和异常NotSerializableException：com.hazelcast.map.impl.proxy.MapProxyImpl

1 个答案: