我正在广播一个hashmap，并且能够在执行器节点上检索它的值。但是当我尝试在转换函数的call（）方法中unpersist（）这个广播的hashmap时，我得到以下异常：
org.apache.spark.SparkException: Error sending message [message = RemoveBroadcast(6,false)]
Caused by: java.io.NotSerializableException: scala.concurrent.impl.Promise$DefaultPromise
Serialization stack:
- object not serializable (class: scala.concurrent.impl.Promise$DefaultPromise, value: scala.concurrent.impl.Promise$DefaultPromise@453ec620)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.rpc.netty.NettyRpcEnv.serialize(NettyRpcEnv.scala:255)
at org.apache.spark.rpc.netty.RemoteNettyRpcCallContext.send(NettyRpcCallContext.scala:64)
at org.apache.spark.rpc.netty.NettyRpcCallContext.reply(NettyRpcCallContext.scala:32)
at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:99)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:104)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
代码如下：
public class SprinterKafkaStream实现Serializable {
private static final long serialVersionUID = 172210347397861748L;
/* KAFKA Parameters for streaming interface */
Map<String, String> inputKafkaParams = null;
Broadcast<HashMap<String, Double>> hamperrateLookupBroadcast = null;
static JavaStreamingContext jsc ;
private Config appConfig ;
private static Broadcast<HashMap<String, KafkaProducerSer>> appWiseKafkaProducerSerBroadcast = null;
private ProcessRDD processor;
changeRefDataRequest.foreachRDD(new VoidFunction<JavaRDD<String>>() {
private static final long serialVersionUID = 1L;
private AWPEvent event;
@Override
public void call(JavaRDD<String> rdd) throws Exception
{
/*Kafka producer instance to connect to response topic*/
final KafkaProducerSer producer = appWiseKafkaProducerSerBroadcast.getValue().get(appConfig.getAppId());
SprinterLogger.infoLog_and_Console("Producer " + producer.toString());
rdd.foreachPartition(new VoidFunction<Iterator<String>>() { //exception issue
private static final long serialVersionUID = 1L;
@Override
public void call(Iterator<String> changeReqItr) throws Exception
{
if(changeReqItr != null)
{
while(changeReqItr.hasNext())
{
try
{
String request = changeReqItr.next();
SprinterLogger.infoLog_and_Console("****Request****" + request);
if(request !=null)
{
event = AWPXmlParser.getReferenceDataRefreshInfo(request);
SprinterLogger.infoLog_and_Console("****AWPevent****" + event);
//check for application id & instance name
//if ((event.getAppID().equals(appConfig.getAppId())) && (event.getProcessID().startsWith(appConfig.getInstanceName())))
if ((event.getAppID().equals(appConfig.getAppId())) && (event.getProcessID().startsWith(appConfig.getRegion())))
{
Process process = event.getProcess();
SprinterLogger.infoLog_and_Console("Process created" + process.isInput());
if (process.isInput())
{
List<Attribute> attributes = process.getAttributes();
if(attributes.isEmpty())
{
SprinterLogger.infoLog_and_Console("Attributes List is empty");
}
for (Attribute attribute : attributes)
{
SprinterLogger.infoLog_and_Console("Attribute info: " + attribute.getName().toString() + " " + attribute.getType().toString() + " " + attribute.getValue().toString());
if (CommonPropertiesConstants.HAMPERRATE_CHANGE_ATTR_NAME.equals(attribute.getName()))
{
SprinterLogger.infoLog_and_Console("Inside hamper rate update");
Set set = hamperrateLookupBroadcast.value().entrySet();
Iterator iterator = set.iterator();
while(iterator.hasNext())
{
Map.Entry mentry = (Map.Entry)iterator.next();
SprinterLogger.infoLog_and_Console("Hamperrate key is: "+ mentry.getKey().toString() + " and Value is: ");
SprinterLogger.infoLog_and_Console(mentry.getValue().toString());
}
hamperrateLookupBroadcast.unpersist(); //exception hamperrateLookupBroadcast
SprinterLogger.infoLog_and_Console("UNPERSISTED");
HashMap<String, Double> hamperrateLookupnMaps=HamperRateLookups.loadHamperRateData(attribute.getValue());
hamperrateLookupBroadcast=jsc.sparkContext().broadcast(hamperrateLookupnMaps);
processor.updateHamperRateLookupBroadcast(hamperrateLookupBroadcast);
SprinterLogger.infoLog_and_Console("\n Refreshing Hamper rate file for application id:"+event.getAppID()+", instance:"+event.getProcessID()+" : DONE\n");
}
}
在这里，我试图取消播放广播的参考数据，然后用更新的数据重播。
有人可以帮忙吗？