Question

这是我的代码：

class FNNode(val name: String)

case class Ingredient(override val name: String, category: String) extends FNNode(name)


val ingredients: RDD[(VertexId, FNNode)] = 
sc.textFile(PATH+"ingr_info.tsv").
      filter(! _.startsWith("#")).
      map(line => line.split('\t')).
      map(x => (x(0).toInt ,Ingredient(x(1), x(2))))

并且在定义这些变量时没有错误。但是，在尝试执行时：

ingredients.take(1)

我得到了

org.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: java.io.InvalidClassException: $iwC$$iwC$Ingredient; no valid constructor
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)

根据答案here，这似乎与序列化问题有关。但是，如果确实是序列化问题，我不知道如何解决这个问题。

我按照他们的方式跟随this书中的代码，所以我认为这至少应该在某个时候起作用？

Answer 1

这解决了我的问题：

<div id="element-box" ondrop="drop(event)" ondragover="allowDrop(event)"><img class="drag-img" id="drag1" src="img/one.jpg" draggable="true"
ondragstart="drag(event)" width="100%" height="100%"/></div>
<div id="element-box" ondrop="drop(event)" ondragover="allowDrop(event)"><img class="drag-img" id="drag2" src="img/two.jpg" draggable="true"
ondragstart="drag(event)" width="100%" height="100%"/></div>
<div id="element-box" ondrop="drop(event)" ondragover="allowDrop(event)"><img class="drag-img" id="drag3" src="img/three.jpg" draggable="true"
ondragstart="drag(event)" width="100%" height="100%"/></div>
<div id="element-box" ondrop="drop(event)" ondragover="allowDrop(event)"><img class="drag-img" id="drag4" src="img/four.jpg" draggable="true"
ondragstart="drag(event)" width="100%" height="100%"/></div>
<div id="element-box" ondrop="drop(event)" ondragover="allowDrop(event)"><img class="drag-img" id="drag5" src="img/five.jpg" draggable="true"
ondragstart="drag(event)" width="100%" height="100%"/></div>

<div id="div1" ondrop="drop(event)" ondragover="allowDrop(event)"></div>

spark

1 个答案: