Scala API的本地环境错误

时间:2018-10-16 14:09:51

标签: apache-flink

当我使用https://ci.apache.org/projects/flink/flink-docs-release-1.6/quickstart/scala_api_quickstart.html推荐的sbt模板创建项目时

sbt new tillrohrmann/flink-project.g8

当我尝试向scala API方法(过滤器,地图等)提供任何匿名函数时,出现以下错误。 ordinals.print()工作正常。我不是flink scala repl中的不是,这只是正常的intellij scala repl。根据{{​​3}},这应该可以,并且如果我要在集成测试中使用它(和StreamExecutionEnvironment),则需要工作。我知道CollectionEnvironment,它工作正常,但是使用StreamExecutionEnvironment并没有帮助。我已经在Windows和Linux的2个intellij环境中进行了尝试,都使用scala 2.11.12,flink 1.6.1和Java 8。

当然,执行环境文档是基于Java的-我需要使scala API正常运行的依赖项或导入项吗?

scala> import org.apache.flink.api.scala.ExecutionEnvironment

val benv = ExecutionEnvironment.getExecutionEnvironment
import org.apache.flink.api.scala.ExecutionEnvironment

scala> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
benv: org.apache.flink.api.scala.ExecutionEnvironment = org.apache.flink.api.scala.ExecutionEnvironment@646427f7

scala> import org.apache.flink.api.scala._
import org.apache.flink.api.scala._

scala> val ordinals = benv.fromCollection(65 until 65 + 26)
ordinals: org.apache.flink.api.scala.DataSet[Int] = org.apache.flink.api.scala.DataSet@40017e98

scala> val capitals = ordinals.map(_.toChar)
capitals: org.apache.flink.api.scala.DataSet[Char] = org.apache.flink.api.scala.DataSet@7323dce7

scala> capitals.print()
org.apache.flink.runtime.client.JobExecutionException: java.lang.RuntimeException: The initialization of the DataSource's outputs caused an error: Could not read the user code wrapper: $anonfun$1
  at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:623)
  at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:235)
  at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
  at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:816)
  at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
  at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
  at org.apache.flink.api.scala.DataSet.print(DataSet.scala:1726)
  ... 28 elided
Caused by: java.lang.RuntimeException: The initialization of the DataSource's outputs caused an error: Could not read the user code wrapper: $anonfun$1
  at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:106)
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
  at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException: Could not read the user code wrapper: $anonfun$1
  at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:290)
  at org.apache.flink.runtime.operators.BatchTask.instantiateUserCode(BatchTask.java:1445)
  at org.apache.flink.runtime.operators.chaining.ChainedMapDriver.setup(ChainedMapDriver.java:39)
  at org.apache.flink.runtime.operators.chaining.ChainedDriver.setup(ChainedDriver.java:90)
  at org.apache.flink.runtime.operators.BatchTask.initOutputs(BatchTask.java:1312)
  at org.apache.flink.runtime.operators.DataSourceTask.initOutputs(DataSourceTask.java:312)
  at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:103)
  ... 2 more
Caused by: java.lang.ClassNotFoundException: $anonfun$1
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:348)
  at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:76)
  at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)
  at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
  at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
  at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
  at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:502)
  at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:489)
  at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:477)
  at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:438)
  at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:288)
  ... 8 more

scala> 

更新:从scalatest内部来看,ExecutionEnvironment.getExecutionEnvironment和StreamExecutionEnvironment.getExecutionEnvironment都可以正常工作,因此我不必担心。

0 个答案:

没有答案