Question

我正在使用apache toree（来自github的版本）。当我试图对postgresql表执行查询时，我得到间歇性的scala编译器错误（当我运行两次相同的单元格时，错误消失了，代码运行正常）。

我正在寻找有关如何调试这些错误的建议。错误看起来很奇怪（它们出现在stdout上的笔记本上）。

error: missing or invalid dependency detected while loading class file 'QualifiedTableName.class'.
Could not access type AnyRef in package scala,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'QualifiedTableName.class' was compiled against an incompatible version of scala.
error: missing or invalid dependency detected while loading class file 'FunctionIdentifier.class'.
Could not access type AnyRef in package scala,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'FunctionIdentifier.class' was compiled against an incompatible version of scala.
error: missing or invalid dependency detected while loading class file 'DefinedByConstructorParams.class'.
...

代码很简单：从postgres表中提取数据集：

%AddDeps org.postgresql postgresql 42.1.4 --transitive
val props = new java.util.Properties();
props.setProperty("driver","org.postgresql.Driver");
val df = spark.read.jdbc(url = "jdbc:postgresql://postgresql/database?user=user&password=password", 
                table = "table", predicates = Array("1=1"), connectionProperties = props)
df.show()

我检查了显而易见的（toree和apache spark使用scala 2.11.8，我用APACHE_SPARK_VERSION = 2.2.0构建apache toree，这与我下载的spark相同）

作为参考，这是我用来设置toree和spark的Dockerfile的一部分：

RUN wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz && tar -zxf spark-2.2.0-bin-hadoop2.7.tgz && chmod -R og+rw /opt/spark-2.2.0-bin-hadoop2.7 && chown -R a1414.a1414 /opt/spark-2.2.0-bin-hadoop2.7
RUN (curl https://bintray.com/sbt/rpm/rpm > /etc/yum.repos.d/bintray-sbt-rpm.repo)
RUN yum -y install --nogpgcheck sbt
RUN (unset http_proxy; unset https_proxy; yum -y install --nogpgcheck java-1.8.0-openjdk-devel.i686)
RUN (git clone https://github.com/apache/incubator-toree && cd incubator-toree && make clean release APACHE_SPARK_VERSION=2.2.0 ; exit 0)
RUN (. /opt/rh/rh-python35/enable; cd /opt/incubator-toree/dist/toree-pip ;python setup.py install)
RUN (. /opt/rh/rh-python35/enable; jupyter toree install --spark_home=/opt/spark-2.2.0-bin-hadoop2.7 --interpreters=Scala)

Answer 1

我有一个类似的问题，但是它似乎可以通过仅重新评估Jupyter笔记本中的单元或重新启动内核然后重新评估单元来解决自身问题。烦人。

Answer 2

正如cchantep的评论所说，你可能使用的Scala版本与用于构建Spark的版本不同。

最简单的解决方案是检查Spark使用哪一个，然后切换到这个，例如在Mac上：

brew switch scala 2.11.8

无法访问Scala包中的AnyRef

2 个答案: