Question

我试图在spark-shell中使用spark SQL读取JSON文件，但出现错误。我是Spark SQL的新手

我如下创建sparksession

import org.apache.spark.sql.SparkSession

 val spark = SparkSession.builder()
  .master("local")
  .appName("Spark SQL Practice")
  .getOrCreate()

import spark.implicits._

之后，我尝试读取JSON文件

val df = spark.read.json("D:/Lectures/11_Nov_day16_spark.day5/people.json")

> scala val df = spark.read.json("D:/Lectures/11_Nov_day16_spark.day5/people.json")

 19/08/06 18:54:53 WARN General: Plugin (Bundle) "org.datanucleus" is
 already registered. Ensure you dont have multiple JAR versions of the
 same plugin in the classpath. The URL
 "file:/C:/spark/jars/datanucleus-core-3.2.10.jar" is already
 registered, and you are trying to register an identical plugin located
 at URL "file:/C:/spark/bin/../jars/datanucleus-core-3.2.10.jar."

 19/08/06 18:54:53 WARN General: Plugin (Bundle)
 "org.datanucleus.store.rdbms" is already registered. Ensure you dont
 have multiple JAR versions of the same plugin in the classpath. The
 URL "file:/C:/spark/bin/../jars/datanucleus-rdbms-3.2.9.jar" is
 already registered, and you are trying to register an identical plugin
 located at URL "file:/C:/spark/jars/datanucleus-rdbms-3.2.9.jar."

 19/08/06 18:54:53 WARN General: Plugin (Bundle)
 "org.datanucleus.api.jdo" is already registered. Ensure you dont have
 multiple JAR versions of the same plugin in the classpath. The URL
 "file:/C:/spark/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already
 registered, and you are trying to register an identical plugin located
 at URL "file:/C:/spark/jars/datanucleus-api-jdo-3.2.6.jar." 

 19/08/06 18:55:00 WARN ObjectStore: Failed to get database default, returning
 NoSuchObjectException

Answer 1

请查看项目的依赖关系树，并删除datanucleus的所有重复依赖项，或将相同的datanucleus-core依赖项添加到plugin-config。

谢谢

Spark SQL error..org.datanucleus”已注册

1 个答案: