Question

我正在尝试从spark连接到SQL数据库，并且使用了以下命令：

scala> import org.apache.spark.sql.SQLContext                                                                                                                                                                      
import org.apache.spark.sql.SQLContext

scala> val sqlcontext = new org.apache.spark.sql.SQLContext(sc)                                                                                                                                                    
warning: there was one deprecation warning; re-run with -deprecation for details
sqlcontext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@2bf4fa1

scala> val dataframe_mysql = sqlcontext.read.format("jdbc").option("url", "jdbc:sqlserver:192.168.103.64/DRE").option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").option("dbtable", "NCentralAlerts")
.option("user", "sqoop").option("password", "hadoop").load()
java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver
  at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:45)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$6.apply(JDBCOptions.scala:79)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$6.apply(JDBCOptions.scala:79)
  at scala.Option.foreach(Option.scala:257)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:79)
  at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:35)
  at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:34)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
  at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
  ... 49 elided

我看到Spark正在寻找SQL驱动程序。我要将这个SQL驱动程序放在哪个目录中？

Answer 1

我从日志中看到您正在尝试使用spark shell运行它。假设您手头有罐子。从spark-shell开始，添加以下内容

spark-shell --jars /path/to/driver.jar

这样，它将被添加到您的类路径中，您将能够使用驱动程序。

希望这会有所帮助

从Spark

1 个答案: