Question

我有：

val sparkBuilder: SparkSession.Builder = SparkSession
  .builder
  .appName("CreateModelDataPreparation")
  .config("spark.master", "local")
implicit val spark: SparkSession = sparkBuilder.getOrCreate()

但是，当我运行程序时，我仍然得到：

org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)

如其他posts中所建议，在Main方法中设置了SparkSession。那些似乎没有解决问题。

这与建议的重复项有所不同，因为我都尝试过：

  def main(argv: Array[String]): Unit = {
    import DeweyConfigs.implicits.da3wConfig

    val commandlineArgs: DeweyReaderArgs = processCommandLineArgs(argv)

    val sparkBuilder: SparkSession.Builder = SparkSession
      .builder
      .appName("CreateModelDataPreparation")
      .master("local")
    implicit val spark: SparkSession = sparkBuilder.config("spark.master", "local").getOrCreate()
    import spark.implicits._
    ...

和

  def main(argv: Array[String]): Unit = {
    import DeweyConfigs.implicits.da3wConfig

    val commandlineArgs: DeweyReaderArgs = processCommandLineArgs(argv)

    val sparkBuilder: SparkSession.Builder = SparkSession
      .builder
      .appName("CreateModelDataPreparation")
      .config("master", "local")
    implicit val spark: SparkSession = sparkBuilder.config("spark.master", "local").getOrCreate()
    import spark.implicits._
    ...

Answer 1

尝试在构建器上添加def encode_string(cat_features): # Load the previously saved encoder with open('./'+cat_features+'_encoder.pickle', 'rb') as file: enc = pickle.load(file) # No fitting, only transform enc_cat_features = enc.transform(cat_features) # Same for OHE with open('./'+cat_features+'_ohe.pickle', 'rb') as file: enc = pickle.load(file) return encoded.transform(enc_cat_features.reshape(-1,1)).toarray()，而不是您提供的config参数。

我本以为他们做了同样的事情，但是我很确定后者能起作用。

SparkSession.Builder失败，显示“必须在配置中设置主URL”：“ spark.master”设置为“ local”

1 个答案: