Question

我正在使用Spark与Scala和Play2.4。首先，我看到了这个帖子： Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects

我很好，如果我们将一个方法传递给Spark，它会尝试序列化整个类。

我的代码，它会更清楚：

class GPSSparkServiceImpl @Inject() (val stepService: StepDbService, val coordinateService: CoordinateService) extends GPSSparkService with Serializable{

  /**
   * Spark config
   * Set Master node and AppName
   */
  val conf = new SparkConf().setAppName("Editus GPS").setMaster("local[2]")

  /**
   * Initialize Spark Context
   */
  val sc = new SparkContext(conf)

  override def execute() = {
    val logData = sc.textFile("file://C://work/csv/gps.csv").cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    println("Lines with a: %s".format(numAs))
  }

  override def generateUserToStep(): Unit = {
    val futureSteps = stepService.findAll()
    futureSteps onSuccess{
      case steps =>
        val data = sc.textFile("file://C://work/csv/gps.csv").cache()
        val result = data.flatMap(line => steps.map(step => (line, step))).filter { tuple =>
          coordinateService.checkProximity(
            coordinateService.coordinateToDistanceInMeters(
              tuple._1.split(";")(1).toDouble, tuple._1.split(";")(2).toDouble, tuple._2.gpsCoordinate.latitude, tuple._2.gpsCoordinate.longitude
            ), tuple._2
          )
        }.count()
        println("result: " + result + " for " + steps.length + " steps")
    }
  }
}

正如您所看到的，我在spark中使用了两种方法：来自checkProximity的{{1}}和CoordinateToDistanceInMeters，它们将被注入CoordinateService，并且此类是可序列化的。类中的所有对象都是可序列化的。

CoordinateServiceImpl：

CoordinateServiceImpl

为什么我面对SparkConf不可序列化，这是在实际的类中？我甚至没有使用它的任何方法。也许我错过了什么......

stacktrace：

class CoordinateServiceImpl @Inject() (val config: Configuration) extends CoordinateService with Serializable{

  override def coordinateToDistanceInMeters(lat1: Double, lng1: Double, lat2: Double, lng2: Double): Double = {
    val earthRadius: Double = 6371000
    val dLat: Double = Math.toRadians(lat2-lat1)
    val dLng: Double = Math.toRadians(lng2-lng1)
    val a: Double = Math.sin(dLat/2) * Math.sin(dLat/2) +
      Math.cos(Math.toRadians(lat1)) * Math.cos(Math.toRadians(lat2)) *
        Math.sin(dLng/2) * Math.sin(dLng/2)
    val c: Double = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a))
    earthRadius * c
  }

  override def checkProximity(distance: Double, step: Step): Boolean =
    distance < step.acceptableProximity
}

Answer 1

如果我使用的是一个对象而不是一个注入的类，它可以很好地工作。

<android.support.design.widget.CoordinatorLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:id="@+id/articleLayout"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:fitsSystemWindows="true"
    tools:context="com.gmail.senokt16.bosphoruschronicle.ArticleActivity">


    <android.support.design.widget.AppBarLayout
        android:id="@+id/articleAppBarLayout"
        android:layout_width="match_parent"
        android:layout_height="256dp"
        android:fitsSystemWindows="true">

        <android.support.design.widget.CollapsingToolbarLayout
            android:id="@+id/articleCollapsingToolbar"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:fitsSystemWindows="true"
            app:contentScrim="@color/primaryColor"
            app:expandedTitleMarginEnd="64dp"
            app:expandedTitleMarginStart="48dp"
            app:layout_scrollFlags="scroll|exitUntilCollapsed"
            app:toolbarId="@+id/article_app_bar">

            <ImageView
                android:id="@+id/articleActivityImage"
                android:layout_width="match_parent"
                android:layout_height="match_parent"
                android:fitsSystemWindows="true"
                app:layout_collapseMode="parallax" />

            <include
                android:id="@+id/article_app_bar"
                layout="@layout/article_app_bar" />
        </android.support.design.widget.CollapsingToolbarLayout>
    </android.support.design.widget.AppBarLayout>

    <android.support.v4.widget.NestedScrollView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_gravity="fill_vertical"
        android:fitsSystemWindows="true"
        app:layout_behavior="@string/appbar_scrolling_view_behavior"
        app:layout_scrollFlags="scroll|enterAlways">

        <LinearLayout
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:orientation="vertical"
            android:paddingTop="24dp">

            <TextView
                android:id="@+id/articleActivityContents"
                android:layout_width="match_parent"
                android:layout_height="wrap_content"
                android:paddingBottom="30dp"
                android:paddingLeft="20dp"
                android:paddingRight="20dp"
                android:paddingTop="5dp" />
        </LinearLayout>
    </android.support.v4.widget.NestedScrollView>

    <android.support.design.widget.FloatingActionButton
        android:id="@+id/articleFAB"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_margin="16dp"
        android:clickable="true"
        android:src="@android:drawable/ic_menu_share"
        app:layout_anchor="@id/articleAppBarLayout"
        app:layout_anchorGravity="bottom|right|end" />


</android.support.design.widget.CoordinatorLayout>

Scala，Play2.4，Spark：由于SparkConf对象

1 个答案: