Question

我是Scala和Spark的新手，并试图建立我发现的一些样本。基本上我试图从数据框中调用一个函数来使用Google API从邮政编码中获取State。我有代码单独工作但不在一起;（这段代码不起作用......

Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type Unit is not supported
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:716)
    at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:654)
    at org.apache.spark.sql.functions$.udf(functions.scala:2837)
    at MovieRatings$.getstate(MovieRatings.scala:51)
    at MovieRatings$$anonfun$4.apply(MovieRatings.scala:48)
    at MovieRatings$$anonfun$4.apply(MovieRatings.scala:47)...
Line 51 starts with def getstate = udf {(zipcode:String)...
...

码

 userDF.createOrReplaceTempView("Users")
    // SQL statements can be run by using the sql methods provided by Spark
   val zipcodesDF = spark.sql("SELECT distinct zipcode, zipcode as state FROM Users")
 //  zipcodesDF.map(zipcodes => "zipcode: " + zipcodes.getAs[String]("zipcode") + getstate(zipcodes.getAs[String]("zipcode"))).show()
  val colNames = zipcodesDF.columns
val cols = colNames.map(cName => zipcodesDF.col(cName))
val theColumn = zipcodesDF("state")
val mappedCols = cols.map(c => 
  if (c.toString() == theColumn.toString()) getstate(c).as("transformed") else c)
  val newDF = zipcodesDF.select(mappedCols:_*).show()
  }
 def getstate = udf {(zipcode:String) => {
val url = "http://maps.googleapis.com/maps/api/geocode/json?address="+zipcode
val result = scala.io.Source.fromURL(url).mkString
val address = parse(result) 
val shortnames = for {
         JObject(address_components) <- address
         JField("short_name", short_name)  <- address_components
          } yield short_name
val state = shortnames(3)
//return state.toString()
val stater = state.toString()

}
  }

Answer 1

感谢您的回复..我想我已经明白了。这是有效的代码。需要注意的一点是Google API有限制，因此一些有效的邮政编码没有状态信息..但对我来说不是问题。

<?php

require_once '../include/DbOperation.php';
require '.././libs/Slim/Slim.php';

\Slim\Slim::registerAutoloader();

$app = new \Slim\Slim();

/**
* Method is working 
*/
$app->get('/testing', function () {
echo 'testing Slim Framework'
 });


/**
 * Method is not working outputing Error 404 
*/

$app->post('/createstudent', function () use ($app) {
verifyRequiredParams(array('name', 'username', 'password'));
$response = array();
$name = $app->request->post('name');
$username = $app->request->post('username');
$password = $app->request->post('password');
$db = new DbOperation();
$res = $db->createStudent($name, $username, $password);
if ($res == 0) {
    $response["error"] = false;
    $response["message"] = "You are successfully registered";
    echoResponse(201, $response);
} else if ($res == 1) {
    $response["error"] = true;
    $response["message"] = "Oops! An error occurred while registereing";
    echoResponse(200, $response);
} else if ($res == 2) {
    $response["error"] = true;
    $response["message"] = "Sorry, this student  already existed";
    echoResponse(200, $response);
}
});



 $app->run();
?>

Spark DF：不支持类型Unit的Schema

1 个答案: