Question

我正在学习如何使用Spark运行机器学习应用程序。我尝试在sublime文本中编写脚本，然后使用spark-submit来运行代码。但是它工作得不好，所以我决定直接在Pyspark中运行所有内容。

这是我的代码：

#ML with MNIST using Spark
import cPickle, gzip
import numpy as np
#import pyspark
#from pyspark import SparkContext
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.classification import NaiveBayes

#sc = SparkContext(appName='PythonWordCount')

f = gzip.open('mnist.pkl.gz', 'rb')
train_set, valid_set, test_set = cPickle.load(f)
f.close()

#set pair aking for train or test
def pair_making(data_set):
    #splitting the two arrays from data_set
    data_set_x, data_set_y = data_set

    return data_set_x, data_set_y


#create training sample and labels
train_x, train_y=pair_making(train_set)

train_points=[]
for i in range(len(train_x)):
    train_points.append(LabeledPoint(train_y[i], train_x[i]))


#training the NB classifier
model=NaiveBayes.train(sc.parallelize(train_points), 1.0)

#using valid set to test, for now
valid_x, valid_y= pair_making(valid_set)

#predict values using valid samples
prediction=model.predict(valid_x)

#checking accuracy
count=0
for i in range(len(valid_y)):
    if prediction[i]==valid_y[i]:
        count+=1
accuracy=float(count)/len(valid_y)
print "accuracy is %f", % accuracy

我遇到的问题是该程序似乎在不间断运行。 30分钟后，我决定打断它并发布到这里。我收到了这条消息（很多警告，不一定是错误）：

15/09/16 16:49:01 WARN AkkaRpcEndpointRef: Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@27349480,BlockManagerId(driver, localhost, 57521))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#78461372]] had already been terminated.
    at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:132)
    at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:299)
    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:444)
    at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:464)
    at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:464)
    at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:464)
    at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
    at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:464)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#78461372]] had already been terminated.
    at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:132)
    at org.apache.spark.rpc.akka.AkkaRpcEndpointRef.ask(AkkaRpcEnv.scala:299)
    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
    ... 14 more

好吧，它很长，这只是警告信息的一次重复。似乎火花一直在请求消息，但不断反弹。

发送邮件时

[Actor[akka://sparkDriver/user/HeartbeatReceiver#78461372]]已被终止。什么是HeartBeatReceiver？

另外，实际上是否有错误？或者我只需要让它运行？我没有在较短的样品上测试它。只是想在继续之前快速与专家核实。有人可以帮忙吗？

谢谢！

在Pyspark中运行NaiveBayes分类器，它似乎经常是超时的＆＃39;

0 个答案: