Question

我手头没有火花群，但是我想摆弄elephas，所以我在try.jupyter.org使用临时实例。

你会在这个问题的底部找到我的所有代码（重复性）和完整的错误日志，但这里是我的问题的简短描述：

当我运行我的代码时，我最终得到了三种类型的错误。第一个是：

this.httpClient.get("http://localhost:3000/users")
.flatMap(users=>{
return Object.values(users)})
.subscribe(
user=>{
this.users.push(user);
});

第二个：

AttributeError: 'Model' object has no attribute 'constraints'

我也看到了第三个错误：

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 9, localhost, executor driver): org.apache.spark.api.python.PythonException

我真的不知道这三个错误中的哪一个是我的代码崩溃，每一个似乎都表明了不同的起源。因此，我想知道我是否收到这些错误：

因为涉及的某些库之间存在错误/不兼容（例如urllib.error.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR和keras之间），因为第一个错误似乎意味着;
因为我的代码中有错误（如果是这样，是什么？），因为第二个错误似乎意味着（见第一个回答here）;
或者因为elephas个实例（如果是这样，我很好奇为什么）我无法做到（因为技术上禁止），因为第三个错误似乎意味着。

我的代码：

try.jupyter.org

我的代码的最后3行生成的完整日志，包括错误日志：

# First, go to try.jupyter.org
# If it says "No Vacancy", wait a bit and refresh
# If not, then select "Welcome to Spark with Python.ipynb"
# Clean all the cells

import pip
pip.main(["install", "--upgrade", "git+https://github.com/maxpumperla/elephas"])
# I sometimes get error 128. If so, shutdown the kernel, open a new try.jupiter.org window, and try again from line 1.
# Do NOT do "pip install elephas", it installs an older build of elephas which is not compatible with keras 1+

pip.main(["install", "--upgrade", "tensorflow"])
import keras
import hyperas, flask, elephas

# start a SparkContext
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName('Elephas_App').setMaster('local[8]')
sc = SparkContext(conf=conf)
sc

# keras model
from keras.models import Sequential
from keras.optimizers import SGD
model = Sequential()

from keras.layers import Dense
model.add(Dense(units = 100 # number of neurons in the first layer
                , activation = 'sigmoid' 
                , input_dim = 30 
                , bias_initializer = keras.initializers.RandomNormal(mean = 0.1, stddev = 0.05) 
                ))
model.add(Dense(units = 10, activation='sigmoid')) 
model.add(Dense(units = 1, activation='sigmoid')) 

# see the neural network
model.summary()

# load dataset
from sklearn import datasets
cancer = datasets.load_breast_cancer()
cancer.data

# dataset into pd.dataframe
import pandas as pd
donnee = pd.concat([pd.DataFrame(data = cancer.data, columns = cancer.feature_names), 
                   pd.DataFrame(data = cancer.target, columns = ["target"])
                      ], axis = 1)

# train/test split
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(donnee.loc[:, donnee.columns != "target"], donnee.target, test_size = 0.25, random_state = 1)

from elephas.utils.rdd_utils import to_simple_rdd
rdd = to_simple_rdd(sc, X_train, y_train)

from elephas.spark_model import SparkModel
from elephas import optimizers as elephas_optimizers

adagrad = elephas_optimizers.Adagrad()
spark_model = SparkModel(sc, model, optimizer=adagrad, frequency='epoch', mode='asynchronous', num_workers=2)
spark_model.train(rdd, nb_epoch=20, batch_size=32, verbose=0, validation_split=0.1)

在jupyter测试笔记本上使用elephas

0 个答案: