Python（boto）TypeError启动Spark Cluster

Question

以下是尝试启动具有十个从属的群集。

12:13:44/sparkup $ec2/spark-ec2 -k sparkeast -i ~/.ssh/myPem.pem \
          -s 10  -z us-east-1a -r us-east-1 launch spark2

这是输出。请注意，使用二月主代码的相同命令已成功。今天我已更新到最新的1.4.0-SNAPSHOT

Setting up security groups...
Searching for existing cluster spark2 in region us-east-1...
Spark AMI: ami-5bb18832
Launching instances...
Launched 10 slaves in us-east-1a, regid = r-68a0ae82
Launched master in us-east-1a, regid = r-6ea0ae84
Waiting for AWS to propagate instance metadata...
Waiting for cluster to enter 'ssh-ready' state.........unable to load cexceptions
TypeError
p0
(S''
p1
tp2
Rp3
(dp4
S'child_traceback'
p5
S'Traceback (most recent call last):\n  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1280, in _execute_child\n    sys.stderr.write("%s %s (env=%s)\\n" %(executable, \' \'.join(args), \' \'.join(env)))\nTypeError\n'
p6
sb.Traceback (most recent call last):
  File "ec2/spark_ec2.py", line 1444, in <module>
    main()
  File "ec2/spark_ec2.py", line 1436, in main
    real_main()
  File "ec2/spark_ec2.py", line 1270, in real_main
    cluster_state='ssh-ready'
  File "ec2/spark_ec2.py", line 869, in wait_for_cluster_state
    is_cluster_ssh_available(cluster_instances, opts):
  File "ec2/spark_ec2.py", line 833, in is_cluster_ssh_available
    if not is_ssh_available(host=dns_name, opts=opts):
  File "ec2/spark_ec2.py", line 807, in is_ssh_available
    stderr=subprocess.STDOUT  # we pipe stderr through stdout to preserve output order
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1328, in _execute_child
    raise child_exception
TypeError

AWS控制台显示实例正在运行。所以目前还不清楚究竟是什么失败了。

任何提示或解决方法都很受欢迎。

更新执行登录命令时会发生同样的错误。这似乎是boto API的问题 - 但群集本身似乎没问题。

ec2/spark-ec2  -i ~/.ssh/sparkeast.pem  login spark2
Searching for existing cluster spark2 in region us-east-1...
Found 1 master, 10 slaves.
Logging into master ec2-54-87-46-170.compute-1.amazonaws.com...
unable to load cexceptions
TypeError
p0
(..  same exception stacktrace as above )

Answer 1

问题是我的yosemite macbook上的python-2.7.6安装似乎已损坏。

我重置PATH和PYTHONPATH指向自定义自制软件安装的python版本，然后boto - 和其他python命令，包括构建spark性能项目 - 工作正常。