Question

我在默认端口10.0.0.60上运行Cassandra。当我启动pyspark时，我会获得默认上下文sc。然而，这指向Cassandra，我相信127.0.0.1。

如何更改它以使其指向10.0.0.60？

[idf@node1 python]$ pyspark
Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec  6 2015, 18:08:32) 
Type "copyright", "credits" or "license" for more information.

IPython 4.1.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
16/05/18 10:40:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/

Using Python version 2.7.11 (default, Dec  6 2015 18:08:32)
SparkContext available as sc, HiveContext available as sqlContext.

In [1]:

Answer 1

执行pyspark时，您可以传递所需的配置：

pyspark --conf spark.cassandra.connection.host=10.0.0.60

或将其添加到SPARK_HOME/conf/spark-defaults.conf。

更改默认的pyspark上下文

1 个答案: