我遵循了这些说明
http://davidssysadminnotes.blogspot.com/2016/01/installing-spark-centos-7.html http://tecadmin.net/setup-hadoop-2-4-single-node-cluster-on-linux/#
适用于CentOS 7.2。当我启动pyspark时，一切似乎都没问题：
[idf@node1 ~]$ pyspark
Python 2.7.11 |Anaconda 4.0.0 (64-bit)| (default, Dec 6 2015, 18:08:32)
Type "copyright", "credits" or "license" for more information.
IPython 4.1.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
16/04/01 16:02:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.6.1
/_/
Using Python version 2.7.11 (default, Dec 6 2015 18:08:32)
SparkContext available as sc, HiveContext available as sqlContext.
In [1]:
但是当我启动spark-shell时，我会收到错误：
[idf@node1 ~]$ spark-shell
16/04/01 15:59:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.1
/_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/04/01 15:59:42 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark-latest/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar."
16/04/01 15:59:42 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark-latest/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar."
16/04/01 15:59:42 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark-latest/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar."
16/04/01 15:59:42 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
Fri Apr 01 15:59:42 EDT 2016 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)
16/04/01 15:59:43 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
----------------------------------------------------------------
Fri Apr 01 15:59:43 EDT 2016:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.10.1.1 - (1458268): instance a816c00e-0153-d368-eecc-000031edf8d8
on database directory /tmp/spark-f22a65d6-4bda-4426-8b80-0f34b39b28dd/metastore with class loader sun.misc.Launcher$AppClassLoader@4e25154f
Loaded from file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar
java.vendor=Oracle Corporation
java.runtime.version=1.8.0_45-b14
user.dir=/home/idf
os.name=Linux
os.arch=amd64
os.version=3.10.0-327.10.1.el7.x86_64
derby.system.home=null
Database Class Loader started - derby.database.classpath=''
16/04/01 16:00:02 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/04/01 16:00:02 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/04/01 16:00:06 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark-latest/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar."
16/04/01 16:00:06 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark-latest/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar."
16/04/01 16:00:06 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/opt/spark-latest/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar."
16/04/01 16:00:06 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.dataDictionary in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.lockManagerJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.dvfJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.javaCompiler in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.replication.slave in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.rawStore.transactionJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.ef in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.rawStore.transactionJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.database in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.NoneAuthentication in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.netServer.autoStart in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.dvfJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.mgmt.null in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.nativeAuthentication in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.lockManagerJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.replication.master in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.dvfCDC in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.access.btree in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.lockManagerJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.uuidJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.cryptographyJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.rawStore.data.genericJ4 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.rawStore.data.genericJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.access in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.jdbc169 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.cryptographyJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.optimizer in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.mgmt.jmx in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.dvfJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.specificAuthentication in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.JNDIAuthentication in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.basicAuthentication in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.data.genericJ4 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.validation in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.data.genericJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.classManagerJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.streams in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.classManagerJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.resourceAdapterJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.rawStore.transactionJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.jdbcJ8 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.jdbcJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.classManagerJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.jdbcJ4 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.log in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.log.readonly in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.access.heap in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.jdbcJ8 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.daemon in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.cacheManagerJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.tcf in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.jdbcJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.jdbcJ4 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.access.uniquewithduplicatenullssort in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.cacheManagerJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.cacheManagerJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.resultSetStatisticsFactory in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.cacheManagerJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.cryptographyJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.rawStore.data.genericJ4 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.database.slave in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.XPLAINFactory in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.resourceAdapterJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.access.sort in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.mgmt.jmx in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.transactionJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.JNDIAuthentication in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.transactionJ1 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.timer in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.lcf in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.rawStore in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.resourceAdapterJ2 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.classes.jdbc169 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.lf in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.jdbcJ8 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.jdbcJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.env.jdk.jdbcJ4 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.nodeFactory in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main] Ignored duplicate property derby.module.lockManagerJ6 in jar:file:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/derby/modules.properties
Fri Apr 01 16:00:06 EDT 2016 Thread[main,5,main]
16/04/01 16:00:06 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
Fri Apr 01 16:00:07 EDT 2016 Thread[main,5,main] Cleanup action starting
java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@3531509c, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
答案 0 :(得分：0)
spark-shell alreeady为您提供了一个带有Hive-support的sqlContext：
[meox@debian-meox: ~/spark-1.6.1-bin-hadoop2.6] # ./bin/spark-shell --master local[4] --packages com.databricks:spark-csv_2.10:1.4.0
Ivy Default Cache set to: /home/meox/.ivy2/cache
The jars for the packages stored in: /home/meox/.ivy2/jars
:: loading settings :: url = jar:file:/home/meox/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.databricks#spark-csv_2.10 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.databricks#spark-csv_2.10;1.4.0 in central
found org.apache.commons#commons-csv;1.1 in list
found com.univocity#univocity-parsers;1.5.1 in list
:: resolution report :: resolve 429ms :: artifacts dl 21ms
:: modules in use:
com.databricks#spark-csv_2.10;1.4.0 from central in [default]
com.univocity#univocity-parsers;1.5.1 from list in [default]
org.apache.commons#commons-csv;1.1 from list in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 3 | 0 | 0 | 0 || 3 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 3 already retrieved (0kB/10ms)
16/04/02 11:02:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/02 11:02:35 INFO SecurityManager: Changing view acls to: meox
16/04/02 11:02:35 INFO SecurityManager: Changing modify acls to: meox
16/04/02 11:02:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(meox); users with modify permissions: Set(meox)
16/04/02 11:02:35 INFO HttpServer: Starting HTTP Server
16/04/02 11:02:35 INFO Utils: Successfully started service 'HTTP class server' on port 40697.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.1
/_/
Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.7.0_95)
Type in expressions to have them evaluated.
Type :help for more information.
16/04/02 11:02:41 INFO SparkContext: Running Spark version 1.6.1
16/04/02 11:02:41 INFO SecurityManager: Changing view acls to: meox
16/04/02 11:02:41 INFO SecurityManager: Changing modify acls to: meox
16/04/02 11:02:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(meox); users with modify permissions: Set(meox)
16/04/02 11:02:42 INFO Utils: Successfully started service 'sparkDriver' on port 41544.
16/04/02 11:02:42 INFO Slf4jLogger: Slf4jLogger started
16/04/02 11:02:42 INFO Remoting: Starting remoting
16/04/02 11:02:42 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.0.50:37295]
16/04/02 11:02:42 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 37295.
16/04/02 11:02:42 INFO SparkEnv: Registering MapOutputTracker
16/04/02 11:02:42 INFO SparkEnv: Registering BlockManagerMaster
16/04/02 11:02:42 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-e5d301cf-bb94-4738-aecd-9d97f0f6174a
16/04/02 11:02:42 INFO MemoryStore: MemoryStore started with capacity 511.5 MB
16/04/02 11:02:42 INFO SparkEnv: Registering OutputCommitCoordinator
16/04/02 11:02:43 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/04/02 11:02:43 INFO SparkUI: Started SparkUI at http://192.168.0.50:4040
16/04/02 11:02:43 INFO HttpFileServer: HTTP File server directory is /tmp/spark-af3ded9a-8e45-4d5a-8d14-a3114dd899aa/httpd-90b07dcf-a8bc-4c32-9e0a-4d698033c7aa
16/04/02 11:02:43 INFO HttpServer: Starting HTTP Server
16/04/02 11:02:43 INFO Utils: Successfully started service 'HTTP file server' on port 45007.
16/04/02 11:02:43 INFO SparkContext: Added JAR file:/home/meox/.ivy2/jars/com.databricks_spark-csv_2.10-1.4.0.jar at http://192.168.0.50:45007/jars/com.databricks_spark-csv_2.10-1.4.0.jar with timestamp 1459587763133
16/04/02 11:02:43 INFO SparkContext: Added JAR file:/home/meox/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar at http://192.168.0.50:45007/jars/org.apache.commons_commons-csv-1.1.jar with timestamp 1459587763134
16/04/02 11:02:43 INFO SparkContext: Added JAR file:/home/meox/.ivy2/jars/com.univocity_univocity-parsers-1.5.1.jar at http://192.168.0.50:45007/jars/com.univocity_univocity-parsers-1.5.1.jar with timestamp 1459587763135
16/04/02 11:02:43 INFO Executor: Starting executor ID driver on host localhost
16/04/02 11:02:43 INFO Executor: Using REPL class URI: http://192.168.0.50:40697
16/04/02 11:02:43 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58692.
16/04/02 11:02:43 INFO NettyBlockTransferService: Server created on 58692
16/04/02 11:02:43 INFO BlockManagerMaster: Trying to register BlockManager
16/04/02 11:02:43 INFO BlockManagerMasterEndpoint: Registering block manager localhost:58692 with 511.5 MB RAM, BlockManagerId(driver, localhost, 58692)
16/04/02 11:02:43 INFO BlockManagerMaster: Registered BlockManager
16/04/02 11:02:43 INFO SparkILoop: Created spark context..
Spark context available as sc.
16/04/02 11:02:44 INFO HiveContext: Initializing execution hive, version 1.2.1
16/04/02 11:02:44 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
16/04/02 11:02:44 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
16/04/02 11:02:44 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/04/02 11:02:44 INFO ObjectStore: ObjectStore, initialize called
16/04/02 11:02:44 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/04/02 11:02:44 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/04/02 11:02:45 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/02 11:02:45 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/02 11:02:47 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/04/02 11:02:48 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:48 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:51 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:51 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:51 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/04/02 11:02:51 INFO ObjectStore: Initialized ObjectStore
16/04/02 11:02:51 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/04/02 11:02:52 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/04/02 11:02:52 INFO HiveMetaStore: Added admin role in metastore
16/04/02 11:02:52 INFO HiveMetaStore: Added public role in metastore
16/04/02 11:02:52 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/04/02 11:02:52 INFO HiveMetaStore: 0: get_all_databases
16/04/02 11:02:52 INFO audit: ugi=meox ip=unknown-ip-addr cmd=get_all_databases
16/04/02 11:02:52 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/04/02 11:02:52 INFO audit: ugi=meox ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/04/02 11:02:52 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:53 INFO SessionState: Created local directory: /tmp/c9aee4a3-05b4-4606-8cc9-eb6e3af28fa1_resources
16/04/02 11:02:53 INFO SessionState: Created HDFS directory: /tmp/hive/meox/c9aee4a3-05b4-4606-8cc9-eb6e3af28fa1
16/04/02 11:02:53 INFO SessionState: Created local directory: /tmp/meox/c9aee4a3-05b4-4606-8cc9-eb6e3af28fa1
16/04/02 11:02:53 INFO SessionState: Created HDFS directory: /tmp/hive/meox/c9aee4a3-05b4-4606-8cc9-eb6e3af28fa1/_tmp_space.db
16/04/02 11:02:53 INFO HiveContext: default warehouse location is /user/hive/warehouse
16/04/02 11:02:53 INFO HiveContext: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
16/04/02 11:02:53 INFO ClientWrapper: Inspected Hadoop version: 2.6.0
16/04/02 11:02:53 INFO ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0
16/04/02 11:02:54 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
16/04/02 11:02:54 INFO ObjectStore: ObjectStore, initialize called
16/04/02 11:02:54 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
16/04/02 11:02:54 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
16/04/02 11:02:54 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/02 11:02:54 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/04/02 11:02:55 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/04/02 11:02:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:56 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
16/04/02 11:02:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/04/02 11:02:56 INFO ObjectStore: Initialized ObjectStore
16/04/02 11:02:57 INFO HiveMetaStore: Added admin role in metastore
16/04/02 11:02:57 INFO HiveMetaStore: Added public role in metastore
16/04/02 11:02:57 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/04/02 11:02:57 INFO HiveMetaStore: 0: get_all_databases
16/04/02 11:02:57 INFO audit: ugi=meox ip=unknown-ip-addr cmd=get_all_databases
16/04/02 11:02:57 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/04/02 11:02:57 INFO audit: ugi=meox ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/04/02 11:02:57 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
16/04/02 11:02:57 INFO SessionState: Created local directory: /tmp/2d5b72da-368c-4c00-8d14-ca8c5cfbf66c_resources
16/04/02 11:02:57 INFO SessionState: Created HDFS directory: /tmp/hive/meox/2d5b72da-368c-4c00-8d14-ca8c5cfbf66c
16/04/02 11:02:57 INFO SessionState: Created local directory: /tmp/meox/2d5b72da-368c-4c00-8d14-ca8c5cfbf66c
16/04/02 11:02:57 INFO SessionState: Created HDFS directory: /tmp/hive/meox/2d5b72da-368c-4c00-8d14-ca8c5cfbf66c/_tmp_space.db
16/04/02 11:02:57 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
如果你看到最后一行说：“16/04/02 11:02:57 INFO SparkILoop：创建了sql上下文（有Hive支持）..”
请记住：