Question

我正在使用Spark 1.5和Cassandra 3.2.1。任何人都可以指定在构建路径中出现连接，查询和插入数据到Cassandra所需的确切jar。

现在我正在使用下面的罐子火花卡桑德拉 - connector_2.10-1.5.0-M3.jar Apache的卡桑德拉 - clientutil-3.2.1.jar 卡桑德拉驱动核-3.0.0-β1-bb1bce4-快照shaded.jar 火花组装1.5.1 hadoop2.0.0-MR1-cdh4.2.0.jar 番石榴18.0.jar 网状-全4.0.23.Final.jar

通过上述罐子，我可以连接到cassandra。我能够截断表，并删除表。但我无法插入任何数据，甚至不能插入简单的插入查询。

以下是代码：

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;

import com.datastax.driver.core.Session;
import com.datastax.spark.connector.cql.CassandraConnector;

public class Test {

public static void main(String[] args) {


    JavaSparkContext ctx = new JavaSparkContext(new SparkConf().setMaster("spark://blr-lt-203:7077").set("spark.cassandra.connection.host", "blr-lt-203").setAppName("testinsert").set("spark.serializer" ,"org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buffer.max" , "1024mb"));

    CassandraConnector connector = CassandraConnector.apply(ctx.getConf());

    Session session = connector.openSession();

    session.execute("insert into test.table1 (name) values ('abcd')") ;
    session.close();
    ctx.stop();

}

}

以下是日志：

16/03/28 21:24:52 INFO BlockManagerMaster: Trying to register BlockManager
16/03/28 21:24:52 INFO BlockManagerMasterEndpoint: Registering   block    manager localhost:50238 with 944.7 MB RAM,BlockManagerId(driver, localhost, 50238)
16/03/28 21:24:52 INFO BlockManagerMaster: Registered BlockManager
16/03/28 21:24:53 INFO NettyUtil: Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
16/03/28 21:24:53 INFO Cluster: New Cassandra host localhost/127.0.0.1:9042 added
16/03/28 21:24:53 INFO CassandraConnector: Connected to Cassandra cluster: Test Cluster

它在这里停留了一段时间，然后用foll例外超时：

Exception in thread "main" com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)

我做错了什么？
请告诉我们所需的罐子是什么，或者是否存在某些版本兼容性问题。
什么是最稳定的spark（1.5）和cassandra（？）

先谢谢

Answer 1

由于谷歌的番石榴库之间的冲突，问题就出现了。

解决方案是遮蔽 spark-cassandra-connector依赖项中存在的guava库。你可以使用maven shade插件来实现。这是我的pom.xml来遮蔽番石榴库。

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0  
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.pc.test</groupId>
<artifactId>casparktest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

 <name>casparktest</name>
<url>http://maven.apache.org</url>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
<dependency>
 <groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>3.8.1</version>
  <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.datastax.spark</groupId>
    <artifactId>spark-cassandra-connector_2.10</artifactId>
    <version>1.5.0</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.0.0-beta1</version>
</dependency>

</dependencies>
<build>

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.3</version>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                 <filters>
    <filter>
        <artifact>*:*</artifact>
        <excludes>
            <exclude>META-INF/*.SF</exclude>
            <exclude>META-INF/*.DSA</exclude>
            <exclude>META-INF/*.RSA</exclude>
        </excludes>
    </filter>
</filters>
                    <relocations>
                        <relocation>
                            <pattern>com.google</pattern>
                            <shadedPattern>com.pointcross.shaded.google</shadedPattern>
                        </relocation>

                    </relocations>
                    <minimizeJar>false</minimizeJar>
                    <shadedArtifactAttached>true</shadedArtifactAttached>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>
</build>

之后你做了一个maven构建，它将生成一个jar，其中包含pom.xml中提到的所有依赖项，同时也会遮挡guava库，使用它们可以提交spark作业。

Spark 1.5和datastax-ddc-3.2.1 Cassandra Dependency Jars？

1 个答案: