使用apache-curator存储纱线上的火花流中的kafka偏移量。异常:用户类引发异常:java.lang.NoSuchMethodError

时间:2019-10-25 10:54:06

标签: apache-spark yarn apache-curator

用户类引发异常:java.lang.NoSuchMethodError:org.apache.curator.framework.api.CreateBuilder.creatingParentContainersIfNeeded()Lorg / apache / curator / framework / api / ProtectACLCreateModeStatPathAndBytesable;

作为本地模型运行,不会抛出异常。

apache-spark:2.2.0

apache-curator:4.0.1,zookeeper-client:3.4.5,zookeeper-server:3.4.5

例外代码:

curator.create().creatingParentsIfNeeded().withMode(CreateMode.PERSISTENT).forPath(path, data)

maven pom:


 <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.curator</groupId>
                    <artifactId>curator-framework</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.curator</groupId>
                    <artifactId>curator-client</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-core_2.11</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-framework</artifactId>
            <version>4.0.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.zookeeper</groupId>
                    <artifactId>zookeeper</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.4.5</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

我尝试在spark-core中排除策展人的依赖性,但仍然有相同的例外。

0 个答案:

没有答案