Question

我遇到与此处描述的完全相同的问题： Spark not working with pureconfig。上述问题的唯一答案似乎是合理的，但我正在使用Maven而不是sbt，而且我没有将发布的解决方案从sbt转换为Maven。

我尝试过以下内容：

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
    <execution>
        <phase>package</phase>
        <goals>
            <goal>shade</goal>
        </goals>
    </execution>
</executions>
<configuration>
    <createDependencyReducedPom>false</createDependencyReducedPom>
    <relocations>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.3.2</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.chuusai:shapeless_2.11:2.0.0</pattern>
            <shadedPattern>com.matek.shaded.com.chuusai:shapeless_2.11:2.0.0</shadedPattern>
        </relocation>
        <relocation>
            <pattern>com.github.pureconfig</pattern>
            <shadedPattern>com.matek.shaded.com.github.pureconfig</shadedPattern>
            <excludes>
                <exclude>com.chuusai:shapeless_2.11:2.3.2</exclude>
            </excludes>
            <includes>
                <include>com.matek.shaded.com.chuusai:shapeless_2.11:2.3.2</include>
            </includes>
        </relocation>
    </relocations>
</configuration>

但毫不奇怪，这不起作用（我甚至不确定它是否正确）。如何指定maven shade插件配置以使其与spark提交一起使用？

Answer 1

我设法解决了这个问题。这实际上是我的错误，模式只是shapeless而不是com.chuusai.shapeless。这很有效：

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <relocations>
                        <relocation>
                            <pattern>shapeless</pattern>
                            <shadedPattern>com.matek.shaded.shapeless</shadedPattern>
                        </relocation>
                    </relocations>
                </configuration>
            </plugin>

Spark with Pureconfig - 正确的maven shade插件配置

1 个答案: