将sql server jar添加到pyspark的类路径后无法查询配置单元

Question

Hive设置正确，我可以在使用<UserControl.Resources> <!-- my dummy values --> <!-- <SolidColorBrush x:Key="BackgroundBrush" Color="DarkGray" /> <SolidColorBrush x:Key="TSBackgroundBrush" Color="Gray" /> <SolidColorBrush x:Key="TSBarBrush" Color="LightGreen" /> <SolidColorBrush x:Key="AccentColorBrush1" Color="Khaki" /> --> <Style TargetType="Rectangle" x:Key="SliderRectangleStyle"> <Setter Property="Stroke" Value="{DynamicResource BackgroundBrush}" /> <Setter Property="Fill" Value="{DynamicResource TSBarBrush}" /> <Style.Triggers> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="0"> <Setter Property="Grid.Column" Value="0" /> </DataTrigger> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="1"> <Setter Property="Grid.Column" Value="2" /> </DataTrigger> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="2"> <Setter Property="Grid.Column" Value="4" /> </DataTrigger> </Style.Triggers> </Style> <Style TargetType="Rectangle" x:Key="LeftFillRectangleStyle"> <Setter Property="Stroke" Value="{DynamicResource BackgroundBrush}" /> <Setter Property="Fill" Value="{DynamicResource AccentColorBrush1}" /> <Setter Property="Margin" Value="2,2,0,2" /> <Style.Triggers> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="0"> <Setter Property="Visibility" Value="Collapsed" /> </DataTrigger> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="1"> <Setter Property="Grid.ColumnSpan" Value="2" /> </DataTrigger> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="2"> <Setter Property="Grid.ColumnSpan" Value="4" /> </DataTrigger> </Style.Triggers> </Style> <Style x:Key="TextLabelStyle" TargetType="TextBlock"> <Style.Triggers> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="0"> <Setter Property="Text" Value="Aus" /> </DataTrigger> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="1"> <Setter Property="Text" Value="Client" /> </DataTrigger> <DataTrigger Binding="{Binding Status, RelativeSource={RelativeSource AncestorType=UserControl}}" Value="2"> <Setter Property="Text" Value="Finance Dept." /> </DataTrigger> </Style.Triggers> </Style> </UserControl.Resources> <Grid> <Grid.ColumnDefinitions> <ColumnDefinition Width="10"/> <ColumnDefinition Width="10"/> <ColumnDefinition Width="10"/> <ColumnDefinition Width="10"/> <ColumnDefinition Width="10"/> <ColumnDefinition Width="auto"/> </Grid.ColumnDefinitions> <Rectangle Margin="5,2,5,2" Grid.Row="0" Grid.Column="0" Grid.ColumnSpan="5" Fill="{DynamicResource BackgroundBrush}" Stroke="{DynamicResource TSBackgroundBrush}" /> <TextBlock Grid.Row="0" Grid.Column="5" Style="{StaticResource TextLabelStyle}" Margin="4,0,4,0" /> <Rectangle Style="{StaticResource LeftFillRectangleStyle}" /> <Rectangle Style="{StaticResource SliderRectangleStyle}" /> <!-- Overlay the slider area with three equal click areas --> <Grid Grid.Column="0" Grid.ColumnSpan="5"> <Grid.ColumnDefinitions> <ColumnDefinition Width="*" /> <ColumnDefinition Width="*" /> <ColumnDefinition Width="*" /> </Grid.ColumnDefinitions> <Grid.Resources> <!-- Because this style has no x:Key, it will apply to all Rectangles in this Grid --> <Style TargetType="Rectangle" > <!-- Uncomment these setters to make sure the clickable areas are where they should be. --> <!-- <Setter Property="Stroke" Value="Red" /> <Setter Property="StrokeThickness" Value="1" /> --> <Setter Property="Fill" Value="Transparent" /> <EventSetter Event="PreviewMouseDown" Handler="StatusSelect_PreviewMouseDown" /> </Style> </Grid.Resources> <Rectangle Grid.Column="0" /> <Rectangle Grid.Column="1" /> <Rectangle Grid.Column="2" /> </Grid> </Grid> 输入repl后使用spark.sql进行查询。我想从sql server读取一个表并将其保存到hive。如果启动repl传递在像pyspark这样的jdbc jar中，我可以从sql server读取。但是现在火花无法进入蜂巢。对现有配置单元表的任何查询都会导致Lzo Codec错误（见下文）。

我想知道如何查询/下拉外部sql server表，然后将其保存到现有的hive表中。

pyspark  --driver-class-path sqljdbc4.jar --jars sqljdbc4.jar

Answer 1

--driver-class-path标志替换您在默认设置中配置的任何值。它不会附加新值。您可能已将LZO jar添加到spark-defaults.conf文件中的类路径中，并且它被忽略，因为您使用该标志覆盖该设置。你应该：

1）在--driver-class-path设置

中包含整个驱动程序类路径

或

2）将SQL JDBC jar添加到spark-defaults.conf文件中spark.driver.extraClassPath设置的类路径中

另外，正如@Tim在评论中指出的那样，你仍然需要使用--jars标志将jar提供给命令行。