Question

我正在使用jceks文件解密密码，并且无法在纱线簇模式下读取加密的密码

我尝试了包括

在内的其他方法

spark-submit --deploy-mode cluster 
--file /localpath/credentials.jceks#credentials.jceks
--conf spark.hadoop.hadoop.security.credential.provider.path=jceks://file////localpath/credentials.jceks test.py

spark1 = SparkSession.builder.appName("xyz").master("yarn").enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate()
x = spark1.sparkContext._jsc.hadoopConfiguration()
x.set("hadoop.security.credential.provider.path", "jceks://file///credentials.jceks")
a = x.getPassword("<password alias>")
passw = ""
for i in range(a.__len__()):
   passw = passw + str(a.__getitem__(i))

我收到以下错误：

attributeError：“ NoneType”对象没有属性“ len ”

当我打印a时，它没有显示

Answer 1

FWIW，如果您尝试将jceks文件放入hdfs，则纱线工作人员将能够在群集模式下运行时找到它，至少它对我有用。希望它对您有用。

tr:nth-child(even) td {
    background: #F1F1F1;
}

tr:nth-child(odd) td {
    background: #FEFEFE;
}


//here is my table
<table id="customDatesTable" class="tableStyle tableMaxWidth">
   <tr>
       <th width='15' align="left">
          Start Date
       </th>
       <th width="23" align="left">
          End Date
       </th>
       <th width="6" align="left">
          Intake#
       </th>
       <th width="10" align="left">
           Intake?
       </th>
   </tr>
   @foreach (var date in Model.CustomDates)
    {
        <tr>
            <td>@date.StartDate</td>
            <td>@date.EndDate</td>
            <td class="paddingCurrentDates">@date.IntakeNumber</td>
            <td class="paddingCurrentDates"><input type="checkbox" 
            checked="@(date.Intake)" id="intakeCheckBox" disabled /></td>
        </tr>
    }
</table>

hadoop fs -put ~/.jceks /user/<uid>/.jceks

这样，您在运行spark-submit时就不需要在参数中指定--files和--conf。希望对您有所帮助。

Answer 2

你可以这样参考jcek

#Choosing jceks from spark staging directory

jceks_location=“jceks://“ + str(“/user/your_user_name/.sparkStaging/“ + str(spark.sparkContest.applicationId) + “/credentials.jceks“)

x.set("hadoop.security.credential.provider.path",jceks_location)

无法在python的纱线簇模式下读取jceks文件

2 个答案: