Question

我正在尝试连接到受Kerberos身份验证保护的HDFS。我有以下详细信息，但不知道如何进行。

User
Password
Realm
HttpFs Url

我尝试了以下代码，但收到身份验证错误：

from hdfs.ext.kerberos import KerberosClient
import requests
import logging

logging.basicConfig(level=logging.DEBUG)

session = requests.Session()
session.verify = False

client = KerberosClient(url='http://x.x.x.x:abcd', session=session, 
mutual_auth='REQUIRED',principal='abcdef@LMNOPQ')

print(client.list('/'))

错误

INFO:hdfs.client:Instantiated   
<KerberosClient(url=http://x.x.x.x:abcd)>.
INFO:hdfs.client:Listing '/'.
DEBUG:hdfs.client:Resolved path '/' to '/'.
DEBUG:hdfs.client:Resolved path '/' to '/'.
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 
DEBUG:urllib3.connectionpool:http://x.x.x.x:abcd "GET /webhdfs/v1/? 
op=LISTSTATUS HTTP/1.1" 401 997
DEBUG:requests_kerberos.kerberos_:handle_401(): Handling: 401
ERROR:requests_kerberos.kerberos_:generate_request_header(): authGSSClientInit() failed:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_kerberos/kerberos_.py", line 213, in generate_request_header
gssflags=gssflags, principal=self.principal)
kerberos.GSSError: ((' No credentials were supplied, or the credentials were unavailable or inaccessible.', 458752), ('unknown mech-code 0 for mech unknown', 0))
ERROR:requests_kerberos.kerberos_:((' No credentials were supplied, or the credentials were unavailable or inaccessible.', 458752), ('unknown mech-code 0 for mech unknown', 0))
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests_kerberos/kerberos_.py", line 213, in generate_request_header
gssflags=gssflags, principal=self.principal)
kerberos.GSSError: ((' No credentials were supplied, or the credentials were unavailable or inaccessible.', 458752), ('unknown mech-code 0 for mech unknown', 0))
DEBUG:requests_kerberos.kerberos_:handle_401(): returning <Response [401]>
DEBUG:requests_kerberos.kerberos_:handle_response(): returning <Response [401]>

我也有密码，但是不知道在哪里提供密码。

Answer 1

比方说，您有原理：hdfs/localhost@HADOOP.COM，您的密钥表文件是：/var/run/cloudera-scm-agent/process/39-hdfs-NAMENODE/hdfs.keytab，如果您想阅读的话在/hadoop_test_data/filecount.csv中已经有一个hdfs csv文件，然后使用以下代码，您将获得带有filecount.csv内容的pandas数据框。

在这里，我使用的是python版本： 3.7.6

import io 
from csv import reader
from krbcontext import krbcontext
import subprocess 
import pandas as pd

try:
    with krbcontext(using_keytab=True,
                    principal='hdfs/localhost@HADOOP.COM',
                    keytab_file='/var/run/cloudera-scm-agent/process/39-hdfs-NAMENODE/hdfs.keytab') as krb:
                    print(krb)
                    print('kerberos authentication successful') 
                    output = subprocess.Popen(["hadoop", "fs", "-cat", "/hadoop_test_data/filecount.csv"], stdout=subprocess.PIPE)
                    stdout,stderr = output.communicate()
                    data = str(stdout,'utf-8').split('\r\n')
                    df = pd.DataFrame( list(reader(data[1:])),columns=data[0].split(','))
                    print(df.shape)
                    print(df)

except Exception as e:
    print("Kerberos authentication unsuccessful")
    print("Detailed error is : "+e)

如果您想了解更多信息，请告诉我。

Answer 2

据我了解，您必须首先使用 kinit 命令进行kerberos身份验证，然后再运行您附加的代码

使用Python使用Kerberos身份验证连接到HDFS

2 个答案: