我正在尝试在jupyter笔记本上执行以下脚本。该脚本接受一些参数,并尝试将数据从SQLServer导入Hadoop。我遇到错误。如果我从linux命令提示符下执行shell脚本,这会很好,但是当我使用子进程调用它时会失败。
代码:
from subprocess import Popen, STDOUT, PIPE
handle = Popen('time bash /user/bin/testshell.sh --options source_format=jdbc,destination=testshelloutput,destination_mode=overwrite --conf spark.sql.warehouse.dir=workspace_dir --options --source_read_option url=jdbc:serverurl--source_read_option "dbtable=(select top 5 * from test_table) as dummy" driver=com.microsoft.sqlserver.jdbc.SQLServerDriver,integratedSecurity=true,domain=corp,authenticationScheme=XXXX,domain=corp,encrypt=true,trustServerCertificate=true,user=XXXX,password=XXXX',shell=True,stdout=PIPE,stderr=STDOUT)
print(handle.stdout.read())
错误:
b'ERROR: ld.so: object \'/usr/lib64/coreutils/libstdbuf.so\' from LD_PRELOAD cannot be preloaded: ignored.\nThu Oct 22 12:32:52 EDT 2020\nProcess limit is \x1b[0;32m1032713\x1b[0m and file limit is \x1b[0;32m65536\x1b[0.......
有人可以帮我吗?