我们有一个非交互式bash shell命令,它执行一些代码并且该代码读取一些环境变量。
#!/bin/bash
export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent"
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"}
export JAVA_HOME=${JAVA_HOME:-"/usr/java/default"}
export LANG="en_US.UTF-8"
export NM_HOST="test.com"
export LOGNAME="root"
export JVM_PID="$$"
export PWD="/opt/hadoop/yarn/local/usercache/root/appcache/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002"
export LOCAL_DIRS="/opt/hadoop/yarn/local/usercache/root/appcache/application_1504729685201_0003"
export SHDP_CONTAINERID="container_e11_1504729685201_0003_01_000002"
export NM_HTTP_PORT="8042"
export LOG_DIRS="/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002"
export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
"
export NM_PORT="45454"
export USER="root"
export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"}
export CLASSPATH="/org/test/import/application.yml:./*"
export HADOOP_TOKEN_FILE_LOCATION="/opt/hadoop/yarn/local/usercache/root/appcache/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/container_tokens"
export NM_AUX_SERVICE_spark_shuffle=""
export LOCAL_USER_DIRS="/opt/hadoop/yarn/local/usercache/root/"
export HADOOP_HOME="/usr/hdp/2.6.1.0-129/hadoop"
export HOME="/home/"
export NM_AUX_SERVICE_spark2_shuffle=""
export CONTAINER_ID="container_e11_1504729685201_0003_01_000002"
export MALLOC_ARENA_MAX="4"
ln -sf "/opt/hadoop/yarn/local/usercache/root/appcache/application_1504729685201_0003/filecache/11/application.yml" "application.yml"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
ln -sf "/opt/hadoop/yarn/local/usercache/root/appcache/application_1504729685201_0003/filecache/12/test.jar" "test.jar"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
# Creating copy of launch script
cp "launch_container.sh" "/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/launch_container.sh"
chmod 640 "/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/launch_container.sh"
# Determining directory contents
echo "ls -l:" 1>"/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/directory.info"
ls -l 1>>"/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/directory.info"
echo "find -L . -maxdepth 5 -ls:" 1>>"/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/directory.info"
find -L . -maxdepth 5 -ls 1>>"/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/directory.info"
echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/directory.info"
find -L . -maxdepth 5 -type l -ls 1>>"/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/directory.info"
exec /bin/bash -c "$JAVA_HOME/bin/java -jar test.jar 1>/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/Container.stdout 2>/opt/hadoop/yarn/log/application_1504729685201_0003/container_e11_1504729685201_0003_01_000002/Container.stderr"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
我们在〜/ .bashrc文件中添加了代码所需的环境变量,但似乎非交互式shell不读取此文件,因此我们的代码无法获取所需的env变量。不幸的是我正在使用的bash脚本是自动生成的,我无法在那里导出我的env变量。有没有其他方法可以使env变量可用于我们的Java代码?感谢。
答案 0 :(得分:0)
可以在/etc/hadoop/conf/yarn-site.xml中配置特别需要的环境变量,那里有一个属性,yarn.nodemanager.admin-env,用于列出应该从中转发的环境变量NodeManager的环境到容器的。 默认值为MALLOC_ARENA_MAX = $ MALLOC_ARENA_MAX,您只需要在逗号分隔列表中附加需要传递的变量: 例如 MALLOC_ARENA_MAX = $ MALLOC_ARENA_MAX,YOUR_VARIABLE = $ YOUR_VARIABLE
这也可以在纱线的纱线现场高级配置部分使用Ambari进行配置。
使用/etc/profile.d/中的脚本设置全局环境变量是一种很好的做法,可供所有用户使用。
参考文献: - 纱线网站默认配置 https://hadoop.apache.org/docs/r0.23.11/hadoop-yarn/hadoop-yarn-common/yarn-default.xml