Question

我正在使用bash shell开发CentOS。

我在代码中没有看到任何错误，但它会导致打印arguments list too long错误。

它不仅适用于aws cp，之后cat /dev/null > $FILE_DIR/dailey_member.csv也会打印相同的消息。

echo " ===== dailey vote by member"
cat /dev/null > $FILE_DIR/dailey_member.csv
QUERY_DAILEY_MEMBER_RS=`mysql -h $MYSQL_URL -u$MYSQL_USER -p$MYSQL_PW $MYSQL_SCHEMA -e "SELECT voteDate, usrId, count(usrId) FROM tbl_vote GROUP BY voteDate, usrId;"`
IFS=$'\n' LINES=($QUERY_DAILEY_MEMBER_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/dailey_member.csv
done
echo "aws s3 cp $FILE_DIR/dailey_member.csv " $S3_PATH # copy output works.
aws s3 cp "$FILE_DIR/dailey_member.csv" $S3_PATH
exit 0

我该如何解决这个问题？

感谢。

FYI，我想做的是......

执行MYSQL查询
解析结果并写为csv文件
发送到s3存储库，以便人们可以下载文件

==============编辑＃1 ================

我将变量设置在文件顶部...

export FILE_DIR="/home/jyoh/applications18/cron/file"
export S3_PATH="s3://juneyoung/applications18/voteLogs/"

结果放在控制台上。

[jyoh@ip-XX-XX-X-XXX cron]$ sh ./daileyVoteByMember.sh 
 ===== dailey vote by member
aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
./daileyVoteByMember.sh: line 27: /usr/bin/aws: arguments list too long
[jyoh@ip-XX-XX-X-XXX cron]$ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
upload: file/dailey_member.csv to s3://juneyoung/applications18/voteLogs/dailey_member.csv

第27行是aws s3 cp "$FILE_DIR/dailey_member.csv" $S3_PATH的位置。

==============编辑＃2 ================

选项-x的结果。

...SKIP...

+ LINE_IDX=265782
+ LINE='20180129    9>2387463256785462364   1'
+ LINE='20180129,9>2387463256785462364,1'
+ echo -e '20180129,9>2387463256785462364,1'
+ for i in '"${!LINES[@]}"'
+ LINE_IDX=265783
+ LINE='20180129    9>3658743656387567834   1'
+ LINE='20180129,9>3658743656387567834,1'
+ echo -e '20180129,9>3658743656387567834,1'
+ echo 'aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/'
+ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
./daileyVoteByMember.sh: line 27: /usr/bin/aws: arguments list too long
+ exit 0

==============编辑＃3 ================

在新的bash文件中复制aws并执行

文件内容

#! /bin/bash

aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/

exit 0

控制台输出

[jyoh@ip-XX-XX-X-XXX cron]$ sh testSh.sh 
upload: file/dailey_member.csv to s3://juneyoung/applications18/voteLogs/

==============编辑＃4 ================

这很好，干净。可能是for和>>文件重定向有问题吗？

function sendS3 () {
   echo "S3 send command : aws s3 cp $1 $S3_PATH"     
   aws s3 cp $1 $S3_PATH
}

echo " ===== daily vote"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #1;" | sed $'s/\t/,/g' > "$FILE_DIR"/daily.csv
sendS3 "$FILE_DIR"/daily.csv


echo " ===== dailey vote by member"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #2;" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_member.csv
sendS3 "$FILE_DIR"/dailey_member.csv


echo " ===== daily vote by ip"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #3" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_ip.csv
sendS3 "$FILE_DIR"/daily_ip.csv


echo " ===== daily vote by member,item"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #4" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_item.csv
sendS3 "$FILE_DIR"/daily_item.csv

exit 0

==============编辑＃5 ================

这是完整的脚本。为了安全起见，我更换了url和一些变量。

#! /bin/bash

export SLACK_URL="SLACK_API_URL"
export MYSQL_URL="MYSQL_CONNECTION_URL"
export MYSQL_USER="MYSQL_USER"
export MYSQL_PW="MYSQL_PW"
export MYSQL_SCHEMA="USER"

export FILE_DIR="FILE_PATH"

export QUERY_DAILY_RS=""
export QUERY_DAILY_MEMBER_RS=""
export QUERY_DAILY_IP_RS=""
export QUERY_DAILY_MEMBER_ITEM_RS=""
export S3_PATH="S3_URL_TO_STORE"

echo -e "Sending vote report to Slack"
echo $MYSQL_URL
echo $MYSQL_USER::$MYSQL_PW


function sendS3 () {
   echo "S3 send command : aws s3 cp $1 $S3_PATH"     
   aws s3 cp $1 $S3_PATH
}


echo " ===== daily vote"
cat /dev/null > $FILE_DIR/daily.csv
QUERY_DAILY_RS=`mysql SELECT QUERY 1;`
IFS=$'\n' LINES=($QUERY_DAILY_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> "$FILE_DIR"/daily.csv
done
sendS3 "$FILE_DIR/daily.csv"



echo " ===== daily vote by member"
cat /dev/null > $FILE_DIR/daily_member.csv
QUERY_DAILY_MEMBER_RS=`mysql SELECT QUERY 2`
IFS=$'\n' LINES=($QUERY_DAILY_MEMBER_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/daily_member.csv
done
sendS3 "$FILE_DIR/daily_member.csv"



echo " ===== daily vote by ip"
cat /dev/null > $FILE_DIR/daily_ip.csv
QUERY_DAILY_IP_RS=`mysql SELECT QUERY 3`
IFS=$'\n' LINES=($QUERY_DAILY_IP_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/daily_ip.csv
done
sendS3 "$FILE_DIR/daily_ip.csv"



echo " ===== daily vote by member,item"
cat /dev/null > $FILE_DIR/daily_member_item.csv
QUERY_DAILY_MEMBER_ITEM_RS=`mysql SELECT QUERY 4`
IFS=$'\n' LINES=($QUERY_DAILY_MEMBER_ITEM_RS)
for i in "${!LINES[@]}"; do
    LINE_IDX=$i
    LINE=${LINES[$LINE_IDX]}
    LINE=${LINE//$'\t'/,}
    echo -e $LINE >> $FILE_DIR/daily_member_item.csv
done

sendS3 "$FILE_DIR/daily_member_item.csv"
curl -X POST -H 'Content-type: application/json' --data '{"text":"Vote data collecting done"}' $SLACK_URL

exit 0

Answer 1

您的代码设置了太多（或太大）的环境变量。

环境变量与命令行参数位于相同的每个进程空间中，因此将变量导出到环境会减少命令行参数的可用空间。（如果您没有在循环中明确export内容，请检查脚本中的echo $-是否包含a，这是一个告诉shell自动导出每个变量集的标志;如果是确实如此，找到你打开该选项的地方，然后禁用它。）

为了证明这一点（使用GNU xargs），您可以运行：

xargs --show-limits </dev/null

输出应该类似于：

Your environment variables take up 2615 bytes
POSIX upper limit on argument length (this system): 257481
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 254866
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647

从第一行的数字中减去第二行的数字，这是你的最大命令行长度（粗略）。第一行的数字太大了？查看您的环境，以及unset个大变量，或者首先不要export。

Bashshell - 在loop-echo命令之后显示的参数太长消息

1 个答案: