上周我使用了用户“ root ”来启动Hadoop的dfs＆amp; mapreduce并运行Embedded-Pig Java代码。它运行正常。 本周我想使用非root用户：charlie 执行相同的任务。 在更改了几个目录的用户权限设置后，现在我可以使用用户“charlie”来启动Hadoop的dfs＆amp; mapreduce没有错误。 但是，当我使用用户“charlie”运行Embedded-Pig Java代码时，它一直在抱怨我设置为 / opt / hdfs /的 hadoop.tmp.dir 的权限 core-stie.xml ：
中的tmp /java.io.FileNotFoundException：/opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml（Permission denied）
我已经检查了以下目录的权限，它们看起来都很好：
bash-3.2$ ls -lt /opt/hdfs/tmp
total 4
drwxr-xr-x 3 charlie comusers 4096 Apr 16 19:30 mapred
bash-3.2$ ls -lt /opt/hdfs/tmp/mapred
total 4
drwxr-xr-x 2 charlie comusers 4096 Apr 16 19:30 local
bash-3.2$ ls -lt /opt/hdfs/tmp/mapred/local
total 0
我需要一些关于我做错事的方向。我搜索了这些关键字但没有找到任何关键字。任何帮助将不胜感激！
我已将猪输出附加如下。希望信息有用。
12/04/16 19:31:28 INFO executionengine.HExecutionEngine: Connecting to hadoop file system at: hdfs://hadoop-namenode:9000
12/04/16 19:31:29 INFO pigstats.ScriptState: Pig features used in the script: HASH_JOIN,GROUP_BY,FILTER,CROSS
12/04/16 19:31:29 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/04/16 19:31:29 INFO mapReduceLayer.MRCompiler: File concatenation threshold: 100 optimistic? false
12/04/16 19:31:30 INFO mapReduceLayer.CombinerOptimizer: Choosing to move algebraic foreach to combiner
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MRCompiler$LastInputStreamingOptimizer: Rewrite: POPackage->POForEach to POJoinPackage
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size before optimization: 11
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 map-reduce splittees.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 2 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 0 out of total 2 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 1 map-reduce splittees.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: Merged 1 out of total 3 MR operators.
12/04/16 19:31:30 INFO mapReduceLayer.MultiQueryOptimizer: MR plan size after optimization: 10
12/04/16 19:31:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:30 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:30 INFO pigstats.ScriptState: Pig script settings are added to the job
12/04/16 19:31:30 WARN pigstats.ScriptState: unable to read pigs manifest file
12/04/16 19:31:30 INFO mapReduceLayer.JobControlCompiler: mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: Setting up multi store job
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=957600
12/04/16 19:31:35 INFO mapReduceLayer.JobControlCompiler: Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO mapReduceLayer.MapReduceLauncher: 1 map-reduce job(s) waiting for submission.
12/04/16 19:31:35 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
12/04/16 19:31:35 INFO input.FileInputFormat: Total input paths to process : 1
12/04/16 19:31:35 INFO util.MapRedUtil: Total input paths to process : 1
12/04/16 19:31:35 INFO util.MapRedUtil: Total input paths (combined) to process : 1
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: 0% complete
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: job null has failed! Stop running all dependent jobs
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: 100% complete
12/04/16 19:31:36 WARN mapReduceLayer.Launcher: There is no log file to write to.
12/04/16 19:31:36 ERROR mapReduceLayer.Launcher: Backend error message during job submission
java.io.FileNotFoundException: /opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:180)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:176)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:234)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1216)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:92)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:373)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:662)
12/04/16 19:31:36 ERROR pigstats.SimplePigStats: ERROR 2997: Unable to recreate exception from backend error: java.io.FileNotFoundException: /opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission denied)
12/04/16 19:31:36 ERROR pigstats.PigStatsUtil: 1 map reduce job(s) failed!
12/04/16 19:31:36 WARN pigstats.ScriptState: unable to read pigs manifest file
12/04/16 19:31:36 INFO pigstats.SimplePigStats: Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2 charlie 2012-04-16 19:31:30 2012-04-16 19:31:36 HASH_JOIN,GROUP_BY,FILTER,CROSS
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A events,events1,grouped MULTI_QUERY Message: java.io.FileNotFoundException: /opt/hdfs/tmp/mapred/local/localRunner/job_local_0001.xml (Permission denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:180)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:176)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:234)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1216)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1197)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:92)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:373)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:662)
Input(s):
Failed to read data from "/grapevine/analysis/recommendation/input/article_based/all_grapevine_events.txt"
Output(s):
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null -> null,null,
null -> null,
null -> null,
null -> null,null,
null -> null,
null -> null,null,
null -> null,
null -> null,
null -> null,
null
12/04/16 19:31:36 INFO mapReduceLayer.MapReduceLauncher: Failed!
答案 0 :(得分：0)
答案已在评论中公布：
使用chmod -R youruser以递归方式更改文件夹的所有者 foldername它不会给出错误