Question

我们通常能够在＆＃34; / var / log / hadoop-yarn / containers＆＃34;中查看纱线容器日志。路径。虽然我能够看到成功作业的日志，但我无法查看失败作业的日志。节点管理器日志显示已删除的日志。

日志：

    public class DownloadDialog extends Dialog implements View.OnClickListener{

public Context c;
public Button download, delete;
private ProgressBar pb;
ProgressTask progressTask;
private int downloadStatus;
private String downloadLink;
private int downloadID

public DownloadDialog(Context a, int downloadId) {
    super(a);
    this.c = a;
    this.downloadId = downloadId
}

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    requestWindowFeature(Window.FEATURE_NO_TITLE);
    setContentView(R.layout.download_dialog);
    download = (Button) findViewById(R.id.downloaddialot_downloadbtn);
    delete = (Button) findViewById(R.id.downloaddialot_deletebtn);
    download.setOnClickListener(this);
    delete.setOnClickListener(this);

    pb = (ProgressBar)findViewById(R.id.progressBar);
    pb.setMax(100);
    pb.setProgress(0);

    //database is opend at mainActivity it's static
    downloadStatus=Integer.parseInt(MainActivity.prDb.intSearch(downloadId));// detects download status --> 0 is "notDownloadedYet" and
    // 1 is "downloading" and 2 is "downloaded"
    downloadLink=  MainActivity.puDb.intSearch(downloadId);//detects download link

    progressTask = new ProgressTask();


    if(downloadStatus==1){
            pb.setProgress(MainActivity.downloadedFile2SizePercent);//this code line works every 2nd and after dialog invoking
            progressTask.execute(true);
            Toast.makeText(c,"test task progress for 2nd started", Toast.LENGTH_SHORT).show();//this code line works every 2nd and afterdialog invoking
    }

}

@Override
public void onClick(View v) {

    switch (v.getId()) {
        case R.id.downloaddialot_downloadbtn:
                    FileDownloader.start(downloadLink); // download task starts here
                    progressTask.execute(true);
                    Toast.makeText(c,"download task progress for 1nd started", Toast.LENGTH_SHORT).show();
            break;

        case R.id.downloaddialot_deletebtn:
            if(downloadStatus==2){
                // delete codes
            }
            break;
    }
}


public  class ProgressTask extends AsyncTask<Boolean, Integer, Boolean> {
    @Override
    protected void onPreExecute() {
        super.onPreExecute();
    }

    @Override
    protected Boolean doInBackground(Boolean... params) {

        while (MainActivity.downloadedFile2SizePercent!=100){
              publishProgress(MainActivity.downloadedFile2SizePercent);
        }
        if(MainActivity.downloadedFile2SizePercent==100){
            publishProgress(MainActivity.downloadedFile2SizePercent);
        }

        return true;
    }

    @Override
    protected void onProgressUpdate(Integer... values) {
        pb.setProgress(values[0]);
    }

    @Override
    protected void onPostExecute(Boolean aBoolean) {
        super.onPostExecute(aBoolean);
        downloadStatus=2; //also saves in database by download listeners in mainActivity
    }

}

}

这是我的yarn-site.xml的片段。

有人可以建议需要修改哪些配置以保留失败作业的日志？

2017-07-13 14:16:04,170 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor (DeletionService #1): Deleting path : /var/log/hadoop-yarn/containers/application_1234567890_12345/container_11234567890_12345_11_0000
01/stdout
2017-07-13 14:16:04,180 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl (LogAggregationService #6093): renaming /var/log/hadoop-yarn/apps/hadoop/logs/application_1234567890_12345/xx.xx.xx.xx_8041.tmp to /var/log/hadoop-yarn/apps/hadoop/logs/application_1234567890_12345/xx.xx.xx.xx_8041
2017-07-13 14:16:04,181 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor (DeletionService #3): Deleting path : /var/log/hadoop-yarn/containers/application_1234567890_12345
2017-07-13 14:16:06,048 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Stopping resource-monitoring for container_11234567890_12345_11_0000

Answer 1

日志聚合完成后，日志将移至HDFS，通常是HDFS上的/ app-logs。

在the documentation

中检查以下设置

yarn.nodemanager.remote-app-log-dir 通常，HDFS上的/ app-logs，但在您的情况下，它设置为/ var / log / hadoop-yarn / apps，此目录在HDFS上是否存在？看来是本地目录值错误地放在这里。

其他可能有用的设置：

yarn.log-aggregation-enable：如果启用$ {yarn.log-aggregation-enable}，则NodeManager将立即将所有容器日志串联到一个文件中，并将它们上传到$ {yarn.nodemanager.remote-app-log-dir} / $ { user.name} / logs /并将其从本地userlogs目录中删除

yarn.nodemanager.delete.debug-delay-sec：应用程序完成后的秒数，节点管理器的DeletionService将删除应用程序的本地化文件目录和日志目录。要诊断Yarn应用程序问题，请将此属性的值设置得足够大（例如，设置为600 = 10分钟）以允许检查这些目录。更改属性的值后，必须重新启动nodemanager才能使其生效。

Hadoop Yarn容器日志丢失

1 个答案: