qstat和长工作名称

时间:2014-09-29 16:02:41

标签: linux sungridengine bash4

如何让qstat给我完整的工作名称?

我知道qstat -r提供了有关任务的详细信息,但它太多了,并且包含了资源要求。

qstat -r输出如下:

 131806 0.25001 tumor_foca ajalali      qw    09/29/2014 15:49:41                                    1 2-100:1
       Full jobname:     tumor_focality-TCGA-THCA-ratboost_linear_svc
       Hard Resources:   distribution=wheezy (0.000000)
                         h_rt=72000 (0.000000)
                         mem_free=15G (0.000000)
                         h_vmem=15G (0.000000)
                         h_stack=256M (0.000000)
       Soft Resources:   
 131807 0.25001 vital_stat ajalali      qw    09/29/2014 15:49:41                                    1 2-100:1
       Full jobname:     vital_status-TCGA-LGG-ratboost_linear_svc
       Hard Resources:   distribution=wheezy (0.000000)
                         h_rt=72000 (0.000000)
                         mem_free=15G (0.000000)
                         h_vmem=15G (0.000000)
                         h_stack=256M (0.000000)
       Soft Resources:   

现在我唯一的选择是根据需要grep输出:

$ qstat -r | grep "Full jobname" -B1
--
 131806 0.25001 tumor_foca ajalali      qw    09/29/2014 15:49:41                                    1 2-100:1
       Full jobname:     tumor_focality-TCGA-THCA-ratboost_linear_svc
--
 131807 0.25001 vital_stat ajalali      qw    09/29/2014 15:49:41                                    1 2-100:1
       Full jobname:     vital_status-TCGA-LGG-ratboost_linear_svc

我可以做得更好,以获得更好的输出吗?

9 个答案:

答案 0 :(得分:32)

这有点乱,但它可以作为命令历史记录中的简单解决方案。所有标准工具。输出几乎与普通qstat调用的输出相同,但是你不会得到标题:

<强>一衬垫:

qstat -xml | tr '\n' ' ' | sed 's#<job_list[^>]*>#\n#g' \
  | sed 's#<[^>]*>##g' | grep " " | column -t

命令说明:

将作业列为XML:

qstat -xml

删除所有换行符:

tr '\n' ' '

在列表中的每个作业条目之前添加换行符:

sed 's#<job_list[^>]*>#\n#g'

删除所有XML内容:

sed 's#<[^>]*>##g'

哈克在最后添加换行符:

grep " "

Columnize:

column -t

示例输出

351996  0.50502  ProjectA_XXXXXXXXX_XXXX_XXXXXX                user123  r   2015-06-25T15:38:41  xxxxx-sim01@xxxxxx02.xxxxx.xxx  1
351997  0.50502  ProjectA_XXX_XXXX_XXX                         user123  r   2015-06-25T15:39:26  xxxxx-sim01@xxxxxx23.xxxxx.xxx  1
351998  0.50502  ProjectA_XXXXXXXXXXXXX_XXXX_XXXX              user123  r   2015-06-25T15:40:26  xxxxx-sim01@xxxxxx14.xxxxx.xxx  1
351999  0.50502  ProjectA_XXXXXXXXXXXXXXXXX_XXXX_XXXX          user123  r   2015-06-25T15:42:11  xxxxx-sim01@xxxxxx19.xxxxx.xxx  1
352001  0.50502  ProjectA_XXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXX    user123  r   2015-06-25T15:42:11  xxxxx-sim01@xxxxxx11.xxxxx.xxx  1
352008  0.50501  runXXXX69                                     usr1     r   2015-06-25T15:49:04  xxxxx-sim01@xxxxxx17.xxxxx.xxx  1
352009  0.50501  runXXXX70                                     usr1     r   2015-06-25T15:49:04  xxxxx-sim01@xxxxxx01.xxxxx.xxx  1
352010  0.50501  runXXXX71                                     usr1     r   2015-06-25T15:49:04  xxxxx-sim01@xxxxxx06.xxxxx.xxx  1
352011  0.50501  runXXXX72                                     usr1     r   2015-06-25T15:49:04  xxxxx-sim01@xxxxxx21.xxxxx.xxx  1
352012  0.50501  runXXXX73                                     usr1     r   2015-06-25T15:49:04  xxxxx-sim01@xxxxxx13.xxxxx.xxx  1
352013  0.50501  runXXXX74                                     usr1     r   2015-06-25T15:49:04  xxxxx-sim01@xxxxxx11.xxxxx.xxx  1

答案 1 :(得分:8)

这个脚本效果很好。它看起来像是来自剑桥。 http://www.hep.ph.ic.ac.uk/~dbauer/grid/myqstat.py

对于Python 3:

#!/usr/bin/python
import xml.dom.minidom
import os
import sys
import string    

f=os.popen('qstat -u \* -xml -r')

dom=xml.dom.minidom.parse(f)


jobs=dom.getElementsByTagName('job_info')
run=jobs[0]

runjobs=run.getElementsByTagName('job_list')


def fakeqstat(joblist):
    for r in joblist:
        try:
            jobname=r.getElementsByTagName('JB_name')[0].childNodes[0].data
            jobown=r.getElementsByTagName('JB_owner')[0].childNodes[0].data
            jobstate=r.getElementsByTagName('state')[0].childNodes[0].data
            jobnum=r.getElementsByTagName('JB_job_number')[0].childNodes[0].data
            jobtime='not set'
            if(jobstate=='r'):
                jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
            elif(jobstate=='dt'):
                jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
            else:
                jobtime=r.getElementsByTagName('JB_submission_time')[0].childNodes[0].data

            print(jobnum, '\t', jobown.ljust(16), '\t', jobname.ljust(16),'\t', jobstate,'\t',jobtime)
        except Exception as e:
            print(e)

fakeqstat(runjobs)

对于Python 2:

#!/usr/bin/python
import xml.dom.minidom
import os
import sys
import string
#import re


f=os.popen('qstat -u \* -xml -r')

dom=xml.dom.minidom.parse(f)


jobs=dom.getElementsByTagName('job_info')
run=jobs[0]

runjobs=run.getElementsByTagName('job_list')


def fakeqstat(joblist):
        for r in joblist:
                jobname=r.getElementsByTagName('JB_name')[0].childNodes[0].data
                jobown=r.getElementsByTagName('JB_owner')[0].childNodes[0].data
                jobstate=r.getElementsByTagName('state')[0].childNodes[0].data
                jobnum=r.getElementsByTagName('JB_job_number')[0].childNodes[0].data
                jobtime='not set'
                if(jobstate=='r'):
                        jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
                elif(jobstate=='dt'):
                        jobtime=r.getElementsByTagName('JAT_start_time')[0].childNodes[0].data
                else:
                        jobtime=r.getElementsByTagName('JB_submission_time')[0].childNodes[0].data



                print  jobnum, '\t', jobown.ljust(16), '\t', jobname.ljust(16),'\t', jobstate,'\t',jobtime


fakeqstat(runjobs)

答案 2 :(得分:4)

也许是一个更简单的解决方案:将SGE_LONG_JOB_NAMES设置为-1,qstat将计算出name列的大小:

export SGE_LONG_JOB_NAMES=-1
qstat -u username

适合我。

干杯!

答案 3 :(得分:3)

我目前正在编写自己的qstat包装器,以获得干净,实用且可自定义的输出。

这是github repository。该项目已经增长太多,无法将代码粘贴到此消息中。

它带有一个安装程序,对Python 2.7和3都应该没有任何问题(如果需要,安装脚本会进行修改)。 qjobs -h为可用选项提供了一些帮助。我将在接下来的日子里在github wiki上写一个更完整的文档。

我会尽可能经常更新此消息以坚持项目的当前状态。请随时在这里(或在github上)发表评论以询问功能/报告问题。

在不久的将来,我将尝试添加一个完全交互模式,以便更轻松地浏览作业列表。当然,经典文本输出仍然可用(通过电子邮件发送输出或快速检查挂起/正在运行的作业可能很有用。)

示例输出

命令qjobs给出:

5599109   short_name        r    2015-06-25 10:27:39   queue1
5599110   jobName           r    2015-06-25 10:35:39   queue2
5599111   a_long_job_name   qw   2015-06-25 10:40:39
5599112   foo               qw   2015-06-25 10:40:39
5599113   bar               qw   2015-06-25 10:40:39
5599114   baz               qw   2015-06-25 10:40:39
5599115   beer              qw   2015-06-25 10:40:39

tot: 7

r: 2   qw: 5

命令qjobs -o给出:

tot: 7

r: 2   qw: 5

命令qjobs -o inek -t给出(e是从开始/子时间开始经过的时间,格式可以使用Python的Format Spec. Mini-Language自定义; k是完整的队列名称,域):

5598985   SpongeBob        522:02 (21.75 days)   queue1@node23.domain.fake
5598987   ping_java        521:47 (21.74 days)   queue1@node39.domain.fake
5598988   run3.14          521:46 (21.74 days)   queue2@node40.domain.fake
5598990   strange_job_42   521:42 (21.74 days)   queue3@node36.domain.fake
5598991   coffee-maker     521:39 (21.74 days)   queue2@node34.domain.fake
5598992   dumbtask         521:29 (21.73 days)   queue1@node14.domain.fake

qjobs -i提供了可用&#39;项目的完整列表。每个项目都可用:

  • 列输出(带-o ITEMS);
  • 作为计算作业和产生总产出的标准,-t(例如-t s按州计算,如前两个示例所示);
  • 作为使用-s对作业进行排序的标准,默认为-s ips,表示作业列表按ID排序,然后按优先级排序,最后按状态排序。

qjobs -i的结果是:

i: job id
p: job priority
n: job name
o: job owner
s: job state
t: job start/submission time
e: elapsed time since start/submission
q: queue name without domain
d: queue domain
k: queue name with domain
r: requested queue(s)
l: number of slots used

答案 4 :(得分:1)

感谢JLT的简单代码。我已经扩展了一点以满足我的需求,让它看起来很漂亮。

示例输出:

Job ID             Job Name                   Owner   Status  
------  ------------------------------------  ------  ------  
201716  AtacSilN100400K                       mtsige  R       
201771  IsoOnGrap400K                         mtsige  R       
202067  AtacOnSilica400K                      mtsige  R       
202100  AtacGrapN100400K                      mtsige  R       
202135  AtacOnSilc400K                        mtsige  R       
202145  AtacOnGrap400K                        mtsige  R       
202152  AtacOnGraphN3360K                     mtsige  R       
202161  AtacticSilicaN10                      mtsige  R       
202163  AtacGrapN10                           mtsige  R       
202169  AtacSilcN10                           mtsige  R       
202192  wallpmma07                            am110   R       
202193  wallpmma03                            am110   R       
202194  att03wpm_95solps                      am110   R       
202202  AtacticSilicaN3                       mtsige  R       
203260  8test18_trop_2p                       ico     R       
203359  parseAll_Bob/Sub951By50/Cyl20A_atom1  oge1    R       
203360  parseAll_Bob/Sub951By50/Cyl30A_atom1  oge1    R       
203361  parseAll_Bob/Sub951By50/Cyl30A_atom2  oge1    R      

代码:

#!/opt/bin/python3
import os
import xml.etree.ElementTree as ET

#Fields
fields=['Job_Id','Job_Name','Job_Owner','job_state']
names=['Job ID','Job Name','Owner','Status']

#Get job info
f = os.popen('qstat -x')
tree = ET.parse(f)
root = tree.getroot()
n_fields=len(fields)
jobs=[[job.find(field).text for field in fields] for job in root]
max_lengths=[len(name) for name in names]
sep='  '

#Identify max characer length per field
for j in jobs:
    for i in range(n_fields):
            #Chop off anything after and including '@' or '.' from all fields
            if j[i].find('@')>0:
                    j[i]=j[i][:j[i].find('@')]
            if j[i].find('.')>0:
                    j[i]=j[i][:j[i].find('.')]
            if(len(j[i])>max_lengths[i]):
                    max_lengths[i]=len(j[i])

#Field names
for i in range(n_fields):
    print('{s:^{length}}'.format(s=names[i],length=max_lengths[i]),end=sep)
print()

#Dashes
for i in range(n_fields):
    print('-'*max_lengths[i],end=sep)
print()

#Jobs
for j in jobs:
    for i in range(n_fields):
            if j[i].find('@')>0:
                    j[i]=j[i][:j[i].find('@')]
            print('{s:<{length}}'.format(s=j[i],length=max_lengths[i]),end=sep)
    print()

答案 5 :(得分:0)

对我来说,物理化学家的脚本没有用,所以我用xml.tree.ElementTree模块编写了一个非常简单的脚本,我认为它比xml.dom.minidom

更容易
import os
import xml.etree.ElementTree as ET
f = os.popen('qstat -x')
tree = ET.parse(f)
root = tree.getroot()
print "Job_Id   walltime state     nodes       Job_Name"
print "------   -------- ----- --------------- --------------------------"
for job in root:
    print job.find('Job_Id').text, " ",
    print job.find('resources_used').find('walltime').text, " ",
    print job.find('job_state').text, " ",
    print job.find('Resource_List').find('nodes').text, " ",
    print job.find('Job_Name').text

答案 6 :(得分:0)

糟糕的KISS解决方案:

qstat -xml -f -u \* | fgrep JB_name | wc -l

答案 7 :(得分:0)

python代码

import xmltodict
import subprocess as sp
import pandas as pd

qstat_xml = sp.check_output(['qstat','--xml'], stderr=sp.STDOUT)  # read xml
stat_dict = xmltodict.parse(qstat_xml) # convert to dict
job_list = stat_dict['Data']['Job'] # select job_list
job_df = pd.DataFrame(job_list) # convert to dataframe
print('columns', job_df.columns) # print available columns
column_list = ['Job_Id', 'Job_Name']
selection_df = job_df[column_list]  # select columns
print(selection_df)

答案 8 :(得分:0)

如果您只想要名称:

qstat -f | grep 'Job_Name'

输出示例:

Job_Name = File.output
Job_Name = file.out