如何在特定的“完成”状态下获得工作

时间:2019-02-15 04:44:56

标签: python google-bigquery

我正在尝试确定各种工作状态。 Bigquery提供了我知道的三种状态:DONE, PENDING, and RUNNING。但是,我正在尝试根据以下条件获取状态:

  • 完成
  • 待处理
  • 成功
  • 错误
  • 已取消
  • 跑步

我将如何以不太“昂贵”的方式执行此操作,因为我以一种“长轮询”方式迭代大约100个结果,大约每十秒钟一次。目前我正在做类似的事情:

jobs = [job for job in self.bq_client.list_jobs(project=PROJECT_ID]
if state is not None:
    jobs = [job for job in jobs if job.state == state]

如果状态为“完成”,“正在运行”或“正在挂起”之一,则上述方法适用。但是我将如何覆盖其他州?

1 个答案:

答案 0 :(得分:2)

状态跟踪作业进度,如果您需要成功/失败信息,则需要查看response中的errorResult。对于成功的工作,它将为None,对于已取消的工作,您将得到{u'reason': u'stopped', u'message': u'Job execution was cancelled: User requested cancellation'}。我用来测试的代码:

from google.cloud import bigquery
client = bigquery.Client()

project = "[PROJECT-ID]"
states = ["RUNNING", "PENDING", "SUCCESSFUL", "CANCELLED", "FAILED"]


def returnState(job):
  if job.state == "DONE":
    if job.error_result is None:
      return "SUCCESSFUL"
    elif job.error_result['reason'] == u'stopped':
      return "CANCELLED"
    else:
      return "FAILED"
  else:
    return job.state


jobs = [job for job in client.list_jobs(project=project, max_results=10)]

for state in states:
  matching_jobs = [job for job in jobs if returnState(job) == state]

  for job in matching_jobs:
    print "Job ID: {0}, State: {1}, Error Result: {2}".format(job.job_id, state, job.error_result)

这将打印出如下内容:

$ python bq-status.py
Job ID: bquijob_..., State: SUCCESSFUL, Error Result: None
Job ID: bquijob_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: job_..., State: SUCCESSFUL, Error Result: None
Job ID: scheduled_query_..., State: SUCCESSFUL, Error Result: None
Job ID: bquijob_..., State: SUCCESSFUL, Error Result: None
Job ID: bquijob_..., State: CANCELLED, Error Result: {u'reason': u'stopped', u'message': u'Job execution was cancelled: User requested cancellation'}
Job ID: bquijob_..., State: FAILED, Error Result: {u'reason': u'invalidQuery', u'message': u'Syntax error: Illegal input character "\\\\" at [2:18]', u'location': u'query'}

请记住,加载作业可能会成功,但是允许一些maxBadRecords以便errorResult不会为空,等等。