Question

使用BigQueryOperator将查询发布到BigQuery时，我面临挑战。气流版本：1.10.6

一段代码

args = {
    'owner': 'Airflow',
    'start_date': datetime(2019, 12, 17),
    'retries': 1,
}

dag = DAG(
    dag_id='my_dag',
    default_args=args,
    schedule_interval="@daily",
)

run_bq_cmd = BigQueryOperator (
    task_id='execute_bq_cmd',
    sql='SELECT 1 FROM `project.dataset.table` WHERE user=@user',
    destination_dataset_table=destination_tbl,
    use_legacy_sql=False,
    write_disposition='WRITE_APPEND',
    create_disposition='CREATE_IF_NEEDED',
    allow_large_results=True,
    query_params=[
        {
            "name": "user",
            "parameterType": { "type": "STRING" },
            "parameterValue": { "value": "Amol"}
        }
    ],
    dag=dag,
)

运行此dag后，我得到以下错误提示。

[2019-12-19 11:54:33,528] {taskinstance.py:1058} ERROR - BigQuery job failed. Final error was: {u'reason': u'invalidQuery', u'message': u'Syntax error: Unexpected identifier "S" at [1:1]', u'location': u'query'}. The job was: {u'status': {u'state': u'DONE', u'errors': [{u'reason': u'invalidQuery', u'message': u'Syntax error: Unexpected identifier "S" at [1:1]', u'location': u'query'}], u'errorResult': {u'reason': u'invalidQuery', u'message': u'Syntax error: Unexpected identifier "S" at [1:1]', u'location': u'query'}}, u'kind': u'bigquery#job', u'statistics': {u'endTime': u'1576756473354', u'creationTime': u'1576756473345', u'startTime': u'1576756473354'}, u'jobReference': {u'projectId': u'converged-dev-datahub', u'location': u'US', u'jobId': u'job_1qEYq2LaJVfSNSpGZ8e25bKOvpu3'}, u'etag': u'Q9k1LNPcvJ91NjuTD9Yo0w==', u'selfLink': u'https://bigquery.googleapis.com/bigquery/v2/projects/converged-dev-datahub/jobs/job_1qEYq2LaJVfSNSpGZ8e25bKOvpu3?location=US', u'configuration': {u'query': {u'useLegacySql': False, u'destinationTable': {u'projectId': u'converged-dev-datahub', u'tableId': u'fishhook_01_report', u'datasetId': u'dataset_for_amol'}, u'priority': u'INTERACTIVE', u'writeDisposition': u'WRITE_APPEND', u'allowLargeResults': True, u'queryParameters': [{u'parameterType': {u'type': u'STRING'}, u'parameterValue': {u'value': u'fishhook'}, u'name': u'partner'}], u'createDisposition': u'CREATE_IF_NEEDED', u'query': u'S'}, u'jobType': u'QUERY'}, u'id': u'converged-dev-datahub:US.job_1qEYq2LaJVfSNSpGZ8e25bKOvpu3', u'user_email': u'123350035192-compute@developer.gserviceaccount.com'}
Traceback (most recent call last):
  File "/root/miniconda2/lib/python2.7/site-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/root/miniconda2/lib/python2.7/site-packages/airflow/contrib/operators/bigquery_operator.py", line 268, in execute
    for s in self.sql]
  File "/root/miniconda2/lib/python2.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line 913, in run_query
    return self.run_with_configuration(configuration)
  File "/root/miniconda2/lib/python2.7/site-packages/airflow/contrib/hooks/bigquery_hook.py", line 1344, in run_with_configuration
    format(job['status']['errorResult'], job))

虽然很奇怪，但是我只能看到查询的开始字符（即SELECT的“ S”）传递给BigQuery。

在气流日志中，我可以正确看到查询

[2019-12-19 11:54:33,083] {bigquery_operator.py:218} INFO - Executing: SELECT 1 FROM `project.dataset.table` WHERE user=@user

在指定BigQueryOperator时是否缺少某些内容？

使用Airflow的BigQueryOperator无法将查询传递给BigQuery

0 个答案: