Question

我正在尝试使用python将本地CSV上传到Google大查询

def uploadCsvToGbq(self,table_name):


    load_config = {
    'destinationTable': {
    'projectId': self.project_id,
    'datasetId': self.dataset_id,
    'tableId': table_name
    }
    }

    load_config['schema'] = {
    'fields': [
    {'name':'full_name', 'type':'STRING'},
    {'name':'age', 'type':'INTEGER'},
    ]
    }
    load_config['sourceFormat'] = 'CSV'

    upload = MediaFileUpload('sample.csv',
                     mimetype='application/octet-stream',
                     # This enables resumable uploads.
                     resumable=True)
    start = time.time()
    job_id = 'job_%d' % start
    # Create the job.
    result = bigquery.jobs.insert(
    projectId=self.project_id,
    body={
    'jobReference': {
    'jobId': job_id
    },
    'configuration': {
    'load': load_config
    }
    },
    media_body=upload).execute()

    return result

当我运行它时会抛出错误，如

“NameError：全局名称'MediaFileUpload'未定义”

是否需要任何模块请帮忙。

Answer 1

pip install --upgrade google-api-python-client

然后在你的python文件顶部写：

from googleapiclient.http import MediaFileUpload

但是你要小心一些括号。更好写：

result = bigquery.jobs().insert(projectId=PROJECT_ID, body={'jobReference': {'jobId': job_id},'configuration': {'load': load_config}}, media_body=upload).execute(num_retries=5)

顺便说一句，您要上传所有CSV行，包括定义列的顶行。

Answer 2

MediaFileUpload类位于http.py中。见https://google-api-python-client.googlecode.com/hg/docs/epy/apiclient.http.MediaFileUpload-class.html

Answer 3

上传到GBQ中的csv文件的最简单方法之一是通过熊猫。只需将csv文件导入到熊猫（pd.read_csv（））。然后从熊猫到GBQ（df.to_gbq（full_table_id，project_id = project_id））。

import pandas as pd
import csv
df=pd.read_csv('/..localpath/filename.csv')
df.to_gbq(full_table_id, project_id=project_id)

或者您可以使用客户端api

from google.cloud import bigquery
import pandas as pd
df=pd.read_csv('/..localpath/filename.csv')
client = bigquery.Client()
dataset_ref = client.dataset('my_dataset')
table_ref = dataset_ref.table('new_table')
client.load_table_from_dataframe(df, table_ref).result()

如何使用python将本地CSV上传到谷歌大查询

3 个答案: