Question

I'm trying to send a csv to bigquery. The columns in the file are delimited with # and newlines are delimited with \n. But I'm getting this error from BigQuery:

RuntimeError: [{u'reason': u'invalid', u'message': u'Too many errors encountered.'}, {u'reason': u'invalid', u'message': u'Too many values in row starting at position: 0.', u'location': u'file-00000000'}]

This is my code:

bigquery_client = bigquery.Client.from_service_account_json(CREDENTIALS_BIGQUERY, 'xxx-yyy')

dataset_name = 'pixel_logs'
table_name = 'full_logs'
source_file_name = '/tmp/logstash-bq/text.csv'
dataset = bigquery_client.dataset(dataset_name)
table = dataset.table(table_name)

table.reload()

with open(source_file_name,'rb') as source_file:
    job = table.upload_from_file(source_file,encoding='UTF-8'
        ,source_format='text/csv'
        ,field_delimiter='#'
        ,allow_quoted_newlines=True)

This is my file:

2017-04-10T01:40:24.329Z#9decaf95-20a5-428e-a3ca-50485edb9f9f#1-fg4fuqed-j0hzl5q2#product_pageview#1#http://dev.xxx.com.br/xxx/#http://dev.xxx.com.br/#1491758180677#UTF-8#1600x900#1600x236#24#xxx%#Chrome%2057#false#Mozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_11_3)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/57.0.2987.133%20Safari/537.36\n2017-04-10T01:40:45.069Z#9decaf95-20a5-428e-a3ca-50485edb9f9f#1-fg4fuqed-j0hzl5q2#product_pageview#1#http://dev.xxx.com.br/#http://dev.xxx.com.br/#1491758180677#UTF-8#1600x900#1600x236#24#xxx#Chrome%2057#false#Mozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_11_3)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/57.0.2987.133%20Safari/537.36\n

Python and BigQuery are not recognizing \n as new line in the file. How can I fix this?

Answer 1

csv是从logstash输出创建的。我刚修复了logstash csv输出。 Python不是问题所在。

Python not recognizing new lines in csv files

1 个答案: