如何使用bq load命令加载多个AVRO文件

Question

我正在尝试根据此文档将多个AVRO文件加载到大查询中：

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro

根据文档，执行此操作的命令是：

bq --location=US load --source_format=AVRO [DATASET].[TABLE_NAME] "gs://mybucket/00/*.avro","gs://mybucket/01/*.avro"

我为搜索文件创建了一个脚本，并安装了如下命令：

bq load --source_format=AVRO --noreplace foo.bar$123456  "gs://mybucket/foo/36.avro", "gs://mybucket/foo_bar/01.avro", "gs://mybucket/bar/211.avro"

但这仅在我有一个这样的文件时起作用：

bq load --source_format=AVRO --noreplace foo.bar$123456 "gs://mybucket/foo/36.avro"

当我尝试对多个文件使用命令时，错误是：

Too many positional args, still have ["gs://mybucket/foo_bar/01.avro"]

它是我的脚本来创建命令：

def create_command_bq_load(buckets):
    for x, bucket in enumerate(buckets):
        command =  'bq load --source_format=AVRO --noreplace %s.%s_%s$%s' % (datasetname,  bucket['product'], bucket['event'],  bucket['data_partition'])
        if bucket['files']:
            command_file = ''
            for x in range(len(bucket['files'])):    
                command_file = '%s "%s",' % (command_file, bucket['files'][x])   
                command_file = command_file
            commands.append((command + ' ' + command_file)[:-1])
    return commands

有帮助吗？

Answer 1

解决了，我的错误是两个文件之间的空格字符'。正确的方法是：

var query = await _db.Countries .Include(c => c.Cities) .Include(c => c.Mountains) .Include(c => c.Rivers) .Where(c => c.Cities.Any(city => city.Population > 10000) || c.Mountains.Any(mountain => mountain.Heigh > 1000) || c.River.Any(river => river.Length > 100000)) .Where(c => c.Cities.Any() || c.Mountains.Any() || c.Rivers.Any()) .ToListAsync(); var totalCount = query.Count(); var countries = query .Select(country => new CountryDto { CountryName = country.Name, CityDtos = country.Citites .Select(city => new CityDto { Name = city.Name, }), MountainDtos = country.Mountains .Select(mountain => new MountainDto { Name = mountain.Name, }), RiverDtos = country.Rivers .Select(river => new RiverDto { Name = river.Name, }), }) .ToList();