Question

我正在尝试在Big Query中加载嵌套的json数据。

以下是我正在使用的数据和架构..

scehma - ＆gt;

[{＆＃34;名称＆＃34;：＆＃34;种类＆＃34;＆＃34;类型＆＃34;：＆＃34;串＆＃34;}，{＆＃34;名称＆＃34 ;：＆＃34;全名＆＃34;＆＃34;类型＆＃34;：＆＃34;串＆＃34;}，{＆＃34;名称＆＃34;：＆＃34;年龄＆＃34;，＆＃34;类型＆＃34;：＆＃34;整数＆＃34;}，{＆＃34;名称＆＃34;：＆＃34; citiesLived＆＃34;＆＃34;类型＆＃34;：＆＃34;记录＆＃34;＆＃34;字段＆＃34;：[{＆＃34;名称＆＃34;：＆＃34;地方＆＃34;＆＃34;类型＆＃34;：＆＃34;串＆＃34 ;}，{＆＃34;名称＆＃34;：＆＃34; numberOfYears＆＃34;＆＃34;类型＆＃34;：＆＃34;整数＆＃34;}]}]

数据 - ＆GT;它存在于gc_data / load_data.json

中

{＆＃34; kind＆＃34;：＆＃34; person＆＃34;，＆＃34; fullName＆＃34;：＆＃34; John Doe＆＃34;，＆＃34; age＆＃34;： 22，＆＃34; citiesLived＆＃34;：[{＆＃34; place＆＃34;：＆＃34; Seattle＆＃34;，＆＃34; numberOfYears＆＃34;：5}，{＆＃34; place＆＃ 34;：＆＃34;斯德哥尔摩＆＃34;，＆＃34; numberOfYears＆＃34;：6}]} {＆＃34;善良＆＃34;：＆＃34;人＆＃34;，＆＃34; fullName＆＃34;：＆＃34; Jane Austen＆＃34;，＆＃34;年龄＆＃34;：24，＆＃34; citiesLived＆＃34;：[{＆＃34; place＆＃34;：＆＃34; Los Angeles＆＃34;，＆＃34; numberOfYears＆＃34;：2}，{＆＃34; place＆＃34; ：＆＃34; Tokyo＆＃34;，＆＃34; numberOfYears＆＃34;：2}]}

try {
    bigquery.datasets().insert(PROJECT_ID, dataset).execute();
} catch (IOException e) {
    System.out.println(e);
}

  // Set where you are importing from (i.e. the Google Cloud Storage paths).
  List<String> sources = new ArrayList<String>();
  sources.add("gs://gc_data/json_test_new_flat.json");
  loadConfig.setSourceUris(sources);
  loadConfig.setSourceFormat("NEWLINE_DELIMITED_JSON");
  //loadConfig.setFieldDelimiter("\n");

  // Describe the resulting table you are importing to:
  TableReference tableRef = new TableReference();
  tableRef.setDatasetId("myDataset");
  tableRef.setTableId("myTableJSONNew");
  tableRef.setProjectId(projectId);
  loadConfig.setDestinationTable(tableRef);

  List<TableFieldSchema> fields = new ArrayList<TableFieldSchema>();
  TableFieldSchema fieldKind = new TableFieldSchema();
  fieldKind.setName("kind");
  fieldKind.setType("STRING");
  TableFieldSchema fieldFullName = new TableFieldSchema();
  fieldFullName.setName("fullName");
  fieldFullName.setType("STRING");

  TableFieldSchema fieldAge = new TableFieldSchema();
  fieldAge.setName("age");
  fieldAge.setType("INTEGER");


  TableFieldSchema fieldJSON = new TableFieldSchema();
  fieldJSON.setName("citiesLived");
  fieldJSON.setType("RECORD");

  // this is for record
  List<TableFieldSchema> listOfJSonSchema = new ArrayList<TableFieldSchema>();
  TableFieldSchema fieldPlace = new TableFieldSchema();
  fieldPlace.setName("place");
  fieldPlace.setType("STRING");

  TableFieldSchema fieldnumberOfYears = new TableFieldSchema();
  fieldnumberOfYears.setName("numberOfYears");
  fieldnumberOfYears.setType("INTEGER");
  listOfJSonSchema.add(fieldPlace);
  listOfJSonSchema.add(fieldnumberOfYears);
  //



  fieldJSON.setFields(listOfJSonSchema);

  fields.add(fieldKind);
  fields.add(fieldFullName);
  fields.add(fieldAge);
  fields.add(fieldJSON);
  TableSchema schema = new TableSchema();
  schema.setFields(fields);  // This is to set delimiter

  loadConfig.setSchema(schema);

  Insert insert = bigquery.jobs().insert(projectId, job);
  insert.setProjectId(projectId);
  JobReference jobRef =  insert.execute().getJobReference();
  System.out.println(jobRef.toPrettyString());

Answer 1

您已经定义了citiesLived记录，但看起来您没有指出记录是可重复的，这是您的示例数据所指示的。尝试在citiesLived字段上使用setMode（“REPEATED”）。

无法在BigQuery中加载嵌套的Json数据

1 个答案: