Question

我在Databricks中有一个DataFrame，我想用它在Cosmos中创建图形，DataFrame中的一行等于Cosmos中的1个顶点。

当我写Cosmos时，我看不到顶点上的任何属性，只是生成的ID。

获取数据：

data = spark.sql("select * from graph.testgraph")

配置：

writeConfig = {
 "Endpoint" : "******",
 "Masterkey" : "******",
 "Database" : "graph",
 "Collection" : "TestGraph",
 "Upsert" : "true",
 "query_pagesize" : "100000",
 "bulkimport": "true",
 "WritingBatchSize": "1000",
 "ConnectionMaxPoolSize": "100",
 "partitionkeydefinition": "/id"
}

写给宇宙：

data.write.
  format("com.microsoft.azure.cosmosdb.spark").
  options(**writeConfig).
  save()

Answer 1

下面是将记录插入到cosmos DB中的工作代码。转到以下站点，单击下载选项，然后选择uber.jar https://search.maven.org/artifact/com.microsoft.azure/azure-cosmosdb-spark_2.3.0_2.11/1.2.2/jar然后添加您的依赖项

spark-shell --master yarn --executor-cores 5 --executor-memory 10g --num-executors 10 --driver-memory 10g --jars“ path / to / jar / dependency / azure-cosmosdb- spark_2.3.0_2.11-1.2.2-uber.jar”-打包“ com.google.guava：guava：18.0，com.google.code.gson：gson：2.3.1，com.microsoft.azure：azure -documentdb：1.16.1“

import org.apache.spark.sql.types._
import org.apache.spark.sql.Row

val data = Seq(
Row(2, "Abb"),
Row(4, "Bcc"),
Row(6, "Cdd")
)

val schema = List(
StructField("partitionKey", IntegerType, true),
StructField("name", StringType, true)
)

val DF = spark.createDataFrame(
spark.sparkContext.parallelize(data),
StructType(schema)
)

val writeConfig = Map("Endpoint" -> "https://*******.documents.azure.com:443/",
"Masterkey" -> "**************",
"Database" -> "db_name",
"Collection" -> "collection_name",
"Upsert" -> "true",
"query_pagesize" -> "100000",
"bulkimport"-> "true",
"WritingBatchSize"-> "1000",
"ConnectionMaxPoolSize"-> "100",
"partitionkeydefinition"-> "/partitionKey")

DF.write.format("com.microsoft.azure.cosmosdb.spark").mode("overwrite").options(writeConfig).save()

从Databricks（Apache Spark）写入Cosmos DB Graph API

1 个答案: