问题:这是我的第一个数据工厂,我可以/应该将多个数据复制管道复制到一个数据工厂中的一个数据集中吗?还是我需要为每个复制作业创建一个数据工厂。
如果我应该执行多个数据复制管道,那我在做错什么,那会使一个工作正常,而另一个写入同一数据集会失败?
详细信息 在Azure数据工厂中,我像从4个集成运行时链接的服务器中提取数据,然后将其更新到Azure突触服务器中。我为每个链接服务器设置了一个数据集,为天蓝色突触服务器设置了一个数据集。由于时区不同,每个流水线拉动都需要在不同的时间发生。第一条管道运作良好。其他管道失败,说明:
对目标副本Company2生产数据的操作失败:ErrorCode = SqlFailedToConnect,'Type = Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, 消息=无法连接到SQL数据库:'azure-synapse-sql01.database.windows.net', 数据库:“ GroupedDataDW”,用户:“ GroupedDataDWSQLAdmin”。检查链接的服务配置是否正确,并确保SQL数据库防火墙允许集成运行时访问。 provider:TCP提供程序,错误:0-连接尝试失败,因为一段时间后连接方未正确响应,或者连接建立失败,因为连接的主机未能响应。), Source = .Net SqlClient数据提供程序,SqlErrorNumber = 10060,Class = 20,ErrorCode = -2146232060,State = 0
同一数据集写入GroupedDataDW正在与Company1一起使用。
我尝试将指向GroupedDataDW的天蓝色数据接收器数据集复制为新数据集,但遇到相同的错误。
这是简单管道的JSON。它通过查询从COMPANY服务器中提取数据,该查询产生相同的数据粒度,并从GroupedDataDW中删除数据并刷新该数据作为终止和填充操作。我为每个公司创建了该管道的副本,但是只有第一个有效。
我本来是使用授权的身份为azure数据库设置授权的,这对于comapny1来说效果很好。我已将其更改为管理员帐户,以期解决托管身份的某些问题。
"name": "Copy Company2 Production Data",
"properties": {
"activities": [
{
"name": "Copy Data from Company2 to GroupedDataDW",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [
{
"name": "Source",
"value": "Company2SourceDataset"
},
{
"name": "Destination",
"value": "fact.PortfolioProduction"
}
],
"typeProperties": {
"source": {
"type": "SqlServerSource",
"sqlReaderQuery": "select Portfolio,\tDivisionName,\tCONVERT(date, CONVERT(varchar(8), DateKey), 112) as Date....<redacted>"
},
"sink": {
"type": "AzureSqlSink",
"preCopyScript": "Delete From [fact].[PortfolioProduction] where PortfolioName = 'Company2';",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [
{
"source": {
"name": "Portfolio",
"type": "String"
},
"sink": {
"name": "PortfolioName",
"type": "String"
}
},
{
"source": {
"name": "DivisionName",
"type": "String"
},
"sink": {
"name": "DivisionName",
"type": "String"
}
},
{
"source": {
"name": "Date",
"type": "Date"
},
"sink": {
"name": "Date",
"type": "DateTime"
}
},
{
"source": {
"name": "Written Premium",
"type": "Decimal"
},
"sink": {
"name": "Written Premium",
"type": "Decimal"
}
},
{
"source": {
"name": "Wholesale Comm",
"type": "Decimal"
},
"sink": {
"name": "Wholesale Commission",
"type": "Decimal"
}
},
{
"source": {
"name": "Retail Comm",
"type": "Decimal"
},
"sink": {
"name": "Retail Commission",
"type": "Decimal"
}
},
{
"source": {
"name": "Fees",
"type": "Decimal"
},
"sink": {
"name": "Fees",
"type": "Decimal"
}
}
]
}
},
"inputs": [
{
"referenceName": "Company2SourceDataset",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "GroupedDataDataset_for_Company2",
"type": "DatasetReference"
}
]
}
],
"annotations": []
},
"type": "Microsoft.DataFactory/factories/pipelines"
}```
Thanks for your time,
Mark