Azure DataFactory链活动

时间:2017-02-08 23:35:43

标签: azure azure-data-factory

我对DataFactory非常陌生,并且在执行复制功能之前遇到了如何正确创建将执行存储过程的管道的问题。

存储过程只是目标表的TRUNCATE,用作第二个活动中的输出数据集。

从DataFactory文档中,它告诉我要先执行存储过程,将proc的“输出”指定为第二个活动的“输入”。

但是,存储过程中没有真正的“输出”。为了使其“正常工作”,我克隆了第二个活动的输出,更改了它的名称并使其external=false使其超过了配置错误,但这显然是一个完全的问题。

对我来说没有意义,至少在这个存储过程执行TRUNCATE动作的情况下,为什么甚至需要定义一个输出。

但是,当我尝试使用存储过程的输出作为附加输入时,我收到有关重复表名的错误。

如何在运行复制活动之前让TRUNCATE存储的proc活动成功执行(并完成)?

这是管道代码:

{
    "name": "Traffic CRM - System User Stage",
    "properties": {
        "description": "Move System User to Stage",
        "activities": [
            {
                "type": "SqlServerStoredProcedure",
                "typeProperties": {
                    "storedProcedureName": "dbo.usp_Truncate_Traffic_Crm_SystemUser",
                    "storedProcedureParameters": {}
                },
                "outputs": [
                    {
                        "name": "Smart App - usp Truncate System User"
                    }
                ],
                "policy": {
                    "timeout": "01:00:00",
                    "concurrency": 1,
                    "retry": 3
                },
                "scheduler": {
                    "frequency": "Day",
                    "interval": 1
                },
                "name": "Smart App - SystemUser Truncate"
            },
            {
                "type": "Copy",
                "typeProperties": {
                    "source": {
                        "type": "SqlSource",
                        "sqlReaderQuery": "select * from [dbo].[Traffic_Crm_SystemUser]"
                    },
                    "sink": {
                        "type": "SqlSink",
                        "writeBatchSize": 0,
                        "writeBatchTimeout": "00:00:00"
                    },
                    "translator": {
                        "type": "TabularTranslator",
                        "columnMappings": "All columns mapped here"
                    }
                },
                "inputs": [
                    {
                        "name": "Traffic CRM - SytemUser Stage"
                    }
                ],
                "outputs": [
                    {
                        "name": "Smart App - System User Stage Production"
                    }
                ],
                "policy": {
                    "timeout": "1.00:00:00",
                    "concurrency": 1,
                    "executionPriorityOrder": "NewestFirst",
                    "style": "StartOfInterval",
                    "retry": 3,
                    "longRetry": 0,
                    "longRetryInterval": "00:00:00"
                },
                "scheduler": {
                    "frequency": "Day",
                    "interval": 1
                },
                "name": "Activity-0-[dbo]_[Traffic_Crm_SystemUser]->[dbo]_[Traffic_Crm_SystemUser]"
            }
        ],
        "start": "2017-01-19T14:30:57.309Z",
        "end": "2099-12-31T05:00:00Z",
        "isPaused": false,
        "hubName": "stagingdatafactory1_hub",
        "pipelineMode": "Scheduled"
    }
}

1 个答案:

答案 0 :(得分:2)

您的SP活动输出数据集即“名称”:应为下一个活动输入“智能应用程序 - usp截断系统用户”。如果您对数据集中的内容感到困惑,只需创建一个虚拟数据集,如下所示

{
    "name": "DummySPDS",
    "properties": {
        "published": false,
        "type": "SqlServerTable",
        "linkedServiceName": "SQLServerLS",
        "typeProperties": {
            "tableName": "dummyTable"
        },
        "availability": {
            "frequency": "Hour",
            "interval": 1
        },
        "IsExternal":"True"
    }
}

这是完整的管道代码

{
    "name": "Traffic CRM - System User Stage",
    "properties": {
        "description": "Move System User to Stage",
        "activities": [
            {
                "type": "SqlServerStoredProcedure",
                "typeProperties": {
                    "storedProcedureName": "dbo.usp_Truncate_Traffic_Crm_SystemUser",
                    "storedProcedureParameters": {}
                },
                "inputs": [
                    {
                        "name": "DummySPDS"
                    }
                ],
                "outputs": [
                    {
                        "name": "Smart App - usp Truncate System User"
                    }
                ],
                "policy": {
                    "timeout": "01:00:00",
                    "concurrency": 1,
                    "retry": 3
                },
                "scheduler": {
                    "frequency": "Day",
                    "interval": 1
                },
                "name": "Smart App - SystemUser Truncate"
            },
            {
                "type": "Copy",
                "typeProperties": {
                    "source": {
                        "type": "SqlSource",
                        "sqlReaderQuery": "select * from [dbo].[Traffic_Crm_SystemUser]"
                    },
                    "sink": {
                        "type": "SqlSink",
                        "writeBatchSize": 0,
                        "writeBatchTimeout": "00:00:00"
                    },
                    "translator": {
                        "type": "TabularTranslator",
                        "columnMappings": "All columns mapped here"
                    }
                },
                "inputs": [
                    {
                        "name": "Smart App - usp Truncate System User"
                    }
                ],
                "outputs": [
                    {
                        "name": "Smart App - System User Stage Production"
                    }
                ],
                "policy": {
                    "timeout": "1.00:00:00",
                    "concurrency": 1,
                    "executionPriorityOrder": "NewestFirst",
                    "style": "StartOfInterval",
                    "retry": 3,
                    "longRetry": 0,
                    "longRetryInterval": "00:00:00"
                },
                "scheduler": {
                    "frequency": "Day",
                    "interval": 1
                },
                "name": "Activity-0-[dbo]_[Traffic_Crm_SystemUser]->[dbo]_[Traffic_Crm_SystemUser]"
            }
        ],
        "start": "2017-01-19T14:30:57.309Z",
        "end": "2099-12-31T05:00:00Z",
        "isPaused": false,
        "hubName": "stagingdatafactory1_hub",
        "pipelineMode": "Scheduled"