Question

我使用Azure Data Factory将本地数据传输到Azure Data Lake商店。复制数据后，我在上传的文件上运行U-SQL脚本，将其转换为新的.csv文件。如果从Visual Studio或直接从Data Lake Analytics运行，我的U-SQL作业运行正常。

但是如果在Azure Data Factory中添加和活动。复制数据后，此U-SQL脚本活动立即失败。我尝试了很多方法，但无法解决问题。它给了我以下错误。

我的U-SQL活动的JSON定义是

{
    "name": "Transform Data",
    "description": "This will transform work space data.",
    "type": "DataLakeAnalyticsU-SQL",
    "dependsOn": [
        {
            "activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
            "dependencyConditions": [
                "Completed"
            ]
        }
    ],
    "policy": {
        "timeout": "7.00:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureOutput": false
    },
    "typeProperties": {
        "scriptPath": "Scripts/Script.usql",
        "scriptLinkedService": {
            "referenceName": "Destination_DataLakeStore_lc0",
            "type": "LinkedServiceReference"
        }
    },
    "linkedServiceName": {
        "referenceName": "AzureDataLakeAnalyticsForDF",
        "type": "LinkedServiceReference"
    }
}

整个管道的JSON

{
    "name": "CopyPipeline_d26",
    "properties": {
        "activities": [
            {
                "name": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
                "type": "Copy",
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false
                },
                "typeProperties": {
                    "source": {
                        "type": "FileSystemSource",
                        "recursive": false
                    },
                    "sink": {
                        "type": "AzureDataLakeStoreSink",
                        "copyBehavior": "MergeFiles"
                    },
                    "enableStaging": false,
                    "cloudDataMovementUnits": 0,
                    "enableSkipIncompatibleRow": true
                },
                "inputs": [
                    {
                        "referenceName": "workplace_groups_info_2018_03_19_09_32_csv_i_lc0",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "workplace_groups_info_2018_03_19_09_32_csv_o_lc0",
                        "type": "DatasetReference"
                    }
                ]
            },
            {
                "name": "Transform Data",
                "description": "This will transform work space data.",
                "type": "DataLakeAnalyticsU-SQL",
                "dependsOn": [
                    {
                        "activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
                        "dependencyConditions": [
                            "Completed"
                        ]
                    }
                ],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false
                },
                "typeProperties": {
                    "scriptPath": "Scripts/Script.usql",
                    "scriptLinkedService": {
                        "referenceName": "Destination_DataLakeStore_lc0",
                        "type": "LinkedServiceReference"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureDataLakeAnalyticsForDF",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "parameters": {
            "windowStart": {
                "type": "String"
            },
            "windowEnd": {
                "type": "String"
            }
        }
    }
}

Answer 1

我通过使用AppService创建运行时解决了这个问题。我按照以下步骤操作。

我在Active Directory中创建了一个WebApp。
也将Azure Data Lake权限分配给该WebApp。
在该应用中创建一个公钥并记下它。它永远不会再显示出来。
注意该WebApp的应用程序ID。
打开Azure Data Lake Analytics，并为在Active Directory中创建的WebApp分配参与者角色。
在创建运行时，将Application ID用作服务主体ID 和公钥作为WebApp的服务主体密钥。

工作正常。：）

U-SQL活动未在Azure Data Factory上运行

1 个答案: