Question

我想遍历stfp文件夹中的每个文件，并检查它是否是新文件，然后将新文件复制到Data Lake上现在，我有以下代码，但我认为这是不正确的。在第二个@item()活动中没有使用GetLastModifyfromFile来引用循环中的最后日期，而是引用了一个完全不同的数据集，称为SrcLocalFile。

{
"name": "IncrementalloadfromSingleFolder",
"properties": {
    "activities": [
        {
            "name": "GetFileList",
            "type": "GetMetadata",
            "policy": {
                "timeout": "7.00:00:00",
                "retry": 0,
                "retryIntervalInSeconds": 30,
                "secureOutput": false
            },
            "typeProperties": {
                "dataset": {
                    "referenceName": "SrcLocalDir",
                    "type": "DatasetReference"
                },
                "fieldList": [
                    "childItems"
                ]
            }
        },
        {
            "name": "ForEachFile",
            "type": "ForEach",
            "dependsOn": [
                {
                    "activity": "GetFileList",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "typeProperties": {
                "items": {
                    "value": "@activity('GetFileList').output.childItems",
                    "type": "Expression"
                },
                "activities": [
                    {
                        "name": "GetLastModifyfromFile",
                        "type": "GetMetadata",
                        "policy": {
                            "timeout": "7.00:00:00",
                            "retry": 0,
                            "retryIntervalInSeconds": 30,
                            "secureOutput": false
                        },
                        "typeProperties": {
                            "dataset": {
                                "referenceName": "SrcLocalFile",
                                "type": "DatasetReference"
                            },
                            "fieldList": [
                                "lastModified"
                            ]
                        }
                    },
                    {
                        "name": "IfNewFile",
                        "type": "IfCondition",
                        "dependsOn": [
                            {
                                "activity": "GetLastModifyfromFile",
                                "dependencyConditions": [
                                    "Succeeded"
                                ]
                            }
                        ],
                        "typeProperties": {
                            "expression": {
                                "value": "@and(less(activity('GetLastModifyfromFile').output.lastModified, pipeline().parameters.current_time), greaterOrEquals(activity('GetLastModifyfromFile').output.lastModified, pipeline().parameters.last_time))",
                                "type": "Expression"
                            },
                            "ifTrueActivities": [
                                {
                                    "name": "CopyNewFiles",
                                    "type": "Copy",
                                    "policy": {
                                        "timeout": "7.00:00:00",
                                        "retry": 0,
                                        "retryIntervalInSeconds": 30,
                                        "secureOutput": false
                                    },
                                    "typeProperties": {
                                        "source": {
                                            "type": "FileSystemSource",
                                            "recursive": false
                                        },
                                        "sink": {
                                            "type": "BlobSink"
                                        },
                                        "enableStaging": false,
                                        "dataIntegrationUnits": 0
                                    },
                                    "inputs": [
                                        {
                                            "referenceName": "SrcLocalFile",
                                            "type": "DatasetReference"
                                        }
                                    ],
                                    "outputs": [
                                        {
                                            "referenceName": "TgtBooksBlob",
                                            "type": "DatasetReference"
                                        }
                                    ]
                                }
                            ]
                        }
                    }
                ]
            }
        }
    ],
    "parameters": {
        "current_time": {
            "type": "String",
            "defaultValue": "2018-04-01T00:00:00Z"
        },
        "last_time": {
            "type": "String",
            "defaultValue": "2018-03-01T00:00:00Z"
        }
    },
    "folder": {
        "name": "IncrementalLoadSingleFolder"
    }
},
"type": "Microsoft.DataFactory/factories/pipelines"

}

Answer 1

只是一个想法-我看不到您的数据集定义，但是...

是否应将路径和文件名作为参数传递给数据集？

即在路径和文件的数据集定义中添加2个参数（例如pathparam和fileparam）。在数据集的fileName和folderName设置中使用这些参数作为@dataset（）。pathparam和@dataset（）。fileparam。

在上面的代码中，将输入的新参数传递给数据集输入的新“ parameters”部分，其pathparam和fileparam等于您从先前活动中检索到的文件夹和子项。

注意-存在一个错误，即数据集名称中不能包含空格。

循环浏览文件夹目录中的每个文件并检查日期Azure Data Factory V2-错误的代码

1 个答案: