Question

.net核心项目。我正在使用如下.net核心代码调用一些ADF（Azure数据工厂）管道。

 public async Task<string> RunADFPipeline(DataFactoryManagementClient dataFactoryManagementClient, Dictionary<string,object> keyValuePairs, ADFClient aDFClient, string pieplineName)
        {
            CreateRunResponse runResponse = dataFactoryManagementClient.Pipelines.CreateRunWithHttpMessagesAsync(aDFClient.ResourceGroupName, aDFClient.DataFactoryName, pieplineName, parameters: keyValuePairs).Result.Body;
            return runResponse.RunId;
        }

此管道将运行大约五分钟，并且管道会将一些数据写入Azure sql Db。现在我的要求是从sql db获取数据。我对此有一些疑问。当管道完成执行时，如何知道我的代码？我在下面尝试过。

 public async Task<object> GetPipelineInfoAsync(DataFactoryManagementClient dataFactoryManagementClient, ADFClient aDFClient, string runId)
        {
            var info = await dataFactoryManagementClient.PipelineRuns.GetAsync(aDFClient.ResourceGroupName, aDFClient.DataFactoryName, runId);
            return new
            {
                RunId = info.RunId,
                PipelineName = info.PipelineName,
                InvokedBy = info.InvokedBy.Name,
                LastUpdated = info.LastUpdated,
                RunStart = info.RunStart,
                RunEnd = info.RunEnd,
                DurationInMs = info.DurationInMs,
                Status = info.Status,
                Message = info.Message
            };
        }

通过将第一次调用收到的RunId传递给上述方法，我可以获得其状态。但是我等不及执行完毕。目的是ADF管道会将一些数据写入db，并将这些数据发送回UI。但是我等不及在当前通话中。我打算为此使用SignalR。 ADF管道完成执行后，我可以调用GetPipelineInfoAsync方法，如果状态为成功，则可以转到db并获取详细信息。我面临的唯一问题是在adf管道完成执行之前，我无法阻塞主线程。有人可以帮我解决这个问题吗？任何帮助将不胜感激。谢谢

Answer 1

您可以使用管理客户端的query by factory扩展API，为管道名称和状态（并说一个时间跨度）传入过滤器：

var filters = new PipelineRunFilterParameters
{
    Filters =
    {
        new PipelineRunQueryFilter
        {
            Operand = "PipelineName",
            OperatorProperty = "Equals",
            Values = new List<string> { "my_pipeline", "my_other_pipeline" }
        },
        new PipelineRunQueryFilter
        {
            Operand = "Status",
            OperatorProperty = "Equals",
            Values = new List<string> { "InProgress", "Queued" }
        }
    },
    LastUpdatedBefore = DateTime.Now,
    LastUpdatedAfter = (DateTime.Now.AddDays(-1))
};

while ((_adfClient.PipelineRuns.QueryByFactory(_resourceGroup, _dataFactoryName, filters).Value.Count) >= _maxConcurrentPipelines)
{
    Task.Delay(1000).Wait();
}

创建运行和此API时，您还可以使用客户端的响应ID在单个管道上等待：

while ((pipelineRun = _adfClient.PipelineRuns.Get(_resourceGroup, _dataFactoryName, response.RunId)).Status == "InProgress" || pipelineRun.Status == "Queued")
{
    Task.Delay(1000).Wait();
}

如何等待数据工厂管道完成执行？

1 个答案: