Question

我有一个自定义处理器（NiFi 1.8.0），该处理器已经根据需要修改了传入流文件。但是，在将文件传输到传出关系之前，我想检查该关系的背压是否接近其阈值。如果是这样，我计划将流文件发送到另一个关系，该关系连接到PutFile处理器，并在该关系中将其写入磁盘。

我知道我可以获取传入队列的数量和大小。但是我不知道如何从传出关系的连接中获取数量和大小。

Answer 1

有一个名为-SiteToSiteStatusReportingTask的控制器服务，该服务实质上发送Nifi中发生的每个事件的状态。

如果您查看data structure it returns，您会发现它在检测背压方面几乎没有非常有用的属性-

// fields for connections
{ "name" : "sourceId", "type" : ["string", "null"]},
{ "name" : "sourceName", "type" : ["string", "null"]},
{ "name" : "destinationId", "type" : ["string", "null"]},
{ "name" : "destinationName", "type" : ["string", "null"]},
{ "name" : "maxQueuedBytes", "type" : ["long", "null"]},
{ "name" : "maxQueuedCount", "type" : ["long", "null"]},
{ "name" : "queuedBytes", "type" : ["long", "null"]},
{ "name" : "backPressureBytesThreshold", "type" : ["long", "null"]},
{ "name" : "backPressureObjectThreshold", "type" : ["long", "null"]},
{ "name" : "backPressureDataSizeThreshold", "type" : ["string", "null"]},
{ "name" : "isBackPressureEnabled", "type" : ["string", "null"]},

您可以使用此信息得出所需的信息。 Refer this article for more details on implementation

Answer 2

我最终从ProcessGroupStatus对象找到了连接：

String myProcessorId = this.getIdentifier();
int queuedCount = 0;
float queuedBytes = 0;
ProcessGroupStatus processGroupStatus = ((EventAccess) getControllerServiceLookup().getControllerStatus();

  if (processGroupStatus.getConnectionStatus() != null {
    Collection < CollectionStatus > groupConnections = processGroupStatus.getConnectionStatus();

    // Now have to iterate through groupConnections to find the one where the connection's source ID = myProcessorId and  
    // the connection's name = 'normal output' (this is the name of a relationship I added)

    ArrayList connections = new ArrayList <> (groupConnections);
    for (Object processorConnection : connections) {
     ConnectionStatus connection = (ConnectionStatus) processorConnection;

     if (connection.getName().equals("normal output") && connections.getSourceId.equals(myProcessorId)) {
      // Now I can grab the current count and size of the 'normal output' relationship
      // The back pressure threshold values can be grabbed from the connection as well
      queuedCount = connection.getQueuedCount();
      queuedBytes = connection.getQueuedBytes();
      break;
     }
    }
   }

以上内容仅从父级组中检索连接。如果您要查找的连接包含在子组中，则需要遍历子组：

ProcessGroupStatus processGroupStatus = ((EventAccess) getControllerServiceLookup().getControllerStatus();
ArrayList childProcessorGroups = new ArrayList < > (processGroupStatus.getProcessGroupStatus());
for (Object childProcessorGroup : childProcessorGroups) {
 ProcessGroupStatus childProcessGroupStatus = (ProcessGroupStatus) childProcessorGroup;
 Collection < CollectionStatus > groupConnections = childProcessGroupStatus.getConnectionStatus();
 // Then iterate through groupConnections as above
}

NiFi getControllerServiceLookup（）确实显示了“ allConnections”变量，其中包含所有组中所有处理器之间的所有连接。但是似乎并没有吸气剂。如果有一个吸气剂，您不必担心要寻找哪个组的连接。您可以简单地遍历“ allConnections”并查找与您的处理器ID和关系名称匹配的连接。

在自定义处理器中监控背压计数和大小

2 个答案: