Question

我正在实现RichParallelSourceFunction，它通过SFTP读取文件。 RichParallelSourceFunction继承SourceFunction的cancel（）和RichFunction（）的close（）。据我了解，cancel（）和close（）都在拆卸源之前被调用。因此，在这两种方法中，我都必须添加逻辑以停止读取文件的无限循环。

当我将源的并行度设置为1并从IDE运行Flink作业时，Flink运行时会在调用start（）之后立即停止stop（），并且整个作业都将停止。我没想到这一点。

当我将源的并行性设置为1并在集群中运行Flink作业时，该作业照常运行。如果我将源的并行性保留为默认值（在我的情况下为4），则作业照常运行。

使用Flink 1.7。


public class SftpSource<TYPE_OF_RECORD>
    extends RichParallelSourceFunction<TYPE_OF_RECORD>
{
    private final SftpConnection mConnection;
    private boolean mSourceIsRunning;

    @Override 
    public void open(Configuration parameters) throws Exception
    {
        mConnection.open();
    }

    @Override 
    public void close()
    {
        mSourceIsRunning = false;
    }


    @Override
    public void run(SourceContext<TYPE_OF_RECORD> aContext)
    {
        while (mSourceIsRunning)
        {
            synchronized ( aContext.getCheckpointLock() )
            {
                // use mConnection
                // aContext.collect() ...
            }

            try
            {
                Thread.sleep(1000);
            }
            catch (InterruptedException ie)
            {
                mLogger.warn("Thread error: {}", ie.getMessage() );
            }
        }

        mConnection.close();
    }


    @Override
    public void cancel()
    {
        mSourceIsRunning = false;
    }
}

所以我有变通办法，问题更多是关于理论。如果并行度为1且作业是从IDE（即从命令行）运行的，为什么要调用close（）？另外，在RichParallelSourceFunction中close（）和cancel（）是否也一样？

Answer 1

我认为javadocs不仅是不言自明的：

Gracefully Stopping Functions
Functions may additionally implement the {@link org.apache.flink.api.common.functions.StoppableFunction} interface. "Stopping" a function, in contrast to "canceling" means a graceful exit that leaves the state and the emitted elements in a consistent state.

-SourceFunction.cancel

Cancels the source. Most sources will have a while loop inside the run(SourceContext) method. The implementation needs to ensure that the source will break out of that loop after this method is called.
A typical pattern is to have an "volatile boolean isRunning" flag that is set to false in this method. That flag is checked in the loop condition.

When a source is canceled, the executing thread will also be interrupted (via Thread.interrupt()). The interruption happens strictly after this method has been called, so any interruption handler can rely on the fact that this method has completed. It is good practice to make any flags altered by this method "volatile", in order to guarantee the visibility of the effects of this method to any interruption handler.

-SourceContext.close

This method is called by the system to shut down the context.

注意，您可以取消SourceFunction，但停止SourceContext

Answer 2

如果并行度为1且作业从以下位置运行，为什么要调用close（） IDE。

在最后一次调用主要工作方法（例如，映射或联接）之后，调用

close。此方法可用于清理工作。它将被称为与并行性中定义的数字无关。

此外，在RichParallelSourceFunction中close（）和cancel（）是否也一样？

它们不是同一回事，看看它的描述。

Cancels the source. Most sources will have a while loop inside the run(SourceContext) method. The implementation needs to ensure that the source will break out of that loop after this method is called.

https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/source/SourceFunction.html#cancel--

以下链接可以帮助您了解任务生命周期： https://ci.apache.org/projects/flink/flink-docs-stable/internals/task_lifecycle.html#operator-lifecycle-in-a-nutshell

Answer 3

我在代码中发现了一个错误。这是解决方法

public void open(Configuration parameters) throws Exception
{
    mConnection.open();
    mSourceIsRunning = true;
}

现在直到我决定停止工作流程时才调用close（），在这种情况下，先调用cancel（），然后再调用close（）。我仍然想知道并行性如何影响行为。

Flink RichParallelSourceFunction-close（）vs cancel（）

3 个答案: