Question

I am using concurrend append method from the class Core in Azure to store data to Azure Data lake.Below is the code and the exception which I got.I am getting this exception rarely not always.Could anyone guide me?...





public void invoke(String value)  {
        BitfinexSingletonClass obj = null;
        try {
            obj = BitfinexSingletonClass.getInstance();
        } catch (IOException e1) {
            slf4jLogger.info(e1.getMessage());
        }
        ADLStoreClient client = obj.getADLStoreClient();
        byte[] myBuffer = (value + "\n").getBytes();

        RequestOptions opts = new RequestOptions();

        opts.retryPolicy = new ExponentialBackoffPolicy();

        OperationResponse resp = new OperationResponse();
        slf4jLogger.info("" + value);
        slf4jLogger
                .info("...............Writing.........above......BITFINEX_DSHBTC_ORDER..Data............................ToADLake............");
        Core.concurrentAppend(BITFINEX_DSHBTC_ORDER, myBuffer, 0, myBuffer.length, true, client, opts, resp);
        slf4jLogger.info("...............BITFINEX_DSHBTC_ORDER...Data...Successfully....written.....to...AzureDataLake............");
        if (!resp.successful) {
            try {
                throw client.getExceptionFromResponse(resp, "BITFINEX_DSHBTC_ORDER data is not written to ADL");
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

    }

com.microsoft.azure.datalake.store.ADLException：操作CONCURRENTAPPEND失败，异常java.net.SocketTimeoutException：读取超时上次遇到5次尝试后抛出的异常[java.net.UnknownHostException，java.net.UnknownHostException，java.net.UnknownHostException，java.net.SocketTimeoutException，java.net.SocketTimeoutException] 在com.microsoft.azure.datalake.store.ADLStoreClient.getExceptionFromResponse（ADLStoreClient.java:1124）在co.biz.yobit.sink.YobitLtcbtcTickerADLSink.invoke（YobitLtcbtcTickerADLSink.java:41）在org.apache.flink.streaming.api.operators.StreamSink.processElement（StreamSink.java:38） at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput（StreamInputProcessor.java:185）在org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run（OneInputStreamTask.java:63）在org.apache.flink.streaming.runtime.tasks.StreamTask.invoke（StreamTask.java:261）在org.apache.flink.runtime.taskmanager.Task.run（Task.java:665）在java.lang.Thread.run（Thread.java:748）引起：java.net.SocketTimeoutException：读取超时在java.net.SocketInputStream.socketRead0（本机方法）在java.net.SocketInputStream.socketRead（SocketInputStream.java:116）在java.net.SocketInputStream.read（SocketInputStream.java:171）在java.net.SocketInputStream.read（SocketInputStream.java:141） at sun.security.ssl.InputRecord.readFully（InputRecord.java:465） at sun.security.ssl.InputRecord.read（InputRecord.java:503） at sun.security.ssl.SSLSocketImpl.readRecord（SSLSocketImpl.java:973） at sun.security.ssl.SSLSocketImpl.readDataRecord（SSLSocketImpl.java:930） at sun.security.ssl.AppInputStream.read（AppInputStream.java:105）在java.io.BufferedInputStream.fill（BufferedInputStream.java:246）在java.io.BufferedInputStream.read1（BufferedInputStream.java:286）在java.io.BufferedInputStream.read（BufferedInputStream.java:345）在sun.net.www.http.HttpClient.parseHTTPHeader（HttpClient.java:735）在sun.net.www.http.HttpClient.parseHTTP（HttpClient.java:678） at sun.net.www.protocol.http.HttpURLConnection.getInputStream0（HttpURLConnection.java:1569） at sun.net.www.protocol.http.HttpURLConnection.getInputStream（HttpURLConnection.java:1474）在java.net.HttpURLConnection.getResponseCode（HttpURLConnection.java:480） at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode（HttpsURLConnectionImpl.java:338）在com.microsoft.azure.datalake.store.HttpTransport.makeSingleCall（HttpTransport.java:292）在com.microsoft.azure.datalake.store.HttpTransport.makeCall（HttpTransport.java:91）在com.microsoft.azure.datalake.store.Core.concurrentAppend（Core.java:210）在co.biz.yobit.sink.YobitLtcbtcTickerADLSink.invoke（YobitLtcbtcTickerADLSink.java:37） ......还有6个

Answer 1

上述错误通常是运行代码的主机与Azure Data Lake Store之间不可靠的网络状况的结果。正如评论中所证实的那样，主机正在跨越WAN连接的地理区域运行。因此，这些错误是预期的，如果您看到这些错误，则应重试

建议Flink群集在与Azure Data Lake Store位于同一区域的VM上运行。在该配置中，您将看不到这些网络错误。

当我运行flink作业将数据存储到Azure Data Lake时，我得到以下异常。有人可以指导我吗？

1 个答案: