Question

从Google Storage下载数据时，我收到大量SocketTimeoutException。什么是增加超时的最佳方法？

用例

使用Map / Reduce从多个服务器并行下载24个文件（总共10GB）。

源代码

  Storage storage = new Storage.Builder(
                new NetHttpTransport(),
                new JacksonFactory(),
                new GoogleCredential().setAccessToken(accessToken))
            .setApplicationName("FooBar")
            .build();

  Storage.Objects.Get getObject = storage.objects().get(bucket, fn);
  getObject.getMediaHttpDownloader().setDirectDownloadEnabled(true);
  getObject.executeMediaAndDownloadTo(outputStream);

堆栈跟踪

java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at sun.net.www.MeteredStream.read(MeteredStream.java:134)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3052)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3046)
    at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:261)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:209)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:553)
    at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:4494)

修改

仅当我从日本下载数据时才会发生这种情况。来自美国东部和美国西部的一切都很好。

Answer 1

查找名为“CommandTimeout”的变量，或者那种性质的变量。通常用于确定连接在标记错误之前可以持续多长时间。

“ConnectionTimeout”通常意味着程序可以尝试连接到您的源多长时间，这不是您需要更改的内容，因为连接发生得很快，下载速度很慢。

Answer 2

尝试以下方法......

在linux框中设置代理。

创建存储时 - 新的NetHttpTransport（）提供了自己的方式来创建套接字并在传输中添加代理。

public HttpClient myHttpClient() throws Exception {
    SchemeRegistry schemeRegistry = new SchemeRegistry();
    //SetRegisrty for both HTTP and HTTPS - Check google for this.
    schemeRegistry.register(new Scheme("http", PlainSocketFactory
            .getSocketFactory(), YOUR_PROXY_PORT));
    schemeRegistry.register(new Scheme("https", SSLSocketFactory
            .getSocketFactory(), 443));

    HttpParams params = new BasicHttpParams();
    HttpConnectionParams.setConnectionTimeout(params, 30 * 1000);  // SET the timeout
    HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
    ClientConnectionManager connManager = new ThreadSafeClientConnManager(
            params, schemeRegistry);
    DefaultHttpClient httpClient = new DefaultHttpClient(connManager,
                params);
    try {
        int proxyPort = YOUR_PROXY_PORT;
        String proxyHost = "YOUR_PROXT_HOST_NAME";
        if (proxyPort > 0 && proxyHost != null && proxyHost.length() > 0) {
            System.setProperty("https.proxyHost", proxyHost);
            System.setProperty("https.proxyPort", proxyPort + "");
            System.setProperty("http.proxyHost", proxyHost);
            System.setProperty("http.proxyPort", proxyPort + "");
            HttpHost proxy = new HttpHost(proxyHost, proxyPort);
            httpClient.getParams().setParameter(
                    ConnRoutePNames.DEFAULT_PROXY, proxy);
        }
    } catch (NullPointerException e) {
        System.out.println("Proxy error here");
    }
    return httpClient;
}

public static HttpTransport myNetHttpTransport()
        throws Exception {
    return new ApacheHttpTransport(myHttpClient());
}

使用.setTransport(myNetHttpTransport())代替NetHttpTransport()。

我们花了很长时间在这上面。但截至目前，这似乎有效。如果有任何帮助，请告诉我......

从Google Storage下载数据时出现SocketTimeoutException

2 个答案: