Question

我正在尝试从谷歌BigQuery接收一些数据，并在我的eclipse控制台中显示结果以开始使用BigQuery和Java。基本上，我遵循谷歌提供的说明，我在这里找到： https://cloud.google.com/bigquery/docs/reference/libraries

所以，我的示例Java代码看起来像google的快速启动示例：

import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.Dataset;
import com.google.cloud.bigquery.DatasetInfo;

public class QuickstartSample {
  public static void main(String... args){
        // Instantiates a client
        BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();

        // The name for the new dataset
        String datasetName = "bigquery-public-data:hacker_news.comments";

        // Prepares a new dataset
        Dataset dataset = null;
        DatasetInfo datasetInfo = DatasetInfo.newBuilder(datasetName).build();

        // Creates the dataset
        dataset = bigquery.create(datasetInfo);

        System.out.printf("Dataset %s created.%n", dataset.getDatasetId().getDataset());
  } 
}

但是当我运行此代码时，我收到以下错误：

HTTP ERROR 500

Problem accessing /getbigquery. Reason: 
    Could not get the access token.


Caused by:
com.google.cloud.bigquery.BigQueryException: Could not get the access token.
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:86)
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:141)
    at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:172)
    at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:169)
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
    at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:169)
    at testpackage.dto.QuickstartSample.main(QuickstartSample.java:27)
    at testpackage.getbigquery.doPost(getbigquery.java:27)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:134)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:122)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

我尝试使用Default Credentials进行身份验证，因此我在google SDK控制台中运行了“gcloud auth application-default login”。虽然我可以在SDK中登录，但是eclipse给了我之前显示的错误。

然后我尝试了一些其他的事情，例如将我的令牌作为Json文件下载并在我的应用程序中使用java进行反复操作或手动设置我的环境变量但是它不能正常工作。

有谁知道我做错了什么？任何帮助表示赞赏。

Answer 1

听起来你正在点击a bug in the App Engine SDK，阻止服务使用gcloud 应用程序默认凭据。有a workaround：重命名（或删除）google-cloud-sdk/platform/google_appengine/google/appengine/tools/java/lib/shared/appengine-remote-api.jar。

您还需要配置GCP项目以用于存储和访问BigQuery数据集。可以通过在启动配置的 Arguments 选项卡中设置com.google.appengine.application.id系统属性来配置此项目（通过运行＆gt;运行配置，然后添加-Dcom.google.appengine.application.id=projectID到 Arguments＆gt; VM Arguments 部分）。但是，如果您要从其他项目（例如bigquery-public-data数据集）访问数据集，则需要按如下方式显式配置BigQuery构建器：

BigQueryOptions.newBuilder()
  .setProjectId("projectID")
  .build().getService();

最后，您还必须在appengine-api-1.0-sdk添加以下内容，以包含pom.xml相关性：

<dependency>
  <groupId>com.google.appengine</groupId>
  <artifactId>appengine-api-1.0-sdk</artifactId>
  <version>1.9.50</version>
</dependency>

（或任何更新的版本）

那就是说， Eclipse的Eclipse工具应该可以更容易地指定默认的应用程序凭据和项目，因此我提交了ticket to improve this situation。

Answer 2

以下内容应添加到您的maven项目的pom.xml中

    <!--Google Bigquery-->
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>0.8.0-beta</version>
    </dependency>

在您的Google Cloud Platform Console中创建一个服务帐户 - ＆gt; IAM-＆gt;具有必要权限的服务帐户。您将被允许下载JSON密钥文件。

    import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
    import com.google.api.client.http.javanet.NetHttpTransport;
    import com.google.api.client.json.jackson2.JacksonFactory;
    import com.google.api.client.util.Data;
    import com.google.api.services.bigquery.model.*;
    import com.google.api.services.bigquery.Bigquery;
    import com.google.api.services.bigquery.BigqueryScopes;
    import java.io.ByteArrayInputStream;


    //Read the content of the service account key file in following variable. Please take appropriate security measures into the account.
    String credentials = "Content of JSON secret file";

    //Create bigquery client
    NetHttpTransport transport = new NetHttpTransport();
    JacksonFactory jsonFactory = new JacksonFactory();
    GoogleCredential credential = GoogleCredential.fromStream(new ByteArrayInputStream(credentials.getBytes()));
    if (credential.createScopedRequired) credential = credential.createScoped(BigqueryScopes.all);
    Bigquery bigqueryClient  = new Bigquery.Builder(transport, jsonFactory, credential).setApplicationName("Your Application Name").build();

    String projectId = "Your-project-id";
    String datasetId = "NewDatasetName";

    Dataset dataset = new Dataset();

    DatasetReference datasetRef = new DatasetReference();
    datasetRef.setProjectId(projectId);
    datasetRef.setDatasetId(datasetId);

    dataset.setDatasetReference(datasetRef);

    try {
         bigqueryClient.datasets().insert(projectId, dataset).execute();
    } catch (Exception e){
     //Handle exceptions
    }

如何使用Eclipse和Java从Google BigQuery接收数据？

2 个答案: