使用Google Cloud Dataflow如何在GCE Compute实例上使用正确的凭据运行？

时间：2015-07-07 12:36:53

标签： google-compute-engine google-cloud-dataflow

我是谷歌云数据流的新手，从下面的问题可能很明显。

我已经编写了一个数据流应用程序，可以在本地和GCE实例上使用我的个人凭据使其无故障运行。但是，我似乎无法使用计算引擎实例的服务凭据或使用API​​＆amp; amp;控制台的AUTH部分。我跑的时候总是得到401未经授权的错误。

这是我尝试过的......

1）创建虚拟机，授予对存储，数据存储，SQL和计算引擎的访问权限。据我所知，这应该创建了一个CI特定服务帐户，它是服务器的默认凭据。这些应该与在此实例上运行应用程序时使用用户身份验证的方式相同。在这里，我得到了401.我的问题是......我在哪里可以看到这个据称创建的服务帐户？或者我只是依赖它存在于某个地方？

2）在API＆amp ;;中创建服务凭证开发人员控制台的Auth部分。然后使用cloud auth activate-service-account并通过将命令指向我下载的凭证json文件来激活该帐户。有点像使用gcloud auth登录时的OAUTH往返。在这里，我也得到了401。

3）最后一件事是使用第2步中与GCE实例分开的服务凭证，然后创建一个实现CredentialFactory接口的对象，并将其传递给PipelineOptions。但是，当它运行时，应用程序崩溃，并显示错误，表示它正在寻找一个方法，fromOptions，而不是在CredentialFactory界面中。如何配置选项，凭证工厂的外观以及随后的堆栈跟踪。

我很乐意利用上述3种方法中的任何一种来使用服务凭证，如果我可以让它们中的任何一种起作用的话。您可以提供有关我做错事的任何见解，我离开的步骤，其他未经探索的选项将不胜感激。文档有点脱节。如果有明确的分步指南，那么链接就足够了。到目前为止，我自己找到的东西几乎没有帮助。

如果我可以提供任何其他信息，请告诉我。

这里有一些可能有用的代码，以及我在使用凭证工厂运行代码时获得的堆栈跟踪。

选项设置代码如下所示：

GcrDataflowPipelineOptions options = PipelineOptionsFactory.fromArgs(args)
        .withValidation()
        .as(GcrDataflowPipelineOptions.class);
options.setKind("Counties");
options.setCredentialFactoryClass(GoogleCredentialProvider.class);

GoogleCredentialProvider.java

请注意，我在创建服务帐户（重命名）时下载的json文件是从我的应用类路径加载的资源。

public class GoogleCredentialProvider implements CredentialFactory {

    @Override
    public Credential getCredential() throws IOException, GeneralSecurityException {
        final String env = System.getProperty("gcr_dataflow_env", "local");
        Properties props = new Properties();
        ClassLoader loader = this.getClass().getClassLoader();
        props.load(loader.getResourceAsStream(env + "-gcr-dataflow.properties"));
        final String credFileName = props.getProperty("gcloud.dataflow.service.account.file");
        InputStream credStream = loader.getResourceAsStream(credFileName);
        GoogleCredential credential = GoogleCredential.fromStream(credStream);
        return credential;
    }

}

堆栈跟踪：

java.lang.RuntimeException: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:268)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:123)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:120)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.visitTransform(DirectPipelineRunner.java:684)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:200)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:196)
    at com.google.cloud.dataflow.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:99)
    at com.google.cloud.dataflow.sdk.Pipeline.traverseTopologically(Pipeline.java:208)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.run(DirectPipelineRunner.java:640)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:354)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:76)
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:149)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.run(GcrDataflowApp.java:65)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.main(GcrDataflowApp.java:49)
Caused by: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.build(InstanceBuilder.java:161)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:180)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:175)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:288)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:127)
    at com.sun.proxy.$Proxy42.getGcpCredential(Unknown Source)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.getDatastore(DatastoreIO.java:335)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:320)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:186)
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:259)
    ... 13 more
java.lang.RuntimeException: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
2015-07-03 09:55:42,519 | main | DEBUG | co.sc.gc.da.ap.GcrDataflowApp | destroying
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:268)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:123)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:120)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.visitTransform(DirectPipelineRunner.java:684)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:200)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:196)
    at com.google.cloud.dataflow.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:99)
    at com.google.cloud.dataflow.sdk.Pipeline.traverseTopologically(Pipeline.java:208)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.run(DirectPipelineRunner.java:640)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:354)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:76)
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:149)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.run(GcrDataflowApp.java:65)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.main(GcrDataflowApp.java:49)
Caused by: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.build(InstanceBuilder.java:161)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:180)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:175)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:288)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:127)
    at com.sun.proxy.$Proxy42.getGcpCredential(Unknown Source)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.getDatastore(DatastoreIO.java:335)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:320)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:186)
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:259)
    ... 13 more

1 个答案:

答案 0 :(得分：1)

您可能没有正确的凭据。从GCE执行数据流作业时，附加到实例的服务帐户将用于DataFlow的验证。

创建机器时是否这样做了？

相关问题
最新问题