Question

我想用应用程序监视我的数据流作业。我正在开发的应用程序是一个nodejs应用程序，理想情况下它将存在像@google-cloud/bigquery这样的包，而不是dataflow。我完全清楚我可能无法开始工作，如果它不是模板工作，但它应该是列出工作或获取工作信息的简单方法。

更新

我发现了这个规范，https://dataflow.googleapis.com/ $ discovery / rest？version = v1b3，但我不明白列表操作的location是什么。该规范与此页面相关联：https://cloud.google.com/dataflow/docs/reference/rest/

Answer 1

我自己找到了解决方案。有一个repo基本上有gcloud的所有API：https://github.com/google/google-api-nodejs-client

在我发现我可以轻松地做我想要的之后：

'use strict';

var google = require('googleapis');
var dataflow = google.dataflow('v1b3');

google.auth.getApplicationDefault(function (err, authClient, projectId) {
    if (err) {
        throw err;
    }

    // The createScopedRequired method returns true when running on GAE or a local developer
    // machine. In that case, the desired scopes must be passed in manually. When the code is
    // running in GCE or a Managed VM, the scopes are pulled from the GCE metadata server.
    // See https://cloud.google.com/compute/docs/authentication for more information.
    if (authClient.createScopedRequired && authClient.createScopedRequired()) {
        // Scopes can be specified either as an array or as a single, space-delimited string.
        authClient = authClient.createScoped([
            'https://www.googleapis.com/auth/compute'
        ]);
    }

    // Fetch the list of GCE zones within a project.
    // NOTE: You must fill in your valid project ID before running this sample!
    var compute = google.compute({
        version: 'v1',
        auth: authClient
    });

    var result = dataflow.projects.jobs.list({
        'projectId': projectId,
        'auth': authClient
    }, function (err, result) {
        console.log(err, result);
    });
});

Answer 2

后代。。。有一种方法可以在没有客户端库的情况下执行此操作，但是它需要从服务帐户凭据生成jwt，并将jwt交换为访问令牌以执行Dataflow模板。本示例使用Cloud_Bigtable_to_GCS_Avro模板：

import axios from "axios";
import jwt from "jsonwebtoken";
import mem from "mem";

const loadCredentials = mem(function() {
  // This is a string containing service account credentials
  const serviceAccountJson = process.env.GOOGLE_APPLICATION_CREDENTIALS;
  if (!serviceAccountJson) {
    throw new Error("Missing GCP Credentials");
  }

  const credentials = JSON.parse(serviceAccountJson.replace(/\n/g, "\\n").replace(/\r/g, "\\r").replace(/\t/g, "\\t"));

  return {
    projectId: credentials.project_id,
    privateKeyId: credentials.private_key_id,
    privateKey: credentials.private_key,
    clientEmail: credentials.client_email,
  };
});

interface ProjectCredentials {
  projectId: string;
  privateKeyId: string;
  privateKey: string;
  clientEmail: string;
}

function generateJWT(params: ProjectCredentials) {
  const scope = "https://www.googleapis.com/auth/cloud-platform";
  const authUrl = "https://www.googleapis.com/oauth2/v4/token";
  const issued = new Date().getTime() / 1000;
  const expires = issued + 60;

  const payload = {
    iss: params.clientEmail,
    sub: params.clientEmail,
    aud: authUrl,
    iat: issued,
    exp: expires,
    scope: scope,
  };

  const options = {
    keyid: params.privateKeyId,
    algorithm: "RS256",
  };

  return jwt.sign(payload, params.privateKey, options);
}

async function getAccessToken(credentials: ProjectCredentials): Promise<string> {
  const jwt = generateJWT(credentials);
  const authUrl = "https://www.googleapis.com/oauth2/v4/token";
  const params = {
    grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
    assertion: jwt,
  };
  try {
    const response = await axios.post(authUrl, params);
    return response.data.access_token;
  } catch (error) {
    console.error("Failed to get access token", error);
    throw error;
  }
}

function buildTemplateParams(projectId: string, table: string) {
  return {
    jobName: `[job-name]`,
    parameters: {
      bigtableProjectId: projectId,
      bigtableInstanceId: "[table-instance]",
      bigtableTableId: table,
      outputDirectory: `[gs://your-instance]`,
      filenamePrefix: `${table}-`,
    },
    environment: {
      zone: "us-west1-a" // omit or define your own,
      tempLocation: `[gs://your-instance/temp]`,
    },
  };
}

async function backupTable(table: string) {
  console.info(`Executing backup template for table=${table}`);
  const credentials = loadCredentials();
  const { projectId } = credentials;
  const accessToken = await getAccessToken(credentials);
  const baseUrl = "https://dataflow.googleapis.com/v1b3/projects";
  const templatePath = "gs://dataflow-templates/latest/Cloud_Bigtable_to_GCS_Avro";
  const url = `${baseUrl}/${projectId}/templates:launch?gcsPath=${templatePath}`;
  const template = buildTemplateParams(projectId, table);
  try {
    const response = await axios.post(url, template, {
      headers: { Authorization: `Bearer ${accessToken}` },
    });
    console.log("GCP Response", response.data);
  } catch (error) {
    console.error(`Failed to execute template for ${table}`, error.message);
  }
}

async function run() {
  await backupTable("my-table");
}

try {
  run();
} catch (err) {
  process.exit(1);
}

如何针对带有节点的gcloud上的数据流API执行简单的HTTP请求？

2 个答案: