Question

大家好，我正在使用terraform创建胶水作业。现在，AWS Glue现在支持在Apache Spark 2.4.3（使用Python 3）上运行ETL作业的功能。我要使用此功能。但是每当我进行更改时，都会引发错误。我在用 aws-cli / 1.16.184。 Terraform v0.12.6 AWS提供商2.29

resource "aws_glue_job" "aws_glue_job_foo" {
glue_version = "1"
name = "job-name"
description = "job-desc"
role_arn = data.aws_iam_role.aws_glue_iam_role.arn
max_capacity = 1
max_retries = 1
connections = [aws_glue_connection.connection.name]
timeout = 5

command {
name = "pythonshell"
script_location = "s3://bucket/script.py"
python_version = "3"
}

default_arguments = {
"--job-language" = "python"
"--ENV" = "env"
"--ROLE_ARN" = data.aws_iam_role.aws_glue_iam_role.arn
}

execution_property {
max_concurrent_runs = 1
}
}

但这给我带来了错误，

错误：不支持的参数此处不应使用名为“ glue_version”的参数。

Answer 1

此Terraform问题已resolved。

Terraform aws_glue_job现在接受一个glue_version argument。

上一个答案

在Terraform python_version块中有或没有command的情况下，我必须转到AWS控制台以编辑作业并设置“胶水版本”。没有此手动步骤，我的工作就会失败。

解决方法＃1

此问题为reported and debated和includes a workaround。

resource "aws_glue_job" "etl" {
  name     = "${var.job_name}"
  role_arn = "${var.iam_role_arn}"

  command {
    script_location = "s3://${var.bucket_name}/${aws_s3_bucket_object.script.key}"
  }

  default_arguments = {
    "--enable-metrics" = ""
    "--job-language" = "python"
    "--TempDir" = "s3://${var.bucket_name}/TEMP"
  }

  # Manually set python 3 and glue 1.0
  provisioner "local-exec" {
    command = "aws glue update-job --job-name ${var.job_name} --job-update 'Command={ScriptLocation=s3://${var.bucket_name}/${aws_s3_bucket_object.script.key},PythonVersion=3,Name=glueetl},GlueVersion=1.0,Role=${var.iam_role_arn},DefaultArguments={--enable-metrics=\"\",--job-language=python,--TempDir=\"s3://${var.bucket_name}/TEMP\"}'"
  }
}

解决方法＃2

这里是different workaround。

resource "aws_cloudformation_stack" "network" {
  name = "${local.name}-glue-job"

  template_body = <<STACK
{
  "Resources" : {
    "MyJob": {
      "Type": "AWS::Glue::Job",
      "Properties": {
        "Command": {
          "Name": "glueetl",
          "ScriptLocation": "s3://${local.bucket_name}/jobs/${var.job}"
        },
        "ExecutionProperty": {
         "MaxConcurrentRuns": 2
        },
        "MaxRetries": 0,
        "Name": "${local.name}",
        "Role": "${var.role}"
      }
    }
  }
}
STACK
}

Answer 2

看起来terraform使用python_version而不是glue_version

通过使用python_version = "3"，您应该使用胶水版本1.0。胶水0.9版不支持python 3。

Answer 3

此版本已在version 2.34.0 of the Terraform AWS provider中发布。

Glue_version和python_version在Terraform中不起作用

3 个答案:

上一个答案

解决方法＃1

解决方法＃2