Question

Unable to write spark dataframe into snowflake table.

21/06/07 09:04:05 INFO JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
21/06/07 09:04:05 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7690638d{/static/sql,null,AVAILABLE,@Spark}
21/06/07 09:04:05 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/06/07 09:04:12 ERROR SnowflakeWriter: net.snowflake.client.jdbc.SnowflakeSQLException: SQL execution internal error:
Processing aborted due to error 370001:4107598406; incident 4986515.
    at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowException(SnowflakeUtil.java:99)
    at net.snowflake.client.core.StmtUtil.execute(StmtUtil.java:410)
    at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:373)
    at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:197)
    at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:149)
    at net.snowflake.client.core.SFStatement.execute(SFStatement.java:531)
    at net.snowflake.client.jdbc.SnowflakeStatementV1.executeInternal(SnowflakeStatementV1.java:204)
    at net.snowflake.client.jdbc.SnowflakeStatementV1.execute(SnowflakeStatementV1.java:239)
    at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$executeInterruptibly$1.apply(SnowflakeJDBCWrapper.scala:261)
    at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$executeInterruptibly$1.apply(SnowflakeJDBCWrapper.scala:261)
    at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$3.apply(SnowflakeJDBCWrapper.scala:283)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

21/06/07 09:04:12 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
21/06/07 09:04:12 INFO SparkContext: Invoking stop() from shutdown hook

Answer 1

内部错误 370001 与权限/身份验证相关。您能否使用导致错误的代码中使用的连接属性运行示例？

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
public class SnowflakeJDBCExample {
  public static void main(String[] args) throws Exception {
    String jdbcUrl = "jdbc:snowflake://xy12345.snowflakecomputing.com/";

    Properties properties = new Properties();
    properties.put("user", "peter");
    properties.put("password", "test");
    properties.put("account", "xy12345");
    properties.put("warehouse", "mywh");
    properties.put("db", "mydb");
    properties.put("schema", "public");

    // get connection
    System.out.println("Create JDBC connection");
    Connection connection = DriverManager.getConnection(jdbcUrl, properties);
    System.out.println("Done creating JDBC connection\n");
    // create statement
    System.out.println("Create JDBC statement");
    Statement statement = connection.createStatement();
    System.out.println("Done creating JDBC statement\n");
    // create a table
    System.out.println("Create my_variant_table table");
    statement.executeUpdate("create or replace table my_variant_table(json VARIANT)");
    statement.close();
    System.out.println("Done creating demo table\n");

    connection.close();
    System.out.println("Close connection\n");
  }
}

Spark java程序将数据帧写入雪花表

1 个答案: