Question

我们的Dataflow管道中的DoFn包含一个Random字段指向SecureRandom实例的类型，当使用{{1}在Dataflow服务中运行时，该字段无法反序列化}}。（下面的堆栈跟踪）

我们使用默认的ctor创建DataflowPipelineRunner，这恰好可以将使用SecureRandom作为其sun.security.provider.Sun的实例提交回来（请参阅SecureRandom#getProvider）。 java.security.Provider扩展了SecureRandom，这是可序列化的。

数据流服务在尝试反序列化此类时会阻塞，因为它无法创建Random。

仔细观察堆栈跟踪，我发现反序列化是通过sun.security.provider.Sun发生的，现在我的理论是这个类加载器不允许加载com.google.apphosting.runtime.security.UserClassLoader类，或者至少加载sun.*类sun.* 1}} class。

java.lang.IllegalArgumentException: unable to deserialize com.example.Example@13e88d
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:73)
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.clone(SerializableUtils.java:88)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$Bound.<init>(ParDo.java:683)
    [...]
    Caused by: java.lang.ClassNotFoundException: sun.security.provider.Sun
    at com.google.apphosting.runtime.security.UserClassLoader.loadClass(UserClassLoader.java:442)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:375)
    at java.lang.Class.forName0(Native Method)
    [...]

Answer 1

问题是sun.security.provider.Sun没有出现在App Engine JRE白名单中，因此类加载器无法实例化它的实例：

https://cloud.google.com/appengine/docs/java/jrewhitelist

但幸运的是，你仍然可以在同一环境中说new SecureRandom()。

要解决此问题，我们在具有Random字段的类中添加了自定义de / serialization挂钩。简化示例：

class Example implements Serializable {

  // See comments on {@link #writeObject} for why this is transient.
  // Should be treated as final, but can't be declared as such.
  private transient Random random;

  //
  // [Guts of the class go here...]
  //

  /**
   * Serialization hook to handle the transient Random field.
   */
  private void writeObject(ObjectOutputStream out) throws IOException {
    out.defaultWriteObject();
    if (random instanceof SecureRandom) {
      // Write a null to tell readObject() to create a new
      // SecureRandom during deserialization; null is safe to use
      // as a placeholder because the constructor disallows null
      // Randoms.
      //
      // The dataflow cloud environment won't deserialize
      // SecureRandom instances that use sun.security.provider.Sun
      // as their Provider, because it's a system
      // class that's not on the App Engine whitelist:
      // https://cloud.google.com/appengine/docs/java/jrewhitelist
      out.writeObject(null);
    } else {
      out.writeObject(random);
    }
  }

  /**
   * Deserialization hook to initialize the transient Random field.
   */
  private void readObject(ObjectInputStream in)
      throws IOException, ClassNotFoundException {
    in.defaultReadObject();
    Object newRandom = in.readObject();
    if (newRandom == null) {
      // writeObject() will write a null if the original field was
      // SecureRandom; create a new instance to replace it. See
      // comments in writeObject() for background.
      random = new SecureRandom();
      random.nextDouble(); // force seeding
    } else {
      random = (Random) newRandom;
    }
  }
}

在Google App Engine中运行Google Cloud Dataflow管道时出现“ClassNotFoundException：sun.security.provider.Sun”

1 个答案: