将Spark Streaming状态保存到外部DB中

Question

如果对spark应用程序代码有任何更改，Spark Streaming checkpoint将无法正常工作......所以我想将状态信息显式保存到像cassandra这样的外部数据库中。

如何定期将火花流状态刷新到外部数据库或磁盘？我不知道如何读取所有状态RDD信息并写入DB /磁盘。有人可以给我java例子吗？

Answer 1

You might find Spark's DataSources API documentation useful for saving state; it contains Java examples.

Also there are many stores that can be used in conjunction with Spark. Here are most of them:

• SnappyData
• Cassandra
• MemSQL
• HBase (one of many)
• MongoDB
• Redis
• Ignite
• Couchbase