用于分配计算任务，这是更好的芹菜或火花

Question

问题：计算任务可以轻松并行。但它需要实时响应。

可以有两种方法。 1.使用Celery：从头开始并行运行 2.使用Spark：与spark框架并行运行作业

我认为在可扩展性方面，火花更好。但是，作为网络应用程序的后端，它是否可以使用Spark？

Answer 1

除上述答案外，还有其他方面需要确定。

1. 如果有的话，与现有的大数据堆栈集成。
2. 摄取数据管道
3. 您提到了＆＃34; Web应用程序的后端＆＃34;。我假设它用于读取操作。任何批处理应用程序的响应时间可能不适合任何Web应用程序。

流式传输的选择可以帮助您更快地将数据导入群集。但它不能保证Web应用程序所需的响应时间。您需要查看HBase和Solr（如果您正在搜索）。

Spark无疑比其他批处理框架更好，更快。在流式传输中，可能还有其他一些。如上所述，您应该考虑选择的参数。

Answer 2

Celery :- is really a good technology for distributed streaming And its supports Python language . Which is it self strong in computation and easy to write. The streaming application in Celery supports so many features as well . Its little over head on CPU.

Spark- Its supports various programming language Java,Scala,Python. its not pure streaming its micro batch streaming as per the Spark documentation

If your task can only be full filled by streaming and you dont need the SQl like feature . Then Celery will be the best. But you need various feature along with streaming then SPark will be better . In that case you can take scenario you application will generate the data in how many batches within second .