Question

我正在开发一个Spark应用程序，我习惯将Spring作为依赖注入框架。现在我遇到了问题，处理部分使用了Spring的@Autowired功能，但它被Spark序列化和反序列化。

所以下面的代码让我陷入困境：

Processor processor = ...; // This is a Spring constructed object
                           // and makes all the trouble
JavaRDD<Txn> rdd = ...; // some data for Spark
rdd.foreachPartition(processor);

处理器看起来像这样：

public class Processor implements VoidFunction<Iterator<Txn>>, Serializeable {
    private static final long serialVersionUID = 1L;

    @Autowired // This will not work if the object is deserialized
    private transient DatabaseConnection db;

    @Override
    public void call(Iterator<Txn> txns) {
        ... // do some fance stuff
        db.store(txns);
    }
}

所以我的问题是：甚至可以将Spring与Spark结合使用吗？如果没有，那么做这样的事情最优雅的方式是什么？任何帮助表示赞赏！

Answer 1

来自问题提示者：已添加：要直接干扰反序列化部分而不修改自己的类，请使用parapluplu <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>1.5.2.RELEASE</version> <relativePath/>  </parent>之后的spring-spark project。这个项目在春天被反序列化时会自动装配你的bean。

修改

要使用Spark，您需要以下设置（也见this repository）：

Spring Boot + Spark：

<dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <exclusions> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency>  <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>2.1.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency>  <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.11</artifactId> <version>2.1.0</version> </dependency>  <dependency> <groupId>org.codehaus.janino</groupId> <artifactId>commons-compiler</artifactId> <version>2.7.8</version> </dependency>  <dependency> <groupId>org.slf4j</groupId> <artifactId>log4j-over-slf4j</artifactId> <version>1.7.25</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.5</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <version>1.6.4</version> </dependency> </dependencies>

...

@SpringBootApplication public class SparkExperimentApplication { public static void main(String[] args) { SpringApplication.run(SparkExperimentApplication.class, args); } }

然后你需要像Spring Boot一样的应用程序类：

@Configuration @PropertySource("classpath:application.properties") public class ApplicationConfig { @Autowired private Environment env; @Value("${app.name:jigsaw}") private String appName; @Value("${spark.home}") private String sparkHome; @Value("${master.uri:local}") private String masterUri; @Bean public SparkConf sparkConf() { SparkConf sparkConf = new SparkConf() .setAppName(appName) .setSparkHome(sparkHome) .setMaster(masterUri); return sparkConf; } @Bean public JavaSparkContext javaSparkContext() { return new JavaSparkContext(sparkConf()); } @Bean public SparkSession sparkSession() { return SparkSession .builder() .sparkContext(javaSparkContext().sc()) .appName("Java Spark SQL basic example") .getOrCreate(); } @Bean public static PropertySourcesPlaceholderConfigurer propertySourcesPlaceholderConfigurer() { return new PropertySourcesPlaceholderConfigurer(); } }

然后是将所有这些绑定在一起的配置

SparkSession

然后您可以使用/** * Created by achat1 on 9/23/15. * Just an example to see if it works. */ @Component public class WordCount { @Autowired private SparkSession sparkSession; public List<Count> count() { String input = "hello world hello hello hello"; String[] _words = input.split(" "); List<Word> words = Arrays.stream(_words).map(Word::new).collect(Collectors.toList()); Dataset<Row> dataFrame = sparkSession.createDataFrame(words, Word.class); dataFrame.show(); //StructType structType = dataFrame.schema(); RelationalGroupedDataset groupedDataset = dataFrame.groupBy(col("word")); groupedDataset.count().show(); List<Row> rows = groupedDataset.count().collectAsList();//JavaConversions.asScalaBuffer(words)).count(); return rows.stream().map(new Function<Row, Count>() { @Override public Count apply(Row row) { return new Count(row.getString(0), row.getLong(1)); } }).collect(Collectors.toList()); } }类与Spark SQL进行通信：

public class Word { private String word; public Word() { } public Word(String word) { this.word = word; } public void setWord(String word) { this.word = word; } public String getWord() { return word; } } public class Count { private String word; private long count; public Count() { } public Count(String word, long count) { this.word = word; this.count = count; } public String getWord() { return word; } public void setWord(String word) { this.word = word; } public long getCount() { return count; } public void setCount(long count) { this.count = count; } }

参考这两个类：

@RequestMapping("api") @Controller public class ApiController { @Autowired WordCount wordCount; @RequestMapping("wordcount") public ResponseEntity<List<Count>> words() { return new ResponseEntity<>(wordCount.count(), HttpStatus.OK); } }

然后你可以看看它返回正确的数据：

[{"word":"hello","count":4},{"word":"world","count":1}]

说

var cleanArr:Array = new Array(); for (var i:int = 0; i < arr.length; i++) { cleanArr[i][0] = arr[i].substring(0, 29); cleanArr[i][1] = arr[i].substring(29, int.MAX_VALUE); trace(cleanArr[i]); }

将Spring与Spark结合使用

1 个答案: