Question

目前，我正致力于火花流项目。刚开始，我仍然是spark-kafka-yarn-cloudera的新人。要尝试（或查看）程序的结果，目前我必须构建项目的jar，将其上传到集群然后spark-submit，我认为这种方式效率不高。

我可以从IDE远程运行这个程序吗？我使用scala-IDE。我寻找一些代码，但仍未找到合适的代码

我的环境： - Cloudera 5.8.2 [OS redhat 7.2，kerberos 5，spark_2.1，scala 2.11] - Windows 7

Answer 1

按照以下步骤对您的应用进行单元测试。

为wondows下载winutils设置HADOOP_HOME环境变量
为Sparkstreaming提供精确的kafka经纪人网址和主题名称
确保设置了正确的偏移管理属性。
使用Intellij IDE（SCALA IDE也没关系）。只需在scala应用程序运行时运行。

val kafkaParams =地图（＆＃34; metadata.broker.list＆＃34; - ＆GT; ＆＃34; 168.172.72.128：9092＆＃34 ;, ConsumerConfig.AUTO_OFFSET_RESET_CONFIG - ＆gt; ＆＃34;最小＆＃34 ;, ＆＃34; group.id＆＃34; - ＆GT; UUID.randomUUID（）。的toString（））

val topicSet = Set（＆＃34; test＆＃34;）//主题名称 val kafkaStream = KafkaUtils .createDirectStream [String，String，StringDecoder，StringDecoder]（ssc，kafkaParams，topicSet） //创建BSON数据结构并将数据加载到MongoDB Collection中 kafkaStream.foreachRDD（ rdd =＆gt; {//业务逻辑代码}）

Answer 2

我按照本教程http://blog.antlypls.com/blog/2017/10/15/using-spark-sql-and-spark-streaming-together/

以下是我的代码：

import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.streaming.kafka010._
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
import org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe

import scala.collection.mutable.ListBuffer
import org.apache.spark.SparkConf
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds
import org.apache.spark.sql.types.{StringType, StructType, TimestampType}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.count

object SparkKafkaExample {

  def main(args: Array[String]): Unit =
  {

  val brokers = "broker1.com:9092,broker2.com:9092," +
    "broker3.com:9092,broker4.com:9092,broker5.com:9092"
  // Create Spark Session
  val spark = SparkSession
    .builder()
    .appName("KafkaSparkDemo")
    .master("local[*]")
    .getOrCreate()

  import spark.implicits._

  // Create Streaming Context and Kafka Direct Stream with provided settings and 10 seconds batches
  val ssc = new StreamingContext(spark.sparkContext, Seconds(10))

  var kafkaParams = Map(
    "bootstrap.servers" -> brokers,
    "key.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer",
    "value.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer",
    "group.id" -> "test",
    "security.protocol" -> "SASL_PLAINTEXT",
    "sasl.kerberos.service.name" -> "kafka",
    "auto.offset.reset" -> "earliest")

  val topics = Array("sparkstreaming")
  val stream = KafkaUtils.createDirectStream[String, String](
    ssc,
    PreferConsistent,
    Subscribe[String, String](topics, kafkaParams))

  // Define a schema for JSON data
  val schema = new StructType()
    .add("action", StringType)
    .add("timestamp", TimestampType)

  // Process batches:
  // Parse JSON and create Data Frame
  // Execute computation on that Data Frame and print result
  stream.foreachRDD { (rdd, time) =>
    val data = rdd.map(record => record.value)
    val json = spark.read.schema(schema).json(data)
    val result = json.groupBy($"action").agg(count("*").alias("count"))
    result.show
  }

  ssc.start
  ssc.awaitTermination

}
}

因为我的集群使用kerberos，所以我将此配置文件（kafka_jaas.conf）传递给我的IDE（Eclipse - ＆gt; on VM Arguments）

-Djava.security.auth.login.config=kafka-jaas.conf

kafka-jaas.conf内容：

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="user.keytab"
    serviceName="kafka"
    principal="user@HOST.COM";
};
Client {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="user.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="zookeeper"
   principal="user@HOST.COM";
};

Spark Yarn Remote提交

2 个答案: