Given below is my code and I am monitoring a folder stored on the MapR sandbox (I have also tried using my local file system, still didn't work) I am copying/moving (tried both) new files into the folder but get NO OUTPUT.
The program is not streaming and files
object Log_Stream extends Serializable{
val timeout = 10 // Terminate after N seconds
val batchSeconds = 2 // Size of batch intervals
def main(args: Array[String]): Unit = {
val offsetReset = "earliest"
val batchInterval = "2"
val pollTimeout = "1000"
val FolderPath = "maprfs:///user/vipulrajan/logs" //
val sparkConf = new SparkConf().setAppName("SensorStream").setMaster("local[2]").set("spark.testing.memory", "536870912")
val ssc = new StreamingContext(sparkConf, Seconds(batchInterval.toInt))
val messages = ssc.textFileStream(FolderPath)
println("message values received")
//val myData = values.map(x =>{parseSensor(x)})
messages.print()
// Start the computation
ssc.start()
// Wait for the computation to terminate
ssc.awaitTermination()
}
}
I have also tried renaming and editing them but to no avail. I am not submitting the code using spark submit, I am running it directly from Eclipse.