我在谷歌云平台上使用spark。
显然我正在从文件系统gs://<bucket>/dir/file
读取文件,但是日志输出提示
FileNotFoundException:`gs:/ bucket / dir / file(不存在此类文件或目录)
缺少的/
显然是问题所在。我该如何解决这个问题?
这是我的代码:
val files = Array(("call 1","gs://<bucket>/google-cloud-dataproc-metainfo/test/123.wav"))
val splitAudioFiles = sc.parallelize(files.map(x => splitAudio(x, 5, sc)))
def splitAudio(path: (String, String), interval: Int, sc: SparkContext): (String, Seq[(String,Int)]) = {
val stopWords = sc.broadcast(loadTxtAsSet("gs://<bucket>/google-cloud-dataproc-metainfo/test/stopword.txt")).value
val keyWords = sc.broadcast(loadTxtAsSet("gs://<bucket>/google-cloud-dataproc-metainfo/test/KeywordList.txt")).value
val file = new File((path._2))
val audioTitle = path._1
val fileFormat: AudioFileFormat = AudioSystem.getAudioFileFormat(file)
val format = fileFormat.getFormat
答案 0 :(得分:2)
您似乎正在使用AudioSystem.getAudioFileFormat(URL),它不支持gs:// URI。相反,您需要使用Hadoop FileSystem接口来获取文件的InputStream并使用AudioSystem.getAudioFileFormat(InputStream)。我认为这样的事情应该有效:
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.Configuration;
val sc: SparkContext = ...
val urls : RDD[URL] = ...
val formats : RDD[AudioFileFormat] = urls.map(url => {
val asUri = url.toURI
val conf = new Configuration()
val hadoopPath = new Path(asUri)
val hadoopFs = hadooPath.getFileSystem(conf)
val inputStream = hadoopFs.open(hadoopPath)
AudioSystem.getAudioFileFormat(inputStream)
})