def uploadFile(ftpClient: FTPClient, localPath: String) = {
println(s"uploadFile:localPath: $localPath")
val localFile = new File(localPath)
val fileName = localFile.getName
println(s"uploading file ${localPath} to ftp file $fileName")
val fileInputStream = new FileInputStream(localPath)
val uploadStatus = ftpClient.storeFile(fileName, fileInputStream)
println(s"upload status of file $fileName = $uploadStatus")
fileInputStream.close()
uploadStatus
}
这是通过本地文件,而不是通过hdfs。
答案 0 :(得分:0)
您可以创建InputStream
来读取您的HDFS文件,如下所示:
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
val path: Path = // ...
val conf: Configuration = // ...
val inStream: InputStream = FileSystem.get(conf).open(path)
继续使用本地文件完成的操作。当然,这将通过执行此代码的机器流式传输所有数据,因此即使文件是分布式的,上传也是单线程的。