是否可以将hdfs中的文件上传到ftp?

时间:2016-03-11 10:57:04

标签: scala

def uploadFile(ftpClient: FTPClient, localPath: String) = {
  println(s"uploadFile:localPath: $localPath")
  val localFile = new File(localPath)
  val fileName = localFile.getName
  println(s"uploading file ${localPath} to ftp file $fileName")
  val fileInputStream = new FileInputStream(localPath)
  val uploadStatus = ftpClient.storeFile(fileName, fileInputStream)
  println(s"upload status of file $fileName = $uploadStatus")
  fileInputStream.close()
  uploadStatus
}

这是通过本地文件,而不是通过hdfs。

1 个答案:

答案 0 :(得分:0)

您可以创建InputStream来读取您的HDFS文件,如下所示:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}

val path: Path = // ...
val conf: Configuration = // ...
val inStream: InputStream = FileSystem.get(conf).open(path)

继续使用本地文件完成的操作。当然,这将通过执行此代码的机器流式传输所有数据,因此即使文件是分布式的,上传也是单线程的。