我运行了一个火花作业,最终保存了一个Parquet文件,并且工作成功完成。但是,我只指定了文件的名称,并没有指定HDFS路径。有没有办法打印出火花写入文件的默认HDFS路径?我看了sc._conf.getAll()
,但似乎没有任何有用的东西。
答案 0 :(得分:2)
AFAIK这是其中一种方式(除了简单的命令方式是hadoop fs -ls -R | grep -i yourfile
)....
下面是示例scala代码片段....(如果你想在python或java中这样做,你可以模拟相同的api调用) 获取镶木地板文件列表。并过滤它们如下....
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileStatus, FileSystem, Path}
import org.apache.hadoop.io.{BytesWritable, Text}
import org.apache.spark.{SparkConf, SparkContext}
//other imports here
lazy val sparkConf = new SparkConf()
lazy val sc = SparkContext.getOrCreate(sparkConf)
lazy val fileSystem = FileSystem.get(sc.hadoopConfiguration)
val fileSystem = listChaildStatuses(fileSystem , new Path("yourbasepathofHDFS")) // normally hdfs://server/user like this...
val allparquet = fileSystem.filter(_.getPath.getName.endsWith(".parquet"))
// now you can print these parquet files out of which your files will be present and you can know the base path...
支持方法如下所示
/**
* Get [[org.apache.hadoop.fs.FileStatus]] objects for all Chaild children (files) under the given base path. If the
* given path points to a file, return a single-element collection containing [[org.apache.hadoop.fs.FileStatus]] of
* that file.
*/
def listChaildStatuses(fs: FileSystem, basePath: Path): Seq[FileStatus] = {
listChaildStatuses(fs, fs.getFileStatus(basePath))
}
/**
* Get [[FileStatus]] objects for all Chaild children (files) under the given base path. If the
* given path points to a file, return a single-element collection containing [[FileStatus]] of
* that file.
*/
def listChaildStatuses(fs: FileSystem, baseStatus: FileStatus): Seq[FileStatus] = {
def recurse(status: FileStatus): Seq[FileStatus] = {
val (directories, leaves) = fs.listStatus(status.getPath).partition(_.isDirectory)
leaves ++ directories.flatMap(f => listChaildStatuses(fs, f))
}
if (baseStatus.isDirectory) recurse(baseStatus) else Seq(baseStatus)
}