我希望能够在hdfs文件系统上读/写图像并利用hdfs局部性。
我有一组图像,每个图像由
组成我想在hdfs文件系统上创建一个存档,并使用spark来分析存档。现在,我正在努力克服在hdfs文件系统上存储数据的最佳方式,以便能够充分利用spark + hdfs结构。
据我所知,最好的方法是创建一个sequenceFile包装器。我有两个问题:
答案 0 :(得分:7)
我找到了一个有效的解决方案:使用pyspark 1.2.0二进制文件完成这项工作。它被标记为实验性的,但我能够使用openCV的正确组合读取tiff图像。
import cv2
import numpy as np
# build rdd and take one element for testing purpose
L = sc.binaryFiles('hdfs://localhost:9000/*.tif').take(1)
# convert to bytearray and then to np array
file_bytes = np.asarray(bytearray(L[0][1]), dtype=np.uint8)
# use opencv to decode the np bytes array
R = cv2.imdecode(file_bytes,1)
请注意pyspark的帮助:
binaryFiles(path, minPartitions=None)
:: Experimental
Read a directory of binary files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI as a byte array. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file.
Note: Small files are preferred, large file is also allowable, but may cause bad performance.