指定Parquet属性pyspark

Question

如何在PySpark中指定Parquet Block Size和Page Size？我到处搜索但找不到函数调用或导入库的任何文档。

Answer 1

根据spark-user archives

sc.hadoopConfiguration.setInt("dfs.blocksize", some_value)
sc.hadoopConfiguration.setInt("parquet.block.size", some_value)

所以在PySpark中

sc._jsc.hadoopConfiguration().setInt("dfs.blocksize", some_value)
sc._jsc.hadoopConfiguration().setInt("parquet.block.size", some_value)