源代码:
val FILES_MAX_PARTITION_BYTES = SQLConfigBuilder("spark.sql.files.maxPartitionBytes")
.doc("The maximum number of bytes to pack into a single partition when reading files.")
.longConf
.createWithDefault(128 * 1024 * 1024) // parquet.block.size
增加/减少时读取是否有任何性能提升?
在阅读orc
个文件时是否需要设置256MB?
此外,如何正确设置