我在使用R数据框调用createOrReplaceTempView时收到此警告：
createOrReplaceTempView (as.Data.Frame(products), "prod")
我应该忽略这个警告吗？这效率很低？
谢谢！
答案 0 :(得分：-1)
这些只是警告。如果要尝试避免它们，请在注册临时表并对数据执行某些功能之前对数据进行重新分区并对其执行操作。重新分配将导致洗牌。
例如，
set.seed(123)
df<- data.frame(thing1=rnorm(100000), thing2=rep("ThisIsAString", 100000), stringsAsFactors = FALSE)
sdf<- SparkR::createDataFrame(df) # Warnings for me
SparkR::getNumPartitions(sdf) # 1 partition
sdf<- SparkR::repartition(sdf, numPartitions=4L) # repartition, will cause a shuffle
SparkR::getNumPartitions(sdf) # spark now knows to repartition the data, this will happen once an action is called on the data, i.e. counting the rows
SparkR::cache(sdf) # Nothing has happened yet
SparkR::nrow(sdf) # Now cause the repartition and a count to happen. # Will be warned
SparkR::createOrReplaceTempView(sdf, "sdfTable") # Make a temp table as you have in your example
res<- SparkR::sql("SELECT thing1, thing2 FROM sdfTable WHERE thing1> 0.5") # SQL
SparkR::nrow(res) # no warnings, 31002 observations found.
SparkR::getNumPartitions(res) # 4 partitions in the result