I am trying to filter a dataframe like below
'
val industry_df = industry_df0.filter( col("classification_type").===("GICS") )
'
Me getting error
java.lang.AssertionError: assertion failed: unsafe symbol Unstable (child of <none>) in runtime reflection universe
at scala.reflect.internal.Symbols$Symbol.<init>(Symbols.scala:205)
at scala.reflect.internal.Symbols$TypeSymbol.<init>(Symbols.scala:3030)
at scala.reflect.internal.Symbols$ClassSymbol.<init>(Symbols.scala:3222)
1)我在这里做错了什么?如何解决?
2)当我尝试执行$(“ classification_type”)。===(“” GICS“)
I am getting $ not found error , what should I include to use $ on dataframe.
3)我正在尝试将连接查询与spark-sql一起使用,但不起作用,如何调试它?当然,此查询LOGIC适用于oracle db。
'
val result_df = sparkSession.sql("""
select a.company_id, a.data_date, a.model_id, b.domicile_country_code, d.two_digit_iso_code, d.capiq_geo_name
from model_lnk a join company_filter_vals b on a.company_id=b.company_id
join reg_lnk c
join country d on c.ctry_geo_code=d.geo_code
where a.model_id in (3020,3030)
and b.model_group_id in (select model_group_id from model where model_id in (3020,3030))
""" )
result_df.show(2)
'