Question

我想使用Scala和可能的Spark处理1987年的路透数据集。我可以看到我下载的文件是.sgm格式。我之前从未见过这种情况，但是执行了more：

$ more reut2-003.sgm 
<!DOCTYPE lewis SYSTEM "lewis.dtd">
<REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="19419" NEWID="3001">
<DATE> 9-MAR-1987 04:58:41.12</DATE>
<TOPICS><D>money-fx</D></TOPICS>
<PLACES><D>uk</D></PLACES>
<PEOPLE></PEOPLE>
<ORGS></ORGS>
<EXCHANGES></EXCHANGES>
<COMPANIES></COMPANIES>
<UNKNOWN> 
&#5;&#5;&#5;RM
&#22;&#22;&#1;f0416&#31;reute
b f BC-U.K.-MONEY-MARKET-SHO   03-09 0095</UNKNOWN>
<TEXT>&#2;
<TITLE>U.K. MONEY MARKET SHORTAGE FORECAST AT 250 MLN STG</TITLE>
<DATELINE>    LONDON, March 9 - </DATELINE><BODY>The Bank of England said it forecast a
shortage of around 250 mln stg in the money market today.
    Among the factors affecting liquidity, it said bills
maturing in official hands and the treasury bill take-up would
drain around 1.02 billion stg while below target bankers'
balances would take out a further 140 mln.
    Against this, a fall in the note circulation would add 345
mln stg and the net effect of exchequer transactions would be
an inflow of some 545 mln stg, the Bank added.
 REUTER
&#3;</BODY></TEXT>
</REUTERS>

我们可以看到它看起来很简单。

由于我不想编写自己的解析器，所以我的问题是，是否有一些简单的方法可以使用某些库在Scala / Spark中对此进行解析？

Answer 1

问：由于我不想编写自己的解析器，所以我的问题是使用一些库在Scala / Spark中对此进行解析的简单方法？

AFAIK没有此类api。您必须对其进行映射和解析（清除其中的特殊字符）。转换为多列。

我以下面的方式尝试了...但是您的xml从数据框中显示为损坏的记录。

进一步的指针：https://github.com/databricks/spark-xml

import java.io.File

import org.apache.commons.io.FileUtils
import org.apache.spark.sql.{SQLContext, SparkSession}

/**
  * Created by Ram Ghadiyaram
  */
object SparkXmlWithDtd {
  def main(args: Array[String]) {

    val spark = SparkSession.builder.
      master("local")
      .appName(this.getClass.getName)
      .getOrCreate()
    spark.sparkContext.setLogLevel("ERROR")
    val sc = spark.sparkContext
    val sqlContext = new SQLContext(sc)
    val str =
      """
        |<!DOCTYPE lewis SYSTEM "lewis.dtd">
        |
        |<REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="19419" NEWID="3001">
        |<DATE> 9-MAR-1987 04:58:41.12</DATE>
        |<TOPICS><D>money-fx</D></TOPICS>
        |<PLACES><D>uk</D></PLACES>
        |<PEOPLE></PEOPLE>
        |<ORGS></ORGS>
        |<EXCHANGES></EXCHANGES>
        |<COMPANIES></COMPANIES>
        |<UNKNOWN>
        |&#5;&#5;&#5;RM
        |&#22;&#22;&#1;f0416&#31;reute
        |b f BC-U.K.-MONEY-MARKET-SHO   03-09 0095</UNKNOWN>
        |<TEXT>&#2;
        |<TITLE>U.K. MONEY MARKET SHORTAGE FORECAST AT 250 MLN STG</TITLE>
        |<DATELINE>    LONDON, March 9 - </DATELINE><BODY>The Bank of England said it forecast a
        |shortage of around 250 mln stg in the money market today.
        |    Among the factors affecting liquidity, it said bills
        |maturing in official hands and the treasury bill take-up would
        |drain around 1.02 billion stg while below target bankers'
        |balances would take out a further 140 mln.
        |    Against this, a fall in the note circulation would add 345
        |mln stg and the net effect of exchequer transactions would be
        |an inflow of some 545 mln stg, the Bank added.
        | REUTER
        |&#3;</BODY></TEXT>
        |</REUTERS>
      """.stripMargin
    val f = new File("sgmtest.sgm")
    FileUtils.writeStringToFile(f, str)
    val xml_df = spark.read.
      format("com.databricks.spark.xml")
      .option("rowTag", "REUTERS")
      .load(f.getAbsolutePath)
    xml_df.printSchema()
    xml_df.createOrReplaceTempView("XML_DATA")
    spark.sql("SELECT * FROM XML_DATA").show(false)
    xml_df.show(false)

  }

}

结果：

root
 |-- _corrupt_record: string (nullable = true)

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|_corrupt_record                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
 9-MAR-1987 04:58:41.12
money-fx
uk





RM
f0416reute
b f BC-U.K.-MONEY-MARKET-SHO   03-09 0095

U.K. MONEY MARKET SHORTAGE FORECAST AT 250 MLN STG
    LONDON, March 9 - The Bank of England said it forecast a
shortage of around 250 mln stg in the money market today.
    Among the factors affecting liquidity, it said bills
maturing in official hands and the treasury bill take-up would
drain around 1.02 billion stg while below target bankers'
balances would take out a further 140 mln.
    Against this, a fall in the note circulation would add 345
mln stg and the net effect of exchequer transactions would be
an inflow of some 545 mln stg, the Bank added.
 REUTER

|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|_corrupt_record                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
 9-MAR-1987 04:58:41.12
money-fx
uk





RM
f0416reute
b f BC-U.K.-MONEY-MARKET-SHO   03-09 0095

U.K. MONEY MARKET SHORTAGE FORECAST AT 250 MLN STG
    LONDON, March 9 - The Bank of England said it forecast a
shortage of around 250 mln stg in the money market today.
    Among the factors affecting liquidity, it said bills
maturing in official hands and the treasury bill take-up would
drain around 1.02 billion stg while below target bankers'
balances would take out a further 140 mln.
    Against this, a fall in the note circulation would add 345
mln stg and the net effect of exchequer transactions would be
an inflow of some 545 mln stg, the Bank added.
 REUTER

|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

如何使用Scala读取sgm文件

1 个答案: