Question

我们有出版物的数据，我们希望将它们标记为各种类别。我们已将它们存储到HBase中，然后我们将它们保存到sparkrdd中，然后使用scala代码对其进行标记。 HBase数据的样本如下所示：

PubEntity:Abstract                  timestamp=1476537886382, value=not found                                                                  
 PubEntity:Affiliations              timestamp=1476537886382, value=[]                                                                         
 PubEntity:Article_Title             timestamp=1476537886382, value=Formate assay in body fluids: application in methanol poisoning.           
 PubEntity:Author                    timestamp=1476537886382, value=[{'LastName': 'Makar', 'ForeName': 'A B', 'author_name': 'A B Makar', 'Init
                                     ials': 'AB', 'author_affiliation': 'not found'}, {'LastName': 'McMartin', 'ForeName': 'K E', 'author_name'
                                     : 'K E McMartin', 'Initials': 'KE', 'author_affiliation': 'not found'}, {'LastName': 'Palese', 'ForeName':
                                      'M', 'author_name': 'M Palese', 'Initials': 'M', 'author_affiliation': 'not found'}, {'LastName': 'Tephly
                                     ', 'ForeName': 'T R', 'author_name': 'T R Tephly', 'Initials': 'TR', 'author_affiliation': 'not found'}]  
 PubEntity:Journal_Title             timestamp=1476537886382, value=Biochemical medicine                                                       
 PubEntity:PMID                      timestamp=1476537886382, value=1                                                                          
 PubRemaining:Countries              timestamp=1476537886382, value=[]                                                                         
 PubRemaining:Created_At             timestamp=1476537886382, value=170812800.0                                                                
 PubRemaining:DOI                    timestamp=1476537886382, value=not found                                                                  
 PubRemaining:Date_Created           timestamp=1476537886382, value=19760116                                                                   
 PubRemaining:ISO_Abbreviation       timestamp=1476537886382, value=Biochem Med                                                                
 PubRemaining:ISSN                   timestamp=1476537886382, value=0006-2944                                                                  
 PubRemaining:Pub_Date               timestamp=1476537886382, value=01 Jun, 1975                                                               
 PubRemaining:Year                   timestamp=1476537886382, value=1975

我们已经能够在

的第一个答案中解释火花中RDD

How to read from hbase using spark 这是代码：

import org.apache.hadoop.hbase.client.{HBaseAdmin, Result}
import org.apache.hadoop.hbase.{ HBaseConfiguration, HTableDescriptor }
import org.apache.hadoop.hbase.mapreduce.TableInputFormat
import org.apache.hadoop.hbase.io.ImmutableBytesWritable

import org.apache.spark._

object HBaseRead {
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("HBaseRead").setMaster("local[2]").set("spark.driver.allowMultipleContexts", "true").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    val sc = new SparkContext(sparkConf)
    val conf = HBaseConfiguration.create()
    val tableName = "name_of_the_database"

    System.setProperty("user.name", "hdfs")
    System.setProperty("HADOOP_USER_NAME", "hdfs")
    conf.set("hbase.master", "localhost:60000")
    conf.setInt("timeout", 120000)
    conf.set("hbase.zookeeper.quorum", "localhost")
    conf.set("zookeeper.znode.parent", "/hbase-unsecure")
    conf.set(TableInputFormat.INPUT_TABLE, tableName) 

    val admin = new HBaseAdmin(conf)
    if (!admin.isTableAvailable(tableName)) {
      val tableDesc = new HTableDescriptor(tableName)
      admin.createTable(tableDesc)
    }

    val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[ImmutableBytesWritable], classOf[Result])
    println(" Number of Records found : " + hBaseRDD.count())
    println(hBaseRDD.first())
    sc.stop()
  }
}

在Scala中运行此代码块之后，我得到了：defined module HBaseRead 然后我做了HBaseRead.main（Array（）），它输出找到的记录数并读取第一条记录：

（31，keyvalues = {1 / PubEntity：Abstract / 1476537886382 / Put / vlen = 9 / mvcc = 0,1 / Entity：Affiliations / 1476537886382 / Put / vlen = 2 / mvcc = 0,1 / Entity：Article_Title / 1476537886382 / Put / vlen = 64 / mvcc = 0,1 / Entity：Author / 1476537886382 / Put / vlen = 497 / mvcc = 0,1 / Entity：Journal_Title / 1476537886382 / Put / vlen = 20 / mvcc = 0,1 /实体：PMID / 1476537886382 / Put / vlen = 1 / mvcc = 0,1 / Remaining：Countries / 1476537886382 / Put / vlen = 2 / mvcc = 0,1 / Remaining：Created_At / 1476537886382 / Put / vlen = 11 / mvcc = 0,1 /剩余：DOI / 1476537886382 / Put / vlen = 9 / mvcc = 0,1 /剩余：Date_Created / 1476537886382 / Put / vlen = 8 / mvcc = 0,1 / Remaining：ISO_Abbreviation / 1476537886382 / Put / vlen = 11 / mvcc = 0,1 /剩余：ISSN / 1476537886382 / Put / vlen = 9 / mvcc = 0,1 /剩余：Pub_Date / 1476537886382 / Put / vlen = 12 / mvcc = 0,1 /剩余：年/ 1476537886382 /放/ VLEN = 4 / MVCC = 0}）

现在在这个输出中，你会看到vlen = 12 / mvcc = 0。在检查数据时，我发现vlen是每个单词/数字的长度。我无法弄清楚mvcc的用途。我们希望输出显示单词/数字而不是vlen = 4。此外，我们希望阅读这些条目并在其中查找某些单词和短语并相应地标记它们。全部在斯卡拉。

任何有用的在线资源的链接都将受到高度赞赏。

标记SparkRDD

0 个答案: