从MongoDB迁移到HBase

时间:2013-03-05 08:06:18

标签: mongodb hadoop hbase

您好我是HBase数据库的新手。我下载了一些twitter数据并存储到MongoDB中。现在我需要将这些数据转换为HBase以加速Hadoop处理。但我无法创建它的计划。在这里,我将Twitter数据转换为JSON格式 -

{
"_id" : ObjectId("512b71e6e4b02a4322d1c0b0"),
"id" : NumberLong("306044618179506176"),
"source" : "<a href=\"http://www.facebook.com/twitter\" rel=\"nofollow\">Facebook</a>",
"user" : {
    "name" : "Dada Bhagwan",
    "location" : "India",
    "url" : "http://www.dadabhagwan.org",
    "id" : 191724440,
    "protected" : false,
    "timeZone" : null,
    "description" : "Founder of Akram Vignan - Practical Spiritual Science of Self Realization",
    "screenName" : "dadabhagwan",
    "geoEnabled" : false,
    "profileImageURL" : "http://a0.twimg.com/profile_images/1647956820/M_DSC_0034_normal.jpg",
    "biggerProfileImageURL" : "http://a0.twimg.com/profile_images/1647956820/M_DSC_0034_bigger.jpg",
    "profileImageUrlHttps" : "https://si0.twimg.com/profile_images/1647956820/M_DSC_0034_normal.jpg",
    "profileImageURLHttps" : "https://si0.twimg.com/profile_images/1647956820/M_DSC_0034_normal.jpg",
    "biggerProfileImageURLHttps" : "https://si0.twimg.com/profile_images/1647956820/M_DSC_0034_bigger.jpg",
    "miniProfileImageURLHttps" : "https://si0.twimg.com/profile_images/1647956820/M_DSC_0034_mini.jpg",
    "originalProfileImageURLHttps" : "https://si0.twimg.com/profile_images/1647956820/M_DSC_0034.jpg",
    "followersCount" : 499,
    "profileBackgroundColor" : "EEE4C1",
    "profileTextColor" : "333333",
    "profileLinkColor" : "990000",
    "lang" : "en",
    "profileSidebarFillColor" : "FCF9EC",
    "profileSidebarBorderColor" : "CBC09A",
    "profileUseBackgroundImage" : true,
    "showAllInlineMedia" : false,
    "friendsCount" : 1,
    "favouritesCount" : 0,
    "profileBackgroundImageUrl" : "http://a0.twimg.com/profile_background_images/396759326/dadabhagwan-twitter.jpg",
    "profileBackgroundImageURL" : "http://a0.twimg.com/profile_background_images/396759326/dadabhagwan-twitter.jpg",
    "profileBackgroundImageUrlHttps" : "https://si0.twimg.com/profile_background_images/396759326/dadabhagwan-twitter.jpg",
    "profileBannerURL" : null,
    "profileBannerRetinaURL" : null,
    "profileBannerIPadURL" : null,
    "profileBannerIPadRetinaURL" : null,
    "miniProfileImageURL" : "http://a0.twimg.com/profile_images/1647956820/M_DSC_0034_mini.jpg",
    "originalProfileImageURL" : "http://a0.twimg.com/profile_images/1647956820/M_DSC_0034.jpg",
    "utcOffset" : -1,
    "contributorsEnabled" : false,
    "status" : null,
    "createdAt" : NumberLong("1284700143000"),
    "profileBannerMobileURL" : null,
    "profileBannerMobileRetinaURL" : null,
    "profileBackgroundTiled" : false,
    "statusesCount" : 1713,
    "verified" : false,
    "translator" : false,
    "listedCount" : 6,
    "followRequestSent" : false,
    "descriptionURLEntities" : [ ],
    "urlentity" : {
        "url" : "http://www.dadabhagwan.org",
        "start" : 0,
        "end" : 26,
        "expandedURL" : "http://www.dadabhagwan.org",
        "displayURL" : "http://www.dadabhagwan.org"
    },
    "rateLimitStatus" : null,
    "accessLevel" : 0
},
"contributors" : [ ],
"geoLocation" : null,
"place" : null,
"favorited" : false,
"retweet" : false,
"retweetedStatus" : null,
"retweetCount" : 0,
"userMentionEntities" : [ ],
"retweetedByMe" : false,
"currentUserRetweetId" : -1,
"possiblySensitive" : false,
"urlentities" : [
    {
        "url" : "http://t.co/gR1GohGjaj",
        "start" : 113,
        "end" : 135,
        "expandedURL" : "http://fb.me/2j2HKHJrM",
        "displayURL" : "fb.me/2j2HKHJrM"
    }
],
"hashtagEntities" : [ ],
"mediaEntities" : [ ],
"truncated" : false,
"inReplyToStatusId" : -1,
"text" : "Spiritual Quote of the Day :\n\n‘I am Chandubhai’ is an illusion itself and from that are \nkarmas charged. When... http://t.co/gR1GohGjaj",
"inReplyToUserId" : -1,
"inReplyToScreenName" : null,
"createdAt" : NumberLong("1361801697000"),
"rateLimitStatus" : null,
"accessLevel" : 0
}

这里如何将数据划分为列和列族?我想创建一个"twitter" column-family,其中包含source, getlocation, place, retweet etc...和另一个"user" column-family,其中包含name, location etc...(用户的数据)。即每个内层子文档的新列族。

这种方法是否正确?现在,我将如何区分urlentity "user" column-family"twitter" column-family

以及如何处理包含子文档列表的键(例如urlentity

1 个答案:

答案 0 :(得分:3)

有许多方法可以在HBase中对此进行建模,包括将所有内容存储在一个列中,每个子实体都有一个不同的表,以及其他几个用于“索引”的表。

一般来说,您可以根据读写访问模式对hbase中的数据进行建模。示例列族存储在磁盘上的不同文件中。将数据划分为两个列族的原因是,在很多情况下,您需要来自一个而不是另一个的数据。等

有关HBAse架构设计的精彩演示,来自HBaseCon 2012的Ian Varley您可以找到幻灯片here和视频here