Question

我想在Pig中解析一串复杂的JSON。具体来说，我希望Pig将我的JSON数组理解为一个包而不是单个chararray。我发现可以使用Twitter's Elephant Bird或Mozilla's Akela库来解析复杂的JSON。（我发现了一些额外的库，但我不能使用基于Loader的方法，因为我使用HCatalog Loader从Hive加载数据。）

但问题是我的数据结构; Map结构的每个值都包含复杂JSON的值部分。例如，

1. My table looks like  (WARNING: type of 'complex_data' is not STRING, a MAP of <STRING, STRING>!)
TABLE temp_table
(
    user_id BIGINT COMMENT 'user ID.',
    complex_data MAP <STRING, STRING> COMMENT 'complex json data'
)
COMMENT 'temp data.'
PARTITIONED BY(created_date STRING)
STORED AS RCFILE;


2. And 'complex_data' contains (a value that I want to get is marked with two *s, so basically #'d'#'f' from each PARSED_STRING(complex_data#'c')  )
{ "a": "[]", 
  "b": "\"sdf\"", 
  "**c**":"[{\"**d**\":{\"e\":\"sdfsdf\"
                      ,\"**f**\":\"sdfs\"
                      ,\"g\":\"qweqweqwe\"},
             \"c\":[{\"d\":21321,\"e\":\"ewrwer\"},
                   {\"d\":21321,\"e\":\"ewrwer\"},
                   {\"d\":21321,\"e\":\"ewrwer\"}]
            },
            {\"**d**\":{\"e\":\"sdfsdf\"
                      ,\"**f**\":\"sdfs\"
                      ,\"g\":\"qweqweqwe\"},
             \"c\":[{\"d\":21321,\"e\":\"ewrwer\"},
                   {\"d\":21321,\"e\":\"ewrwer\"},
                   {\"d\":21321,\"e\":\"ewrwer\"}]
            },]"
}

3. So, I tried... (same approach for Elephant Bird)

REGISTER '/path/to/akela-0.6-SNAPSHOT.jar';
DEFINE JsonTupleMap com.mozilla.pig.eval.json.JsonTupleMap();

data = LOAD temp_table USING org.apache.hive.hcatalog.pig.HCatLoader();
values_of_map = FOREACH data GENERATE complex_data#'c' AS attr:chararray;    -- IT WORKS

-- dump values_of_map shows correct chararray data per each row
-- eg) ([{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
         {"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
         {"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... }])
       ([{"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
         {"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... },
         {"d":{"e":"sdfsdf","f":"sdfs","g":"sdf"},... }]) ...

attempt1 = FOREACH data GENERATE JsonTupleMap(complex_data#'c');   -- THIS LINE CAUSE AN ERROR 
attempt2 = FOREACH data GENERATE JsonTupleMap(CONCAT(CONCAT('{\\"key\\":', complex_data#'c'), '}');   -- IT ALSO DOSE NOT WORK

我猜想＆＃34;尝试1＆＃34;失败，因为该值不包含完整的JSON。但是，当我CONCAT喜欢＆＃34; attempt2＆＃34;时，我会生成额外的\ mark。（所以每一行都以 {\＆＃34; key \＆＃34;：）开头。我不确定这些额外的标记是否会破坏解析规则。无论如何，我想解析给定的JSON字符串，以便Pig可以理解。如果您有任何方法或解决方案，请随时告诉我。

Answer 1

我终于使用jyson库jython UDF解决了我的问题。我知道我可以使用JAVA或其他语言来解决它。但是，我认为jyon与jyson是这个问题最简单的答案。

解析Hadoop中包含的复杂Json字符串

1 个答案: