Question

twitter的推文存储在hadoop中的hdfs中。这些推文需要进行情感分析。 hdfs中的推文为avro格式，因此需要使用Json loader进行处理。但是在Pig脚本中，HDFS中的推文无法读取。更改jar文件后，pig脚本显示失败消息

通过Pig脚本使用以下以下jar文件失败。

注册'/home/cloudera/Desktop/elephant-bird-hadoop-compat-4.17.jar';

注册'/home/cloudera/Desktop/elephant-bird-pig-4.17.jar';

注册'/home/cloudera/Desktop/json-simple-3.1.0.jar';

这是另一组jar文件，通过它们它们不会失败，但是数据也没有被读取。

注册'/home/cloudera/Desktop/elephant-bird-hadoop-compat-4.17.jar';

注册'/home/cloudera/Desktop/elephant-bird-pig-4.17.jar';

注册'/home/cloudera/Desktop/json-simple-1.1.jar';

这是我使用的所有猪脚本命令：

tweets = LOAD '/user/cloudera/OutputData/tweets' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS myMap;

B = FOREACH tweets GENERATE myMap#'id' as id ,myMap#'tweets' as tweets;

tokens = foreach B generate id, tweets, FLATTEN(TOKENIZE(tweets)) As word;

dictionary = load ' /user/cloudera/OutputData/AFINN.txt' using PigStorage('\t') AS(word:chararray,rating:int);

word_rating = join tokens by word left outer, dictionary by word using 'replicated';

describe word_rating;

rating = foreach word_rating generate tokens::id as id,tokens::tweets as tweets, dictionary::rating as rate;

word_group = group rating by (id,tweets);

avg_rate = foreach word_group generate group, AVG(rating.rate) as tweet_rating;

positive_tweets = filter avg_rate by tweet_rating>=0;
DUMP positive_tweets;

negative_tweets = filter avg_rate by tweet_rating<=0;

DUMP negative_tweets;

对于第一组jar文件，在上述tweets命令上转储时出错：

输入：   无法从“ / user / cloudera / OutputData / tweets”读取数据

输出：   无法在“ hdfs：//quickstart.cloudera：8020 / tmp / temp-1614543351 / tmp37889715”中生成结果

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1556902124324_0001


2019-05-03 09:59:09,409 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2019-05-03 09:59:09,427 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias tweets. Backend error : org.json.simple.parser.ParseException
Details at logfile: /home/cloudera/pig_1556902594207.log

针对第二组jar文件转储上述tweets命令时出错：

输入：   从“ / user / cloudera / OutputData / tweets”成功读取0条记录（5178477字节）

输出：   已成功将0条记录存储在：“ hdfs：//quickstart.cloudera：8020 / tmp / temp-1614543351 / tmp479037703”

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1556902124324_0002


2019-05-03 10:01:05,417 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2019-05-03 10:01:05,418 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2019-05-03 10:01:05,418 [main] INFO  org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2019-05-03 10:01:05,428 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2019-05-03 10:01:05,428 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1

预期的输出按正向和反向Tweet排序，但出现错误。请帮忙。谢谢。

Answer 1

ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias tweets. Backend error : org.json.simple.parser.ParseException这通常表示Pig脚本中存在语法错误。

LOAD语句中的AS关键字通常需要一个模式。 LOAD语句中的myMap无效。

有关JsonLoader的示例，请参见https://stackoverflow.com/a/12829494/8886552。

使用hadoop和Pig的Twitter数据的情感分析

1 个答案: