“sqoop-merge --merge-key”和“sqoop --import --merge-key”之间的区别

时间:2017-03-29 03:42:41

标签: hadoop merge sqoop bigdata

ITradesman tradie = Tradesman.Factory(); // in reality i know it's a plumber, but in the real world you won't know who's on the other side of the tradie assignment. tradie.Work(); // and then tradie will do the work of a plumber, or electrician etc. depending on what type of tradesman he is. The foreman doesn't need to know anything, apart from telling the anonymous tradie to get to Work()!! sqoop-merge --merge-key之间的差异只是sqoop --import --merge-key会先将增量数据保存到hdfs然后与旧数据合并,而sqoop-merge --merge-key将加载新数据并直接将数据与旧数据合并?

如果我的旧数据太大而无法合并,哪种更新旧数据的方式更好?

现在我的旧数据大约是40GB,当我使用sqoop --import --merge-key来更新数据时,我需要更长时间才能直接加载所有数据。由于RAM的使用,有时会导致mapreduce作业失败。那么,我可以进行一些优化来解决这个问题吗?

0 个答案:

没有答案