fiware - Cosmos中的用户HDFS配额管理

据我们所知，FIWARE Lab（cosmos.lab.fiware.org）中的每个Cosmos用户在HDFS中最多可以使用5GB。
尽管如此，我们在运行map-reduce Hadoop作业时得到public static final String FONT = "resources/fonts/NotoNaskhArabic-Regular.ttf"; public static final String ARABIC = "\u0627\u0644\u0633\u0639\u0631 \u0627\u0644\u0627\u062c\u0645\u0627\u0644\u064a"; public void createPdf(String dest) throws IOException, DocumentException { Document document = new Document(); PdfWriter.getInstance(document, new FileOutputStream(dest)); document.open(); Font f = FontFactory.getFont(FONT, BaseFont.IDENTITY_H, BaseFont.EMBEDDED); PdfPTable table = new PdfPTable(1); Phrase phrase = new Phrase(); Chunk chunk = new Chunk("test value"); phrase.add(chunk); phrase.add(new Chunk(ARABIC, f)); PdfPCell cell = new PdfPCell(phrase); cell.setUseDescender(true); cell.setRunDirection(PdfWriter.RUN_DIRECTION_RTL); table.addCell(cell); document.add(table); document.close(); }，尽管作业生成的数据并不超过5GB配额。

如果我们在执行map-reduce作业期间监控HDFS使用情况，我们会得到以下输出：

Command: "while true; do date; hadoop fs -count -q . ; sleep 20; done"
Format:
DATE
QUOTA  REMAINING_QUOTA SPACE_QUOTA    REMAINING_SPACE_QUOTA DIR_COUNT  FILE_COUNT CONTENT_SIZE   FILE_NAME

jue jul 28 18:50:12 CEST 2016
        none             inf      5368709120      1197734302           19           46         1389627219 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:50:34 CEST 2016
        none             inf      5368709120      2678747494           16           26          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:50:57 CEST 2016
        none             inf      5368709120      2678747494           16           26          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:51:20 CEST 2016
        none             inf      5368709120      2678747494           16           26          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:51:44 CEST 2016
        none             inf      5368709120      2678747494           16           26          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:07 CEST 2016
        none             inf      5368709120      2678747494           16           26          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:28 CEST 2016
        none             inf      5368709120      1198032544           22           35         1389528792 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:52:50 CEST 2016
        none             inf      5368709120      1197738517           19           39         1389625814 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:11 CEST 2016
        none             inf      5368709120      2678747494           16           27          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:35 CEST 2016
        none             inf      5368709120      2678747494           16           27          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:53:59 CEST 2016
        none             inf      5368709120      2678747494           16           27          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:54:22 CEST 2016
        none             inf      5368709120      2678747494           16           27          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:54:46 CEST 2016
        none             inf      5368709120      2678747494           16           27          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:09 CEST 2016
        none             inf      5368709120      2477420902           17           28          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:31 CEST 2016
        none             inf      5368709120      1197738514           19           39         1389625815 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:55:55 CEST 2016
        none             inf      5368709120      1197738514           20           48         1389625815 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:56:17 CEST 2016
        none             inf      5368709120      2678747506           16           28          895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:56:40 CEST 2016
        none             inf      5368709120      2678747506           16           28          895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:04 CEST 2016
        none             inf      5368709120      2678747506           16           28          895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:28 CEST 2016
        none             inf      5368709120      2678747506           16           28          895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:57:51 CEST 2016
        none             inf      5368709120      2678747506           16           28          895957138 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:13 CEST 2016
        none             inf      5368709120      1198032556           16           37         1389528788 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:34 CEST 2016
        none             inf      5368709120      1197738742           19           40         1389625760 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:58:56 CEST 2016
        none             inf      5368709120      2678747494           16           29          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:59:20 CEST 2016
        none             inf      5368709120      2678747494           16           29          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 18:59:43 CEST 2016
        none             inf      5368709120      2678747494           16           29          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:07 CEST 2016
        none             inf      5368709120      2678747494           16           29          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:31 CEST 2016
        none             inf      5368709120      2678747494           16           29          895957142 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:00:54 CEST 2016
        none             inf      5368709120      1076586601           22           38         1228684181 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:01:18 CEST 2016
        none             inf      5368709120      1197724648           19           41         1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:01:41 CEST 2016
        none             inf      5368709120      1197724648           19           41         1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:05 CEST 2016
        none             inf      5368709120      1197724648           19           41         1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:29 CEST 2016
        none             inf      5368709120      1197724648           19           41         1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:02:53 CEST 2016
        none             inf      5368709120      1197724648           19           41         1389630437 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:14 CEST 2016
        none             inf      5368709120       364004107           19           46         1667537284 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:36 CEST 2016
        none             inf      5368709120       197959591           20           48         1722885456 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:03:57 CEST 2016
        none             inf      5368709120       201060881           18           44         1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:04:19 CEST 2016
        none             inf      5368709120       201060881           18           44         1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:04:40 CEST 2016
        none             inf      5368709120       201060881           18           44         1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:05:02 CEST 2016
        none             inf      5368709120       201060881           18           44         1722549413 hdfs://cosmosmaster-gi/user/rbarriuso
jue jul 28 19:05:23 CEST 2016
        none             inf      5368709120       201060881           18           44         1722549413 hdfs://cosmosmaster-gi/user/rbarriuso

一段时间后，执行完成此异常：

16/07/28 19:03:11 INFO mapred.JobClient: Task Id : attempt_201604111313_157784_r_000006_0, Status : FAILED
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/rbarriuso is exceeded: quota=5368709120 diskspace consumed=5.0g
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3778)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3640)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2846)
    at org.apache.ha...

正如您在上面的日志末尾所看到的，最大HDFS用量对应于1.722.549.413字节和201.060.881字节的可用配额（根据DSQuotaExceededException），它们并不总和5GB的可用用户空间此外，占用的空间与剩余的自由空间不匹配。

如何计算剩余的配额空间？
有没有办法避免hadoop fs -count -q？

提前致谢。

Cosmos中的用户HDFS配额管理

1 个答案: