Git gc使用了大量内存,即使我限制它

时间:2017-02-11 11:26:13

标签: git memory

我在共享主机(uberspace.de)上运行git并在执行git pushgit gc时达到内存限制。他们报告的内存消耗大约为1500 MB。

所以我按照

中的建议

https://stackoverflow.com/a/8761464/288568

它似乎工作了一次,但经过大量的提交后它再也无济于事了,内存消耗再次大约为1500MB,git pack进程被杀死了。

我提交了一个120 MB的SQL转储,只有很少的更改(每次大约5行) - 所以打包的存储库只有大约150 MB,但解压缩大约3 GB(我点击了bitbucket的限制)。 / p>

我看到哪个pmap打开了很多包对象 - 它是否会增加高内存?怎么能限制?

最后我得到" pack-objects死于信号15"和来自主持人的邮件,我的过程被杀了。

sh-4.1$ git config --list|grep pack
pack.windowmemory=25m
pack.packsizelimit=25m
pack.threads=1
pack.deltecachesize=25m
sh-4.1$ git gc
Counting objects: 2586, done.
^Zmpressing objects:  15% (163/1085)   
[1]+  Stopped(SIGTSTP)        git gc
sh-4.1$ bg
[1] git gc &
sh-4.1$ pmap 14190
14190:   git pack-objects --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago --local --delta-base-offset .git/objects/pack/.tmp-14187-pack
0000000000400000   1848K r-x--  /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007ce000     36K rw---  /home/foo/.toast/pkg/git/v2.11.1/1/root/libexec/git-core/git
00000000007d7000    280K rw---    [ anon ]
0000000000da1000   1680K rw---    [ anon ]
00007f735dc5d000 161648K rw---    [ anon ]
00007f7377555000 149580K rw---    [ anon ]
00007f73915ee000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.pack
00007f73990b6000 125592K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.pack
00007f73a0b5c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.pack
00007f73a8624000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.pack
00007f73b00ec000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.pack
00007f73b7bb4000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.pack
00007f73bf67c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.pack
00007f73c7144000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.pack
00007f73cec0c000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.pack
00007f73d66d4000 125348K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.pack
00007f73de13d000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.pack
00007f73e5c05000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.pack
00007f73ed6cb000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.pack
00007f73f5191000 120872K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.pack
00007f73fc79b000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.pack
00007f7404263000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.pack
00007f740bd29000 125720K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.pack
00007f74137ef000 120868K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.pack
00007f741adf8000 125728K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.pack
00007f74228c0000    484K rw---    [ anon ]
00007f7422939000  96840K r----  /usr/lib/locale/locale-archive
00007f74287cb000      8K r-x--  /lib64/libdl-2.12.so
00007f74287cd000   2048K -----  /lib64/libdl-2.12.so
00007f74289cd000      4K r----  /lib64/libdl-2.12.so
00007f74289ce000      4K rw---  /lib64/libdl-2.12.so
00007f74289cf000   1576K r-x--  /lib64/libc-2.12.so
00007f7428b59000   2048K -----  /lib64/libc-2.12.so
00007f7428d59000     16K r----  /lib64/libc-2.12.so
00007f7428d5d000      8K rw---  /lib64/libc-2.12.so
00007f7428d5f000     16K rw---    [ anon ]
00007f7428d63000     92K r-x--  /lib64/libpthread-2.12.so
00007f7428d7a000   2048K -----  /lib64/libpthread-2.12.so
00007f7428f7a000      4K r----  /lib64/libpthread-2.12.so
00007f7428f7b000      4K rw---  /lib64/libpthread-2.12.so
00007f7428f7c000     16K rw---    [ anon ]
00007f7428f80000     28K r-x--  /lib64/librt-2.12.so
00007f7428f87000   2044K -----  /lib64/librt-2.12.so
00007f7429186000      4K r----  /lib64/librt-2.12.so
00007f7429187000      4K rw---  /lib64/librt-2.12.so
00007f7429188000   1768K r-x--  /usr/lib64/libcrypto.so.1.0.1e
00007f7429342000   2044K -----  /usr/lib64/libcrypto.so.1.0.1e
00007f7429541000    108K r----  /usr/lib64/libcrypto.so.1.0.1e
00007f742955c000     48K rw---  /usr/lib64/libcrypto.so.1.0.1e
00007f7429568000     16K rw---    [ anon ]
00007f742956c000     84K r-x--  /lib64/libz.so.1.2.3
00007f7429581000   2044K -----  /lib64/libz.so.1.2.3
00007f7429780000      4K r----  /lib64/libz.so.1.2.3
00007f7429781000      4K rw---  /lib64/libz.so.1.2.3
00007f7429782000    128K r-x--  /lib64/ld-2.12.so
00007f74297a6000    464K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.pack
00007f742981a000     52K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.pack
00007f7429827000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.pack
00007f7429828000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.pack
00007f7429829000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.pack
00007f742982a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.pack
00007f742982b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.pack
00007f742982c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.pack
00007f742982d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.pack
00007f742982e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.pack
00007f742982f000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.pack
00007f7429831000     20K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.pack
00007f7429836000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.pack
00007f7429838000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.pack
00007f742983a000      8K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.pack
00007f742983c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.pack
00007f742983d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.pack
00007f742983e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.pack
00007f742983f000     20K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.pack
00007f7429875000      4K rw---    [ anon ]
00007f7429876000    400K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.pack
00007f74298da000    512K rw---    [ anon ]
00007f742995a000     44K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-004517110da3d0eaeea50b4eace06633eab4294c.idx
00007f7429965000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-4c28c08cbf0279f71e3c0a97cc91639fd6a8b308.idx
00007f7429966000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-ee7d89bc7f8e5cd3e36c3655d967dfc8af3e6c7b.idx
00007f7429967000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fcebe7c0d63e44015ecc186ed74d45cb998c22ef.idx
00007f7429968000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-0381d5fab5a4f18e02eb18c3e12c5585702c9564.idx
00007f7429969000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d497de57bd0e623e5338eb553e0940ea2442274f.idx
00007f742996a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-f26f972bc47c3a18cf47493e680c90fa3d5c9046.idx
00007f742996b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-33af25946a2fcb1c3e284b3093ac0eafa440cf46.idx
00007f742996c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-30fe739ba7e266ddf6f09a29fc2aa49eedd9e90b.idx
00007f742996d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-56f5137267429f72a6e57ec70a637dc944cfeec7.idx
00007f742996e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fb13617b92c223a9d017935c5ed27c9c8b0e62de.idx
00007f742996f000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-aeb01e8db0b4d43181626d8b7a412e813a4a8757.idx
00007f7429970000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d4bc55fff1083c9601bafeb49abf378cdc6f0727.idx
00007f7429971000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-564815c1b986e15fa385daf51d6ecc9f44a07679.idx
00007f7429972000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e104e0bffca85b6346ef588c8c63d6ddadc567e3.idx
00007f7429973000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5fc5649f65f8596a6b3dfd7c71d47d34e7b150e5.idx
00007f7429974000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5eeddad5db2c0a44f33a3c41cd9216a7af0417df.idx
00007f7429975000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-937dae9140ac11d33831a697f9347f98baa25894.idx
00007f7429976000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-fc2614aa21f9284e0b76b6cc3ad9a90a0bae371e.idx
00007f7429977000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-1468b8f7d29c999297b2411f9b94d86b0cd9965d.idx
00007f7429978000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-739869658456c4a7627279fceafa7c7c4c408acb.idx
00007f7429979000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bb77ce8e83062bf2d26a3366b5b4e0f77c3a3810.idx
00007f742997a000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-feabd42e4392c52c8864a0397d10a73d7ada6fc8.idx
00007f742997b000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-dc37fbdaf5ad7f6c317a7c3d5ffecf53daa045b6.idx
00007f742997c000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-26462c3338a2e9302852788bad7b827a98748b65.idx
00007f742997d000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-3ecc274dd42f4783926224457225968b1fe1d95b.idx
00007f742997e000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-69b7829b456d813335b339545583ac9585ba9faa.idx
00007f742997f000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e2d986a0faf6e464ff240b5bb21bb1b446b329f.idx
00007f7429980000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-7e87b7b2f517ff2097026c55e98be6d1cec14587.idx
00007f7429981000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-bbef922979f633183f0bc207b5e52608b36d6434.idx
00007f7429982000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-5c633980ef7d381a4af2e7bf433c1c86154848c1.idx
00007f7429983000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-e84a2620505a10807e64727283f608b9661c39c0.idx
00007f7429984000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-eec773d189c951b5768eb85d33390d3d43db1d9f.idx
00007f7429985000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-c7d6e0b1e4da6a2aaa5eff3cad27f6fcbb38a5c9.idx
00007f7429986000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-25232ec34a92ab095486707f73fe5ad0a92bcb14.idx
00007f7429987000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-188241ed7cb01328a4c32b09b0aff82311e32a1c.idx
00007f7429988000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-be75b796793b738f7566a4be06491cfd3ad9c236.idx
00007f7429989000     24K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-d18bf8e1b6a421fac697607e204da80b6c4c0370.idx
00007f742998f000     20K rw---    [ anon ]
00007f7429998000      4K r----  /var/www/virtual/foo/bar/.git/objects/pack/pack-2ec52679fc048e2db73cedc5bab30680f9b14bc9.idx
00007f7429999000     28K r--s-  /usr/lib64/gconv/gconv-modules.cache
00007f74299a0000      4K rw---    [ anon ]
00007f74299a1000      4K r----  /lib64/ld-2.12.so
00007f74299a2000      4K rw---  /lib64/ld-2.12.so
00007f74299a3000      4K rw---    [ anon ]
00007ffe76939000     84K rw---    [ stack ]
00007ffe7698b000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]
 total          2809104K
sh-4.1$ error: pack-objects died of signal 15
error: failed to run repack

3 个答案:

答案 0 :(得分:3)

我有一些大的存储库,我必须压缩,我没有注意到任何与memrory相关的选项帮助。如果git-pack-objects必须重新包装某些对象,则必须映射大致相同的大内存。管理它的唯一方法是限制它必须重新包装的对象数量。

在你的情况下,对象已经压缩成小包文件(每次推送一个)。所以git gc必须将那些小包文件重新打包成大包。您可以尝试在上一次成功重新打包创建的大包文件旁边创建空pack-<HASH>.keep个文件。那么也许下一个git gc将不会再尝试对这些对象进行分类。

答案 1 :(得分:0)

可能是解决方案

我观察到,当我选择packSizeLimit非常小时,这是非常有效的 - 它增加了.git目录而不是减少它,这也可能导致达到桶位大小限制(未验证)

1)将repo复制到具有更多RAM的另一台计算机并正确打包

2)使用内存限制将其同步回服务器

3)增加到git gc

4)现在在git gc之后,回购并没有像疯了一样增长

5)提交数据库转储

6)repo将其大小翻倍

7)直接进行api

8)尺寸变好

9)现在推

让我们看看它在几天之后会如何发展......

答案 2 :(得分:0)

Git 2.18(2018年第二季度)应该有助于检测并减轻过多的内存消耗。

  

&#34; git gc&#34;在一个大型存储库中需要花费大量时间   默认情况下将所有对象重新打包到一个包中   该命令已被教导假装最大的现有packfile   标有&#34; .keep&#34;这样当物体进入时它就不会受到影响   重新包装其他包装和松散包装。

请参阅commit 5af0504commit 9806f5acommit 8fc6776commit 55dfe13commit ae4e89ecommit ed7e5fccommit e9e33ab(2018年4月15日) Nguyễn Thái Ngọc Duy (pclouds) (由Junio C Hamano -- gitster --合并于commit 30b015b,2018年5月23日)

  

gc --auto:如果没有足够的内存,则排除基本包&#34;重新包装-ad&#34;

     

pack-objects可能是一个大记忆力,特别是在大型回购,   大家都知道。
  为了避免这个问题,将.keep文件粘贴在巨型基础包上的建议也是众所周知的。

     

最近的补丁添加了一个选项来做到这一点,但它必须是   手动配置或激活。
  此补丁允许git gc --auto在认为repack -ad将使用大量内存并因交换或刷新操作系统缓存而开始影响系统时自动激活此模式。

     

gc --auto决定基于包对象的估计来做到这一点   内存使用情况,至少对于堆部分而言是非常准确的   是否适合系统内存的一半(这里的假设是为了   运行许多其他应用程序的桌面环境。

     

如果未配置gc.bigBasePackThreshold,则此机制仅启动。   如果是,则假定用户已经知道他们想要什么。

git gc man page现在声明:

  

如果包的数量超过gc.autoPackLimit的值,   然后是现有的包(标有.keep文件的包除外   或超过gc.bigPackThreshold限制)   使用-A选项合并为单个包   &#39; git repack&#39;。

     

如果估算的内存量不足git repack   运行顺利,gc.bigPackThreshold未设置,最大   包也将被排除(这相当于运行git gc   与--keep-base-pack)。
  将gc.autoPackLimit设置为0将禁用自动合并   包。