有没有办法只将已更改的文件作为新图层添加到docker镜像中 - 无需借助docker commit?

时间:2016-04-11 15:52:37

标签: docker

TL; DR

在图像上运行COPY . /app但源代码稍微过时会创建一个与整个源代码一样大的新图层,即使只有少量字节的更改也是如此。 有没有办法只将已更改的文件作为新图层添加到此docker镜像中 - 而无需借助docker commit?

长版:

将我们的应用程序部署到生产环境时,我们需要将源代码添加到映像中。一个非常简单的Dockerfile用于此:

FROM neam/dna-project-base-debian-php:0.6.0
COPY . /app

由于源代码很大(1.2 GB),这对每次部署都有很大的推动力:

$ docker build -f .stack.php.Dockerfile -t project/project-web-src-php:git-commit-17c279b .
Sending build context to Docker daemon 1.254 GB
Step 0 : FROM neam/dna-project-base-debian-php:0.6.0
 ---> 299c10c416fc
Step 1 : COPY . /app
 ---> 78a30802804a
Removing intermediate container 13b49c323bb6
Successfully built 78a30802804a

$ docker tag -f project/project-web-src-php:git-commit-17c279b tutum.co/project/project-web-src-php:git-commit-17c279b
$ docker login --email=tutum-project@project.com --username=project --password=******** https://tutum.co/v1
WARNING: login credentials saved in /home/dokku/.docker/config.json
Login Succeeded
$ docker push tutum.co/project/project-web-src-php:git-commit-17c279b
The push refers to a repository [tutum.co/project/project-web-src-php] (len: 1)
Sending image list
Pushing repository tutum.co/project/project-web-src-php (1 tags)
Image a604b236bcde already pushed, skipping
Image 1565e86129b8 already pushed, skipping
...
Image 71156b357f2f already pushed, skipping
Image 299c10c416fc already pushed, skipping
78a30802804a: Pushing [=========>                     ] 234.2 MB/1.254 GB

在下一次部署时,我们只想将更改后的文件添加到图像中,但在先前添加的图像上运行COPY . /app时观察和观察实际上需要我们推送价值1.2 GB的源代码AGAIN ,即使我们只更改了几个字节的源代码:

新Dockerfile(.stack.php.git-commit-17c279b.Dockerfile):

FROM project/project-web-src-php:git-commit-17c279b
COPY . /app

更改几个文件后,添加一些文本和代码,然后构建并推送:

$ docker build -f .stack.php.git-commit-17c279b.Dockerfile -t project/project-web-src-php:git-commit-17c279b-with-a-few-changes .
Sending build context to Docker daemon 1.225 GB
Step 0 : FROM project/project-web-src-php:git-commit-17c279b
 ---> 4dc643a45de3
Step 1 : COPY . /app
 ---> ecc7adc194c4
Removing intermediate container cb3e87c6cb7a
Successfully built ecc7adc194c4
$ docker tag -f project/project-web-src-php:git-commit-17c279b-with-a-few-changes tutum.co/project/project-web-src-php:git-commit-17c279b-with-a-few-changes
$ docker push tutum.co/project/project-web-src-php:git-commit-17c279b-with-a-few-changes
The push refers to a repository [tutum.co/project/project-web-src-php] (len: 1)
Sending image list
Pushing repository tutum.co/project/project-web-src-php (1 tags)
Image 1565e86129b8 already pushed, skipping
Image a604b236bcde already pushed, skipping
...
Image fe64bff23cf8 already pushed, skipping
Image 71156b357f2f already pushed, skipping
ecc7adc194c4: Pushing [==>                           ] 68.21 MB/1.225 GB

有一种解决方法可以实现Updating docker images with small changes using commits中描述的小层,包括在映像中启动rsync进程,然后使用docker commit将新内容保存为新层,但是(如该线程中所述) )这是非正统的,因为图像不是从Dockerfile构建的,我们更喜欢不依赖于docker commit的正统解决方案。

有没有办法只将已更改的文件作为新图层添加到此docker镜像中 - 而无需借助docker commit?

Docker版本1.8.3

3 个答案:

答案 0 :(得分:7)

Actually, the solution IS to use COPY . /app as the OP is doing, there is however an open bug causing this not to work as expected on most systems

The only currently feasible workaround to this issue seems to be to use rsync to analyze the differences between the old and new images prior to pushing the new one, then use the changelog output to generate a tar-file containing the relevant changes which is subsequently COPY:ed to a new image layer.

This way, the layer sizes becomes a few bytes or kilobytes for smaller changes instead of 1.2 GB every time.

I put together documentation and scripts to help out with this over at https://github.com/neam/docker-diff-based-layers.

The end results are shown below:

Verify that basing the project images on the revision 1 image tag contents does not lead to desired outcome

Verify that subsequent COPY . /app commands re-adds all files in every layer instead of only the files that have changed:

docker history sample-project:revision-2

Output:

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
4a3115eaf267        3 seconds ago       /bin/sh -c #(nop) COPY dir:61d102421e6692b677   16.78 MB
d4b30af167f4        25 seconds ago      /bin/sh -c #(nop) COPY dir:68b8f374d8731b8ad8   16.78 MB
c898fe1daa44        2 minutes ago       /bin/sh -c apt-get update && apt-get install    10.77 MB
39a8a358844a        4 months ago        /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
b1dacad9c5c9        4 months ago        /bin/sh -c #(nop) ADD file:5afd8eec1dc1e7666d   125.1 MB

Even though we added/changed only a few bytes, all files are re-added and 16.78 MB is added to the total image size.

Also, the file(s) that we removed did not get removed.

Create an image with an optimized layer

export RESTRICT_DIFF_TO_PATH=/app
export OLD_IMAGE=sample-project:revision-1
export NEW_IMAGE=sample-project:revision-2
docker-compose -f rsync-image-diff.docker-compose.yml up
docker-compose -f shell.docker-compose.yml -f process-image-diff.docker-compose.yml up
cd output; docker build -t sample-project:revision-2-processed .; cd ..

Verify that the processed new image has smaller sized layers with the changes:

docker history sample-project:revision-2-processed

Output:

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
1920e750d362        24 seconds ago      /bin/sh -c if [ -s /.files-to-remove.list ];    0 B
1267bf926729        2 minutes ago       /bin/sh -c #(nop) ADD file:5021c627243e841a45   19 B
d04a2181b62a        2 minutes ago       /bin/sh -c #(nop) ADD file:14780990c926e673f2   264 B
d4b30af167f4        7 minutes ago       /bin/sh -c #(nop) COPY dir:68b8f374d8731b8ad8   16.78 MB
c898fe1daa44        9 minutes ago       /bin/sh -c apt-get update && apt-get install    10.77 MB
39a8a358844a        4 months ago        /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
b1dacad9c5c9        4 months ago        /bin/sh -c #(nop) ADD file:5afd8eec1dc1e7666d   125.1 MB

Verify that the processed new image contains the same contents as the original:

export RESTRICT_DIFF_TO_PATH=/app
export OLD_IMAGE=sample-project:revision-2
export NEW_IMAGE=sample-project:revision-2-processed
docker-compose -f rsync-image-diff.docker-compose.yml up

The output should indicate that there are no differences between the images/tags. Thus, the sample-project:revision-2-processed tag can now be pushed and deployed, leading to the same end result but without having to push an unnecessary 16.78M over the wire, leading to faster deploy cycles.

答案 1 :(得分:2)

Docker文件中每层/指令的Docker缓存工作。在这种情况下,该层中使用的文件(构建上下文中的所有内容(.))都会被修改,因此需要重建该层。

如果代码的某些特定部分经常不发生变化,您可以考虑在单独的图层中添加这些部分,或者甚至将这些部分移到基本图像中,然后将其移动到基础图像中。 / p>

FROM mybaseimage
COPY ./directories-that-dont-change-often /somewhere
COPY ./directories-that-change-often /somewhere

根据您的项目,可能需要进行一些规划或重组才能实现,但可能值得这样做。

答案 2 :(得分:0)

我的解决方案:(来自https://github.com/neam/docker-diff-based-layers的想法!)

docker rm -f uniquename 2> /dev/null
docker run --name uniquename -v ~/repo/mycode:/src ${REPO}/${IMAGE}:${BASE} rsync -ar --exclude-from '/src/.dockerignore' --delete /src/ /app/
docker commit uniquename ${REPO}/${IMAGE}:${NEW_TAG}