Question

我们如何在jupyter服务和其他服务（Google云端存储或git存储库）之间同步笔记本？

关于这个问题的一些背景知识：

目前我正在从Google的Datalab迁移到我自己的容器。我的动机是更多地控制数据区域（仅在美国提供的Datalab Beta）和软件包，因为我想使用当前的Tensorflow版本。

根据Google的想法（see github），我构建了自己的Docker镜像，并在Google容器引擎中的Kubernetes群集上运行它。 GCP包可以安装为I have previously explained。 Google使用 node.js服务器将GIT 与datalab实例同步 - 但是我无法在欧盟中使用自我部署的容器运行。

第二次尝试是 GCSFuse驱动程序。从Kubernetes v1.0和Google Container Engine开始，这个不适用于非特权容器。如此完整。

我的Docker文件（基于Google的GCE Datalab图片）：

FROM debian:jessie

# Setup OS and core packages
RUN apt-get clean
RUN echo "deb-src http://ftp.be.debian.org/debian testing main" >> /etc/apt/sources.list && \
apt-get update -y && \
apt-get install --no-install-recommends -y -q \
    curl wget unzip git vim build-essential ca-certificates pkg-config \
    libatlas-base-dev liblapack-dev gfortran \
    libpng-dev libfreetype6-dev libxft-dev \
    libxml2-dev \
    python2.7 python-dev python-pip python-setuptools python-zmq && \
mkdir -p /tools && \
mkdir -p /srcs && \
cd /srcs && apt-get source -d python-zmq && cd

WORKDIR /datalab


# Setup Google Cloud SDK
RUN apt-get install --no-install-recommends -y -q wget unzip git -y
RUN wget -nv https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.zip && \
unzip -qq google-cloud-sdk.zip -d tools && \
rm google-cloud-sdk.zip && \
tools/google-cloud-sdk/install.sh --usage-reporting=false \
    --path-update=false --bash-completion=false \
    --disable-installation-options && \
tools/google-cloud-sdk/bin/gcloud config set --scope=installation \
    component_manager/fixed_sdk_version 0.9.57 && \
tools/google-cloud-sdk/bin/gcloud -q components update \
    gcloud core bq gsutil compute preview alpha beta && \
rm -rf /root/.config/gcloud

# Install FUSE driver for GCE
RUN apt-get install -y lsb-release
RUN echo "deb http://packages.cloud.google.com/apt gcsfuse-jessie main" >     /etc/apt/sources.list.d/gcsfuse.list
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
RUN apt-get update && apt-get install -y gcsfuse
RUN mkdir /datalab/mount

# Setup Python packages
RUN pip install -U \
    tornado==4.2.1 pyzmq==14.4.0 jinja2==2.7.3 \
    jsonschema==2.5.1 py-dateutil==2.2 pytz==2015.4 pandocfilters==1.2.4 pygments==2.0.2 \
    argparse==1.2.1 mock==1.2.0 requests==2.4.3 oauth2client==1.4.12 httplib2==0.9.2 \
    futures==3.0.3 && \
    pip install -U numpy==1.9.2 && \
    pip install -U pandas==0.16.2 && \
    pip install -U scikit-learn==0.16.1 && \
    pip install -U scipy==0.15.1 && \
    pip install -U sympy==0.7.6 && \
    pip install -U statsmodels==0.6.1 && \
    pip install -U matplotlib==1.4.3 && \
    pip install -U ggplot==0.6.5 && \
    pip install -U seaborn==0.6.0 && \
    pip install -U notebook==4.0.2 && \
    pip install -U PyYAML==3.11 && \
    easy_install pip && \
    find /usr/local/lib/python2.7 -type d -name tests | xargs rm -rf

# Path configuration
ENV PATH $PATH:/datalab/tools/google-cloud-sdk/bin
ENV PYTHONPATH /env/python

# IPython configuration
WORKDIR /datalab
RUN ipython profile create default
RUN jupyter notebook --generate-config
ADD ipython.py /root/.ipython/profile_default/ipython_config.py

# Install TensorFlow.
RUN wget -nv https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.7.1-cp27-none-linux_x86_64.whl && \
  pip install --upgrade tensorflow-0.7.1-cp27-none-linux_x86_64.whl && rm tensorflow-0.7.1-cp27-none-linux_x86_64.whl


# Add build artifacts
ADD build/lib/GCPData-0.1.0.tar.gz /datalab/lib/
ADD build/lib/GCPDataLab-0.1.0.tar.gz /datalab/lib/
ADD setup-repo.sh /datalab
ADD setup-env.sh /datalab
ADD run.sh /datalab
RUN chmod 755 /datalab/*

# Install build artifacts
RUN cd /datalab/lib/GCPData-0.1.0 && python setup.py install
RUN cd /datalab/lib/GCPDataLab-0.1.0 && python setup.py install

RUN mkdir /datalab/content
WORKDIR /datalab/content
EXPOSE 6006
EXPOSE 8123
# see https://github.com/ipython/ipython/issues/7062
CMD ["/datalab/run.sh"]

Answer 1

好的，我解决了这个问题：

将保存后挂钩用作explained in a previous post
在钩子中使用几个git命令explained by this blog

以下是（2.）中用于存档的代码。它进入了ipython.py：

import os
from subprocess import check_call
from shlex import split

...

def post_save(model, os_path, contents_manager):
    """post-save hook for doing a git commit / push"""
    if model['type'] != 'notebook':
        return  # only do this for notebooks
    workdir, filename = os.path.split(os_path)
    if filename.startswith('Scratch') or filename.startswith('Untitled'):
        return  # skip scratch and untitled notebooks
    # now do git add / git commit / git push
    check_call(split('git add {}'.format(filename)), cwd=workdir)
    check_call(split('git commit -m "notebook save" {}'.format(filename)), cwd=workdir)
    check_call(split('git push'), cwd=workdir)

c.FileContentsManager.post_save_hook = post_save

我的run.sh使用来自Google Datalab的setup-env.sh和setup-repo.sh，因此依赖于gcloud命令和Kubernetes部署凭据。否则请确保使用凭据扩展您的Dockerfile。

cd /datalab/content
. /datalab/setup-env.sh
. /datalab/setup-repo.sh
if [ $? != "0" ]; then
    exit 1
fi
cd /datalab/content/master_branch  # multiple branches not planed here!
/usr/local/bin/jupyter notebook --ip=* --no-browser --port=8123

将Jupyter与文件共享或已装入的文件夹

1 个答案: