在GAE中使用gcloud-python

时间:2014-11-21 09:22:39

标签: google-app-engine google-cloud-datastore google-cloud-python gcloud-python

我有一堆小的Raspberry Pis运行一些python代码,使用gcloud-python数据存储包直接保存到Datastore(跳过GAE)。这非常有效。我现在想要使用Google App Engine通过网络和移动客户端呈现数据。在我的MacBook上,我使用安装程序安装了GAE,通过pip安装了gcloud。我可以编写一个简单的python脚本并直接从终端执行它,它可以通过gcloud从数据存储区写入和读取 - 这也可以正常工作。

然而,当我尝试将相同的代码合并到GAE中时,它失败了。根据我的研究,我预计这是一个PATH问题,但经过几个小时的各种尝试后,我无法解决这个问题。建议将不胜感激。

我认为这篇文章与我的问题类似:Google App Engine, Change which python version

以下是一些可能相关的信息:

Python版

$ which python
/usr/bin/python

根据引用的Stack Overflow问题,我将GAE的首选项设置为Python路径为/ usr / bin / python。我试过了

$ PATH

$ echo $PATH
/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/bin:/Users/sheridangray/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin
来自Python解释器的

os.path

$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path
<module 'posixpath' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>

gcloud安装

$ pip install --upgrade gcloud
Requirement already up-to-date: gcloud in /Library/Python/2.7/site-packages
...

Python 2.7引用

$ sudo find / -name python2.7
Password:
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Dropbox.app/Contents/Resources/include/python2.7
/Applications/Dropbox.app/Contents/Resources/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib/python2.7
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/usr/bin/python2.7
/usr/lib/python2.7

相关代码

from gcloud import datastore

class MainHandler(webapp.RequestHandler):
  def get(self):
     dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)

GAE日志文件

*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8080 --admin_port=8000
Python command: /usr/bin/python
INFO     2014-11-21 09:02:03,276 devappserver2.py:745] Skipping SDK update check.
INFO     2014-11-21 09:02:03,288 api_server.py:172] Starting API server at: http://localhost:49183
INFO     2014-11-21 09:02:03,292 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO     2014-11-21 09:02:03,294 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO     2014-11-21 09:02:34,319 module.py:709] default: "GET / HTTP/1.1" 200 2
INFO     2014-11-21 09:02:34,455 module.py:709] default: "GET /favicon.ico HTTP/1.1" 200 8348
INFO     2014-11-21 09:02:44,359 module.py:387] Detected file changes:
  /Users/sheridangray/Projects/city-pulse-web/main.py
ERROR    2014-11-21 09:02:46,443 webapp2.py:1552] gcloud
Traceback (most recent call last):
      File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/sheridangray/Projects/city-pulse-web/main.py", line 63, in get
dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 103, in get_dataset
connection = get_connection(client_email, private_key_path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 65, in get_connection
    from gcloud.datastore.connection import Connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/connection.py", line 3, in <module>
    from gcloud import connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 8, in <module>
class Connection(object):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 24, in Connection
USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 311, in get_distribution
if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 197, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
DistributionNotFound: gcloud
INFO     2014-11-21 09:02:46,456 module.py:709] default: "GET / HTTP/1.1" 500 5010
INFO     2014-11-21 09:02:46,514 module.py:709] default: "GET /favicon.ico HTTP/1.1" 304 -

2 个答案:

答案 0 :(得分:4)

您可以在App Engine上运行gcloud-python,但它需要一个 额外的工作。看看我写的骨架project 这使得这个工作。

要涵盖的主要基础是:

获取依赖项

install_gcloud.sh中,pip用于安装gcloud及其pip install --target="application/vendor/" gcloud 使用

在应用程序内部的依赖项
pip

(如other answer中所述,本地安装不会上传 在部署时使用App Engine。)

修改依赖项

--target - 安装pkg_resources.get_distribution集,pytz 将按预期工作(它在堆栈跟踪中失败)。

此外,默认情况下pytz有太多的读取才能正常工作 App Engine,因此使用gae-pytz。结果,一些oauth2client.client 需要修改进口。

此外,为了减少计算引擎检查开销(网络开销), vendor/模块可以修改。

这两项调整都可以在一个commit中找到。

使进口工作

上面的脚本将所有依赖项放在名为import darth darth.vendor('vendor') 和的目录中 appengine_config.py将此添加到导入路径中 Darth Vendor

protobuf

此外,由于google依赖也是其中的一部分 google.appengine.ext.ndb包(就像所有App Engine导入一样,例如 __path__)您需要修改import os import google curr_dir = os.path.abspath(os.path.dirname(__file__)) vendor_dir = os.path.join(curr_dir, 'vendor') google.__path__.append(os.path.join(vendor_dir, 'google')) 与该包相关联:

gcloud-python

CAVEAT(截至2015年1月22日)

请注意在App Engine中使用db 比使用本机App Engine库ndbgcloud-python慢3-5倍。 这是因为那些使用直接RPC到App Engine运行时 {{1}}将使用App Engine外部的HTTP与云进行通信 数据存储API。


注意:我在最初发布后更新了这个,引用了 历史上的前一个point

答案 1 :(得分:1)

Mac上的GAE无法访问Mac上默认位置安装的Python软件包。你需要这样做:

ln -s /Library/Python/2.7/site-packages/.../gcloud /Users/sheridangray/Projects/city-pulse-web/gcloud

(需要用相关路径信息替换...)