Plone挂在Python的waiter.acquire()上并使用100%的CPU

时间:2014-05-28 14:14:26

标签: python python-2.7 plone zope plone-4.x

当内容管理员执行诸如签出,编辑,工作流转换等操作时,我们在Plone实例中遇到了一些瓶颈。到目前为止,我们还无法隔离任何特定的东西,它会间歇性地发生。

对于我自己的测试,我坚持"退房"动作,我在我的故障排除中使用Products.LongRequestLogger。我看到记录结果中的操作需要8秒到78秒,以及用户体验。以下是LongRequestLogger中重复的回溯:

Traceback:
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/ZServer/PubCore/ZServerPublisher.py", line 31, in __init__
    response=b)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/ZPublisher/Publish.py", line 455, in publish_module
    environ, debug, request, response)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.LongRequestLogger-1.1.0-py2.7.egg/Products/LongRequestLogger/patch.py", line 16, in wrapper
    result = wrapper.original(*args, **kw)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/ZPublisher/Publish.py", line 249, in publish_module_standard
    response = publish(request, module_name, after_list, debug=debug)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/ZPublisher/Publish.py", line 138, in publish
    request, bind=1)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/ZPublisher/mapply.py", line 77, in mapply
    if debug is not None: return debug(object,args,context)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/ZPublisher/Publish.py", line 48, in call_object
    result=apply(object,args) # Type s<cr> to step into published object.
  File "/var/db/zope/plone43/zeocluster/src/plone.app.iterate/plone/app/iterate/browser/checkout.py", line 77, in __call__
    wc = policy.checkout(locator())
  File "/var/db/zope/plone43/zeocluster/src/plone.app.iterate/plone/app/iterate/policy.py", line 65, in checkout
    working_copy, relation = copier.copyTo( container )
  File "/var/db/zope/plone43/zeocluster/src/plone.app.iterate/plone/app/iterate/copier.py", line 54, in copyTo
    wc = self._copyBaseline( container )
  File "/var/db/zope/plone43/zeocluster/src/plone.app.iterate/plone/app/iterate/copier.py", line 211, in _copyBaseline
    result = container.manage_pasteObjects( clipboard )
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/OFS/CopySupport.py", line 235, in manage_pasteObjects
    self._setObject(id, ob)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.BTreeFolder2-2.13.3-py2.7.egg/Products/BTreeFolder2/BTreeFolder2.py", line 455, in _setObject
    notify(ObjectAddedEvent(ob, self, id))
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.event-3.5.2-py2.7.egg/zope/event/__init__.py", line 31, in notify
    subscriber(event)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/event.py", line 24, in dispatch
    zope.component.subscribers(event, None)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/_api.py", line 136, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/registry.py", line 321, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.interface-4.1.0-py2.7-linux-x86_64.egg/zope/interface/adapter.py", line 601, in subscribers
    subscription(*objects)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/event.py", line 32, in objectEventNotify
    zope.component.subscribers((event.object, event), None)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/_api.py", line 136, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/registry.py", line 321, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.interface-4.1.0-py2.7-linux-x86_64.egg/zope/interface/adapter.py", line 601, in subscribers
    subscription(*objects)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/OFS/subscribers.py", line 110, in dispatchObjectMovedEvent
    callManageAfterAdd(ob, event.object, event.newParent)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/OFS/subscribers.py", line 143, in callManageAfterAdd
    ob.manage_afterAdd(item, container)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.ATContentTypes-2.1.13-py2.7.egg/Products/ATContentTypes/content/document.py", line 196, in manage_afterAdd
    ATCTContent.manage_afterAdd(self, item, container)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.Archetypes-1.9.4-py2.7.egg/Products/Archetypes/BaseContent.py", line 43, in manage_afterAdd
    BaseObject.manage_afterAdd(self, item, container)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.Archetypes-1.9.4-py2.7.egg/Products/Archetypes/BaseObject.py", line 158, in manage_afterAdd
    Referenceable.manage_afterAdd(self, item, container)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.Archetypes-1.9.4-py2.7.egg/Products/Archetypes/Referenceable.py", line 245, in manage_afterAdd
    r.set(self, rrefs[r.getName()])
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.Archetypes-1.9.4-py2.7.egg/Products/Archetypes/Field.py", line 1987, in set
    tool.addReference(instance, uid, self.relationship, **addRef_kw)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.Archetypes-1.9.4-py2.7.egg/Products/Archetypes/ReferenceEngine.py", line 340, in addReference
    annotation._setObject(rID, referenceObject)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/OFS/ObjectManager.py", line 359, in _setObject
    notify(ObjectAddedEvent(ob, self, id))
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.event-3.5.2-py2.7.egg/zope/event/__init__.py", line 31, in notify
    subscriber(event)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/event.py", line 24, in dispatch
    zope.component.subscribers(event, None)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/_api.py", line 136, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/registry.py", line 321, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.interface-4.1.0-py2.7-linux-x86_64.egg/zope/interface/adapter.py", line 601, in subscribers
    subscription(*objects)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/event.py", line 32, in objectEventNotify
    zope.component.subscribers((event.object, event), None)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/_api.py", line 136, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.component-3.9.5-py2.7.egg/zope/component/registry.py", line 321, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/var/db/zope/plone43/buildout-cache/eggs/zope.interface-4.1.0-py2.7-linux-x86_64.egg/zope/interface/adapter.py", line 601, in subscribers
    subscription(*objects)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/OFS/subscribers.py", line 110, in dispatchObjectMovedEvent
    callManageAfterAdd(ob, event.object, event.newParent)
  File "/var/db/zope/plone43/buildout-cache/eggs/Zope2-2.13.21-py2.7.egg/OFS/subscribers.py", line 143, in callManageAfterAdd
    ob.manage_afterAdd(item, container)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.Archetypes-1.9.4-py2.7.egg/Products/Archetypes/ReferenceEngine.py", line 145, in manage_afterAdd
    rc.catalog_object(self, url)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.Archetypes-1.9.4-py2.7.egg/Products/Archetypes/UIDCatalog.py", line 189, in catalog_object
    ZCatalog.catalog_object(self, obj, uid, **kwargs)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.ZCatalog-2.13.23-py2.7.egg/Products/ZCatalog/ZCatalog.py", line 476, in catalog_object
    update_metadata=update_metadata)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.ZCatalog-2.13.23-py2.7.egg/Products/ZCatalog/Catalog.py", line 320, in catalogObject
    index = self.updateMetadata(object, uid, None)
  File "/var/db/zope/plone43/buildout-cache/eggs/Products.ZCatalog-2.13.23-py2.7.egg/Products/ZCatalog/Catalog.py", line 276, in updateMetadata
    while not data.insert(index, newDataRecord):
  File "/var/db/zope/plone43/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 860, in setstate
    self._setstate(obj)
  File "/var/db/zope/plone43/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 901, in _setstate
    p, serial = self._storage.load(obj._p_oid, '')
  File "/var/db/zope/plone43/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 1270, in load
    return self._storage.load(oid, '')
  File "/var/db/zope/plone43/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/ClientStorage.py", line 833, in load
    data, tid = self._server.loadEx(oid)
  File "/var/db/zope/plone43/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/ServerStub.py", line 176, in loadEx
    return self.rpc.call("loadEx", oid)
  File "/var/db/zope/plone43/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/zrpc/connection.py", line 763, in call
    r_args = self.wait(msgid)
  File "/var/db/zope/plone43/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/zrpc/connection.py", line 791, in wait
    self.replies_cond.wait()
  File "/var/db/zope/plone43/Python-2.7/lib/python2.7/threading.py", line 339, in wait
    waiter.acquire()

在此期间,我注意到它正在使用的客户端在CPU使用率为100%时出现最大值,如果不是完全停止,这会导致站点进入爬行减速状态。我在我们的生产环境中观看了这个,在任何给定时间,它同时拥有5到40个用户。所以我在我们网站的本地安装中对此进行了测试,即使只有一个用户,我也看到了相同的结果。

我们在RedHat 5.10(Tikanga)64位盒子上运行Plone 4.3.2。我的本地虚拟机使用相同的Plone版本运行CentOS 5.10。对于我们的生产环境,我们运行两个服务器,一个专用于运行Zeoserver,第二个运行4个客户端。

我无法弄清楚为什么它悬在这里并占用了这么多资源。任何建议或解决方案将不胜感激!

更新 我们最近从Plone 4.2升级到了Plone 4.3,看起来这个问题要么在那时开始,要么至少显着增加。我们使用自己的自定义主题和内容类型,但它们基于ATContentTypes,我们为他们使用Plone 4.3进行了更新所需的更新

修改 我将ZEO服务器日志记录级别更改为DEBUG,这就是我得到的:

2014-05-28T14:38:35 (unconnected) disconnected
2014-05-28T14:38:44 new transaction
2014-05-28T14:38:44 (192.168.56.102:54486) ('1') lock: transactions waiting: 0
2014-05-28T14:38:44 (192.168.56.102:54486) Preparing to commit transaction: 2 objects, 763 bytes
2014-05-28T14:38:44 new transaction
2014-05-28T14:38:44 (192.168.56.102:54486) ('1') lock: transactions waiting: 0
2014-05-28T14:38:44 (192.168.56.102:54486) Preparing to commit transaction: 1 objects, 123 bytes
2014-05-28T14:39:22 new transaction
2014-05-28T14:39:22 (192.168.56.102:54474) ('1') lock: transactions waiting: 0
2014-05-28T14:39:22 (192.168.56.102:54474) Preparing to commit transaction: 885 objects, 458894 bytes
2014-05-28T14:40:44 new transaction
2014-05-28T14:40:44 (192.168.56.102:54486) ('1') lock: transactions waiting: 0
2014-05-28T14:40:44 (192.168.56.102:54486) Preparing to commit transaction: 1 objects, 123 bytes

根据Client1 LongRequest.log,操作(Check out)在14:39:07开始,在14:39:28结束(这是响应时间更快的一个)。

0 个答案:

没有答案