我不确切地知道问题出在哪里,所以我写这里是为了获得一些提示或线索。我想知道是否有人对此有任何意见或想法。 使用ckan进行收获似乎有效(我能够在开放数据网站上获取数据)但最终其状态保持“正在进行”#34;因此它仍然受阻并且我无法再收获其他数据。谢谢
以下是fetch_consumer.log
中的日志2016-09-02 13:05:17,824 INFO [ckanext.harvest.queue] Received harvest object id: bd9dbc78-0b81-4566-87b4-61cdaf783b46
2016-09-02 13:05:17,840 DEBUG [ckanext.mydata.harvester] In Mydata Harvest fetch_stage
2016-09-02 13:05:17,845 DEBUG [ckanext.harvest.harvesters.ckanharvester] Using config: {u'force_all': False, u'user': u'admin'}
2016-09-02 13:05:17,956 DEBUG [ckanext.mydata.harvester] In Mydata Harvest import_stage
2016-09-02 13:05:17,957 DEBUG [ckanext.harvest.harvesters.ckanharvester] Using config: {u'force_all': False, u'user': u'admin'}
2016-09-02 13:05:17,990 DEBUG [ckanext.mydata.harvester] Package id d64484b6-1447-4c9e-a8dd-fde132842de0 doesn't exist, moving extras to base metadata
2016-09-02 13:05:17,995 INFO [ckanext.mydata.harvester] Package with GUID d64484b6-1447-4c9e-a8dd-fde132842de0 does not exist, let's create it
2016-09-02 13:05:18,292 INFO [ckanext.mydata.harvester] Attempting to upload resource 29b781c7-06d6-4a7d-bee4-09d12f2e29cf
2016-09-02 13:05:22,097 INFO [ckanext.mydata.harvester] Resource 29b781c7-06d6-4a7d-bee4-09d12f2e29cf has to be created
2016-09-02 13:05:26,046 INFO [ckanext.mydata.harvester] Resource 29b781c7-06d6-4a7d-bee4-09d12f2e29cf has been created
来自gather_consumer.log的日志:
2016-09-02 13:04:43,226 DEBUG [ckanext.mydata.harvester] Delta packages to remove locally: [u'9f058088-056f-433a-b9f4-c61b2fb07230']
2016-09-02 13:04:43,226 DEBUG [ckanext.mydata.harvester] Candidate package_id for removal: u'9f058088-056f-433a-b9f4-c61b2fb07230'
/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:80: SAWarning: Usage of the 'related attribute set' operation is not currently supported within the execution stage of the flush process. Results may not be consistent. Consider using alternative event listeners or connection-level operations instead.
sess._flush_warning("related attribute set")
/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:38: SAWarning: Usage of the 'collection append' operation is not currently supported within the execution stage of the flush process. Results may not be consistent. Consider using alternative event listeners or connection-level operations instead.
sess._flush_warning("collection append")
/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py:1919: SAWarning: Attribute history events accumulated on 1 previously clean instances within inner-flush event handlers have been reset, and will not result in database updates. Consider using set_committed_value() within inner-flush event handlers to avoid this warning.
self._flush(objects)
2016-09-02 13:04:43,905 DEBUG [ckanext.harvest.queue] Received from plugin gather_stage: 6 objects (first: [u'b5ca80a0-6301-47c0-a592-338f6adfddd2'] last: [u'bd9dbc78-0b81-4566-87b4-61cdaf783b46'])
2016-09-02 13:04:43,908 DEBUG [ckanext.harvest.queue] Sent 6 objects to the fetch queue