我正在尝试使用python字典中的cStringIO读取一个非常大的字符串流:
def stream_read(self, path):
try:
# create a string stream from the contents at 'path'
# note: the string at self._my_dict[path] is 7MB in size
stream = StringIO.StringIO(self._my_dict[path])
while True:
# buffer size is 128kB, or 128 * 1024
buf = stream.read(self.buffer_size)
if buf != '':
yield buf
else:
raise StopIteration
except KeyError:
raise IOError("Could not get content")
在我的测试套件中,我通过首先测试stream_write,断言数据存在于该路径,然后调用stream_read来测试此函数:
def test_stream(self):
filename = self.gen_random_string()
# test 7MB
content = self.gen_random_string(7 * 1024 * 1024)
# test stream write
io = StringIO.StringIO(content)
self._storage.stream_write(filename, io)
io.close()
self.assertTrue(self._storage.exists(filename))
# test read / write
data = ''
for buf in self._storage.stream_read(filename):
data += buf
self.assertEqual(content, data)
然而在我的测试套件中,我正在捕获AssertionError:
======================================================================
FAIL: test_stream (test_swift_storage.TestSwiftStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/bacongobbler/.../test/test_local_storage.py", line 44, in test_stream
self.assertEqual(content, data)
AssertionError: '[squelched]' != '<cStringIO.StringI object at 0x3148e70>'
----------------------------------------------------------------------
Ran 28 tests in 20.495s
FAILED (failures=1)
它看起来与an issue I posted last week有关,但我仍然不太清楚为什么stream
在这种情况下将{{1}}设置为生成器作为字符串。
如果有人想仔细查看源代码,那么全部都在https://github.com/bacongobbler/docker-registry/blob/106-swift-storage/test/utils/mock_swift_storage.py
答案 0 :(得分:2)
在调用StringIO
时,您只需 self._storage.stream_write(filename, io)
对象:
def put_content(self, path, content, chunk=None):
path = self._init_path(path)
try:
self._swift_container[path] = content
except Exception:
raise IOError("Could not put content")
其中content
是您传入的io
对象。
稍后,您再次将该文件对象传递给StringIO
:
stream = StringIO.StringIO(self.get_content(path))
这会在str()
上调用self.get_content(path)
,存储cStringIO.StringI()
实例的字符串表示形式:
>>> from cStringIO import StringIO
>>> str(StringIO('test data'))
'<cStringIO.StringI object at 0x1074ea470>'
您的阅读代码运行正常,您的编写模拟需要实际获取StringIO
对象的 out 数据。
.read()
来电将在此处进行:
def put_content(self, path, content, chunk=None):
path = self._init_path(path)
try:
self._swift_container[path] = content.read()
except Exception:
raise IOError("Could not put content")