I have a program that returns rows of allowed arrays (as follows) which are stored in a file (say output.txt), using python program.py>output.txt on the windows 7 command line:
0 , 0 , 1 , 1 , 6
0 , 0 , 1 , 2 , 1
0 , 0 , 1 , 4 , 1
0 , 0 , 1 , 6 , 1
0 , 0 , 2 , 0 , 0
I later use output.txt as input for another program, which reads it row by row into an array. The output.txt file is about 1.5 GB. I can use 7-zip to reduce it to 33 MB.
I risk running out of memory for the larger output data sizes. I'm aware that we can read zipped files in Python so is it possible to get zipped outputs as well?
My program has multiple nested loops, so I can make a small zip file after a cycle in the innermost loop and append it to another which gradually becomes larger as the loop traverses ahead.