Question

我想在将数据写入CSV文件时最大程度地减少内存使用。

对于更大的表，即使它是临时的，它也会使用更多的内存。

有人可以建议如何减少内存使用吗？

也许我可以将操作分离为更大的表，写入更多文件，然后合并它们，但是我还没有尝试过，也许我缺少明显的东西。

以下是当前使用的代码：

 @autoreleasepool {
    NSOutputStream *csvStream = [[NSOutputStream alloc] initToMemory];
    [csvStream open];

    CHCSVWriter *writer = [[CHCSVWriter alloc] initWithOutputStream:csvStream encoding:NSUTF8StringEncoding delimiter:';'];
    NSArray *keySortDescriptors = @[[NSSortDescriptor sortDescriptorWithKey:@"self" ascending:YES]];
    if (writeHeader==YES) {
        //> write header
        NSMutableDictionary *firstRow = [[self sharedUploadManager].modifiedRows firstObject];
        if (firstRow==nil) {
            result = NO;
            return result;
        }

        NSArray *orderedKeys = [[firstRow allKeys] sortedArrayUsingDescriptors:keySortDescriptors];
        for (NSString *columnName in  orderedKeys) {
            [writer writeField:columnName];
        }
    }
    [writer finishLine];

    @autoreleasepool {
        //> write the rows
        for (NSMutableDictionary *row in [self sharedUploadManager].modifiedRows) {

            NSArray *orderedKeys = [[row allKeys] sortedArrayUsingDescriptors:keySortDescriptors];

            for (NSString *key in orderedKeys ) {

                NSString *field = [row objectForKey:key];
                if ([field isKindOfClass:[NSNull class]]) {
                    [writer writeField:nil];
                } else {
                    [writer writeField:field];
                }
            }

            //> finish the line
            [writer finishLine];
        }
    }

    [writer closeStream];

    NSData *buffer = [csvStream propertyForKey:NSStreamDataWrittenToMemoryStreamKey];
    NSString *output = [[NSString alloc] initWithData:buffer encoding:NSUTF8StringEncoding];

    if (![[NSFileManager defaultManager] fileExistsAtPath:csvPath]) {
        [[NSFileManager defaultManager] createFileAtPath:csvPath contents:nil attributes:nil];
    }

    BOOL res = [[output dataUsingEncoding:NSUTF8StringEncoding] writeToFile:csvPath atomically:NO];

    if (!res) {
        NSLog(@"Error Creating CSV File path = %@", csvPath);
    } else{
        NSLog(@"Data saved! File path = %@", csvPath);

    }
}

我之前也尝试过这种逻辑-稍微干净一点，但结果相同：

NSOutputStream *csvStream = [[NSOutputStream alloc] initToFileAtPath:csvPath append:YES];
[csvStream open];

CHCSVWriter *writer = [[CHCSVWriter alloc] initWithOutputStream:csvStream encoding:NSUTF8StringEncoding delimiter:';'];

if (writeHeader==YES) {
    //> write header
    NSMutableDictionary *firstRow = [rows firstObject];
    if (firstRow==nil) {
        result = NO;
        return result;
    }

    NSArray *orderedKeys = [[firstRow allKeys] sortedArrayUsingDescriptors:@[[NSSortDescriptor sortDescriptorWithKey:@"self" ascending:YES]]];

    for (NSString *columnName in  orderedKeys) {
        [writer writeField:columnName];
    }
    [writer finishLine];
}


//> write the rows
for (NSMutableDictionary *row in rows) {

    NSArray *orderedKeys = [[row allKeys] sortedArrayUsingDescriptors:@[[NSSortDescriptor sortDescriptorWithKey:@"self" ascending:YES]]];

    for (NSString *key in orderedKeys ) {

        NSString *field = [row objectForKey:key];
        if ([field isKindOfClass:[NSNull class]]) {
            [writer writeField:nil];
        }
        else {
            [writer writeField:field];
        }

    }
    //> finish the line
    [writer finishLine];
}
[writer closeStream];

Answer 1

如果在创建大型CSV文件时不想使用大量内存，请不要创建基于内存的输出流。创建到实际文件的输出流。然后，CSV数据将被写入文件而不是存储器。那么该文件可以是千兆字节，并且使用的内存很少。

这还有一个好处，即不需要访问缓冲区数据，无需从缓冲区创建一个字符串（现在将内存使用量增加了一倍），然后将该字符串写入文件。

NSOutputStream *csvStream = [NSOutputStream outputStreamToFileAtPath:csvPath append:NO];
[csvStream open];
CHCSVWriter *writer = [[CHCSVWriter alloc] initWithOutputStream:csvStream encoding:NSUTF8StringEncoding delimiter:';'];

// write your CSV entries

[writer closeStream];

就是这样。无需其他代码即可创建文件。

除了这些更改之外，您还需要更改使用自动释放池的位置。它应该位于外部for循环之内。

//> write the rows
for (NSMutableDictionary *row in [self sharedUploadManager].modifiedRows) {
    @autoreleasepool {
        NSArray *orderedKeys = [[row allKeys] sortedArrayUsingDescriptors:keySortDescriptors];

        for (NSString *key in orderedKeys ) {

            NSString *field = [row objectForKey:key];
            if ([field isKindOfClass:[NSNull class]]) {
                [writer writeField:nil];
            } else {
                [writer writeField:field];
            }
        }

        //> finish the line
        [writer finishLine];
    }
}

这将确保每行之后清除自动释放对象的内存。

CHCSVWriter的内存使用量，用于写入更大的CSV文件

1 个答案: