Question

我有一组大文件，必须分成100MB部分。我遇到的问题是行被 ^ B ASCII（或\ u002）字符终止。

因此，我需要能够获得100MB的部分（显然加上或减去几个字节），这也是行结束的原因。

示例文件：

... 000111222333 NNN ^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... NNN ^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B000111222333 ... nnn的^ B

“线”的大小可能会有所不同。

我知道分裂和csplit，但无法将我的头包裹在两者之间。

#!/bin/bash
split -b 100m filename                              #splitting by size
csplit filename “/$(echo -e “\u002”)/+1” “{*}”      #splitting by context

关于我如何能够保持线条完整的100MB块的任何建议？作为旁注，我无法将行结尾更改为 \ n ，因为这会损坏文件，因为 ^ B 之间的数据必须维护新行字符如果存在。

Answer 1

以下内容将在本机bash中实现您的拆分逻辑 - 执行速度不是很快，但它可以在任何地方运行bash而无需运行第三方工具：

#!/bin/bash

prefix=${1:-"out."}                        # first optional argument: output file prefix
max_size=${2:-$(( 1024 * 1024 * 100 ))}    # 2nd optional argument: size in bytes

cur_size=0                                 # running count: size of current chunk
file_num=1                                 # current numeric suffix; starting at 1
exec >"$prefix$file_num"                   # open first output file

while IFS= read -r -d $'\x02' piece; do    # as long as there's new input...
  printf '%s\x02' "$piece"                 # write it to our current output file      
  cur_size=$(( cur_size + ${#piece} + 1 )) # add its length to our counter
  if (( cur_size > max_size )); then       # if our counter is over our maximum size...
    (( ++file_num ))                       # increment the file counter
    exec >"$prefix$file_num"               # open a new output file
    cur_size=0                             # and reset the output size counter
  fi
done

if [[ $piece ]]; then  # if the end of input had content without a \x02 after it...
  printf '%s' "$piece" # ...write that trailing content to our output file.
fi

依赖于dd的版本（GNU版本，此处;可以更改为可移植版），但对于大型输入，它应该更快：

#!/bin/bash

prefix=${1:-"out."}                        # first optional argument: output file prefix

file_num=1                                 # current numeric suffix; starting at 1
exec >"$prefix$file_num"                   # open first output file

while true; do
  dd bs=1M count=100                       # tell GNU dd to copy 100MB from stdin to stdout
  if IFS= read -r -d $'\x02' piece; then   # read in bash to the next boundary
    printf '%s\x02' "$piece"               # write that segment to stdout
    exec >"$prefix$((++file_num))"         # re-open stdout to point to the next file
  else
    [[ $piece ]] && printf '%s' "$piece"   # write what's left after the last boundary
    break                                  # and stop
  fi
done

# if our last file is empty, delete it.
[[ -s $prefix$file_num ]] || rm -f -- "$prefix$file_num"

按bash中的上下文和大小拆分文件

1 个答案: