大型CSV文件处理

时间:2018-10-04 11:57:19

标签: python python-3.x shell audio

这可能是重复的问题。我已经搜索了很多,但没有找到答案。

我正在进行音频分析,其中需要将主要音频文件分解成多个音频文件,每个音频文件应包含一个句子。

当我花费大约3秒钟的音频,而我正在执行的过程大约需要5分钟时,但是如果采用更长的音频长度,则完成该过程所需的时间会大大增加。例如5分钟的音频大约需要14个小时。 首先,我创建了一个包含时间(以秒为单位)与振幅的csv文件,然后设置了一个阈值,其中小于 i 的振幅将为0,大于1的振幅。后来,我检查数字是否连续0的大于另一个阈值 j ,然后花费该位置的时间。因此,我找到了结束句子的时间。 此过程花费的时间太长,因此任何其他方法都将有所帮助。

我的数据集如下:

1.000000000000000000e+00,0.000000000000000000e+00,6.103515625000000000e+01
2.000000000000000000e+00,2.267999999999999969e-05,3.051757811999999959e+01
3.000000000000000000e+00,4.534999999999999779e-05,0.000000000000000000e+00
4.000000000000000000e+00,6.802999999999999748e-05,3.051757811999999959e+01
5.000000000000000000e+00,9.069999999999999558e-05,3.051757811999999959e+01
6.000000000000000000e+00,1.133800000000000020e-04,0.000000000000000000e+00
7.000000000000000000e+00,1.360500000000000001e-04,0.000000000000000000e+00
8.000000000000000000e+00,1.587299999999999931e-04,0.000000000000000000e+00
9.000000000000000000e+00,1.814100000000000131e-04,0.000000000000000000e+00
1.000000000000000000e+01,2.040800000000000112e-04,0.000000000000000000e+00
1.100000000000000000e+01,2.267600000000000041e-04,0.000000000000000000e+00
1.200000000000000000e+01,2.494299999999999751e-04,3.051757811999999959e+01
1.300000000000000000e+01,2.721099999999999951e-04,0.000000000000000000e+00
1.400000000000000000e+01,2.947800000000000203e-04,0.000000000000000000e+00
1.500000000000000000e+01,3.174599999999999861e-04,0.000000000000000000e+00
1.600000000000000000e+01,3.401400000000000061e-04,3.051757811999999959e+01
1.700000000000000000e+01,3.628099999999999771e-04,0.000000000000000000e+00
1.800000000000000000e+01,3.854899999999999972e-04,3.051757811999999959e+01
1.900000000000000000e+01,4.081600000000000224e-04,0.000000000000000000e+00
2.000000000000000000e+01,4.308399999999999882e-04,0.000000000000000000e+00
2.100000000000000000e+01,4.535100000000000134e-04,3.051757811999999959e+01

从CSV文件复制。行数为415449。我只给了您20。我需要检查第一列是否在(1,2,3,...,n)中。当系列中断时,我需要在系列的最后一个数字处获取第2列的值。希望我能解决我的问题

请注意:我需要在shell,python,C,C ++中进行硬编码。

1 个答案:

答案 0 :(得分:0)

现在,我对您有所了解了,使用awk:

awk -v n=6 '            # n as parameter
BEGIN { 
    FS=","              # comma as the field separator
} 
int($1)==$1 && $1<=n {  # if $1 is an integer less than or equal to n
    val=$2              # value of column 2 at the last number of series
}
END {
    print val           # output the value
}' file
1.133800000000000020e-04

更新

$ awk -v i=1 -v j=0 -v k=3 '
BEGIN {
    FS=","
}
$3<i {        # if the value of 3rd column is less than "i" 
    j++       # then "j" will increment by 1
}
j>k {         # when the value of "j" is greater than a value "k" 
    print $1  # It will print the column 1 value
    # exit    # uncomment this if it only needs to print one value
}' file
8.000000000000000000e+00
9.000000000000000000e+00
1.000000000000000000e+01
1.100000000000000000e+01
1.200000000000000000e+01
...