我有一个带有IMU传感器测量值的.txt日志文件，需要将其解析为.CSV文件。加速度计，陀螺仪具有500Hz ODR（输出数据速率）磁粉100Hz，gps 1Hz和baro 1Hz。 Wi-fi，BLE，压力，光等也被记录下来，但大多数都不需要。智能手机应用程序不会按顺序保存所有测量结果。
将200k +行的文件解析到pandas DataFrame花费1000秒钟以上的时间，将时间戳上的DataFrame排序并保存为csv文件。
在DataFrame中的坐标（行=时间戳，列=传感器测量）处分配传感器测量值时，有些需要约40％的运行时间，而另一些则需要+-0.1％的运行时间。
这可能是什么原因？ 不应花费1000秒钟以上。.
ACCE;AppTimestamp(s);SensorTimestamp(s);Acc_X(m/s^2);Acc_Y(m/s^2);Acc_Z(m/s^2);Accuracy(integer)
GYRO;AppTimestamp(s);SensorTimestamp(s);Gyr_X(rad/s);Gyr_Y(rad/s);Gyr_Z(rad/s);Accuracy(integer)
MAGN;AppTimestamp(s);SensorTimestamp(s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Accuracy(integer)
MAGN;AppTimestamp(s);SensorTimestamp(s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Accuracy(integer)
PRES;AppTimestamp(s);SensorTimestamp(s);Pres(mbar);Accuracy(integer)
LIGH;AppTimestamp(s);SensorTimestamp(s);Light(lux);Accuracy(integer)
PROX;AppTimestamp(s);SensorTimestamp(s);prox(?);Accuracy(integer)
HUMI;AppTimestamp(s);SensorTimestamp(s);humi(Percentage);Accuracy(integer)
TEMP;AppTimestamp(s);SensorTimestamp(s);temp(Celsius);Accuracy(integer)
AHRS;AppTimestamp(s);SensorTimestamp(s);PitchX(deg);RollY(deg);YawZ(deg);RotVecX();RotVecY();RotVecZ();Accuracy(int)
GNSS;AppTimestamp(s);SensorTimeStamp(s);Latit(deg);Long(deg);Altitude(m);Bearing(deg);Accuracy(m);Speed(m/s);SatInView;SatInUse
WIFI;AppTimestamp(s);SensorTimeStamp(s);Name_SSID;MAC_BSSID;RSS(dBm);
BLUE;AppTimestamp(s);Name;MAC_Address;RSS(dBm);
BLE4;AppTimestamp(s);MajorID;MinorID;RSS(dBm);
SOUN;AppTimestamp(s);RMS;Pressure(Pa);SPL(dB);
RFID;AppTimestamp(s);ReaderNumber(int);TagID(int);RSS_A(dBm);RSS_B(dBm);
IMUX;AppTimestamp(s);SensorTimestamp(s);Counter;Acc_X(m/s^2);Acc_Y(m/s^2);Acc_Z(m/s^2);Gyr_X(rad/s);Gyr_Y(rad/s);Gyr_Z(rad/s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Roll(deg);Pitch(deg);Yaw(deg);Quat(1);Quat(2);Quat(3);Quat(4);Pressure(mbar);Temp(Celsius)
IMUL;AppTimestamp(s);SensorTimestamp(s);Counter;Acc_X(m/s^2);Acc_Y(m/s^2);Acc_Z(m/s^2);Gyr_X(rad/s);Gyr_Y(rad/s);Gyr_Z(rad/s);Mag_X(uT);;Mag_Y(uT);Mag_Z(uT);Roll(deg);Pitch(deg);Yaw(deg);Quat(1);Quat(2);Quat(3);Quat(4);Pressure(mbar);Temp(Celsius)
POSI;Timestamp(s);Counter;Latitude(degrees); Longitude(degrees);floor ID(0,1,2..4);Building ID(0,1,2..3)
MAGN;1.249;343268.933;2.64000;-97.50000;-69.06000;0
GYRO;1.249;343268.934;0.02153;0.06943;0.09880;3
ACCE;1.249;343268.934;-0.24900;0.53871;9.59625;3 GNSS;1.250;1570711878.000;52.225976;5.174543;58.066;175.336;3.0;0.0;23;20
ACCE;1.253;343268.936;-0.26576;0.52674;9.58428;3
GYRO;1.253;343268.936;0.00809;0.06515;0.10002;3
ACCE;1.253;343268.938;-0.29450;0.49561;9.57710;3
GYRO;1.253;343268.938;0.00015;0.06088;0.10613;3
PRES;1.253;343268.929;1011.8713;3
GNSS;1.254;1570711878.000;52.225976;5.174543;58.066;175.336;3.0;0.0;23;20
ACCE;1.255;343268.940;-0.29450;0.49801;9.57710;3
GYRO;1.255;343268.940;-0.00596;0.05843;0.10979;3
ACCE;1.260;343268.942;-0.30647;0.50280;9.55795;3
GYRO;1.261;343268.942;-0.01818;0.05721;0.11529;3
MAGN;1.262;343268.943;2.94000;-97.74000;-68.88000;0
fileContent是上面显示的txt文件的字符串。
def parseValues(line):
valArr = []
valArr = np.fromstring(line[5:], dtype=float, sep=";")
return (valArr)
i = 0
while i < len(fileContent):
if (fileContent[i][:4] == "ACCE"):
vals = parseValues(fileContent[i])
idx = vals[1] - initialSensTS
df.at[idx, 'ax'] = vals[2]
df.at[idx, 'ay'] = vals[3]
df.at[idx, 'az'] = vals[4]
df.at[idx, 'accStat'] = vals[5]
i += 1
请参阅第28行。
Line # Hits Time Per Hit % Time Line Contents
==============================================================
22 1 1.0 1.0 0.0 i = 0
23 232250 542594.0 2.3 0.0 while i < len(fileContent):
24 232249 294337000.0 1267.3 23.8 update_progress(i / len(fileContent))
25 232249 918442.0 4.0 0.1 if (fileContent[i][:4] == "ACCE"):
26 54602 1584625.0 29.0 0.1 vals = parseValues(fileContent[i])
27 54602 316968.0 5.8 0.0 idx = vals[1] - initialSensTS
28 54602 504189480.0 9233.9 40.8 df.at[idx, 'ax'] = vals[2]
29 54602 8311109.0 152.2 0.7 df.at[idx, 'ay'] = vals[3]
30 54602 4901983.0 89.8 0.4 df.at[idx, 'az'] = vals[4]
31 54602 4428239.0 81.1 0.4 df.at[idx, 'accStat'] = vals[5]
32 54602 132590.0 2.4 0.0 i += 1
答案 0 :(得分：0)
这不会解决您有关时间戳排序等问题的部分，但是应该可以有效替代'ACCE'解析代码。
import pandas as pd
import collections as colls
logs_file_path = '../resources/imu_logs_raw.txt'
msmt_type_dict = colls.defaultdict(list)
with open(logs_file_path, 'r') as file_1:
for line in file_1:
curr_measure_type, *rest_str = line.split(';')
rest_str[-1] = rest_str[-1].strip()
msmt_type_dict[curr_measure_type].append(rest_str)
acce_df = pd.DataFrame(data=msmt_type_dict['ACCE'], columns=['app_timestamp', 'sensor_timestamp', 'acc_x', 'acc_y', 'acc_z', 'accuracy'])
如果您可以提供更多信息/上下文，我希望了解一下时间戳排序方面。