许多文件需要分两个循环处理，并且同一文件可以多次加载。如何在不添加重复记录的情况下将唯一记录追加到R SQLite数据库中？
问题是覆盖= T仅保存最后一个数据帧，append = T每次运行程序都添加所有行。
参考：https://stackoverflow.com/a/51922038/9410024。主键应为文件名+方向+里程。次要键为文件名，方向，位置，里程。
# Collect thousands of traffic data files for analysis
library(DBI)
setwd('C://')
db <- dbConnect(RSQLite::SQLite(), "trarr.sqlite")
for (fn in list.files(pattern = "\\.OUT$")) {
lines <- readLines( fn )
end = 0
for (direction in c(1,2)) {
start = end + 1
start = start + min(grep(" \\*{4} DIRECTION", lines[start:length(lines)])) + 3
end = start + min(grep("^$", lines[start:length(lines)])) -2
datalist = lapply(fn, function(x)read.table(x, skip=start, nrows=end-start,
col.names=c('POSITION', 'CHAINAGE', 'OVERTAKINGS', 'COMMENCED', 'SPEED',
'SPEED_SD','PERC_FOLL', 'NUMBER', 'PASSED', 'CARS', 'CARTOW', 'RIGIDS',
'SGLART', 'DBLART', 'RDTRAIN', 'MSDIF')))
# Bind list into a dataframe
datafr = do.call("rbind", datalist)
# Add Metadata Fields
datafr = cbind(FILENAME = fn, DIRECTION = direction, datafr)
# Append to trarr database pob (point observaton) table
dbWriteTable(conn = db, name = "pob", datafr, overwrite=F, append=T, row.names=FALSE)
}
}
# Rows in table
print(dbGetQuery(db, 'SELECT Count(*) FROM pob'))
print(dbGetQuery(db, 'SELECT DISTINCT filename, direction FROM pob'))
# print(dbGetQuery(db, 'SELECT DISTINCT chainage, perc_foll, direction FROM pob'))
dbDisconnect(db)
Sample.OUT数据文件：
TRARR 7.00 25/05/2018 16:36:13
highflow_seed693305.TRF AND SGH Yarram Sale.ROD AND --------- - PBS VEHS COMBINATION (TRARR 7.00 )
TRAFFIC PARAMETERS SPECIFIED AT INPUT:
BASIC TIME UNIT (SECS) = 1.0
TIME OF SIMULATION (SECS) = 30000.0
SETTLING DOWN TIME (SECS) = 15000.0
RANDOM SEED NUMBER = 693305.0
% FOLLOWING, DIRECTION 1 = 60.0
% FOLLOWING, DIRECTION 2 = 60.0
STREAM DIR1 DIR2
FLOW FLOW TOTAL
(VEH/H) (VEH/H)
CARS 700.0 700.0 1400.0
CAR TOW 70.0 70.0 140.0
RIGIDS 50.0 50.0 100.0
SGLARTC 60.0 60.0 120.0
DBLARTC 30.0 30.0 60.0
RDTRAIN 10.0 10.0 20.0
TOTAL 920.0 920.0 1840.0
ACTUAL FLOWS - DIRECTION 1: 925.1 VEH/H
- DIRECTION 2: 927.8 VEH/H
- COMBINED: 1852.9 VEH/H
ACTUAL COMPLETION TIME: 33187.0 SEC
MAXIMUM NUMBER OF VEHICLES ON ROAD: 1777
-------------------------------------------------------
**** DIRECTION 1 ****
POINT OBSERVATIONS: POSITIONS MEASURED FROM START IN DIRECTION OF TRAVEL
POSITION ROAD FILE OVERTAKINGS COMMENCED SPEED (KM/H) %FOLL NUMBER PASSED OBS POINT MEAN SPEED BY CATEGORY
M CHAINAGE M TOTAL NO./HOUR MEAN S.D. TOTAL NO./HOUR CARS CARTOW RIGIDS SGLART DBLART RDTRAIN MSDIF
2000.0 186000.0 0 0.0 89.0 4.9 72.7 7709 925.1 88.7 90.8 90.6 90.5 87.7 81.3 9.4
9668.0 193668.0 76 9.1 87.4 3.0 90.4 7709 925.1 87.4 87.6 87.5 87.5 87.7 88.6 1.2
17335.0 201335.0 5 0.6 86.6 2.5 94.0 7709 925.1 86.6 86.8 86.8 86.6 86.4 85.6 1.2
25003.0 209003.0 8 1.0 86.3 2.4 95.3 7709 925.1 86.3 86.4 86.4 86.3 86.1 87.4 1.2
32671.0 216671.0 0 0.0 86.1 2.3 96.3 7709 925.1 86.1 86.2 86.3 86.1 86.0 86.2 0.3
40338.0 224338.0 2 0.2 83.6 4.2 96.9 7709 925.1 83.7 83.8 83.7 83.6 82.6 78.3 5.5
48006.0 232006.0 4 0.5 86.7 2.9 96.6 7709 925.1 86.6 86.7 87.0 86.8 87.6 88.3 1.7
55674.0 239674.0 2 0.2 82.1 5.6 96.5 7709 925.1 82.0 82.0 82.4 82.0 83.4 81.8 1.7
63342.0 247342.0 0 0.0 86.6 3.0 96.5 7709 925.1 86.5 86.5 86.7 86.3 88.2 89.4 3.1
71010.0 255010.0 0 0.0 85.3 2.5 98.1 7709 925.1 85.2 85.2 85.4 85.5 86.1 87.5 2.3
TOTAL NUMBER VEHICLES COMMENCING OVERTAKING = 97
MINIMUM, MAXIMUM and AVERAGE CONSERVATION 7709 7709 7709
DIR1 INTERVAL OBSERVATIONS BETWEEN 2000.M AND 71010.M ( 69010.M )
VEHICLE TRAVEL TIME JOURNEY SPEED %TIME ___________OVERTAKEN VEHICLES___________ PETROL DIESEL TOTAL NUMBER
CATEGORY MEAN S.D. MEAN S.D. SPENT TOTAL NO./HOUR TOTAL NO./HOUR RATE CONS. CONS. NUMBER VEHICLES
SEC SEC KM/H KM/H FOLL. OF OF BY BY # ML/VEH ML/VEH VEHICLES PER HOUR
CARS 2945.4 72.6 84.4 2.1 95.0 83 10.0 118 14.2 0.000 9343.7 0.0 5889 706.7
CARTOW 2941.6 67.4 84.5 1.9 95.3 7 0.8 0 0.0 0.000 11864.3 0.0 553 66.4
RIGIDS 2942.2 69.9 84.5 2.0 96.3 11 1.3 0 0.0 0.000 0.0 17846.2 466 55.9
SGLART 2943.4 77.2 84.5 2.2 94.9 11 1.3 0 0.0 0.000 0.0 28027.2 489 58.7
DBLART 2945.3 76.0 84.4 2.1 93.6 4 0.5 0 0.0 0.000 0.0 35838.3 235 28.2
RDTRAIN 2976.8 66.5 83.5 1.8 81.1 2 0.2 0 0.0 0.000 0.0 44048.3 77 9.2
ALL 2945.1 72.5 84.4 2.1 94.9 118 14.2 0.000 9560.1 26705.1 7709 925.1
# OVERTAKING RATE IS VEHICLES OVERTAKEN/VEHICLE/KM
No. Vehicles COMMENCING OVERTAKING per TOTAL No. Vehicles 0.013
No. Vehicles OVERTAKEN per TOTAL No. Vehicles 0.015
RATIO of No. Vehicles OVERTAKEN to No. Vehicles OVERTAKING 1.2
**** DIRECTION 2 ****
POINT OBSERVATIONS: POSITIONS MEASURED FROM START IN DIRECTION OF TRAVEL
POSITION ROAD FILE OVERTAKINGS COMMENCED SPEED (KM/H) %FOLL NUMBER PASSED OBS POINT MEAN SPEED BY CATEGORY
M CHAINAGE M TOTAL NO./HOUR MEAN S.D. TOTAL NO./HOUR CARS CARTOW RIGIDS SGLART DBLART RDTRAIN MSDIF
2000.0 255000.0 0 0.0 86.4 6.2 73.5 7732 927.8 86.8 86.9 86.6 84.1 83.5 78.0 8.8
9668.0 247332.0 53 6.4 54.7 7.2 96.7 7732 927.8 54.7 54.8 55.1 55.0 55.2 49.1 6.1
17335.0 239665.0 0 0.0 84.6 4.3 95.0 7732 927.8 84.6 84.5 84.5 84.4 85.1 85.7 1.3
25003.0 231997.0 0 0.0 76.8 6.3 97.3 7732 927.8 77.2 76.9 75.6 76.4 73.2 68.0 9.2
32671.0 224329.0 16 1.9 87.6 3.0 91.4 7732 927.8 87.4 87.6 87.6 87.5 89.6 90.7 3.2
40338.0 216662.0 2 0.2 86.3 2.7 94.7 7732 927.8 86.3 86.4 86.4 86.2 86.5 88.2 2.1
48006.0 208994.0 2 0.2 86.1 2.4 96.1 7732 927.8 86.0 86.2 86.1 86.0 86.0 88.3 2.3
55674.0 201326.0 0 0.0 85.5 2.4 96.9 7732 927.8 85.4 85.5 85.5 85.4 85.4 87.0 1.6
63342.0 193658.0 0 0.0 85.4 2.4 97.3 7732 927.8 85.3 85.4 85.3 85.3 85.3 86.9 1.6
71010.0 185990.0 0 0.0 85.0 3.7 97.1 7732 927.8 85.1 85.0 84.9 85.4 84.6 76.7 8.7
TOTAL NUMBER VEHICLES COMMENCING OVERTAKING = 73
MINIMUM, MAXIMUM and AVERAGE CONSERVATION 7732 7732 7732
DIR2 INTERVAL OBSERVATIONS BETWEEN 2000.M AND 71010.M ( 69010.M )
VEHICLE TRAVEL TIME JOURNEY SPEED %TIME ___________OVERTAKEN VEHICLES___________ PETROL DIESEL TOTAL NUMBER
CATEGORY MEAN S.D. MEAN S.D. SPENT TOTAL NO./HOUR TOTAL NO./HOUR RATE CONS. CONS. NUMBER VEHICLES
SEC SEC KM/H KM/H FOLL. OF OF BY BY # ML/VEH ML/VEH VEHICLES PER HOUR
CARS 3077.0 93.0 80.8 2.4 95.5 186 22.3 298 35.8 0.001 9171.9 0.0 5846 701.5
CARTOW 3072.7 91.2 80.9 2.4 96.1 33 4.0 0 0.0 0.000 11698.5 0.0 617 74.0
RIGIDS 3076.8 89.7 80.8 2.3 96.0 26 3.1 1 0.1 0.000 0.0 17734.1 406 48.7
SGLART 3073.7 95.2 80.9 2.5 95.8 45 5.4 0 0.0 0.000 0.0 28541.5 516 61.9
DBLART 3072.6 83.5 80.9 2.2 92.3 3 0.4 0 0.0 0.000 0.0 36601.3 263 31.6
RDTRAIN 3107.5 96.6 80.0 2.5 72.9 6 0.7 0 0.0 0.000 0.0 44286.5 84 10.1
ALL 3076.6 92.6 80.8 2.4 95.3 299 35.9 0.001 9413.1 27796.4 7732 927.8
# OVERTAKING RATE IS VEHICLES OVERTAKEN/VEHICLE/KM
No. Vehicles COMMENCING OVERTAKING per TOTAL No. Vehicles 0.010
No. Vehicles OVERTAKEN per TOTAL No. Vehicles 0.039
RATIO of No. Vehicles OVERTAKEN to No. Vehicles OVERTAKING 4.1
* INTERVAL INFORMATION FOR BOTH DIRECTIONS COMBINED *
( ASSUMES MATCHING LENGTHS OF 69010.M )
VEHICLE TRAVEL TIME JOURNEY SPEED %TIME ___________OVERTAKEN VEHICLES___________ PETROL DIESEL TOTAL NUMBER
CATEGORY MEAN S.D. MEAN S.D. SPENT TOTAL NO./HOUR TOTAL NO./HOUR RATE CONS. CONS. NUMBER VEHICLES
SEC SEC KM/H KM/H FOLL. OF OF BY BY # ML/VEH ML/VEH VEHICLES PER HOUR
CARS 3011.0 106.2 82.6 2.9 95.3 269 32.3 416 49.9 0.001 9258.1 0.0 11735 1408.2
CARTOW 3010.7 104.0 82.6 2.8 95.7 40 4.8 0 0.0 0.000 11776.8 0.0 1170 140.4
RIGIDS 3004.9 104.2 82.8 2.8 96.2 37 4.4 1 0.1 0.000 0.0 17794.0 872 104.6
SGLART 3010.3 108.6 82.6 2.9 95.4 56 6.7 0 0.0 0.000 0.0 28291.3 1005 120.6
DBLART 3012.5 102.2 82.6 2.8 92.9 7 0.8 0 0.0 0.000 0.0 36241.3 498 59.8
RDTRAIN 3045.0 106.0 81.7 2.8 76.8 8 1.0 0 0.0 0.000 0.0 44172.6 161 19.3
ALL 3011.0 106.0 82.6 2.9 95.1 417 50.0 0.000 9486.5 27251.2 15441 1852.9
** FREE SPEED DISTRIBUTIONS: DESIRED SPEEDS IGNORE ROAD CHARACTERISTICS;
UNIMPEDED SPEEDS TAKE ACCOUNT OF ROAD SPEED INDICES, BUT NOT GRADES OR TRAFFIC.
VEHICLE DESIRED UNIMPEDED NUMBER
CATEGORY SPEED SPEED
MEAN S.D. MEAN S.D.
CARS 106.1 4.0 93.8 3.5 11735
CARTOW 100.9 4.1 96.2 3.9 1170
RIGIDS 101.9 3.9 97.2 3.8 872
SGLART 99.0 4.0 95.5 3.9 1005
DBLART 99.2 3.1 95.6 2.9 498
RDTRAIN 92.5 4.9 92.1 4.9 161
ALL 104.6 4.8 94.3 3.7 15441
**** FREQUENCY DISTRIBUTIONS TAKEN AT THE END-OF-INTERVAL OBSERVING POINTS ****
FREQUENCY DISTRIBUTION OF SPEEDS, AS A CUMULATIVE PERCENTAGE:
1 2 3 4 5 6 7 8 9 0
31- 40 0 0 0 0 0 0 0 0 0 0
41- 50 0 0 0 0 0 0 0 0 0 0
51- 60 0 0 0 0 0 0 0 0 0 0
61- 70 0 0 0 0 0 0 0 0 0 0
71- 80 0 0 0 0 1 2 3 4 6 10
81- 90 15 21 31 44 59 72 84 91 95 98
91-100 99 100 100 100 100 100 100 100 100 100
101-110 100 100 100 100 100 100 100 100 100 100
111-120 100 100 100 100 100 100 100 100 100 100
121-130 100 100 100 100 100 100 100 100 100 100
FREQUENCY OF BUNCH SIZES:
A VEHICLE IS TAKEN TO BE FOLLOWING IF IT IS AT ITS CHOSEN FOLLOWING DISTANCE
BASED ON VEHICLE LENGTH AND HEADWAY DISTANCE AND TIME
A SINGLE VEHICLE HAS A BUNCH SIZE OF 1.
1 2 3 4 5 6 7 8 9 10+
DIRECTION 1 49 49 51 38 30 44 39 32 31 269
DIRECTION 2 79 64 65 48 50 39 31 45 37 302
TOTAL 128 113 116 86 80 83 70 77 68 571
DISTRIBUTION OF THE NUMBER OF VEHICLES OVERTAKEN BY ONE VEHICLE,
TAKEN FROM THE CHANGES OF ORDER BETWEEN ARRIVAL AND EXIT.
0 1 2 3 4 5 6 7 8 9 10+
DIRECTION 1 7630 62 8 4 0 3 1 1 0 0 0
DIRECTION 2 7681 14 6 3 3 3 0 4 2 2 14
TOTAL 15311 76 14 7 3 6 1 5 2 2 14
NUMBERS OF VEHICLES AND AVERAGE SPEEDS AT GENERATION
VEH TYPE NAME NUMBER AVE SPD
1 PBS 4B Vehicles 0 0.00
2 Triple Road Train (TypeII) 12 92.87
3 Triple Road Train 19 93.55
4 Double Road Train (TypeI) 378 92.41
5 PBS 2B A-Double 13 97.49
6 B-double heavy load 109 99.45
7 B-double heavy load (low power) 1004 99.16
8 B-double light load 164 98.79
9 (single) Artic heavy load 2002 98.97
10 (single) Artic light load 667 99.00
11 3/4 axle Rigid heavy load (or Bus) 20 103.05
12 3/4 axle Rigid light load (or Bus) 554 101.97
13 2 axle Rigid 1683 101.95
14 Car Towing 3050 100.93
15 Unaggressive Car 4565 104.77
16 Low powered Car 2998 105.15
17 Typical SUV / Sedan 18408 106.19
18 Small Hatchback 4649 107.37
AVERAGE PROPERTIES OF FREE and FOLLOWING VEHICLES PASSING OBSERVATION POINTS
DIRECTION 1
TYPE NAME FREE SPEED FOLLOWING SPEED %FREE(TIME) %FOLL(TIME) %FREE(DIST) %FOLL(DIST)
1 PBS 4B Vehicles - - - - - -
2 Triple Road Train (TypeII) 88.4 85.8 40.00 60.00 40.76 59.24
3 Triple Road Train 86.9 85.1 32.86 67.14 33.33 66.67
4 Double Road Train (TypeI) 85.3 85.5 63.68 36.32 63.62 36.38
5 PBS 2B A-Double 87.7 85.6 12.00 88.00 12.26 87.74
6 B-double heavy load 90.6 85.2 8.00 92.00 8.47 91.53
7 B-double heavy load (low power) 88.5 85.8 13.64 86.36 14.00 86.00
8 B-double light load 92.0 86.1 6.43 93.57 6.84 93.16
9 (single) Artic heavy load 90.4 85.8 8.34 91.66 8.75 91.25
10 (single) Artic light load 90.4 85.5 7.64 92.36 8.04 91.96
11 3/4 axle Rigid heavy load (or Bus) -- 86.8 0.00 100.00 0.00 100.00
12 3/4 axle Rigid light load (or Bus) 94.6 86.2 2.67 97.33 2.92 97.08
13 2 axle Rigid 92.5 85.9 4.80 95.20 5.14 94.86
14 Car Towing 91.6 85.8 6.47 93.53 6.88 93.12
15 Unaggressive Car 87.9 85.8 9.59 90.41 9.80 90.20
16 Low powered Car 89.1 85.7 7.23 92.77 7.50 92.50
17 Typical SUV / Sedan 89.2 85.7 5.03 94.97 5.22 94.78
18 Small Hatchback 88.9 85.8 4.36 95.64 4.51 95.49
DIRECTION 2
TYPE NAME FREE SPEED FOLLOWING SPEED %FREE(TIME) %FOLL(TIME) %FREE(DIST) %FOLL(DIST)
1 PBS 4B Vehicles - - - - - -
2 Triple Road Train (TypeII) 79.1 84.2 93.33 6.67 93.02 6.98
3 Triple Road Train 76.4 83.1 50.00 50.00 47.92 52.08
4 Double Road Train (TypeI) 78.0 84.4 69.87 30.13 68.18 31.82
5 PBS 2B A-Double 79.3 83.0 40.00 60.00 38.99 61.01
6 B-double heavy load 83.2 81.2 9.57 90.43 9.77 90.23
7 B-double heavy load (low power) 77.3 82.4 20.00 80.00 19.01 80.99
8 B-double light load 84.5 81.5 10.57 89.43 10.91 89.09
9 (single) Artic heavy load 84.2 81.3 7.60 92.40 7.85 92.15
10 (single) Artic light load 86.3 81.2 7.26 92.74 7.67 92.33
11 3/4 axle Rigid heavy load (or Bus) 90.4 82.7 2.50 97.50 2.73 97.27
12 3/4 axle Rigid light load (or Bus) 88.9 81.9 4.57 95.43 4.94 95.06
13 2 axle Rigid 87.9 81.3 4.84 95.16 5.21 94.79
14 Car Towing 89.9 81.6 4.41 95.59 4.84 95.16
15 Unaggressive Car 87.7 81.3 7.70 92.30 8.25 91.75
16 Low powered Car 89.3 81.3 6.87 93.13 7.49 92.51
17 Typical SUV / Sedan 89.3 81.6 4.53 95.47 4.94 95.06
18 Small Hatchback 88.9 81.5 4.26 95.74 4.63 95.37
AVERAGE PROPERTIES OF ALL FREE AND FOLLOWING VEHICLES ON THE ROAD
DIRECTION 1
TYPE NAME FREE SPEED FOLLOWING SPEED %FREE(TIME) %FOLL(TIME) %FREE(DIST) %FOLL(DIST)
1 PBS 4B Vehicles - - - - - -
2 Triple Road Train (TypeII) 83.3 83.0 41.99 58.01 42.06 57.94
3 Triple Road Train 83.3 83.5 31.41 68.59 31.37 68.63
4 Double Road Train (TypeI) 83.2 83.9 61.23 38.77 61.01 38.99
5 PBS 2B A-Double 86.9 83.7 12.10 87.90 12.50 87.50
6 B-double heavy load 89.1 83.6 6.18 93.82 6.56 93.44
7 B-double heavy load (low power) 86.4 84.0 11.75 88.25 12.04 87.96
8 B-double light load 91.8 84.8 4.34 95.66 4.67 95.33
9 (single) Artic heavy load 89.2 84.3 5.72 94.28 6.03 93.97
10 (single) Artic light load 90.0 83.9 5.17 94.83 5.53 94.47
11 3/4 axle Rigid heavy load (or Bus) -- 84.7 0.00 100.00 0.00 100.00
12 3/4 axle Rigid light load (or Bus) 93.7 84.5 1.16 98.84 1.29 98.71
13 2 axle Rigid 91.9 84.2 2.89 97.11 3.14 96.86
14 Car Towing 90.3 84.2 4.06 95.94 4.34 95.66
15 Unaggressive Car 87.6 84.3 8.52 91.48 8.83 91.17
16 Low powered Car 88.7 84.1 6.05 93.95 6.35 93.65
17 Typical SUV / Sedan 88.6 82.3 4.22 95.78 4.53 95.47
18 Small Hatchback 88.0 84.5 3.64 96.36 3.78 96.22
DIRECTION 2
TYPE NAME FREE SPEED FOLLOWING SPEED %FREE(TIME) %FOLL(TIME) %FREE(DIST) %FOLL(DIST)
1 PBS 4B Vehicles - - - - - -
2 Triple Road Train (TypeII) 79.9 83.8 92.11 7.89 91.76 8.24
3 Triple Road Train 80.1 79.3 38.74 61.26 38.99 61.01
4 Double Road Train (TypeI) 79.5 81.1 65.20 34.80 64.77 35.23
5 PBS 2B A-Double 82.1 79.8 41.08 58.92 41.78 58.22
6 B-double heavy load 83.6 79.9 6.65 93.35 6.94 93.06
7 B-double heavy load (low power) 80.3 81.3 16.88 83.12 16.70 83.30
8 B-double light load 86.9 81.2 7.11 92.89 7.57 92.43
9 (single) Artic heavy load 86.6 80.7 5.52 94.48 5.89 94.11
10 (single) Artic light load 89.2 80.7 4.46 95.54 4.90 95.10
11 3/4 axle Rigid heavy load (or Bus) 91.9 80.8 1.08 98.92 1.23 98.77
12 3/4 axle Rigid light load (or Bus) 90.6 81.2 2.33 97.67 2.59 97.41
13 2 axle Rigid 89.0 80.6 2.83 97.17 3.11 96.89
14 Car Towing 90.6 81.0 2.45 97.55 2.74 97.26
15 Unaggressive Car 87.3 80.8 6.55 93.45 7.04 92.96
16 Low powered Car 88.7 80.6 5.25 94.75 5.74 94.26
17 Typical SUV / Sedan 87.7 78.8 3.41 96.58 3.79 96.21
18 Small Hatchback 87.5 80.8 3.31 96.69 3.57 96.43