使用批处理窗口合并1300万行不起作用？

时间：2017-12-03 10:41:15

标签： windows bash csv batch-file

我有大约1000个带有标题的csv文件。我试图在Windows中使用批处理脚本合并所有文件。这是每个csv文件的内容，分隔符是（|），每个csv文件有超过200万行，总数约为1300万行。 enter image description here

在 bash 中非常简单方便，我使用此命令合并所有文件，只考虑一个标头。 

sed 1d mpage*.csv | sort -u | sort -r > DEST_FILE

在Windows中我发现了以下内容，

@echo off
setlocal
set first=1
set fileName="mergefiles.csv"
>%fileName% (
  for %%F in (*.csv) do (
    if not "%%F"==%fileName% (
      if defined first (
        type "%%F"
        set "first="
      ) else more +1 "%%F"
    )
  )
)

当我运行批次时，需要更多 1小时，最后我没有得到任何结果。 在bash（Linux）中只需2分钟。 我需要帮助。我不确定批次的哪一部分不正确！？如有任何帮助，我将不胜感激。

2 个答案:

答案 0 :(得分：0)

您可以像这样使用PowerShell：

12-03 16:41:11.099 2211-2211/? I/art: Late-enabling -Xcheck:jni
12-03 16:41:11.199 2211-2211/raghuveer.singh.bhardwaj.blogapp W/DynamiteModule: Local module descriptor class for com.google.firebase.auth not found.
12-03 16:41:11.203 2211-2211/raghuveer.singh.bhardwaj.blogapp W/DynamiteModule: Local module descriptor class for com.google.firebase.auth not found.
12-03 16:41:11.212 2211-2211/raghuveer.singh.bhardwaj.blogapp I/BiChannelGoogleApi: [FirebaseAuth: ] No Fallback module; NOT setting up for lazy initialization
12-03 16:41:11.222 2211-2211/raghuveer.singh.bhardwaj.blogapp D/FirebaseAuth: Notifying id token listeners about user ( sLZMm3S1o8TESyFjjy11mYEdVTG3 ).
12-03 16:41:11.223 2211-2232/raghuveer.singh.bhardwaj.blogapp W/DynamiteModule: Local module descriptor class for com.google.firebase.auth not found.
12-03 16:41:11.232 2211-2211/raghuveer.singh.bhardwaj.blogapp D/FirebaseApp: com.google.firebase.crash.FirebaseCrash is not linked. Skipping initialization.
12-03 16:41:11.242 2211-2232/raghuveer.singh.bhardwaj.blogapp I/FirebaseAuth: [FirebaseAuth:] Loading module via FirebaseOptions.
12-03 16:41:11.242 2211-2232/raghuveer.singh.bhardwaj.blogapp I/FirebaseAuth: [FirebaseAuth:] Preparing to create service connection to gms implementation
12-03 16:41:11.260 2211-2211/raghuveer.singh.bhardwaj.blogapp V/FA: Registered activity lifecycle callback
12-03 16:41:11.261 2211-2211/raghuveer.singh.bhardwaj.blogapp I/FirebaseInitProvider: FirebaseApp initialization successful
12-03 16:41:11.283 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Collection enabled
12-03 16:41:11.284 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: App package, google app id: raghuveer.singh.bhardwaj.blogapp, 1:967774314306:android:75a87ded59f6ecc8
12-03 16:41:11.285 2211-2235/raghuveer.singh.bhardwaj.blogapp I/FA: App measurement is starting up, version: 11717
12-03 16:41:11.286 2211-2235/raghuveer.singh.bhardwaj.blogapp I/FA: To enable debug logging run: adb shell setprop log.tag.FA VERBOSE
12-03 16:41:11.286 2211-2235/raghuveer.singh.bhardwaj.blogapp I/FA: To enable faster debug mode event logging run:
                                                                      adb shell setprop debug.firebase.analytics.app raghuveer.singh.bhardwaj.blogapp
12-03 16:41:11.286 2211-2235/raghuveer.singh.bhardwaj.blogapp D/FA: Debug-level message logging enabled
12-03 16:41:11.299 2211-2211/raghuveer.singh.bhardwaj.blogapp W/art: Before Android 4.1, method android.graphics.PorterDuffColorFilter android.support.graphics.drawable.VectorDrawableCompat.updateTintFilter(android.graphics.PorterDuffColorFilter, android.content.res.ColorStateList, android.graphics.PorterDuff$Mode) would have incorrectly overridden the package-private method in android.graphics.drawable.Drawable
12-03 16:41:11.302 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Connecting to remote service
12-03 16:41:11.309 2211-2211/raghuveer.singh.bhardwaj.blogapp V/FA: onActivityCreated
12-03 16:41:11.309 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Connection attempt already in progress
12-03 16:41:11.462 2211-2211/raghuveer.singh.bhardwaj.blogapp I/DynamiteModule: Considering local module com.google.android.gms.firebase_database:4 and remote module com.google.android.gms.firebase_database:6
12-03 16:41:11.462 2211-2211/raghuveer.singh.bhardwaj.blogapp I/DynamiteModule: Selected remote version of com.google.android.gms.firebase_database, version >= 6
12-03 16:41:11.476 2211-2211/raghuveer.singh.bhardwaj.blogapp W/ResourcesManager: Asset path '/system/framework/com.android.media.remotedisplay.jar' does not exist or contains no resources.
12-03 16:41:11.476 2211-2211/raghuveer.singh.bhardwaj.blogapp W/ResourcesManager: Asset path '/system/framework/com.android.location.provider.jar' does not exist or contains no resources.
12-03 16:41:11.526 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Connection attempt already in progress
12-03 16:41:11.526 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Activity resumed, time: 11707769
12-03 16:41:11.535 2211-2235/raghuveer.singh.bhardwaj.blogapp I/FA: Tag Manager is not found and thus will not be used
12-03 16:41:11.543 2211-2211/raghuveer.singh.bhardwaj.blogapp I/ViewRootImpl: CPU Rendering VSync enable = true
12-03 16:41:11.543 2211-2235/raghuveer.singh.bhardwaj.blogapp D/FA: Logging event (FE): screen_view(_vs), Bundle[{firebase_event_origin(_o)=auto, firebase_screen_class(_sc)=MainActivity, firebase_screen_id(_si)=-7835563879800379359}]
12-03 16:41:11.546 2211-2241/raghuveer.singh.bhardwaj.blogapp D/OpenGLRenderer: Use EGL_SWAP_BEHAVIOR_PRESERVED: true
12-03 16:41:11.565 2211-2211/raghuveer.singh.bhardwaj.blogapp D/Atlas: Validating map...
12-03 16:41:11.574 2211-2211/raghuveer.singh.bhardwaj.blogapp D/FirebaseApp: Notifying auth state listeners.
12-03 16:41:11.574 2211-2211/raghuveer.singh.bhardwaj.blogapp D/FirebaseApp: Notified 1 auth state listeners.
12-03 16:41:11.577 2211-2211/raghuveer.singh.bhardwaj.blogapp I/ViewRootImpl: CPU Rendering VSync enable = true
12-03 16:41:11.633 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Connection attempt already in progress
12-03 16:41:11.638 2211-2241/raghuveer.singh.bhardwaj.blogapp I/Adreno-EGL: <qeglDrvAPI_eglInitialize:379>: EGL 1.4 QUALCOMM build: Nondeterministic_AU_msm8909_LA.BR.1.2.5_RB2__release_AU (I9d3821c5ab)
                                                                            OpenGL ES Shader Compiler Version: E031.25.03.04
                                                                            Build Date: 02/24/16 Wed
                                                                            Local Branch: mybranch18408715
                                                                            Remote Branch: quic/LA.BR.1.2.5_rb2.32
                                                                            Local Patches: NONE
                                                                            Reconstruct Branch: NOTHING
12-03 16:41:11.639 2211-2241/raghuveer.singh.bhardwaj.blogapp I/OpenGLRenderer: Initialized EGL, version 1.4
12-03 16:41:11.652 2211-2241/raghuveer.singh.bhardwaj.blogapp D/OpenGLRenderer: Enabling debug mode 0
12-03 16:41:11.945 2211-2235/raghuveer.singh.bhardwaj.blogapp D/FA: Connected to remote service
12-03 16:41:11.945 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Processing queued up service tasks: 4
12-03 16:41:13.556 2211-2211/raghuveer.singh.bhardwaj.blogapp I/ViewRootImpl: CPU Rendering VSync enable = true
12-03 16:41:13.633 2211-2241/raghuveer.singh.bhardwaj.blogapp V/RenderScript: Application requested CPU execution
12-03 16:41:13.640 2211-2241/raghuveer.singh.bhardwaj.blogapp V/RenderScript: 0xb7a6a9d8 Launching thread(s), CPUs 4
12-03 16:41:14.419 2211-2211/raghuveer.singh.bhardwaj.blogapp D/MainActivity: Cooking
12-03 16:41:14.421 2211-2211/raghuveer.singh.bhardwaj.blogapp D/MainActivity: Programming
12-03 16:41:14.433 2211-2211/raghuveer.singh.bhardwaj.blogapp I/Timeline: Timeline: Activity_idle id: android.os.BinderProxy@f2c1b9f time:11295907
12-03 16:41:17.363 2211-2235/raghuveer.singh.bhardwaj.blogapp V/FA: Inactivity, disconnecting from the service

答案 1 :(得分：0)

您可以使用SET /P命令和当前文件的重定向将列标签分配给变量。然后，您可以将此变量用作与FIND命令的反向匹配。正如我所说，这仍然比你的BASH脚本慢。

@echo off
setlocal enabledelayedexpansion
set "first=1"
set "fileName=mergefiles.csv"
>%fileName% (
    for %%F in (*.csv) do (
        if not "%%F"=="%fileName%" (
            set /p "header="<"%%F"
            if defined first (
                type "%%F"
                set "first="
            ) else (
                type "%%F" |find /V "!header!"
            )
        )
    )
)
endlocal
相关问题
最新问题