批处理脚本:更快地将ECHO输出到FOR循环中的csv文件?

时间:2017-02-17 02:10:09

标签: csv batch-file for-loop

我编写了一个脚本,用于将文本文件中的分号分隔符替换为逗号,但它也会在第9列的文本周围添加引号,因为第9列包含逗号。当我输出到新文本文件时它处理速度非常慢,大约需要4-5分钟,它正在读取的文本文件是50MB。有没有更快的方式或更有效的方法来做到这一点?这是我的FOR循环:

FOR /f "usebackq tokens=1-9* delims=;" %%a IN ("%FILENAME%") DO (
SET C10=%%j
ECHO(%%a,%%b,%%c,%%d,%%e,%%f,%%g,%%h,"%%i",!C10:;=,! >> "%MYPATH%\Filename %MMDDYYYY%.csv")

或者我应该学习python .....

谢谢。

3 个答案:

答案 0 :(得分:6)

使脚本运行"更快" 的一件事是避免为每次写操作打开和关闭输出文件

>> "%MYPATH%\Filename %MMDDYYYY%.csv" (
    FOR /f "usebackq tokens=1-9* delims=;" %%a IN ("%FILENAME%") DO (
        SET C10=%%j
        ECHO(%%a,%%b,%%c,%%d,%%e,%%f,%%g,%%h,"%%i",!C10:;=,!
    )
)

不是重定向每一行,而是重定向完整的for命令。

答案 1 :(得分:4)

如果要更改分隔符以在Excel中正确打开csv:无需执行此操作。有一个(未记录的)技巧告诉Excel,使用什么字符作为分隔符:

(echo Sep=;) > new.csv
type old.csv >> new.csv
move /y new.csv old.csv

注意:Excel会读取Sep=;,并使用该分隔符导入文件的其余部分,但它不属于传播文件的一部分,因此如果再次使用Excel保存,{{1}丢失(Excel使用Sep=;,作为分隔符,具体取决于它的安装语言。

答案 2 :(得分:1)

Squashman最好使用我的parseCSV.bat utility进行转换。该实用程序是一个混合脚本,它使用批处理和JScript。该实用程序比任何纯批处理解决方案都快得多,并且它可以在任何Windows机器上从XP开始本地运行 - 不需要第三方exe。

该实用程序实际上是为了在批处理文件中通过FOR / F方便地解析CSV文件而生成的。版本1.0有一个问题,使得使用该实用程序转换CSV格式以用于其他用途不太理想 - 转义的引用文字未转义,因此""成为"。此转换对于FOR / F解析很有用,但它不是有效的CSV格式。

我修改了该实用程序,以便选择保留引号文字的转义。现在,您可以安全地使用以下内容快速从分号转换为逗号分隔符。

call parseCSV

因为parseCSV本身就是一个批处理脚本,所以如果在另一个批处理脚本中使用该命令,则必须使用@if (@X)==(@Y) @end /* harmless hybrid line that begins a JScrpt comment ::************ Documentation *********** ::parseCSV.bat version 1.2 ::: :::parseCSV [/option]... ::: ::: Parse stdin as CSV and write it to stdout in a way that can be safely ::: parsed by FOR /F. All columns will be enclosed by quotes so that empty ::: columns may be preserved. It also supports delimiters, newlines, and ::: escaped quotes within quoted values. Two consecutive quotes within a ::: quoted value are converted into one quote by default. ::: ::: Available options: ::: ::: /I:string = Input delimiter. Default is a comma (,) ::: ::: /O:string = Output delimiter. Default is a comma (,) ::: ::: The entire option must be quoted if specifying poison character ::: or whitespace literals as a delimiters for /I or /O. ::: ::: Examples: pipe = "/I:|" ::: space = "/I: " ::: ::: Standard JScript escape sequences can also be used. ::: ::: Examples: tab = /I:\t or /I:\x09 ::: backslash = /I:\\ ::: ::: /E = Encode output delimiter literal within value as \D ::: Encode newline within value as \N ::: Encode backslash within value as \S ::: ::: /D = escape exclamation point and caret for Delayed expansion ::: ! becomes ^! ::: ^ becomes ^^ ::: ::: /L = treat all input quotes as quote Literals ::: ::: /Q:QuoteOutputFormat ::: ::: Controls output of Quotes, where QuoteOutputFormat may be any ::: one of the following: ::: ::: L = all columns quoted, quote Literals output as " (Default) ::: E = all columns quoted, quote literals Escaped as "" ::: N = No columns quoted, quote literals output as " ::: ::: The /Q:E and /Q:N options are useful for transforming data for ::: purposes other than parsing by FOR /F ::: ::: /U = Write unix style lines with newline (\n) instead of the default ::: Windows style of carriage return and linefeed (\r\n). ::: :::parseCSV /? ::: ::: Display this help ::: :::parseCSV /V ::: ::: Display the version of parseCSV.bat ::: :::parseCSV.bat was written by Dave Benham. Updates are available at the original :::posting site: http://www.dostips.com/forum/viewtopic.php?f=3&t=5702 ::: ::************ Batch portion *********** @echo off if "%~1" equ "/?" ( setlocal disableDelayedExpansion for /f "delims=: tokens=*" %%A in ('findstr "^:::" "%~f0"') do echo(%%A exit /b 0 ) if /i "%~1" equ "/V" ( for /f "delims=:" %%A in ('findstr /bc:"::%~nx0 version " "%~f0"') do echo %%A exit /b 0 ) cscript //E:JScript //nologo "%~f0" %* exit /b 0 ************ JScript portion ***********/ var args = WScript.Arguments.Named, stdin = WScript.Stdin, stdout = WScript.Stdout, escape = args.Exists("E"), literalQ = args.Exists("L"), escapeQ = (args.Item("Q")&&args.Item("Q").toUpperCase()=="E"), quoteCol = (args.Item("Q")&&args.Item("Q").toUpperCase()=="N") ? '' : '"', delayed = args.Exists("D"), inDelim = args.Item("I") ? eval('"'+args.Item("I")+'"') : ",", outDelim = args.Item("O") ? eval('"'+args.Item("O")+'"') : ",", newline = args.Exists("U") ? "\n" : "\r\n", quote = false, ln, c, n, out; while (!stdin.AtEndOfStream) { ln=stdin.ReadLine(); out=""; if (!quote) stdout.Write(quoteCol); for (n=0; n<ln.length; n++ ) { c=ln.charAt(n); if (c == '"') { if (literalQ) { if (escapeQ) c+='"'; } else if (quote && ln.charAt(n+1) == '"') { n++; if (escapeQ) c+='"'; } else { quote=!quote; continue; } } else if (c == inDelim && !quote) c=quoteCol+outDelim+quoteCol; else if (escape) { if (c == outDelim) c="\\D"; if (c == "\\") c="\\S"; } else if (delayed) { if (c == "!") c="^!"; if (c == "^") c="^^"; } out+=c; } out += (quote) ? ((escape) ? "\\N" : newline) : quoteCol+newline; stdout.Write(out); }

我能够在2.5分钟内使用parseCSV.bat转换53MB文件。

以下是parseCSV的代码。但是,我不保证将此代码保持最新。我建议您从DosTips post获取代码。在那里,您还可以找到该实用程序其他功能的良好描述。

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import javafx.application.Application;
import javafx.event.ActionEvent;
import javafx.event.EventHandler;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.layout.Pane;
import javafx.scene.shape.Line;
import javafx.stage.Stage;

/**
 *
 * @author Sedrick
 */
public class DrawTriangle extends Application {

    @Override
    public void start(Stage primaryStage) {

        //I changed the points to make processing easier. I also add 100 to each point so that the output looks better.
        String[] points = {"100.350:100.890", "300.825:300.330", "300.915:100.575", "200.385:300.690", 
                           "100.350:200.295", "400.545:100.845", "200.250:100.710", "400.140:300.060",
                           "200.250:300.150", "300.465:100.755"};

        Set<String> distinctPoints = new HashSet(Arrays.asList(points));//Keep only distinct points
        List<String> distincePointsList = new ArrayList<String>(distinctPoints); //Make the distinct points gettable


        Button btn = new Button();
        btn.setText("Press me!");


        Pane root = new Pane();
        root.getChildren().add(btn);

        btn.setOnAction(new EventHandler<ActionEvent>() {

            @Override
            public void handle(ActionEvent event) {
                for(int i = 0; i < distincePointsList.size() - 3; i++)
                {
                    for(int t = i + 1; t < distincePointsList.size() - 2; t++)
                    {
                        for(int s = t + 1; s < distincePointsList.size() - 1; s++)
                        {
                            root.getChildren().addAll(drawTriangle(distincePointsList.get(i), distincePointsList.get(t), distincePointsList.get(s))); //get the three points and return a triangle made by the lines drawn from point to point
                        }
                    }
                }
            }
        });
        Scene scene = new Scene(root, 1080, 720);

        primaryStage.setTitle("Hello World!");
        primaryStage.setScene(scene);
        primaryStage.show();
    }

    /**
     * @param args the command line arguments
     */
    public static void main(String[] args) {
        launch(args);
    }


    //Creates drawing of lines that form triangle. returns an list of the 3 lines
    static List drawTriangle(String p1, String p2, String p3)
    {
        List<Line> lineHolder = new ArrayList();
        String[] tempP1 = p1.split(":");
        String[] tempP2 = p2.split(":");
        String[] tempP3 = p3.split(":");

        Line line1 = new Line();
        line1.setStartX(Float.parseFloat(tempP1[0]));
        line1.setStartY(Float.parseFloat(tempP1[1]));
        line1.setEndX(Float.parseFloat(tempP2[0]));
        line1.setEndY(Float.parseFloat(tempP2[1]));
        lineHolder.add(line1);

        Line line2 = new Line();
        line2.setStartX(Float.parseFloat(tempP2[0]));
        line2.setStartY(Float.parseFloat(tempP2[1]));
        line2.setEndX(Float.parseFloat(tempP3[0]));
        line2.setEndY(Float.parseFloat(tempP3[1]));
        lineHolder.add(line2);

        Line line3 = new Line();
        line3.setStartX(Float.parseFloat(tempP3[0]));
        line3.setStartY(Float.parseFloat(tempP3[1]));
        line3.setEndX(Float.parseFloat(tempP1[0]));
        line3.setEndY(Float.parseFloat(tempP1[1]));
        lineHolder.add(line3);

        return lineHolder;
    }
}