从Excel VBA到Web服务的有效数据传输

时间:2018-09-24 14:13:24

标签: excel vba excel-vba large-data

我有一个很大的工作表(约25万行,22列,约40MB普通数据),必须将其内容传输到Intranet API。格式无所谓。问题是:访问

之类的数据时
Const ROWS = 250000
Const COLS = 22

Dim x As Long, y As Long
Dim myRange As Variant
Dim dummyString As String
Dim sb As New cStringBuilder

myRange = Range(Cells(1, 1), Cells(ROWS, COLS)).Value2

For x = 1 To ROWS
    For y = 1 To COLS
        dummyString = myRange(x, y) 'Runtime with only this line: 1.8s
        sb.Append dummyString 'Runtime with this additional line 163s
    Next
Next

我得到了一个很棒的2D数组,但是我无法有效地收集数据以进行HTTP导出。 在数组上进行X / Y循环并访问myRange[x, y]的运行时间大于1分钟。我找不到能够帮助获取2D数组的内爆/编码内容的数组方法。 我当前的解决方法是误用剪贴板(Workaround for Memory Leak when using large string),该方法运行速度很快,但是在我看来却是一个肮脏的解决方法,并且存在一个主要问题:我得到的值的格式为“ .Value”而不是“ .Value2”,所以我必须在使用之前再次转换服务器站点上的数据,例如取消格式化货币单元格使其浮动。

处理数据数组的另一个想法是什么?

1 个答案:

答案 0 :(得分:1)

我的想法是,您将创建两个字符串数组ABA的大小为1 to ROWSB的大小为1 to COLUMNS。当您遍历myRange数组中的每一行时,用该行中每一列的值填充B中的每个元素。在该行的最后一列之后,然后移至下一行,请加入数组B并分配给A中的行。对于这种大小的循环,仅将必要的内容放入循环本身中。最后,您将加入A。将项目分配给cstr()时,可能需要使用B

Matschek(OP)能够根据上述内容编写代码,但是对于其他人来说,代码本身可能像这样:

Option Explicit

Private Sub concatenateArrayValues()

    Const TOTAL_ROWS As Long = 250000
    Const TOTAL_COLUMNS As Long = 22

    Dim inputValues As Variant
    inputValues = ThisWorkbook.Worksheets("Sheet1").Range("A1").Resize(TOTAL_ROWS, TOTAL_COLUMNS).Value2

    ' These are static string arrays, as OP's use case involved constants.
    Dim outputArray(1 To TOTAL_ROWS) As String ' <- in other words, array A
    Dim interimArray(1 To TOTAL_COLUMNS) As String ' <- in other words, array B

    Dim rowIndex As Long
    Dim columnIndex As Long

    ' We use constants below when specifying the loop's limits instead of Lbound() and Ubound()
    ' as OP's use case involved constants.
    ' If we were using dynamic arrays, we could call Ubound(inputValues,2) once outside of the loop
    ' And assign the result to a Long type variable
    ' To avoid calling Ubound() 250k times within the loop itself.

    For rowIndex = 1 To TOTAL_ROWS
        For columnIndex = 1 To TOTAL_COLUMNS
            interimArray(columnIndex) = inputValues(rowIndex, columnIndex)
        Next columnIndex
        outputArray(rowIndex) = VBA.Strings.Join(interimArray, ",")
    Next rowIndex

    Dim concatenatedOutput As String
    concatenatedOutput = VBA.Strings.Join(outputArray, vbNewLine)

    Debug.Print concatenatedOutput

    ' My current machine isn't particularly great
    ' but the code above ran and concatenated values in range A1:V250000
    ' (with each cell containing a random 3-character string) in under 4 seconds.

End Sub