Question

我有一个Powershell脚本，可以在本地计算机上正常工作。我的脚本通过从blob storage中进行选择将数据上传到Azure sql data warehouse。

当我上传小桌子时，它工作正常。但是，当我尝试从大表上载时，它会重新启动脚本，而不会出现任何错误。

所以我真的不知道为什么它会在某些时候重新开始。对我来说，它闻起来有点超时或类似的味道。但是我找不到我们的位置。而且没有错误消息。

try
{

   #Remove files to cleanup
   $deletefilelist = Get-AzStorageBlob -Container "XXXXXXX" -Context $context

    foreach ($listfiles in $deletefilelist)  
    {
      $removefile = $listfiles.Name
      if ($removefile -ne $null)
      {
        Write-Output "removeing file $removefile"

        Remove-AzStorageBlob -Blob $removefile -Container "XXXXXXX"-Context $context -Force
      }
    }

  #SQL connection settings
  $SqlConnection = New-Object System.Data.SqlClient.SqlConnection
  $SqlConnection.ConnectionString = "Server = $SQLServer; Database = $SQLDBName; Integrated Security = False; User ID = $uid; Password = $pwd;"
  $SqlConnection.Open()

  #MetaData CMDs
  $SqlCmdMeta = New-Object System.Data.SqlClient.SqlCommand
  $SqlAdapterMeta = New-Object System.Data.SqlClient.SqlDataAdapter

  #Datasets
  $DataSetMeta = New-Object System.Data.DataSet
  $DataSet = New-Object System.Data.DataSet

  #Getting metadata from table in Az DWH
  $SqlMetaQuery = "Select FullTableName,TableName,Category,MapSourceSystem,SQLStatement from XXXXXXX"

  $SqlCmdMeta.Connection = $SqlConnection
  $SqlCmdMeta.CommandTimeout = 0
  $SqlCmdMeta.CommandText = $SqlMetaQuery

  $SqlAdapterMeta.SelectCommand = $SqlCmdMeta

  $SqlAdapterMeta.Fill($DataSetMeta)
  $metadataoutput = $DataSetMeta.Tables[0]  

  Write-Output "Preparing settings and upload to blob.."
  #Build the blob file name and sql
  $Todaydate = Get-Date -Format "yyyyMMddHHmmss"
  foreach ($row in $metadataoutput) {
    $category = $row.Category
    $mapsourcesystem = $row.MapSourceSystem

    $csvfileName = $category + "_" + $row.TableName + "_$mapsourcesystem" + "_$Todaydate"


    #Check if it is a fact table. If it is a fact table then the SQLStatement should have an attached mapsourcesystem to the where clause
    $SqlFilename = $row.TableName
    if ($category -eq "Fact") {
      $SqlQuery = $row.SQLStatement + " '$mapsourcesystem'"
    }
    else {
      $SqlQuery = $row.SQLStatement 
    }
    $SqlCmd.CommandTimeout = 0      
    $SqlCmd.Connection = $SqlConnection
    $SqlCmd.CommandText = $SqlQuery


    $SqlAdapter.SelectCommand = $SqlCmd
    $SqlAdapter.Fill($DataSet)
    Write-Output "Rows Detected"

    #Check if table is empty
    if ($DataSet.Tables[0].Rows.Count -eq 0) {
      Write-Output $csvfileName" does not contain any rows in SQL Database and therefore is a CSV file not created"
    }

    else {
      Write-Host "Building and uploading blob file to storage.."
      #Create the blob file and upload it to blob storage
      $file = New-TemporaryFile
      $DataSet.Tables[0] | Export-Csv -Path $file -Delimiter ";" -Encoding UTF8 -NoTypeInformation 
      Set-AzStorageBlobContent -File $file -Container "XXXXXXX" -Context $context -ClientTimeoutPerRequest -1 -ServerTimeoutPerRequest -1 -Force -blob $csvfileName".csv"

      Write-Output $ErrorActionPreference
      Write-Output $csvfileName " Has been uploaded to the Blob storage with success!"

      Write-Output "Deleting $csvfileName in temporary enviorment"
      #Remove-Item -Path $Env:Temp -Recurse -Force -ErrorAction SilentlyContinue
      $DataSet.Reset()
    }

  }
}

Answer 1

我认为这与fair share的限制有关。有关更多信息，请参见this堆栈溢出线程中的Joe的答案。

希望这会有所帮助！

Azure Runbook重新启动脚本没有错误

1 个答案: