什么是SSIS包部署和更改跟踪最佳实践?

时间:2009-10-15 10:05:25

标签: sql sql-server deployment ssis

我的SQL数据库Schema和Data有一个非常强大的开发环境(一切都是源代码控制,部署是自动化的等等),但是当涉及到SSIS包时,这个过程非常繁琐且完全是手动的。 / p>

  • 如何自动化SSIS包升级过程(现在,我手动将包复制到每个服务器,然后使用Management Studio的“SSIS包升级向导”)

  • 如何部署我的时间表(现在,我手动完成)

  • 如何部署我的SQL代理作业(现在,我手动完成)

  • 如何在我的TeamSystem存储库中获取我的计划和工作(现在,我没有使用源代码控制)

  • 我是否可以/应该做些什么来改进我对SSIS包的工作方式?

我正在使用MS SQL Server 2008 SP1,但我想一些建议在其他版本上也很有用。

6 个答案:

答案 0 :(得分:3)

第1点(自动部署):

可以使用dtutil.exe从命令行部署SSIS包。

以下示例将本地文件c:\test.dtsx部署到名为server1的{​​{1}}上的MSDB存储:

test_package

如果您可以计算出要使用的参数组合,它将对所有各种SSIS存储选项执行导入/导出的任意组合。

这可能与您无关,但这不会做什么(向导会这样做)是复制配置文件;你需要一个单独的批处理步骤。

在第2点和第3点(SQL Agent的源代码控制):

正如其他人所说,SQL代理作业和日程安排可以编写为T-SQL(在Management Studio中,右键单击作业并选择dtutil.exe /FILE C:\test.dtsx /dests server1 /COPY SQL;test_package ),保存在源代码管理中,并像任何其他T一样部署-SQL脚本。

我们的数据库源代码控制树设置为每个生产服务器一个文件夹,每个用户数据库包含一个子文件夹以及一个名为Script job to...的额外子文件夹,其中包含服务器级对象(如登录,链接服务器和SQL代理)的脚本工作。

(这与您的问题无关,但我们将SSIS包存储在他们自己的树中,因为它们可以应用于多个服务器)

我们不使用TFS,所以我无法评论将脚本放入其中的具体细节。

答案 1 :(得分:2)

我们所做的就是将所有内容保存在源代码管理中。我们的SSIS包保存为DTSX文件,并将它们放入我们的源代码管理中。我们使用所有内容的源代码控制版本来推广QA和Prod。我们使用我们为管理SSIS包的元数据而设置的数据库中的配置表,对SSIS包进行了所有配置。这通常包括我们所有的连接字符串以及我们可能希望在运行时设置的一些变量。我们还在此数据库中设置了所有日志记录。其中一个优点是上传正在更改的软件包变得容易得多,因为通常不需要更改配置表,所需要的只是将修改后的dtsx文件加载到它将运行的位置。即使我们需要在配置表中添加新的引用,也可以在表中插入并轻松编写升级过程脚本。

您还可以使用配置xml文件轻松更改配置(这些也可以存储在源代码管理中),但我们更喜欢在ETL数据库中包含存储元数据的所有内容。

答案 2 :(得分:1)

对于第1点,请参阅此站点 http://jessicammoss.blogspot.com/2008/05/ssis-configuration-to-configuration-to.html

前一段时间我有同样的问题,这个网站帮了很多忙。 我很好奇其他几点以及我的团队像你一样手动部署它们,但我想我会分享我所知道的。

答案 3 :(得分:1)

  1. 我不知道升级包是否可以自动化。
  2. 部署SSIS包create a Manifest file
  3. 您可以write SQL scripts to create SQL schedules and jobs然后使用某些vb.net脚本或其他内容调用此脚本。
  4. 要将SQL脚本添加到TFS,请打开SQL Server Management Studio>文件>新>项目> SQL Server脚本。将脚本添加到Queries文件夹,然后您可以将此解决方案添加到TFS。我从未尝试过这个。相反,我手动将脚本文件复制到TFS。
  5. 在我看来,创建一个完整的部署实用程序来部署SSIS包是没有意义的,因为这是一项非常罕见的活动。当我们将软件产品分发给非技术用户时,部署/安装应该很简单,以便用户友好。但是SSIS包总是由DBA或者已经知道如何使用它的人处理......

答案 4 :(得分:0)

关于脚本编写对象,您可以通过DMO,SMO,PowerShell,VBScript和任何其他方法来完成。我仍然使用我编写的古老的VB DMO脚本,该脚本基于来自多个网站的代码并通过SSIS脚本任务执行代码。设置执行包的SQL Server代理作业,并添加一个签入代码的步骤。我不熟悉TFS,但是我已经用Visual SourceSafe和Perforce完成了这个,没有任何问题。诀窍在于您只需要学习如何进行无人参与的命令行检入,即合并更改(更新,删除,创建)。下面是我古老的VB.NET脚本任务。请记住,您可以使用变量替换部分内容,使其更具动态性。

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic 2008.
' The ScriptMain is the entry point class of the script.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.IO
Imports System.Text.RegularExpressions


<System.AddIn.AddIn("ScriptMain", Version:="1.0", Publisher:="", Description:="")> _
Partial Class ScriptMain

Private Sub ScriptMain_Startup(ByVal sender As Object, ByVal e As System.EventArgs)

End Sub

Private Sub ScriptMain_Shutdown(ByVal sender As Object, ByVal e As System.EventArgs)
    Try
        ' Unlock variables from the read-only and read-write variable collection properties
        If (Dts.Variables.Count <> 0) Then
            Dts.Variables.Unlock()
        End If
    Catch ex As Exception
    End Try
End Sub

Enum ScriptResults
    Success = DTSExecResult.Success
    Failure = DTSExecResult.Failure
End Enum
Private Function CleanFileName(ByVal FileName As String) As String


    FileName = Regex.Replace(FileName, "[/\\:?=*<>]", "-")
    Return FileName

End Function







Public Sub Main()

    ''THESE MUST BE SET AND THE AGENT ACCOUNT MUST HAVE WRITE ACCESS TO THE DRIVE SELECTED
    Dim FolderName = "\\Your\Folder\Hierarchy\"
    Dim ServerName = "your-server-name"

    Dim ScriptOptions As Integer
    ScriptOptions = 1 ' Generate Drop
    ScriptOptions = ScriptOptions Or 4 ' Generate Default (Create)
    ScriptOptions = ScriptOptions Or 34 ' Generate Permissions (Database & Object)

    Dim ScriptOptions_Jobs As Integer
    ScriptOptions_Jobs = 1 ' Generate Drop
    ScriptOptions_Jobs = ScriptOptions_Jobs Or 1203765415 ' Other stuff


    Dim ScriptOptions_Tables As Integer
    ScriptOptions_Tables = 1 ' Generate Drop
    ScriptOptions_Tables = ScriptOptions_Tables Or 4 ' Generate Default (Create)
    ScriptOptions_Tables = ScriptOptions_Tables Or 34 ' Generate Permissions (Database & Object)
    ScriptOptions_Tables = ScriptOptions_Tables Or 73736 ' Generate Indexes

    Dim EXCLUDE_LIST = "master/tempdb/model/msdb"

    ''Standardize the end of the folder name to include \.
    If Right(FolderName, 1) <> "\" Then
        FolderName = FolderName & "\"
    End If

    ''Declare a folder that can be deleted.  Delete doesn't like ending with \.
    Dim FolderNameDelete = FolderName
    If Right(FolderNameDelete, 1) = "\" Then FolderNameDelete = _
        Left(FolderNameDelete, Len(FolderNameDelete) - 1)

    'Used for file system tasks.
    Dim oFSO = CreateObject("Scripting.FileSystemObject")

    'Delete if the folder exists.  If you don't delete the folder and populate the scripts,
    'then objects that were deleted from the database won't disappear and the Perforce diff
    'won't recognize that the objects need to be deleted in Perforce.
    If oFSO.FolderExists(FolderName) = True Then
        oFSO.DeleteFolder(FolderNameDelete, True)
    End If

    'Create the folder now so that we have a clean script destination.
    oFSO.CreateFolder(FolderName)
    oFSO.CreateFolder(FolderName & "Databases")
    oFSO.CreateFolder(FolderName & "Jobs")

    'connect to the server instance using trusted credentials so we dont have password stored in a file and we dont have
    'to worry about password changes breaking anything
    Dim oServer = CreateObject("SQLDMO.SQLServer")
    With oServer
        .LoginSecure = True
        .Connect(ServerName)
    End With

    'Script out SQL Server Agent Jobs.
    For Each oJob In oServer.JobServer.Jobs
        oJob.Script(ScriptOptions_Jobs, FolderName & "Jobs\" & CleanFileName(oJob.Name) & ".sql")
    Next


    'loop through all databases and excluding those in the EXCLUDE_LIST above, script out all the stored procedures.
    'You could easily change this to read from a table or to let you pass in a different list.
    For Each oDB In oServer.Databases

        If InStr(1, EXCLUDE_LIST, oDB.Name) = 0 Then

            'each db will get a folder in the main folder (FolderName) that will act as a container for the backup
            'folders we'll create each time we run this
            Dim dbFolder = FolderName & "Databases\" & oDB.Name & "\"

            oFSO.CreateFolder(dbFolder)
            oFSO.CreateFolder(dbFolder & "Stored Procedures")
            oFSO.CreateFolder(dbFolder & "Tables")
            oFSO.CreateFolder(dbFolder & "User-Defined Functions")
            oFSO.CreateFolder(dbFolder & "Views")

            'script out all the non-system procs
            For Each oProc In oDB.StoredProcedures
                If oProc.SystemObject = False Then
                    ''262150= SQLDMOScript_ObjectPermissions + SQLDMOScript_PrimaryObject + SQLDMOScript_OwnerQualify
                    oProc.Script(ScriptOptions, dbFolder & "\Stored Procedures\" & CleanFileName(oProc.Owner) & "_" & CleanFileName(oProc.Name) & ".sql")
                End If
            Next

            'script out all the non-system views
            For Each oView In oDB.Views
                If oView.SystemObject = False Then
                    ''262150= SQLDMOScript_ObjectPermissions + SQLDMOScript_PrimaryObject + SQLDMOScript_OwnerQualify
                    oView.Script(ScriptOptions, dbFolder & "\Views\" & CleanFileName(oView.Owner) & "_" & CleanFileName(oView.Name) & ".sql")
                End If
            Next

            'script out all the non-system user-defined functions
            For Each oUDF In oDB.UserDefinedFunctions
                If oUDF.SystemObject = False Then
                    ''262150= SQLDMOScript_ObjectPermissions + SQLDMOScript_PrimaryObject + SQLDMOScript_OwnerQualify
                    oUDF.Script(ScriptOptions, dbFolder & "\User-Defined Functions\" & CleanFileName(oUDF.Owner) & "." & CleanFileName(oUDF.Name) & ".sql")
                End If
            Next

            'script out all the non-system tables
            For Each oTable In oDB.Tables
                If oTable.SystemObject = False Then
                    ''262150= SQLDMOScript_ObjectPermissions + SQLDMOScript_PrimaryObject + SQLDMOScript_OwnerQualify
                    oTable.Script(ScriptOptions_Tables, dbFolder & "\Tables\" & CleanFileName(oTable.Owner) & "_" & CleanFileName(oTable.Name) & ".sql")
                End If
            Next


        End If
    Next

    'close it all out
    oServer.DisConnect()
    oServer = Nothing
    oFSO = Nothing

    Dts.TaskResult = ScriptResults.Success
End Sub

结束班

答案 5 :(得分:0)

使用Powershell脚本和 dtutil.exe 实用程序。

例如,来自maxt2posh博客的示例源代码,可能有任何错误(在匿名评论中):

#get the location of DTUTIL
$DTSPath = (get-itemproperty -path “HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\100\SSIS\Setup\DTSPath”).’(default)’
#get the names of the packages in the current windows folder
$PackageNames = Get-ChildItem -Name -Filter “*.dtsx”
#The SSIS or Filesystem folder where the packages will be installed
$InstallTargetFolder = “”
#The SSIS or Filesystem folder where the existing packages will be backed up
$BackupFolder = “MattBKUP\”

function RUNINSTALL()
{
cls
Write-Host “.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^.^”
foreach ($name in $PackageNames )
{
InstallPackage $name.TrimEnd().ToString()
Write-Host “………………………………………………………….”
}
Write-Host ” ”
Write-Host ” ”
Write-Host “……………….DONE”

}

#copy all packages in the BackupFolder to the InstallFilder and delete the Backup Folder
#get the list of files in the $BackupFolder
#move them to the $InstallTargetFolder
function RollbackPackages {
Write-Host “ROLLING BACK…………………..”
foreach ($name in $PackageNames )
{
Write-Host “ROLLING BACK PACKAGE ” + $Name
[String]$From = $BackupFolder + $Name.Replace(“.dtsx”,”")
[String]$To = “SQL;” + $InstallTargetFolder + $Name.Replace(“.dtsx”,”")
Write-Host “Restoring PACKAGE From:” + $From + ” TO:” + $To
[String]$Status = dtutil /SQL $From /MOVE $To /quiet

}
}

#Move the Package $Name From $InstallTargetFolder to $BackupFolder
function BackupPackage { param( [String]$Name, [String]$Status )
[String]$From = $InstallTargetFolder + $Name.Replace(“.dtsx”,”")
[String]$To = “SQL;” + $BackupFolder + $Name.Replace(“.dtsx”,”").ToString()
[String]$ToPath = “SQL;” + $BackupFolder
#package exists?
[String]$PackageExists = dtutil /SQL $From /Ex /quiet
if($PackageExists.Contains(“The specified package exists”) )
{
#backup folder exists?
[String]$FolderExists = dtutil /Fe $ToPath /quiet
if($FolderExists.Contains(“The specified folder does not exist”) )
{
[String]$CreatePath = “SQL;\;” + $BackupFolder.TrimEnd(‘\’)
Write-Host “Creating Backup folder:” $CreatePath
#/FC[reate] {SQL | DTS};ParentFolderPath;NewFolderName
[String]$Status = dtutil /FC $CreatePath /quiet
}
Write-Host “BACKING UP PACKAGE From:” $From ” TO:” $To
[String]$Status = dtutil /SQL $From /COPY $To /quiet
}
else
{Write-Host “Backup not required for:” $From}
}

function ValidateInstall {
param([String]$Name,[String]$Status)

if ( $LASTEXITCODE -eq 0)
{
Write-Host “Package Deployment Success ” $Name
}
else
{
if
( $LASTEXITCODE -eq 1) {Write-Host “Package Deployment Failed ” + $Name + ” …*** ERROR *** The utility failed. Error code 1 ” + $status }
elseif
($LASTEXITCODE -eq 4) {Write-Host “Package Deployment Failed ” + $Name + ” …*** ERROR *** The utility cannot locate the requested package. Error code 4 ” + $status }
elseif
($LASTEXITCODE -gt 5) {Write-Host “Package Deployment Failed ” + $Name + ” …*** ERROR *** The utility cannot load the requested package. Error code 5 ” + $status }
elseif
($LASTEXITCODE -gt 6) {Write-Host “Package Deployment Failed ” + $Name + ” …*** ERROR *** The utility cannot resolve the command line because it contains either syntactic or semantic errors. Error code 6 ” + $status }
else
{“Package Deployment Failed ” + $Name + ” …*** ERROR *** Unidentified Error. ” + $status }

RollbackPackages
Write-Host ” ”
Write-Host “ROLLBACK COMPLETE”
Write-Host ” ”
Read-Host “Press enter to continue …”

Exit
}
}

function InstallPackage { param( [String]$Name, [String]$Status )
backuppackage $Name $Status
[String]$To = “SQL;” + $InstallTargetFolder + $Name.Replace(“.dtsx”,”")
$Status = dtutil /FILE $Name /COPY $To /quiet
ValidateInstall $Name $Status
}

参考文献:

Script by Chad Miller

http://maxt2posh.wordpress.com/2010/04/23/deploying-ssis-packages-using-dtutil-exe-with-powershell-part-1%E2%80%A6/

http://billfellows.blogspot.com.es/2010/05/powershell-dtutil-ssisdeploymanifest.html

dtutil Utility

Sql PSX

http://www.katieandemil.com/powershell-dtutil