Speed up Compare-Object for large directory compares

Question

Currently I'm using Compare-Object to compare the contents of two directories.

The directories contain a mix of subfolders, PDF files and some textfiles. The folders can contain upwards of 20,000+ files in 50,000+ folders, 1-2GB all told.

Compare-Object completes in about 4m30s for a compare of that size. I tried using this hashtable method to speed it up, but it produces a lot of false positives due to having some duplicate folder names in there (I think?).

Are there any other options to speed this thing up?

Param(
    [Parameter(Mandatory=$true)][string]$dir1,
    [Parameter(Mandatory=$true)][string]$dir2
)

$sw = [Diagnostics.Stopwatch]::StartNew()

# Directories
$sourcedir = Get-ChildItem -Path $dir1 -Recurse
$destinationdir = Get-ChildItem -Path $dir2 -Recurse

# Differences
$differences = Compare-Object -ReferenceObject $sourcedir -DifferenceObject $destinationdir
$differences

$output = $differences | 
          select @{l='File/Folder';e={$_.InputObject}},
              @{l='Indicator';e={$_.SideIndicator}},
              @{l='Path';e={$_.InputObject.FullName}}
$output

$sw.Stop()

"Found " + $differences.Count + " missing objects in " + $sw.Elapsed