压缩距离矩阵如何工作? (pdist)

时间:2012-10-26 01:21:09

标签: python numpy scipy

scipy.spatial.distance.pdist返回精简距离矩阵。来自the documentation

  

返回一个压缩距离矩阵Y.对于每个和(where),度量dist(u = X [i],v = X [j])被计算并存储在条目ij中。

我认为ij意味着i*j。但我想我可能错了。考虑

X = array([[1,2], [1,2], [3,4]])
dist_matrix = pdist(X)

然后文档说明dist(X[0], X[2])应为dist_matrix[0*2]。但是,dist_matrix[0*2]为0 - 不应该是2.8。

在给定ij的情况下,我应使用什么公式来访问两个向量的相似性?

7 个答案:

答案 0 :(得分:76)

你可以这样看:假设x是m乘以n。一次选择两个m行的itertools.combinations(range(m), 2)行可能是m=3,例如>>> import itertools >>> list(combinations(range(3),2)) [(0, 1), (0, 2), (1, 2)]

d = pdist(x)

因此,如果k,则combinations(range(m), 2))中的x元组会提供与d[k]相关联的>>> x = array([[0,10],[10,10],[20,20]]) >>> pdist(x) array([ 10. , 22.36067977, 14.14213562]) 行的索引。

示例:

dist(x[0], x[1])

第一个元素是dist(x[0], x[2]),第二个元素是dist(x[1], x[2]),第三个元素是>>> squareform(pdist(x)) array([[ 0. , 10. , 22.361], [ 10. , 0. , 14.142], [ 22.361, 14.142, 0. ]]) >>> y = array([[0,10],[10,10],[20,20],[10,0]]) >>> squareform(pdist(y)) array([[ 0. , 10. , 22.361, 14.142], [ 10. , 0. , 14.142, 10. ], [ 22.361, 14.142, 0. , 22.361], [ 14.142, 10. , 22.361, 0. ]]) >>> pdist(y) array([ 10. , 22.361, 14.142, 14.142, 10. , 22.361])

或者您可以将其视为方形距离矩阵的上三角形部分中的元素,将它们串联成一维数组。

E.g。

{{1}}

答案 1 :(得分:27)

压缩距离矩阵到全距离矩阵

pdist返回的压缩距离矩阵可以使用scipy.spatial.distance.squareform转换为全距离矩阵:

>>> import numpy as np
>>> from scipy.spatial.distance import pdist, squareform
>>> points = np.array([[0,1],[1,1],[3,5], [15, 5]])
>>> dist_condensed = pdist(points)
>>> dist_condensed
array([  1.        ,   5.        ,  15.5241747 ,   4.47213595,
        14.56021978,  12.        ])

使用squareform转换为完整矩阵:

>>> dist = squareform(dist_condensed)
array([[  0.        ,   1.        ,   5.        ,  15.5241747 ],
       [  1.        ,   0.        ,   4.47213595,  14.56021978],
       [  5.        ,   4.47213595,   0.        ,  12.        ],
       [ 15.5241747 ,  14.56021978,  12.        ,   0.        ]])

点i,j之间的距离存储在dist [i,j]:

>>> dist[2, 0]
5.0
>>> np.linalg.norm(points[2] - points[0])
5.0

指数缩减指数

可以将用于访问方阵元素的索引转换为压缩矩阵中的索引:

def square_to_condensed(i, j, n):
    assert i != j, "no diagonal elements in condensed matrix"
    if i < j:
        i, j = j, i
    return n*j - j*(j+1)/2 + i - 1 - j

示例:

>>> square_to_condensed(1, 2, len(points))
3
>>> dist_condensed[3]
4.4721359549995796
>>> dist[1,2]
4.4721359549995796

索引的压缩索引

如果没有sqaureform,另一个方向是可能的,这在运行时和内存消耗方面更好:

import math

def calc_row_idx(k, n):
    return int(math.ceil((1/2.) * (- (-8*k + 4 *n**2 -4*n - 7)**0.5 + 2*n -1) - 1))

def elem_in_i_rows(i, n):
    return i * (n - 1 - i) + (i*(i + 1))/2

def calc_col_idx(k, i, n):
    return int(n - elem_in_i_rows(i + 1, n) + k)

def condensed_to_square(k, n):
    i = calc_row_idx(k, n)
    j = calc_col_idx(k, i, n)
    return i, j

示例:

>>> condensed_to_square(3, 4)
(1.0, 2.0)

与方形

的运行时比较
>>> import numpy as np
>>> from scipy.spatial.distance import pdist, squareform
>>> points = np.random.random((10**4,3))
>>> %timeit dist_condensed = pdist(points)
1 loops, best of 3: 555 ms per loop

创建sqaureform结果非常慢:

>>> dist_condensed = pdist(points)
>>> %timeit dist = squareform(dist_condensed)
1 loops, best of 3: 2.25 s per loop

如果我们搜索具有最大距离的两个点,那么在完整矩阵中搜索是O(n)并且在浓缩形式中仅搜索O(n / 2)就不足为奇了:

>>> dist = squareform(dist_condensed)
>>> %timeit dist_condensed.argmax()
10 loops, best of 3: 45.2 ms per loop
>>> %timeit dist.argmax()
10 loops, best of 3: 93.3 ms per loop

在两种情况下获取两点的内容几乎没有时间,但当然计算压缩指数有一些开销:

>>> idx_flat = dist.argmax()
>>> idx_condensed = dist.argmax()
>>> %timeit list(np.unravel_index(idx_flat, dist.shape))
100000 loops, best of 3: 2.28 µs per loop
>>> %timeit condensed_to_square(idx_condensed, len(points))
100000 loops, best of 3: 14.7 µs per loop

答案 2 :(得分:18)

压缩矩阵的矢量对应于方阵的底部三角形区域。要转换该三角形区域中的点,您需要计算三角形中左侧的点数,以及列中上方的数字。

您可以使用以下功能进行转换:

q = lambda i,j,n: n*j - j*(j+1)/2 + i - 1 - j

检查:

import numpy as np
from scipy.spatial.distance import pdist, squareform
x = np.random.uniform( size = 100 ).reshape( ( 50, 2 ) )
d = pdist( x )
ds = squareform( d )
for i in xrange( 1, 50 ):
    for j in xrange( i ):
        assert ds[ i, j ] == d[ q( i, j, 50 ) ]

答案 3 :(得分:4)

如果要访问与方形距离矩阵的第(i,j)个元素对应的pdist元素,则数学如下:假设i < j(否则翻转索引)如果i == j,答案是0.

X = random((N,m))
dist_matrix = pdist(X)

然后第(i,j)个元素是dist_matrix [ind] where

ind = (N - array(range(1,i+1))).sum()  + (j - 1 - i).

答案 4 :(得分:4)

这是上三角版本( i&lt; j ),这对某些人来说一定很有趣:

condensed_idx = lambda i,j,n: i*n + j - i*(i+1)/2 - i - 1

这很容易理解:

  1. 使用i*n + j,您将转到方形矩阵中的位置;
  2. 使用- i*(i+1)/2删除i;
  3. 之前的所有行中的下三角(包括对角线)
  4. 使用- i删除对角线之前的第i行中的位置;
  5. 使用- 1删除对角线上第i行的位置。
  6. 检查:

    import scipy
    from scipy.spatial.distance import pdist, squareform
    condensed_idx = lambda i,j,n: i*n + j - i*(i+1)/2 - i - 1
    n = 50
    dim = 2
    x = scipy.random.uniform(size = n*dim).reshape((n, dim))
    d = pdist(x)
    ds = squareform(d)
    for i in xrange(1, n-1):
        for j in xrange(i+1, n):
            assert ds[i, j] == d[condensed_idx(i, j, n)]
    

答案 5 :(得分:1)

如果有人正在寻找逆变换(即给定元素索引idx,找出哪个(i, j)元素对应),这是一个合理的向量解决方案:

def actual_indices(idx, n):
    n_row_elems = np.cumsum(np.arange(1, n)[::-1])
    ii = (n_row_elems[:, None] - 1 < idx[None, :]).sum(axis=0)
    shifts = np.concatenate([[0], n_row_elems])
    jj = np.arange(1, n)[ii] + idx - shifts[ii]
    return ii, jj

n = 5
k = 10
idx = np.random.randint(0, n, k)
a = pdist(np.random.rand(n, n))
b = squareform(a)

ii, jj = actual_indices(idx, n)]
assert np.allclose(b[ii, jj, a[idx])

我用它来计算矩阵中最近行的索引。

m = 3  # how many closest
lowest_dist_idx = np.argpartition(-a, -m)[-m:]
ii, jj = actual_indices(lowest_dist_idx, n)  # rows ii[0] and jj[0] are closest

答案 6 :(得分:0)

我有同样的问题。而且我发现使用numpy.triu_indices更简单:

import numpy as np
from scipy.spatial.distance import pdist, squareform
N = 10

# Calculate distances
X = np.random.random((N,3))
dist_condensed = pdist(X)

# Get indexes: matrix indices of dist_condensed[i] are [a[i],b[i]]
a,b = np.triu_indices(N,k=1)

# Fill distance matrix
dist_matrix = np.zeros((N,N))
for i in range(len(dist_condensed)):
    dist_matrix[a[i],b[i]] = dist_condensed[i]
    dist_matrix[b[i],a[i]] = dist_condensed[i]

# Compare with squareform output
np.all(dist_matrix == squareform(distances))