从两个嵌套为python中字典值的列表中检索数据的最有效方法

时间:2020-04-12 10:27:48

标签: python list dictionary coordinates euclidean-distance

我有一个三维空间结构坐标字典,

    struc_dict = { 
    'struc1' : [np.array(x,y,z), np.array(x,y,z), np.array(x,y,z), ...],
    'struc2' : [np.array(x,y,z), np.array(x,y,z), np.array(x,y,z), ...], 
    'struc3' : [np.array(x,y,z), np.array(x,y,z), np.array(x,y,z), ...], 
    'struc4' : [np.array(x,y,z), np.array(x,y,z), np.array(x,y,z), ...] } 

例如:

struc_dict = { 
    'struc1' : [[-31.447,  -4.428, -28.285], [-32.558,  -2.108, -29.213], [-31.656,  -4.071, -30.89 ], [-33.899,  -4.504, -29.349]],
    'struc2' : [[-27.487, -15.05,  -31.418], [-29.178, -14.63,  -33.498], [-29.548, -16.754, -31.937], [-30.028, -14.278, -30.977]], 
    'struc3' : [[-16.07,   -2.042, -29.853], [-16.734,  -4.162, -29.905], [-16.279,  -4.438, -28.936], [-16.544,  -4.098, -31.514]]} 

我想找出每个结构之间的最短距离。 因此,我想浏览一下字典,获取一对值并计算最短距离。

我当前的代码看起来像那样,但是它不是很漂亮或没有效率:

import numpy as np
for s1 in struc_dict.keys():
    for s2 in struc_dict.keys():
        # only consider distances between two structures
        if s1 == s2:
            continue
        else:
            # defining an arbitrary max value, necessary for the first comparison?
            min_dist = 10000
            for c1 in struc_dict[s1]:
                for c2 in struc_dict[s2]:
                    # calculates the distance between the two coordinates
                    if np.linalg.norm(np.array(c1)-np.array(c2)) <= min_dist:
                        min_dist = np.linalg.norm(np.array(c1)-np.array(c2))

            print("Min dist between {s1} & {s2} : {min:.3f} units".format(s1=s1, s2=s2, min=min_dist))

示例的输出:

Min dist between struc1 & struc2 : 10.309 units
Min dist between struc1 & struc3 : 14.804 units
Min dist between struc2 & struc1 : 10.309 units
Min dist between struc2 & struc3 : 15.377 units
Min dist between struc3 & struc1 : 14.804 units
Min dist between struc3 & struc2 : 15.377 units

此代码有效,但是两次计算两个结构之间的距离,因为它必须两次通过字典。另外,对于两个结构的第一次比较,我需要一个较大的min_dist起始值,但是有办法解决吗?

通常,必须有一个更优雅的解决方案。谢谢!

2 个答案:

答案 0 :(得分:1)

对于更多优雅解决方案,请考虑itertools.product。考虑下面的简单示例:

import itertools
points = {'A': (1,1), 'B': (2,2), 'C': (3,3)}
def dist(a, b):
    return ((a[0]-b[0])**2+(a[1]-b[1])**2)**0.5
for p1, p2 in itertools.product(points.keys(), repeat=2):
    print('Distance between',p1,'and',p2,'is',dist(points[p1],points[p2]))

输出:

Distance between A and A is 0.0
Distance between A and B is 1.4142135623730951
Distance between A and C is 2.8284271247461903
Distance between B and A is 1.4142135623730951
Distance between B and B is 0.0
Distance between B and C is 1.4142135623730951
Distance between C and A is 2.8284271247461903
Distance between C and B is 1.4142135623730951
Distance between C and C is 0.0

for中的for相比,这可以避免一个嵌套级别。

答案 1 :(得分:0)

从您的第一篇文章开始,尚不清楚各个结构的坐标数是否相同,因此我认为不是。

这是您朴素方法的略微修订版​​,并且是利用NumPy的快速低级矢量化技术进行的第一个改进版本。

import numpy as np

def naive(data):
  res = np.inf
  for k1, v1 in data.items():
    for k2, v2 in data.items():
      if k1 == k2:
        continue
      for c1 in v1:
        for c2 in v2:
          res = np.minimum(res, np.sum((c1 - c2)**2))
  return np.sqrt(res)

def version1(data):
  res = np.inf
  for k1, v1 in data.items():
    for k2, v2 in data.items():
      if k1 == k2:
        continue
      res = np.minimum(res, np.min(np.sum((v1[None, ...] - v2[:, None, :])**2, axis=-1)))
  return np.sqrt(res)

关键点是v1[None, ...] - v2[:, None, :],其中,通过在不同位置的每个结构上添加一条附加轴,我们利用NumPy广播删除了两个内部循环。

对数据进行测试(需要使用IPython,仅需使用timeit的简化接口即可):

struc_dict = { 
    'struc1' : [[-31.447,  -4.428, -28.285], [-32.558,  -2.108, -29.213], [-31.656,  -4.071, -30.89 ], [-33.899,  -4.504, -29.349]],
    'struc2' : [[-27.487, -15.05,  -31.418], [-29.178, -14.63,  -33.498], [-29.548, -16.754, -31.937], [-30.028, -14.278, -30.977]], 
    'struc3' : [[-16.07,   -2.042, -29.853], [-16.734,  -4.162, -29.905], [-16.279,  -4.438, -28.936], [-16.544,  -4.098, -31.514]]}
data = {k: np.array(v) for k,v in struc_dict.items()}
%timeit naive(data)
%timeit version1(data)

输出:

1000 loops, best of 3: 433 µs per loop
10000 loops, best of 3: 55.7 µs per loop

为了更好地评估性能,让我们尝试使用更多数据:

np.random.seed(42)

for n in [10, 20, 50, 100]:
  for max_size in [10, 20, 50, 100]:
    data = {str(i): np.random.normal(size=[np.random.randint(1, max_size), 3])
            for i in range(n)}
    print("Measuring for n=%r and max_size=%r" % (n, max_size))
    %timeit naive(data)
    %timeit version1(data)

输出:

Measuring for n=10 and max_size=10
100 loops, best of 3: 10.6 ms per loop
1000 loops, best of 3: 784 µs per loop
Measuring for n=10 and max_size=20
10 loops, best of 3: 32.5 ms per loop
1000 loops, best of 3: 894 µs per loop
Measuring for n=10 and max_size=50
1 loop, best of 3: 323 ms per loop
1000 loops, best of 3: 1.94 ms per loop
Measuring for n=10 and max_size=100
1 loop, best of 3: 478 ms per loop
100 loops, best of 3: 2.36 ms per loop
Measuring for n=20 and max_size=10
10 loops, best of 3: 43.3 ms per loop
100 loops, best of 3: 3.32 ms per loop
Measuring for n=20 and max_size=20
10 loops, best of 3: 188 ms per loop
100 loops, best of 3: 3.78 ms per loop
Measuring for n=20 and max_size=50
1 loop, best of 3: 1.41 s per loop
100 loops, best of 3: 8.09 ms per loop
Measuring for n=20 and max_size=100
1 loop, best of 3: 4.34 s per loop
100 loops, best of 3: 17.8 ms per loop
Measuring for n=50 and max_size=10
1 loop, best of 3: 341 ms per loop
10 loops, best of 3: 22.6 ms per loop
Measuring for n=50 and max_size=20
1 loop, best of 3: 1.24 s per loop
10 loops, best of 3: 24.3 ms per loop
Measuring for n=50 and max_size=50
1 loop, best of 3: 7.86 s per loop
10 loops, best of 3: 45.9 ms per loop
Measuring for n=50 and max_size=100
1 loop, best of 3: 22.6 s per loop
10 loops, best of 3: 97.1 ms per loop
Measuring for n=100 and max_size=10
1 loop, best of 3: 1.03 s per loop
10 loops, best of 3: 83.5 ms per loop
Measuring for n=100 and max_size=20
1 loop, best of 3: 4 s per loop
10 loops, best of 3: 96.1 ms per loop
Measuring for n=100 and max_size=50
1 loop, best of 3: 27.4 s per loop
10 loops, best of 3: 180 ms per loop
Measuring for n=100 and max_size=100
1 loop, best of 3: 1min 50s per loop
1 loop, best of 3: 447 ms per loop

加速的功能取决于坐标数量(只要您的计算机有足够的可用RAM,就越好)。

进一步的改进可以通过以下方式实现:

  1. 使用np.einsum(爱因斯坦求和)代替np.sum,这是众所周知的速度更快
  2. 如果所有结构都具有相同数量的坐标,则可以通过构建两个四维数组来删除其他两个循环(例如Nx1xMx3和1xNxMx3,其中N是结构数量,M是每个结构的坐标数量)
  3. 结合以上两点!