使用TensorFlow构建简单的图像搜索

Question

我需要使用TensorFlow在我的应用中实现简单的图像搜索。 要求如下：

1. 数据集包含大约一百万张图片，大小相同，每张图片都包含一个唯一对象，仅该对象。
2. 搜索参数是使用可能位于数据集中的某个对象的手机摄像头拍摄的图像。

我设法从相机图片中提取图像并将其拉直成矩形形状，因此，像TinEye这样的反向搜索图像索引器能够找到匹配。

现在我想通过使用TensorFlow来创建基于我的数据集的模型（使每个图像的文件名成为唯一索引）来重现该索引器。

有人能指点教程/代码解释如何在不过多地深入计算机视觉术语的情况下实现这样的事情吗？

非常感谢！

Answer 1

The Wikipedia article on TinEye says that Perceptual Hashing will yield results similar to TinEye's. They reference this detailed description of the algorithm. But TinEye refuses to comment.

---

The biggest issue with the Perceptual Hashing approach is that while it's efficient for identifying the same image (subject to skews, contrast changes, etc.), it's not great at identifying a completely different image of the same object (e.g. the front of a car vs. the side of a car).

TensorFlow has great support for deep neural nets which might give you better results. Here's a high level description of how you might use a deep neural net in TensorFlow to solve this problem:

Start with a pre-trained NN (such as GoogLeNet) or train one yourself on a dataset like ImageNet. Now we're given a new picture we're trying to identify. Feed that into the NN. Look at the activations of a fairly deep layer in the NN. This vector of activations is like a 'fingerprint' for the image. Find the picture in your database with the closest fingerprint. If it's sufficiently close, it's probably the same object.

The intuition behind this approach is that unlike Perceptual Hashing, the NN is building up a high-level representation of the image including identifying edges, shapes, and important colors. For example, the fingerprint of an apple might include information about its circular shape, red color, and even its small stem.

---

You could also try something like this 2012 paper on image retrieval which uses a slew of hand-picked features such as SIFT, regional color moments and object contour fragments. This is probably a lot more work and it's not what TensorFlow is best at.

---

UPDATE

OP has provided an example pair of images from his application:

Here are the results of using the demo on the pHash.org website on that pair of similar images as well as on a pair of completely dissimilar images.

Comparing the two images provided by the OP:

RADISH (radial hash): pHash determined your images are not similar with PCC = 0.518013

DCT hash: pHash determined your images are not similar with hamming distance = 32.000000.

Marr/Mexican hat wavelet: pHash determined your images are not similar with normalized hamming distance = 0.480903.

Comparing one of his images with a random image from my machine:

RADISH (radial hash): pHash determined your images are not similar with PCC = 0.690619.

DCT hash: pHash determined your images are not similar with hamming distance = 27.000000.

Marr/Mexican hat wavelet: pHash determined your images are not similar with normalized hamming distance = 0.519097.

Conclusion

We'll have to test more images to really know. But so far pHash does not seem to be doing very well. With the default thresholds it doesn't consider the similar images to be similar. And for one algorithm, it actually considers a completely random image to be more similar.

Answer 2

https://github.com/wuzhenyusjtu/VisualSearchServer 它是使用TensorFlow和InceptionV3模型进行类似图像搜索的简单实现。该代码实现了两种方法，一种是处理图像搜索的服务器，另一种是基于提取的pool3特性进行最近邻匹配的简单索引器。