Question

在我的应用程序中，有一个用例来查找与其他某个地理点最接近的点。我决定使用内存空间索引，并找到了几个候选对象：jeospatial和Lucene spatial。

我制定了一些基准测试，并惊讶地发现Lucene指数非常慢。这是使用JMH完成的基准测试代码。完整的源代码可以在我的GitHub repository中找到。

@State(Scope.Thread)
public class MyBenchmark {

    // Lucene
    private static final String COORDINATES_FIELD = "coordinates";
    private static final int GEO_PRECISION_LEVEL = 5;
    private static final double NEARBY_RADIUS_DEGREE = DistanceUtils.dist2Degrees(
            50, DistanceUtils.EARTH_MEAN_RADIUS_KM);

    private final Directory directory = new RAMDirectory();
    private final IndexWriterConfig iwConfig = new IndexWriterConfig();
    private IndexWriter indexWriter = null;
    private IndexSearcher indexSearcher = null;
    private final SpatialContext spatialCxt = SpatialContext.GEO;
    private final ShapeFactory shapeFactory = spatialCxt.getShapeFactory();
    private final SpatialStrategy coordinatesStrategy = new RecursivePrefixTreeStrategy(
            new GeohashPrefixTree(spatialCxt, GEO_PRECISION_LEVEL),
            COORDINATES_FIELD);

    // Jeospatial
    private VPTree<SimpleGeospatialPoint> jeospatialPoints = new VPTree<>();

    public MyBenchmark() {
        try {
            indexWriter = new IndexWriter(directory, iwConfig);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Setup
    public void init() throws IOException {
        var r = new Random();
        for (int i = 0; i < 3000; i++) {
            double latitude = ThreadLocalRandom.current().nextDouble(50.4D, 51.4D);
            double longitude = ThreadLocalRandom.current().nextDouble(8.2D, 11.2D);

            Document doc = new Document();
            doc.add(new StoredField("id", r.nextInt()));
            var point = shapeFactory.pointXY(longitude, latitude);
            for (var field : coordinatesStrategy.createIndexableFields(point)) {
                doc.add(field);
            }
            doc.add(new StoredField(coordinatesStrategy.getFieldName(), latitude + ":" + longitude));
            indexWriter.addDocument(doc);

            jeospatialPoints.add(new MyGeospatialPoint(latitude, longitude));
        }
        indexWriter.forceMerge(1);
        indexWriter.close();
        final IndexReader indexReader = DirectoryReader.open(directory);
        indexSearcher = new IndexSearcher(indexReader);
    }

    private SimpleGeospatialPoint createRandomPoint() {
        final double latitude = ThreadLocalRandom.current().nextDouble(50.4D, 51.4D);
        final double longitude = ThreadLocalRandom.current().nextDouble(8.2D, 11.2D);
        return new MyGeospatialPoint(latitude, longitude);
    }

    @Benchmark
    @BenchmarkMode(Mode.Throughput)
    @OutputTimeUnit(TimeUnit.SECONDS)
    @Fork(value = 1)
    @Warmup(iterations = 0)
    @Measurement(iterations = 3)
    public void benchLucene() {
        double latitude = ThreadLocalRandom.current().nextDouble(50.4D, 51.4D);
        double longitude = ThreadLocalRandom.current().nextDouble(8.2D, 11.2D);
        final var spatialArgs = new SpatialArgs(SpatialOperation.IsWithin,
                                                shapeFactory.circle(longitude, latitude, NEARBY_RADIUS_DEGREE));
        final Query q = coordinatesStrategy.makeQuery(spatialArgs);
        try {
            final TopDocs topDocs = indexSearcher.search(q, 1);
            if (topDocs.totalHits == 0) {
                return;
            }
            var doc = indexSearcher.doc(topDocs.scoreDocs[0].doc);
            var coordinates = doc.getField(COORDINATES_FIELD).stringValue();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Benchmark
    @BenchmarkMode(Mode.Throughput)
    @OutputTimeUnit(TimeUnit.SECONDS)
    @Fork(value = 1)
    @Warmup(iterations = 0)
    @Measurement(iterations = 3)
    public void benchJeospatial() {
        var neighbor = jeospatialPoints.getNearestNeighbor(createRandomPoint(), 50 * 1000);
        var n = neighbor.getLatitude();
    }
}

在Lucene中，我正在使用RAMDirectory，但也尝试过MMapDirectory。几乎没有区别。

基准测试结果

# JMH version: 1.21
# VM version: JDK 10, Java HotSpot(TM) 64-Bit Server VM, 10+46
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-10.jdk/Contents/Home/bin/java
# VM options: <none>
# Warmup: <none>
# Measurement: 3 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.sample.MyBenchmark.benchJeospatial

# Run progress: 0,00% complete, ETA 00:01:00
# Fork: 1 of 1
Iteration   1: 77528,657 ops/s
Iteration   2: 81921,096 ops/s
Iteration   3: 83470,405 ops/s


Result "org.sample.MyBenchmark.benchJeospatial":
  80973,386 ±(99.9%) 56230,060 ops/s [Average]
  (min, avg, max) = (77528,657, 80973,386, 83470,405), stdev = 3082,159
  CI (99.9%): [24743,326, 137203,446] (assumes normal distribution)


# JMH version: 1.21
# VM version: JDK 10, Java HotSpot(TM) 64-Bit Server VM, 10+46
# VM invoker: /Library/Java/JavaVirtualMachines/jdk-10.jdk/Contents/Home/bin/java
# VM options: <none>
# Warmup: <none>
# Measurement: 3 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.sample.MyBenchmark.benchLucene

# Run progress: 50,00% complete, ETA 00:00:31
# Fork: 1 of 1
Iteration   1: 997,103 ops/s
Iteration   2: 1087,487 ops/s
Iteration   3: 1077,964 ops/s


Result "org.sample.MyBenchmark.benchLucene":
  1054,184 ±(99.9%) 906,037 ops/s [Average]
  (min, avg, max) = (997,103, 1054,184, 1087,487), stdev = 49,663
  CI (99.9%): [148,147, 1960,221] (assumes normal distribution)


# Run complete. Total time: 00:01:03

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

Benchmark                     Mode  Cnt      Score       Error  Units
MyBenchmark.benchJeospatial  thrpt    3  80973,386 ± 56230,060  ops/s
MyBenchmark.benchLucene      thrpt    3   1054,184 ±   906,037  ops/s

您可以看到Jeospatial的速度提高了约75倍。所以我很好奇，如果这是真的，或者我只是以某种方式错误地配置了Lucene。

Answer 1

这几乎是一年前发布的。以下内容与当时一样重要，但性能要好得多。

不要使用spatial-extras，而要使用LatLonPoint，它会更高效，更直接地使用API。

这就是您所需要的：

// add your points to the document
doc.add(new LatLonPoint(fieldName, lat, lon));

// create your distance query
Query q = LatLonPoint.newDistanceQuery(fieldName, centerLat, centerLon, radiusMeters);

由于空间扩展（在反向索引中使用前缀树）而导致性能问题的原因有很多：

对于每个点，您都将递归四叉树并为该树的每个级别的倒排索引添加一项。这也意味着您的空间分辨率（和性能）取决于四叉树的深度（例如GEO_PRECISION_LEVEL）。
makeQuery与shapeFactory.circle并不是真正的距离搜索。通过相同的四叉树分解处理圆，以创建近似圆的项（四单元格）的集合。然后使用JTS.relate将倒排索引中的项相对于圆的栅格进行检查，这是一项极其昂贵的操作。

另一方面，

LatLonPoint创建块KD树结构，而不使用倒排索引。这是一种数据结构，专为按比例缩放空间和多维数字而设计。它具有更高的空间和时间效率，并且在非常大的数据集上表现更好。

希望这会有所帮助！

Lucene的内存空间索引性能不佳

1 个答案: