Question

当我运行Nutch并且链接不再存在时，我可以运行readdb命令，它会显示有标记为db_gone的网址。

所以我运行SolrClean命令并说：

SolrClean deleting a total of 1 documents

哪个是正确的，但没有从Solr中删除任何内容

帮助？

如果你想查看我的配置，那么我有一个关于如何配置我自己的Solr / Nutch设置的博客Here

修改：

很有可能不仅SolrClean命令不起作用，我觉得它与我的设置有关，删除不是被提交的？

这是为文档发出的删除请求 - 但文档存在：

INFO  - 2013-08-09 15:54:52.729; 
org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={wt=javabin&version=2} 
{delete=file:/C:/Users/alamil/Documents/TextFiles/Y2012.doc 
(-1442903587791306752)]} 0 2

这是整个日志：

 INFO  - 2013-08-09 15:54:51.785; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@f5331a main
INFO  - 2013-08-09 15:54:51.786; org.apache.solr.core.QuerySenderListener; QuerySenderListener sending requests to Searcher@f5331a main{StandardDirectoryReader(segments_7w:549:nrt _6j(4.3.1):C12/11 _6k(4.3.1):C12)}
INFO  - 2013-08-09 15:54:51.787; org.apache.solr.core.QuerySenderListener; QuerySenderListener done.
INFO  - 2013-08-09 15:54:51.787; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO  - 2013-08-09 15:54:51.788; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@f5331a main{StandardDirectoryReader(segments_7w:549:nrt _6j(4.3.1):C12/11 _6k(4.3.1):C12)}
INFO  - 2013-08-09 15:54:51.789; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&wt=javabin&waitFlush=true&version=2} {commit=} 0 903
INFO  - 2013-08-09 15:54:52.053; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/select params={fl=id&q=id:[*+TO+*]&wt=javabin&version=2&rows=1} hits=13 status=0 QTime=1 
INFO  - 2013-08-09 15:54:52.355; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/select params={fl=id&q=id:[*+TO+*]&wt=javabin&version=2&rows=1} hits=13 status=0 QTime=0 
INFO  - 2013-08-09 15:54:52.413; org.apache.solr.core.SolrCore; [collection1] webapp=/solr path=/select params={fl=id,boost,tstamp,digest&start=0&q=id:[*+TO+*]&wt=javabin&version=2&rows=13} hits=13 status=0 QTime=1 
INFO  - 2013-08-09 15:54:52.729; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {delete=[file:/C:/Users/alamil/Documents/TextFiles/Y2012.doc (-1442903587791306752)]} 0 2
INFO  - 2013-08-09 15:54:52.733; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO  - 2013-08-09 15:54:52.835; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
commit{dir=NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@C:\Users\alamil\Documents\Test\solr_home\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ae0e7d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_7w,generation=284,filenames=[_6j_1.del, _6k_Lucene41_0.pos, segments_7w, _6j.nvd, _6j_Lucene41_0.tim, _6j_Lucene41_0.tip, _6k.fdt, _6k.fnm, _6j_Lucene41_0.pos, _6j.nvm, _6k_Lucene41_0.doc, _6k_Lucene41_0.tim, _6k.si, _6j.si, _6k.nvd, _6k.fdx, _6j_Lucene41_0.doc, _6j.fdt, _6k.nvm, _6j.fdx, _6k_Lucene41_0.tip, _6j.fnm]
commit{dir=NRTCachingDirectory(org.apache.lucene.store.SimpleFSDirectory@C:\Users\alamil\Documents\Test\solr_home\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ae0e7d; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_7x,generation=285,filenames=[_6j_1.del, _6k_Lucene41_0.pos, _6j.nvd, _6j_Lucene41_0.tim, _6j_Lucene41_0.tip, _6k.fdt, _6k.fnm, _6j_Lucene41_0.pos, _6j.nvm, _6k_Lucene41_0.doc, _6k_Lucene41_0.tim, _6k.si, _6j.si, _6k.nvd, _6k.fdx, segments_7x, _6j_Lucene41_0.doc, _6j.fdt, _6k.nvm, _6j.fdx, _6k_Lucene41_0.tip, _6j.fnm]
INFO  - 2013-08-09 15:54:52.835; org.apache.solr.core.SolrDeletionPolicy; newest commit = 285[_6j_1.del, _6k_Lucene41_0.pos, _6j.nvd, _6j_Lucene41_0.tim, _6j_Lucene41_0.tip, _6k.fdt, _6k.fnm, _6j_Lucene41_0.pos, _6j.nvm, _6k_Lucene41_0.doc, _6k_Lucene41_0.tim, _6k.si, _6j.si, _6k.nvd, _6k.fdx, segments_7x, _6j_Lucene41_0.doc, _6j.fdt, _6k.nvm, _6j.fdx, _6k_Lucene41_0.tip, _6j.fnm]

Answer 1

转储crawldb并找到状态为db_gone的url，并在solr index中检查这些url。如果您正在进行新的爬网，则不会在Solr索引中找到任何这些URL，因为db_gone URL不会被考虑用于索引。

现在当我们运行solrclean命令时，nutch会在crawldb中找到状态为db_gone的所有url，并从Solr索引中删除所有这些url。如果在solr索引中找到状态为db_gone的任何url，那么只会更改index，否则index将保持不变。

这可能发生在你的情况下。

注意：
如果只是重新抓取，我们可能会发现一些已在首次抓取时已成功抓取并编入索引的网址，但在重新抓取阶段标记为db_gone，因为这些网址不再可用。因此，当我们运行solrclean命令并且solr index将被更改时，这些url将从solr索引中删除。

查看SolrClean类的源代码 http://grepcode.com/file/repo1.maven.org/maven2/org.apache.nutch/nutch/1.3/org/apache/nutch/indexer/solr/SolrClean.java

希望帮助你......

SolrClean不会从Solr中删除

1 个答案: