如何避免抓取共享磁盘而不降低它？

Question

我正在使用Nutch。我计划抓取共享磁盘而不是互联网网站。

我担心的一件事是，抓取它会使磁盘变得非常慢。 如何避免抓取共享磁盘而不降低它？

Answer 1

您可以在conf / nutch-site.xml中设置请求之间的线程数和等待时间。

尝试重写这些属性并将它们设置为您认为合适的值：

<property>
  <name>fetcher.threads.fetch</name>
  <value>10</value>
  <description>The number of FetcherThreads the fetcher should use.
  This is also determines the maximum number of requests that are
  made at once (each FetcherThread handles one connection). The total
  number of threads running in distributed mode will be the number of
  fetcher threads * number of nodes as fetcher has one map task per node.
  </description>
</property>

<property>
  <name>fetcher.threads.per.queue</name>
  <value>1</value>
  <description>This number is the maximum number of threads that
    should be allowed to access a queue at one time.
   </description>
</property>