Question

我有一个场景，其中会有一个包含网站的列表和用于抓取这些网站的代码块。是否可以实现多线程方式，以便每个线程从列表中获取5个或更多网站并独立爬网，并确保它们不会使用由另一个线程收集的同一网站。

List <String> websiteList;

//crawling code block here

Answer 1

例如，您可以使用可由所有感兴趣的消费者共享的BlockingQueue（注意，为清晰起见，跳过了错误处理）：

public static void main(String[] args) throws Exception {
    // for test purposes add 10 integers
    final BlockingQueue<Integer> queue = new LinkedBlockingDeque<Integer>();
    for (int i = 0; i < 10; i++) {
        queue.add(i);    // 
    }

    new Thread(new MyRunnable(queue)).start();
    new Thread(new MyRunnable(queue)).start();
    new Thread(new MyRunnable(queue)).start();

}

static class MyRunnable implements Runnable {
    private Queue<Integer> queue;

    MyRunnable(Queue<Integer> queue) {
        this.queue = queue;
    }

    @Override
    public void run() {
        while(!queue.isEmpty()) {
            Integer data = queue.poll();
            if(data != null) {
                System.out.println(Thread.currentThread().getName() + ": " + data);
            }
        }
    }
}

当Queue为空时，Threads将退出，程序将结束。

Answer 2

正如其他答案所述，根据这样的要求，您应该首先考虑将您的网站保留在java.util.concurrent包中的Java并发抽象数据类型之一，而不是标准列表中。 BlockingQueue的drainTo方法听起来就像您正在寻找的一样，因为您希望线程能够一次占用一堆网站。

Answer 3

您可以使用LinkedBlockingQueue，将所有websiteList放入此队列并在每个线程之间共享此队列。现在所有线程都会在这个队列上进行轮询，这是一个阻塞操作，它确保一个元素只是一个线程获取队列。

类似的东西：

String site;
while((site=queue.poll(timeout, TimeUnit.SECONDS))!=null)
{
//process site
}

Answer 4

您可以尝试使用DoubleBufferedList。这允许您从多个线程向列表中添加列表和条目，并使用多个线程以完全无锁的方式从列表中获取列表。

public class DoubleBufferedList<T> {
  // Atomic reference so I can atomically swap it through.
  // Mark = true means I am adding to it so momentarily unavailable for iteration.
  private AtomicMarkableReference<List<T>> list = new AtomicMarkableReference<>(newList(), false);

  // Factory method to create a new list - may be best to abstract this.
  protected List<T> newList() {
    return new ArrayList<>();
  }

  // Get and replace the current list.
  public List<T> get() {
    // Atomically grab and replace the list with an empty one.
    List<T> empty = newList();
    List<T> it;
    // Replace an unmarked list with an empty one.
    if (!list.compareAndSet(it = list.getReference(), empty, false, false)) {
      // Failed to replace! 
      // It is probably marked as being appended to but may have been replaced by another thread.
      // Return empty and come back again soon.
      return Collections.<T>emptyList();
    }
    // Successfull replaced an unmarked list with an empty list!
    return it;
  }

  // Grab and lock the list in preparation for append.
  private List<T> grab() {
    List<T> it;
    // We cannot fail so spin on get and mark.
    while (!list.compareAndSet(it = list.getReference(), it, false, true)) {
      // Spin on mark - waiting for another grabber to release (which it must).
    }
    return it;
  }

  // Release the list.
  private void release(List<T> it) {
    // Unmark it - should this be a compareAndSet(it, it, true, false)?
    if (!list.attemptMark(it, false)) {
      // Should never fail because once marked it will not be replaced.
      throw new IllegalMonitorStateException("It changed while we were adding to it!");
    }
  }

  // Add an entry to the list.
  public void add(T entry) {
    List<T> it = grab();
    try {
      // Successfully marked! Add my new entry.
      it.add(entry);
    } finally {
      // Always release after a grab.
      release(it);
    }
  }

  // Add many entries to the list.
  public void add(List<T> entries) {
    List<T> it = grab();
    try {
      // Successfully marked! Add my new entries.
      it.addAll(entries);
    } finally {
      // Always release after a grab.
      release(it);
    }
  }

  // Add a number of entries.
  @SafeVarargs
  public final void add(T... entries) {
    // Make a list of them.
    add(Arrays.<T>asList(entries));
  }
}

Answer 5

我建议使用其中一个 3个解决方案：

保持简单

synchronized(list) {
    // get and remove 5 websites from the list
}

如果您可以更改列表类型，则可以使用

BlockingQueue

如果您无法更改列表类型，可以使用

Collections.synchronizedList(list)

共享列表上的多线程

5 个答案: