我的方案如下： 我有5台Linux机器，我在HDFS中有10个（可能超过这个）文件。我的要求是一台机器应该锁定其中一个文件并处理它，而其他机器不应该处理这个文件，而是锁定另一个文件并处理它们。 例如：machine1 - 获取对file2的锁定并对其进行处理 machine2 - 锁定file3并处理它 machine3 - 获取对file1的锁定并对其进行处理
我编写了一个虚拟的多线程java程序来模拟它。但是它无法正常工作：
public class DistributedLock {
private final ZooKeeper zk;
private final String lockBasePath;
private String lockPath;
public DistributedLock(ZooKeeper zk, String lockBasePath) {
this.zk = zk;
this.lockBasePath = lockBasePath;
}
public boolean lock(String lockName) throws IOException {
try {
boolean locked = false;
if(zk.exists(lockBasePath + "/" + lockName, false) == null){
lockPath = zk.create(lockBasePath + "/" + lockName, null,
ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL_SEQUENTIAL);
if(lockPath != null ){
locked =true;
}
}
final Object lock = new Object();
return locked;
} catch (KeeperException e) {
throw new IOException(e);
} catch (InterruptedException e) {
throw new IOException(e);
}
}
public void unlock() throws IOException {
try {
zk.delete(lockPath, -1);
lockPath = null;
} catch (KeeperException e) {
throw new IOException(e);
} catch (InterruptedException e) {
throw new IOException(e);
}
}
}
public class DistributedLockTest {
public static void main(String[] args) throws Exception {
new DistributedLockTest().run();
}
public void run() throws Exception {
Thread t1 = new Thread(new Process(1));
Thread t2 = new Thread(new Process(2));
Thread t3 = new Thread(new Process(3));
Thread t4 = new Thread(new Process(4));
t1.start();
t2.start();
t3.start();
t4.start();
}
class Process implements Runnable {
int id;
List<String> fileNames = new ArrayList<String>();
public Process(int id) {
this.id = id;
for (int i = 1; i < 11; i++) {
fileNames.add("file" + i);
}
}
// @Override
public void run() {
try {
System.out.println("machine " + id + " started");
String resource = "resource";
String path = "/LockDir";
ZooKeeper zooKeeper = new ZooKeeper("127.0.0.1", 2181, null);
if (zooKeeper.exists(path, false) == null) {
zooKeeper.create(path, null, ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
}
DistributedLock lock = new DistributedLock(zooKeeper, path);
String lockedFile;
for (String fileName : fileNames) {
System.out.println("machine " + id + " Acquiring Lock on "+ fileName);
boolean locked = lock.lock(fileName);
if(locked){
System.out.println("machine " + id + "got Lock on "+ fileName);
lockedFile = fileName;
}
else continue;
Thread.sleep(500);
}
System.out.println("machine " + id + " Releasing Lock");
lock.unlock();
System.out.println("machine " + id + " Released Lock");
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
我得到的输出是：
machine 1 started
machine 2 started
machine 3 started
machine 4 started
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
machine 2 Acquiring Lock on file1
machine 1 Acquiring Lock on file1
machine 4 Acquiring Lock on file1
machine 3 Acquiring Lock on file1
machine 1got Lock on file1
machine 3got Lock on file1
machine 2got Lock on file1
machine 4got Lock on file1
machine 1 Acquiring Lock on file2
machine 3 Acquiring Lock on file2
machine 4 Acquiring Lock on file2
machine 2 Acquiring Lock on file2
machine 1got Lock on file2
machine 3got Lock on file2
machine 2got Lock on file2
machine 4got Lock on file2
machine 3 Acquiring Lock on file3
machine 1 Acquiring Lock on file3
machine 2 Acquiring Lock on file3
machine 4 Acquiring Lock on file3
machine 1got Lock on file3
machine 4got Lock on file3
machine 3got Lock on file3
machine 2got Lock on file3
machine 2 Acquiring Lock on file4
machine 4 Acquiring Lock on file4
machine 3 Acquiring Lock on file4
machine 1 Acquiring Lock on file4
machine 4got Lock on file4
machine 2got Lock on file4
machine 3got Lock on file4
machine 1got Lock on file4
machine 4 Acquiring Lock on file5
machine 3 Acquiring Lock on file5
machine 2 Acquiring Lock on file5
machine 1 Acquiring Lock on file5
machine 3got Lock on file5
machine 2got Lock on file5
machine 4got Lock on file5
machine 1got Lock on file5
machine 2 Acquiring Lock on file6
machine 4 Acquiring Lock on file6
machine 3 Acquiring Lock on file6
machine 1 Acquiring Lock on file6
machine 2got Lock on file6
machine 1got Lock on file6
machine 4got Lock on file6
machine 3got Lock on file6
machine 2 Acquiring Lock on file7
machine 4 Acquiring Lock on file7
machine 1 Acquiring Lock on file7
machine 3 Acquiring Lock on file7
machine 4got Lock on file7
machine 2got Lock on file7
machine 1got Lock on file7
machine 3got Lock on file7
machine 4 Acquiring Lock on file8
machine 3 Acquiring Lock on file8
machine 1 Acquiring Lock on file8
machine 2 Acquiring Lock on file8
machine 1got Lock on file8
machine 4got Lock on file8
machine 3got Lock on file8
machine 2got Lock on file8
machine 2 Acquiring Lock on file9
machine 4 Acquiring Lock on file9
machine 3 Acquiring Lock on file9
machine 1 Acquiring Lock on file9
machine 4got Lock on file9
machine 3got Lock on file9
machine 1got Lock on file9
machine 2got Lock on file9
machine 4 Acquiring Lock on file10
machine 3 Acquiring Lock on file10
machine 1 Acquiring Lock on file10
machine 2 Acquiring Lock on file10
machine 2got Lock on file10
machine 4got Lock on file10
machine 1got Lock on file10
machine 3got Lock on file10
machine 4 Releasing Lock
machine 1 Releasing Lock
machine 2 Releasing Lock
machine 3 Releasing Lock
machine 2 Released Lock
machine 1 Released Lock
machine 4 Released Lock
machine 3 Released Lock
这表明每个线程/机器都试图锁定每个文件并获取它。但我想要的是如果一台机器没有锁定特定的机器，它应该尝试锁定另一个文件并处理它。 有什么建议吗？
答案 0 :(得分：2)
我在您的代码中发现了两个错误，第一个错误是您使用CreateMode.EPHEMERAL_SEQUENTIAL作为锁定节点。当你想要使用CreateMode.EPHEMERAL时。 Sequential主要用于不用于锁的队列，它将创建名称类似于file10000000000123 file10000000000124等的节点。因此，您永远不会创建用于检查锁是否为服用。
如果你修复了这个问题，你很可能会在线程之间遇到竞争条件，因为它们首先检查节点是否存在然后创建它。使多个线程可以尝试创建相同的节点，因此我的解决方案如下所示：
public class DistributedLock {
public static final String _LOCK = "lock";
...
public boolean lock(String lockName) throws IOException {
try {
boolean locked = false;
synchronized(_LOCK){
if(zk.exists(lockBasePath + "/" + lockName, false) == null){
lockPath = zk.create(lockBasePath + "/" + lockName, null,
ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.EPHEMERAL);
if(lockPath != null ){
locked =true;
}
}
}
final Object lock = new Object();
return locked;
...
}
...
并且输出看起来像这样，这是我想你想要的：
machine 1 Acquiring Lock on file1
machine 4 Acquiring Lock on file1
machine 2 Acquiring Lock on file1
machine 3 Acquiring Lock on file1
machine 1 got Lock on file1
machine 3 Acquiring Lock on file2
machine 3 got Lock on file2
machine 2 Acquiring Lock on file2
machine 4 Acquiring Lock on file2
machine 2 Acquiring Lock on file3
machine 2 got Lock on file3
...