Question

我想为Spring应用程序创建一个同步文件写入机制。我大约有10000万个json，应该保存在单独的文件中，例如：

来自json的文本“ abc”：{“ id”：“ 1”，“ text”：“ abc”}应该保存到“ 1.json”文件中
来自json的文本“ pofd”：{“ id”：“ 2”，“ text”：“ pofd”}应该保存到“ 2.json”文件中

其他要求：

可以同时写入多个文件
一个文件可以由多个线程同时更新（许多具有相同ID但文本不同的json）

我创建了FileWriterProxy（单个bean），这是保存文件的主要组件。它加载负责写入文件的懒惰FileWriter组件（原型Bean）（具有同步的写入方法）。每个FileWriter对象代表一个单独的文件。我怀疑我的解决方案不是线程安全。让我们考虑以下情形：

有3个线程（线程1，线程2，线程3）要写入同一文件（1.json），它们全部都来自FileWriterProxy组件的写入方法
Thread1正在获取正确的FileWriter
Thread1正在锁定1.json文件的FileWriter
Thread1正在写入1.json文件
Thread1正在完成向文件的写入，并将要从ConcurrentHashMap中删除FileWriter
与此同时，Thread2正在为1.json文件获取FileWriter并等待Thread1释放锁
Thread1正在释放锁并从ConcurrentHashMap中删除FileWriter
现在Thread2可以写入1.json文件（它的FileWriter已从ConcurrentHashMap中删除）
Thread3正在为1.json获取FileWriter（一个新的！旧的FileWriter已被Thread1删除）
由于Thread2和Thread3锁定了不同的FileWriters对象，因此它们同时写入同一文件中

如果我错了，请纠正我。如何解决我的实施方案？

FileWriterProxy：

@Component
public class FileWriterProxy {
    private final BeanFactory beanFactory;
    private final Map<String, FileWriter> filePathsMappedToFileWriters = new ConcurrentHashMap<>();

    public FileWriterProxy(BeanFactory beanFactory) {
        this.beanFactory = beanFactory;
    }

    public void write(Path path, String data) {
        FileWriter fileWriter = getFileWriter(path);
        fileWriter.write(data);
        removeFileWrite(path);
    }

    private FileWriter getFileWriter(Path path) {
        return filePathsMappedToFileWriters.computeIfAbsent(path.toString(), e -> beanFactory.getBean(FileWriter.class, path));
    }

    private void removeFileWrite(Path path) {
        filePathsMappedToFileWriters.remove(path.toString());
    }

}

FileWriterProxyTest：

@RunWith(SpringRunner.class)
@SpringBootTest
public class FileWriterProxyTest {

    @Rule
    public TemporaryFolder temporaryFolder = new TemporaryFolder();
    private static final String FILE_NAME = "filename.txt";
    private File baseDirectory;
    private Path path;

    @Autowired
    private FileWriterProxy fileWriterProxy;

    @Before
    public void setUp() {
        baseDirectory = temporaryFolder.getRoot();
        path = Paths.get(baseDirectory.getAbsolutePath(), FILE_NAME);
    }

    @Test
    public void writeToFile() throws IOException {
        String data = "test";
        fileWriterProxy.write(path, data);
        String fileContent = new String(Files.readAllBytes(path));
        assertEquals(data, fileContent);
    }

    @Test
    public void concurrentWritesToFile() throws InterruptedException {
        Path path = Paths.get(baseDirectory.getAbsolutePath(), FILE_NAME);
        List<Task> tasks = Arrays.asList(
                new Task(path, "test1"),
                new Task(path, "test2"),
                new Task(path, "test3"),
                new Task(path, "test4"),
                new Task(path, "test5"));
        ExecutorService executorService = Executors.newFixedThreadPool(5);
        List<Future<Boolean>> futures = executorService.invokeAll(tasks);

        wait(futures);
    }

    @Test
    public void manyRandomWritesToFiles() throws InterruptedException {
        List<Task> tasks = createTasks(1000);
        ExecutorService executorService = Executors.newFixedThreadPool(5);
        List<Future<Boolean>> futures = executorService.invokeAll(tasks);
        wait(futures);
    }

    private void wait(List<Future<Boolean>> tasksFutures) {
        tasksFutures.forEach(e -> {
            try {
                e.get(10, TimeUnit.SECONDS);
            } catch (Exception e1) {
                e1.printStackTrace();
            }
        });
    }

    private List<Task> createTasks(int number) {
        List<Task> tasks = new ArrayList<>();

        IntStream.range(0, number).forEach(e -> {
            String fileName = generateFileName();
            Path path = Paths.get(baseDirectory.getAbsolutePath(), fileName);
            tasks.add(new Task(path, "test"));
        });

        return tasks;
    }

    private String generateFileName() {
        int length = 10;
        boolean useLetters = true;
        boolean useNumbers = false;
        return RandomStringUtils.random(length, useLetters, useNumbers) + ".txt";
    }

    private class Task implements Callable<Boolean> {
        private final Path path;
        private final String data;

        Task(Path path, String data) {
            this.path = path;
            this.data = data;
        }

        @Override
        public Boolean call() {
            fileWriterProxy.write(path, data);
            return true;
        }
    }
}

配置：

@Configuration
public class Config {

    @Bean
    @Lazy
    @Scope("prototype")
    public FileWriter fileWriter(Path path) {
        return new FileWriter(path);
    }

}

FileWriter：

public class FileWriter {
    private static final Logger logger = LoggerFactory.getLogger(FileWriter.class);

    private final Path path;

    public FileWriter(Path path) {
        this.path = path;
    }

    public synchronized void write(String data) {
        String filePath = path.toString();
        try {
            Files.write(path, data.getBytes());
            logger.info("File has been saved: {}", filePath);
        } catch (IOException e) {
            logger.error("Error occurred while writing to file: {}", filePath);
        }
    }

}

在Java中同时将文件写入机制同步成多个文件

0 个答案: