Question

我有一个REST API，可以收集有关第三方应用程序使用的urls的信息。安装了一个嗅探器，该嗅探器正在寻找HTTP请求，然后调用API，以提供有关所使用的URL和标头的信息。

我想存储尽可能唯一的URL，以便每次搜索用户：

http://localhost/users/49a95b87-083e-475b-9278-bade6f24413b
http://localhost/users/508f2a55-fe5b-4b83-b853-7e829dd366b8
http://localhost/users/af48be64-ad48-4867-ac06-984ce064dbeb

将作为一个条目存储在数据库中（uuid在这里并不重要，因此只有最后或最后一个会保留）。这样做的算法是这样的：

获取请求

检查数据库是否包含相似的网址

如果数据库未返回任何结果，则将URL存储到数据库

控制器在哪里：

    @PreAuthorize("hasAuthority('ROLE_API')")
    @PostMapping(value = "/api/webapp")
    public synchronized ResponseEntity<Status> getWebApp(@RequestBody ServiceDiscovery req) throws InterruptedException {
        return webAppService.processWebAppRequest(req.getWebApp());
    }

服务方法：webAppService.processWebAppRequest

private final static String UUID_PATTERN = "[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}";
    private final static String SIM_NUMBER_PATTERN = "[0-9]{19}";
    private final static String MSISDN_SHORT_PATTERN = "[0-9]{9}";
    private final static String MSISDN_LONG_PATTERN = "[0-9]{11}";

    @Transactional
    public ResponseEntity<Status> processScanWebAppRequest(ServiceDiscovery serviceDiscovery){
        try {
            Optional<WebApp> wa = checkRegexes(serviceDiscovery.getUrl());
            if (wa.isPresent()){
                updateExistingWebApplication(serviceDiscovery, wa.get());
            } else {
                saveNewWebApplication(serviceDiscovery);
            }
        } catch (IncorrectResultSizeDataAccessException ex) {
            return new ResponseEntity<Status>(new Status("Processing error"), HttpStatus.PRECONDITION_FAILED);
        } 
        return new ResponseEntity<Status>(new Status("OK"), HttpStatus.OK);
    }
    //Checkregex function which return Optional.empty it looks for regex in string and then replace finding with regex itself so it can be used in JPA query
    private Optional<WebApp> checkRegexes(String url) {
        String urlToLookFor = url.replaceAll(UUID_PATTERN,UUID_PATTERN);
        urlToLookFor=urlToLookFor.replaceAll(SIM_NUMBER_PATTERN,SIM_NUMBER_PATTERN);
        urlToLookFor=urlToLookFor.replaceAll(MSISDN_LONG_PATTERN,MSISDN_LONG_PATTERN);
        urlToLookFor=urlToLookFor.replaceAll(MSISDN_SHORT_PATTERN,MSISDN_SHORT_PATTERN);

        return waRepository.getWebAppByRegex(urlToLookFor+"$");
    }
    //JPA check for 
    @Query(value="select * from webapp wa where wa.url ~ :url", nativeQuery = true)
    Optional<WebApp> getWebAppByRegex(@Param("url") String url);

在我测试时一切正常。但是在生产环境中，我定期收到大量请求（几天之内一天一次〜5k请求），在该请求中，有多个用于同一端点的UUID正在并行处理，因此当在checkRegexes处进行处理时没有找到，但是一秒钟后有5。

为避免这种情况，我尝试将API设置为synchronized，但没有成功。无论如何，是否可以在不更改客户端的情况下使它正常工作？

使用查询的示例输出是：

db=# select inserted, url from webapp where url ~ 'https://api-gateway.intra.com/users/[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}$' order by inserted;
      inserted       |                                        url
---------------------+-----------------------------------------------------------------------------------
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/cd1da115-a2c1-4722-a381-6d524cbf5c95
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/6b9b2c76-f416-4c8c-a0b0-01d29e976c03
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/0f197568-0d2f-405d-8468-3bbea8b3a8ef
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/32e02581-02b4-4a99-9121-1592b0a67566
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/f1bbb4cf-1336-45ec-814f-77ecb9736c47
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/6b0969a1-cfd7-4cda-9041-14229ddb3c6f
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/1fa4e464-aae6-472e-9a4c-cf4c7d6b4e0f
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/a25cbb9c-49c8-4603-aeb6-68d5a065c58e
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/e9be9949-f866-4765-9d38-582193dd1839
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/ad46db32-82dd-4c8e-9e77-a67e59dfca9a
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/fed07e4a-42e4-44e3-b932-9d0e01a5b535
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/747ff6df-dea5-48db-a15b-b7f43bc48ecf
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/4b6341ad-2584-427d-898a-f24d3623eb32
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/1e75c8bb-fb27-4183-a993-6763fa796e79
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/72b19740-7f61-4a32-88f9-0f7196d47853
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/6c21da91-56eb-498b-91bd-4895be3cfdcc
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/a7d3dc5c-c5de-4fc4-9c56-9eefe7a67d80
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/b9658963-c1f5-45da-b78d-240bc7b2a225
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/e8be9c46-e7d0-4642-9c7b-981663aa552e
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/74886e61-150d-4f35-8af0-c911cfdbf009
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/175840ab-9f4c-43b7-a684-1adeb884af71
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/7655bf87-1f34-4a38-9ad0-11c5c1ddbb6b
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/34406b3d-319d-4ca4-9ca2-d53c104ad703
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/15f40dc6-5852-4f9f-9017-87b20dd326f1
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/de5caa58-cb7b-4dac-9b87-b6f4460072f3
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/b601b86b-ba8e-4768-b013-12d12f74362b
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/a64fd22b-f144-4b70-b3bb-86ee0e7a47c6
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/87672dac-ce7d-4a03-acfc-e694d229c4fe
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/ea6db1e8-461b-459a-b5d0-2f4aec872cdd
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/d4b7cd3c-68bd-42e4-99d0-0f5043428080
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/4dfd42cc-4029-46dc-8867-adafac249345
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/2be89a7a-cf26-4163-bd49-0549bba24cd0
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/3d1f3f34-a11d-4f8a-b5d5-7d8fc58e49f0
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/db1ec2c3-5571-4db0-a3fe-6e12a4b1ae46
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/75f964aa-db57-43d3-a424-99345c8c9997

Answer 1

我建议您同步对数据库的访问。因此，select，save和update操作应处于同步块中。但是它将减少并发到单线程，并且性能下降将是巨大的。因此，您可以使用一些技巧来同步数据库访问，但可以一定程度地节省性能。

您可以通过混杂的url的哈希值同步数据库访问。我使用url_hash %10000来限制锁定对象的数量。当然，它会影响性能，但是比简单同步好10000。

看看代码：

private final static String UUID_PATTERN = "[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}";
private final static String SIM_NUMBER_PATTERN = "[0-9]{19}";
private final static String MSISDN_SHORT_PATTERN = "[0-9]{9}";
private final static String MSISDN_LONG_PATTERN = "[0-9]{11}";

private final Map<Integer, Object> locks = new ConcurrentHashMap<>();

@Transactional
public ResponseEntity<Status> processScanWebAppRequest(ServiceDiscovery serviceDiscovery){
    try {
        String urlToLookFor = normalizeUrl(serviceDiscovery.getUrl());
        int lockHash = urlToLookFor.hashCode() % 10000;
        synchronized (locks.computeIfAbsent(lockHash, integer -> new Object())) {
            Optional<WebApp> wa = checkRegexes(urlToLookFor);
            if (wa.isPresent()){
                updateExistingWebApplication(serviceDiscovery, wa.get());
            } else {
                saveNewWebApplication(serviceDiscovery);
            }
        }
    } catch (IncorrectResultSizeDataAccessException ex) {
        return new ResponseEntity<Status>(new Status("Processing error"), HttpStatus.PRECONDITION_FAILED);
    }
    return new ResponseEntity<Status>(new Status("OK"), HttpStatus.OK);
}

private String normalizeUrl(String url) {
    String urlToLookFor = url.replaceAll(UUID_PATTERN,UUID_PATTERN);
    urlToLookFor=urlToLookFor.replaceAll(SIM_NUMBER_PATTERN,SIM_NUMBER_PATTERN);
    urlToLookFor=urlToLookFor.replaceAll(MSISDN_LONG_PATTERN,MSISDN_LONG_PATTERN);
    urlToLookFor=urlToLookFor.replaceAll(MSISDN_SHORT_PATTERN,MSISDN_SHORT_PATTERN);
    return urlToLookFor;
}

private Optional<WebApp> checkRegexes(String url) {
    return waRepository.getWebAppByRegex(urlToLookFor+"$");
}

解决Rest API调用中的竞争条件

1 个答案: