Spring Kafka和处理中间的重新平衡

时间:2020-01-21 22:27:47

标签: spring apache-kafka transactions spring-kafka

在此链接https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html中,标题为“使用具有特定偏移量的记录的消费”部分引用了一种策略,该策略有效地“按需更新”外部存储中的主题分区偏移量,然后撤销分区(例如重新平衡)只需将任何中断的事务提交到外部存储。

现在,我假设此策略意味着在分区撤销回调中,我不需要处理传入的TopicPartition集合的偏移量,因为任何被中断的“进行中”事务都将保留并将包含需要提交/保存的分区偏移量。

(如果我对此有误,请纠正我。)

那么,既然这是Spring Kafka,并且我正在使用@Transactional服务来保存必要的数据,那么上述策略是否相关/可行?换句话说,我不确定如何恢复/提交标记为@Transactional的内容,因为事务管理器,边界等都是在后台进行处理的。

这甚至是个问题吗?如果是这样,那么实现此策略的最佳方法是什么?手动跟踪交易(在方法和回调之间听起来很糟糕)?

还是我应该在分区撤销时浏览TopicPartition集合并更新分区偏移量?

希望这很有意义,因为我想确保我做对了。

谢谢。

1 个答案:

答案 0 :(得分:0)

2017年9月发布

从卡夫卡的角度来看,那本书很老。具有现代版本;在Kafka中保留偏移量要简单得多;只需确保您的消费者可以处理<?php namespace Sample; require_once '/home/server/public/vendor/autoload.php'; //1. Import the PayPal SDK client that was created in `Set up Server-Side SDK`. use Sample\PayPalClient; use PayPalCheckoutSdk\Orders\OrdersGetRequest; use PayPalCheckoutSdk\Core\PayPalHttpClient; use PayPalCheckoutSdk\Core\SandboxEnvironment; ini_set('error_reporting', E_ALL); // or error_reporting(E_ALL); ini_set('display_errors', '1'); ini_set('display_startup_errors', '1'); class PayPalClient { /** * Returns PayPal HTTP client instance with environment that has access * credentials context. Use this instance to invoke PayPal APIs, provided the * credentials have access. */ public static function client() { return new PayPalHttpClient(self::environment()); } /** * Set up and return PayPal PHP SDK environment with PayPal access credentials. * This sample uses SandboxEnvironment. In production, use LiveEnvironment. */ public static function environment() { $clientId = getenv("CLIENT_ID") ?: "CLIENT_ID-here"; $clientSecret = getenv("CLIENT_SECRET") ?: "CLIENT_SECRET-here"; return new SandboxEnvironment($clientId, $clientSecret); } } class GetOrder { // 2. Set up your server to receive a call from the client /** *You can use this function to retrieve an order by passing order ID as an argument. */ public static function getOrder($orderId) { echo "testtttt ".$orderId; // 3. Call PayPal to get the transaction details $client = PayPalClient::client(); $response = $client->execute(new OrdersGetRequest($orderId)); /** *Enable the following line to print complete response as JSON. */ print json_encode($response->result); print "Status Code: {$response->statusCode}\n"; print "Status: {$response->result->status}\n"; print "Order ID: {$response->result->id}\n"; print "Intent: {$response->result->intent}\n"; print "Links:\n"; foreach($response->result->links as $link) { print "\t{$link->rel}: {$link->href}\tCall Type: {$link->method}\n"; } // 4. Save the transaction in your database. Implement logic to save transaction to your database for future reference. print "Gross Amount: {$response->result->purchase_units[0]->amount->currency_code} {$response->result->purchase_units[0]->amount->value}\n"; // To print the whole response body, uncomment the following line // echo json_encode($response->result, JSON_PRETTY_PRINT); } } /** *This driver function invokes the getOrder function to retrieve *sample order details. * *To get the correct order ID, this sample uses createOrder to create a new order *and then uses the newly-created order ID with GetOrder. */ $request_body = file_get_contents('php://input'); $data = json_decode($request_body); print_r($data); echo 'order id = '.$data->orderID; if (!count(debug_backtrace())) { GetOrder::getOrder($data->orderID, true); } ?> poll()返回的所有记录,以避免完全重新平衡。