Question

我将Pub / Sub订阅逻辑包装在一个订阅方法中，该方法在每次订阅的服务初始化期间都会被调用一次：

    def subscribe(self,
                  callback: typing.Callable,
                  subscription_name: str,
                  topic_name: str,
                  project_name: str = None) -> typing.Optional[SubscriberClient]:
        """Subscribes to Pub/Sub topic and return subscriber client

        :param callback: subscription callback method
        :param subscription_name: name of the subscription
        :param topic_name: name of the topic
        :param project_name: optional project name. Uses default project if not set
        :return: subscriber client or None if testing
        """
        project = project_name if project_name else self.pubsub_project_id
        self.logger.info('Subscribing to project `{}`, topic `{}`'.format(project, topic_name))

        project_path = self.pubsub_subscriber.project_path(project)
        topic_path = self.pubsub_subscriber.topic_path(project, topic_name)
        subscription_path = self.pubsub_subscriber.subscription_path(project, subscription_name)

        # check if there is an existing subscription, if not, create it
        if subscription_path not in [s.name for s in self.pubsub_subscriber.list_subscriptions(project_path)]:
            self.logger.info('Creating new subscription `{}`, topic `{}`'.format(subscription_name, topic_name))
            self.pubsub_subscriber.create_subscription(subscription_path, topic_path)

        # subscribe to the topic
        self.pubsub_subscriber.subscribe(
            subscription_path, callback=callback,
            scheduler=self.thread_scheduler
        )
        return self.pubsub_subscriber

此方法的调用方式如下：

        self.subscribe_client = self.subscribe(
            callback=self.pubsub_callback,
            subscription_name='subscription_topic',
            topic_name='topic'
        )

回调方法可以完成很多工作，发送2封电子邮件，然后确认消息

    def pubsub_callback(self, data: gcloud_pubsub_subscriber.Message):
        self.logger.debug('Processing pub sub message')

        try:
            self.do_something_with_message(data)

            self.logger.debug('Acknowledging the message')
            data.ack()
            self.logger.debug('Acknowledged')
            return

        except:
            self.logger.warning({
                "message": "Failed to process Pub/Sub message",
                "request_size": data.size,
                "data": data.data
            }, exc_info=True)

        self.logger.debug('Acknowledging the message 2')
        data.ack()

当我运行将某些内容推送到订阅时，将运行回调，并打印所有调试消息，包括Acknowledged。但是，该消息保留在Pub / Sub中，回调再次被调用，并且每次重试后都需要花费指数时间。问题是什至导致ack被调用后，该消息仍保留在发布/订阅中的原因是什么？

我有几个这样的订阅，它们都按预期工作。截止日期不是一个选项，回调几乎立即完成，而且无论如何我都使用ack截止日期，无济于事。

当我尝试从连接到该发布订阅的本地运行的应用程序处理这些消息时，它会很好地完成并确认将消息按预期从队列中移出。

因此，该问题仅在已部署的服务（在kubernetes容器内运行）中显现出来
回调执行buck ack看似没有任何作用
从本地运行的脚本（...执行完全相同的操作）或通过GCP UI获取消息的工作符合预期。

有什么想法吗？

Answer 1

在发布/订阅中，确认是尽力而为的，因此重新发送邮件是可能的，但并不常见。

如果您始终收到重复邮件，则可能是由于相同消息内容的重复发布。就发布/订阅而言，这些是不同的消息，并且将被分配不同的消息ID。检查发布/提供的消息ID，以确保您实际上多次收到同一条消息。

有an edge case in dealing with large backlogs of small messages with streaming pull（这是Python客户端库使用的）。如果您正在运行多个订阅同一订阅的客户端，则这种情况可能很重要。

您还可以查看订阅的Stackdriver metrics来查看：

如果成功发送了其通知（subscription/ack_message_count）
如果其积压量正在减少（subscription/backlog_bytes）
如果您的订户缺少确认期限（subscription/streaming_pull_ack_message_operation_count过滤后的response_code != "success"）

如果您没有错过确认截止日期并且积压的订单保持稳定，则应与Google Cloud支持人员联系，提供您的项目名称，订阅名称以及重复的消息ID的示例。他们将能够调查这些重复发生的原因。

Answer 2

我做了一些其他测试，终于找到了问题。

TL; DR：我对所有订阅使用相同的google.cloud.pubsub_v1.subscriber.scheduler.ThreadScheduler。

这是我用来测试的代码片段。这是损坏的版本：

server.py

import concurrent.futures.thread
import os
import time

from google.api_core.exceptions import AlreadyExists
from google.cloud import pubsub_v1
from google.cloud.pubsub_v1.subscriber.scheduler import ThreadScheduler


def create_subscription(project_id, topic_name, subscription_name):
    """Create a new pull subscription on the given topic."""
    subscriber = pubsub_v1.SubscriberClient()
    topic_path = subscriber.topic_path(project_id, topic_name)
    subscription_path = subscriber.subscription_path(
        project_id, subscription_name)

    subscription = subscriber.create_subscription(
        subscription_path, topic_path)

    print('Subscription created: {}'.format(subscription))


def receive_messages(project_id, subscription_name, t_scheduler):
    """Receives messages from a pull subscription."""
    subscriber = pubsub_v1.SubscriberClient()
    subscription_path = subscriber.subscription_path(
        project_id, subscription_name)

    def callback(message):
        print('Received message: {}'.format(message.data))
        message.ack()

    subscriber.subscribe(subscription_path, callback=callback, scheduler=t_scheduler)
    print('Listening for messages on {}'.format(subscription_path))


project_id = os.getenv("PUBSUB_PROJECT_ID")

publisher = pubsub_v1.PublisherClient()
project_path = publisher.project_path(project_id)

# Create both topics
try:
    topics = [topic.name.split('/')[-1] for topic in publisher.list_topics(project_path)]
    if 'topic_a' not in topics:
        publisher.create_topic(publisher.topic_path(project_id, 'topic_a'))
    if 'topic_b' not in topics:
        publisher.create_topic(publisher.topic_path(project_id, 'topic_b'))
except AlreadyExists:
    print('Topics already exists')

# Create subscriptions on both topics
sub_client = pubsub_v1.SubscriberClient()
project_path = sub_client.project_path(project_id)

try:
    subs = [sub.name.split('/')[-1] for sub in sub_client.list_subscriptions(project_path)]
    if 'topic_a_sub' not in subs:
        create_subscription(project_id, 'topic_a', 'topic_a_sub')
    if 'topic_b_sub' not in subs:
        create_subscription(project_id, 'topic_b', 'topic_b_sub')
except AlreadyExists:
    print('Subscriptions already exists')

scheduler = ThreadScheduler(concurrent.futures.thread.ThreadPoolExecutor(10))

receive_messages(project_id, 'topic_a_sub', scheduler)
receive_messages(project_id, 'topic_b_sub', scheduler)

while True:
    time.sleep(60)

client.py

import datetime
import os
import random
import sys
from time import sleep

from google.cloud import pubsub_v1


def publish_messages(pid, topic_name):
    """Publishes multiple messages to a Pub/Sub topic."""
    publisher = pubsub_v1.PublisherClient()
    topic_path = publisher.topic_path(pid, topic_name)

    for n in range(1, 10):
        data = '[{} - {}] Message number {}'.format(datetime.datetime.now().isoformat(), topic_name, n)
        data = data.encode('utf-8')
        publisher.publish(topic_path, data=data)
        sleep(random.randint(10, 50) / 10.0)


project_id = os.getenv("PUBSUB_PROJECT_ID")
publish_messages(project_id, sys.argv[1])

我连接到云发布/订阅，服务器创建了主题和订阅。然后，我为两个主题并行运行了客户端脚本多次。片刻之后，一旦我更改了服务器代码以在receive_messages范围内实例化新的线程调度程序，服务器便清理了两个主题并按预期运行。

令人困惑的是，无论哪种情况，服务器都会为所有消息打印出接收到的消息。

我将其发布到https://github.com/googleapis/google-cloud-python/issues

确认后，GCP消息会保留在发布/订阅中

2 个答案: