Celery任务获取SoftTimeLimitExceeded调用API

Question

我有两个任务，我需要调用两个不同的API来发送短信和电子邮件。

views.py

def ...:
    ...
    send_payment_failed_sms.delay(payment_log.id, 1)
    task_send_booking_failed_email(payment_log.id, 1)  

但是我总是在发送短信的任务中遇到以下错误，但是发送电子邮件的任务工作正常：

[2018-07-12 20:47:03,519: INFO/MainProcess] Received task: payments.tasks.send_payment_failed_sms[ac4f1c2a-f09d-4e79-89c9-dfe1ab26b25d]  
[2018-07-12 20:47:18,548: WARNING/MainProcess] Soft time limit (15s) exceeded for payments.tasks.send_payment_failed_sms[ac4f1c2a-f09d-4e79-89c9-dfe1ab26b25d]
[2018-07-12 20:47:18,563: WARNING/ForkPoolWorker-1] on_failure
[2018-07-12 20:47:18,563: WARNING/ForkPoolWorker-1] ac4f1c2a-f09d-4e79-89c9-dfe1ab26b25d
[2018-07-12 20:47:18,564: WARNING/ForkPoolWorker-1] 'ac4f1c2a-f09d-4e79-89c9-dfe1ab26b25d' failed: SoftTimeLimitExceeded()
[2018-07-12 20:47:18,564: ERROR/ForkPoolWorker-1] Task payments.tasks.send_payment_failed_sms[ac4f1c2a-f09d-4e79-89c9-dfe1ab26b25d] raised unexpected: SoftTimeLimitExceeded()
Traceback (most recent call last):
...
billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()

tasks.py

@shared_task(bind=True, base=MyBaseClassForTask, max_retries=5, default_retry_delay=1 * 60, soft_time_limit=15,
             time_limit=30)
def send_payment_failed_sms(self, payment_id, reason):

    try:
        try:
            payload = ""
            headers = {'content-type': 'application/x-www-form-urlencoded'}
            url = 'https://sms-site.com/apikey=' + api_key + '&to=%s&var1=%s' % (
                mobile, payment_log.id
            )
            response = requests.request("GET", url, data=payload, headers=headers)
            response = response.json()

            if response['Status'] == 'Success':
                print('Sent message'), response['Details']
            else:
                print('Error:', response['Details'])

        except ConnectionError as exc:
            print('connection error @ failed sms')
            raise self.retry(exc=exc)

    except PaymentLog.DoesNotExist:
        ...

此外，它不会重试max_retries，即5次。你能告诉我我做错了吗？还是在调用API时有更好的方法来定义任务？

Answer 1

您有soft_time_limit=15 seconds，这次芹菜提高了SoftTimeLimitExceeded之后，这个技巧可以让您在任务中进行清理

from celery.exceptions import SoftTimeLimitExceeded

@celery.task(soft_time_limit=15, time_limit=20)
def mytask():
    try:
        return do_work()
    except SoftTimeLimitExceeded:
        cleanup_in_a_hurry()