Question

I am in a situation where I have a CRON task on google app engine (using flex environment) that just dies after some time, but I have no trace WHY (checked the GA Logs, nothing, tried try/catch, and explicitly log it - no error).

I have explicitly verified that if I create a cron task that runs for 8 minutes (but doesn't do much - just sleeps and updates database every second), it will run successfully. This is just to prove that CRON jobs can at least run 8 minutes if not more. & I have set up the Express & NodeJS combo up correctly.

This is all fine, but seems that my other cron job dies in 2-3 minutes, so quite fast. It is hitting some kind of limit, but I have no idea how to control for it, or even what limit it is, so all I can do is speculate.

I will tell more about my CRON task. It is basically rapidly querying MongoDB database where every query is quite fast. I've tried the same code locally, and there are no problems.

My speculation is that I am somehow creating too many MongoDB requests at once, and potentially running out of something?

Here's a pseudocode (just to describe what kind of scale data we're talking about - the numbers and flow are exactly the same):

function q1() {
      return await mongoExecute(async (db) => {
        const [l1, l2] = await Promise.all([
            db.collection('Obj1').count({uid1: c1, u2action: 'L'}),
            db.collection('Obj1').count({uid2: c2, u1action: 'L'}),
        ]);

        return l1+l2;
    });
}

for(let i = 0; i < 8000; i++) {
   const allImportantInformation = Promise.all([
       q1(),
       q2(),
       q3(),
       .....
       q10()
   ])

   await mongoDb.saveToServer(document);
}

It is getting somewhere around i=1600 before the CRON job just dies without any explanation. The GA Cron Job panel clearly says the JOB has failed.

Here is also my mongoExecute (which is just a separate module that caches the db object, which hopefully is the correct practice in order to ensure that mongodb pooling works correctly.)

import { MongoClient, Db } from 'mongodb';

let db = null;
let promiseInProgress = null;

export async function mongoExecute<T> (executor: (instance: Db) => T): Promise<T | null> {

    if (!db) {
        if (!promiseInProgress) {
            promiseInProgress = new Promise(async (resolve, reject) => {
                const tempDb = await MongoClient.connect(process.env.MONGODB_URL);
                resolve(tempDb);
            });
        }

        db = await promiseInProgress;
    }
    try {
        const value = await executor(db);
        return value;
    } catch (error) {
        console.log(error);
        return null;
    }
}

What would be the solution? My idea is to basically ensure less requests are made at once (so all the promises would be sequential, and potentially add sleep between each cycle in the FOR.

I don't understand because it works fine up until some specific point (and quite big point, it's definitely different amount, sometimes it is 800, sometimes 1200, etc).

Is there any "running out of TCP connections" scenario happening? Theoretically we shouldn't run out of anything because we don't have much open at any given point.

It seems to be working if I throw 200ms wait between each cycle & I suspect I can figure out solution, all the items don't have to be updated in the same CRON execution, but it is a bit annoying, and I would like to know what's going on.

Is the garbage collector not catching up fast enough, why exactly is GA silently failing my cron task?

Answer 1

我发现了错误所在，并相应地进行了修复。

让我改一下；我不知道该错误是什么，并且在任何时候都没有错误令人沮丧，但是我设法通过将我的nodejs mongodb驱动程序更新到最新版本（从2.xx-> 3.1）来修复（幸运的是）发生了什么。 10）。

我的代码中不再需要睡眠。

Cron job failed without a reason

1 个答案: