Question

我有一些我不拥有的服务器上的heroku日志。我正在编写一个脚本来计算服务器提供页面所需的平均时间，但是我不知道从哪里开始。这是我的日志文件的一部分：

3948    2014-09-02T16:25:56 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[35mDelayed::Backend::ActiveRecord::Job Load (2.0ms)[0m  UPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:25:56.552236', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:25:56.551737' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:25:56.551781') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING * 
3951    2014-09-02T16:27:06 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local3  Info    heroku/router   at=info method=GET path="/backend/account/edit" host=address request_id=65cd953e-c859-4f9a-8478-7438446fa693 fwd="ip" dyno=web.2 connect=1ms service=158ms status=200 bytes=2001 
3955    2014-09-02T16:27:16 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[35mDelayed::Backend::ActiveRecord::Job Load (5.1ms)[0m  UPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:27:16.619709', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:27:16.618922' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:27:16.618983') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING * 
3959    2014-09-02T16:29:41 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[36mDelayed::Backend::ActiveRecord::Job Load (1.9ms)[0m  [1mUPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:29:41.741650', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:29:41.741170' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:29:41.741211') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *[0m 
8152    2014-09-02T14:13:59 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [ef909505-0df5-45c3-860b-6b569997f36e] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36m (2.5ms)[0m  [1mSELECT COUNT(*) FROM "accounts" INNER JOIN "members" ON "members"."account_id" = "accounts"."id" WHERE "accounts"."archived" = 'f' AND "members"."user_id" = 45061424 AND "members"."active" = 't'[0m 
8155    2014-09-02T14:14:08 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [3dd30db2-b0b3-4a71-8235-d6191bcff1ed] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mPlan Load (2.2ms)[0m  [1mSELECT  "plans".* FROM "plans"  WHERE "plans"."id" = $1 LIMIT 1[0m  [["id", 284678023]] 
8159    2014-09-02T14:14:08 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [3dd30db2-b0b3-4a71-8235-d6191bcff1ed] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mCACHE (0.0ms)[0m  [1mSELECT COUNT(*) FROM "accounts" INNER JOIN "members" ON "members"."account_id" = "accounts"."id" WHERE "accounts"."archived" = 'f' AND "members"."user_id" = 45061424 AND "members"."active" = 't'[0m 
8162    2014-09-02T14:14:08 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [3dd30db2-b0b3-4a71-8235-d6191bcff1ed] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mCACHE (0.0ms)[0m  [1mSELECT COUNT(*) FROM "accounts" INNER JOIN "members" ON "members"."account_id" = "accounts"."id" WHERE "accounts"."archived" = 'f' AND "members"."user_id" = 45061424 AND "members"."active" = 't'[0m 
2454    2014-09-02T14:14:08 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [3dd30db2-b0b3-4a71-8235-d6191bcff1ed] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36m (2.8ms)[0m  [1mSELECT "accounts"."name", "accounts"."id" FROM "accounts" INNER JOIN "members" ON "members"."account_id" = "accounts"."id" WHERE "accounts"."archived" = 'f' AND "members"."user_id" = 45061424 AND "members"."active" = 't'[0m 
2456    2014-09-02T14:14:38 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [097ec54d-e1c1-40ee-ac46-82aad810375e] [VISITOR #NEW] [GUEST]   [1m[35mAccount Load (7.1ms)[0m  SELECT  "accounts".* FROM "accounts"  WHERE (accounts.approval_status != 'blocked') AND "accounts"."id" = 3 LIMIT 1 
2460    2014-09-02T14:14:38 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [3ea7ec23-85fd-4f93-81b6-53d5a0687517] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]  
2462    2014-09-02T14:14:38 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   Mailman#send_invitation: processed outbound mail in 1936.2ms 
2464    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[35mCACHE (0.0ms)[0m  SELECT  "jobs".* FROM "jobs"  WHERE "jobs"."id" = $1 LIMIT 1  [["id", 856633841]] 
2465    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[35mCACHE (0.1ms)[0m  SELECT  "jobs".* FROM "jobs"  WHERE "jobs"."id" = $1 LIMIT 1  [["id", 337079003]] 
2466    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[35mCACHE (0.0ms)[0m  SELECT  "jobs".* FROM "jobs"  WHERE "jobs"."id" = $1 LIMIT 1  [["id", 79295084]] 
2469    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mCACHE (0.2ms)[0m  [1mSELECT "jobs".* FROM "jobs"  WHERE "jobs"."account_id" = $1  ORDER BY created_at desc[0m  [["account_id", 196474847]] 
2470    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mCACHE (0.0ms)[0m  [1mSELECT  "jobs".* FROM "jobs"  WHERE "jobs"."id" = $1 LIMIT 1[0m  [["id", 862485972]] 
2471    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mCACHE (0.0ms)[0m  [1mSELECT  "jobs".* FROM "jobs"  WHERE "jobs"."id" = $1 LIMIT 1[0m  [["id", 745483704]] 
2473    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mCACHE (0.1ms)[0m  [1mSELECT  "jobs".* FROM "jobs"  WHERE "jobs"."id" = $1 LIMIT 1[0m  [["id", 126684480]] 
6738    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mCACHE (0.0ms)[0m  [1mSELECT  "jobs".* FROM "jobs"  WHERE "jobs"."id" = $1 LIMIT 1[0m  [["id", 862485973]] 
6739    2014-09-02T14:14:53 2014-09-03T00:48:52Z    4147981 demo-app    ip2 Local7  Info    app/web.2   [05ad9b39-58e7-4242-a6f1-e3efb7037404] [VISITOR #52285924-7c77-478a-8dbe-b43cc2e9dfd0] [USER #45061424]   [1m[36mInvitation Load (3.7ms)[0m  [1mSELECT "invitations".* FROM "invitations"  WHERE "invitations"."account_id" = $1 AND "invitations"."active" = 't'[0m  [["account_id", 196474847]] 
5356    2014-09-02T16:32:36 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[35mDelayed::Backend::ActiveRecord::Job Load (1.8ms)[0m  UPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:32:36.886356', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:32:36.885869' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:32:36.885914') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING * 
5362    2014-09-02T16:33:11 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[36mDelayed::Backend::ActiveRecord::Job Load (5.7ms)[0m  [1mUPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:33:11.949352', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:33:11.948846' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:33:11.948889') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *[0m 
5366    2014-09-02T16:35:04 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local3  Info    heroku/router   at=info method=GET path="/api/accounts/3" host=address request_id=0a99e4f6-cdb3-4163-9ee4-fac2f05d0288 fwd="ip" dyno=web.1 connect=4ms service=22ms status=404 bytes=445 
5369    2014-09-02T16:36:34 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/web.2   [9468b331-ec86-4d15-965a-5b2eb8085fa7] [VISITOR #NEW] [GUEST]   [1m[35mAccount Load (0.9ms)[0m  SELECT  "accounts".* FROM "accounts"  WHERE (accounts.approval_status != 'blocked') AND "accounts"."id" = 3 LIMIT 1 
5372    2014-09-02T16:37:52 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[36mDelayed::Backend::ActiveRecord::Job Load (2.0ms)[0m  [1mUPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:37:52.204583', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:37:52.204085' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:37:52.204130') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *[0m 
5374    2014-09-02T16:38:57 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[35mDelayed::Backend::ActiveRecord::Job Load (2.0ms)[0m  UPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:38:57.258868', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:38:57.258218' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:38:57.258269') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING * 
5382    2014-09-02T16:39:57 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[35mDelayed::Backend::ActiveRecord::Job Load (3.2ms)[0m  UPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:39:57.303114', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:39:57.302631' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:39:57.302674') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING * 
5385    2014-09-02T16:40:17 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[35mDelayed::Backend::ActiveRecord::Job Load (2.1ms)[0m  UPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:40:17.329199', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:40:17.322590' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:40:17.322648') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING * 
5390    2014-09-02T16:40:35 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local3  Info    heroku/router   at=info method=GET path="/api/accounts/3" host=address.com request_id=315458eb-8e7a-423d-8680-cead30dd6634 fwd="ip" dyno=web.2 connect=1ms service=17ms status=404 bytes=444 
5392    2014-09-02T16:42:05 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/web.2   [f420e156-4f00-4799-a3f0-ea67c65285b0] [VISITOR #NEW] [GUEST]   [1m[35mAccount Load (7.2ms)[0m  SELECT  "accounts".* FROM "accounts"  WHERE (accounts.approval_status != 'blocked') AND "accounts"."id" = 3 LIMIT 1 
5394    2014-09-02T16:43:57 2014-09-03T00:48:52Z    4147981 demo-app    ip1 Local7  Info    app/worker.1      [1m[35mDelayed::Backend::ActiveRecord::Job Load (2.1ms)[0m  UPDATE "delayed_jobs" SET locked_at = '2014-09-02 16:43:57.543745', locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2' WHERE id IN (SELECT  "delayed_jobs"."id" FROM "delayed_jobs"  WHERE ((run_at <= '2014-09-02 16:43:57.543237' AND (locked_at IS NULL OR locked_at < '2014-09-02 12:43:57.543283') OR locked_by = 'host:63bff4e5-8601-4620-b87c-75d0e4d0302c pid:2') AND failed_at IS NULL)  ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *

我想我必须找到一个属性才能将heroku / router与app / web.x条目进行分组，并且我相信此属性是ip？

在这里做我想做的正确方法是什么？

非常感谢您

Answer 1

不要重新发明轮子，

已经有很多软件可以实现您的目的，您可以尝试Splunk（免费），我已经使用您的数据对其进行了测试，并且可以正常工作，您应该尝试一下：

如果您可以访问服务器，请尝试Graylog支持Heroku（more info here）：

在这里您还可以查看一些日志管理软件示例：

如何找到服务页面的平均时间？

1 个答案: