Rails Postgres查询无法返回新记录

时间:2019-01-30 23:37:38

标签: ruby-on-rails postgresql activerecord

该Rails代码应该可以防止服务器在20秒内记录重复的记录:

@transit = Transit.new(tag: params[:tag])
if Transit.where(tag: @transit.tag).where("created_at > ?", 20.seconds.ago).first
  logger.warn "Duplicate tag"
else
  @transit.save!
end

但是,这不起作用。我可以在生产数据库(托管在Heroku中)中看到两个不同的记录,它们是用相距10秒的相同标签创建的。

日志显示在第二个请求上执行了正确的查询,但没有返回任何结果,并且仍然保存了一条新记录。

为什么会这样?我认为Postgres的默认隔离级别read_committed可以防止这种情况的发生。不返回任何记录的查询应该错过Rails的SQL缓存。日志显示这两个请求是由Heroku上的同一WEB.1 Dyno处理的,而我的Puma.rb设置为4个工作人员和5个线程。

我想念什么?

这是数据库中的两条记录:

=> #<Transit id: 1080116, tag: 33504, 
             created_at: "2019-01-30 12:36:11", 
             updated_at: "2019-01-30 12:41:23">

=> #<Transit id: 1080115, tag: 33504, 
             created_at: "2019-01-30 12:35:56", 
             updated_at: "2019-01-30 12:35:56">

第一次插入的日志:

30 Jan 2019 07:35:56.203132 <190>1 2019-01-30T12:35:56.050681+00:00 app web.1 - - [1m [36m (0.8ms) [0m [1mBEGIN [0m
30 Jan 2019 07:35:56.203396 <190>1 2019-01-30T12:35:56.055097+00:00 app web.1 - - [1m [35mSQL (1.0ms) [0m INSERT INTO "transits" ("tag", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"
30 Jan 2019 07:35:56.269133 <190>1 2019-01-30T12:35:56.114572+00:00 app web.1 - - [1m [36m (2.0ms) [0m [1mCOMMIT [0m

在插入重复项之前的查询日志:

30 Jan 2019 07:36:12.160359 <190>1 2019-01-30T12:36:11.863973+00:00 app web.1 - - [1m [35mTransit Load (5.1ms) [0m SELECT "transits".* FROM "transits" WHERE "transits"."tag" = 33504 AND created_at > '2019-01-30 12:35:51.846431' ORDER BY "transits"."id" ASC LIMIT 1

这是postgres事务隔离级别,需要明确的是出现此问题后打开的另一个连接:

SHOW default_transaction_isolation;

 default_transaction_isolation 
-------------------------------
 read committed
(1 row)

3 个答案:

答案 0 :(得分:0)

防止Rails中重复的一种方法是进行验证: Correct way of prevent duplicate records in Rails

但是,您的条件更复杂,因为它涉及跨多个行。 我相信您的标准是,如果最近的运输记录创建时间少于20秒,则不允许输入运输记录。是吗?

在这里提到试图强制执行一个约束,该约束涉及从许多行查看数据,这是不希望的: SQL Sub queries in check constraint

触发器可用于在数据库级别强制执行约束。 一个人可能会抓住一个例外。 不确定是否有一个名为HairTrigger的宝石。

从这里获取想法: https://karolgalanciak.com/blog/2016/05/06/when-validation-is-not-enough-postgresql-triggers-for-data-integrity/

带有Postgresql触发器的示例:

bin/rails generate model transit tag:text
rails generate migration add_validation_trigger_for_transit_creation

class AddValidationTriggerForTransitCreation < ActiveRecord::Migration[5.2]
  def up
    execute <<-CODE
      CREATE FUNCTION validate_transit_create_time() returns trigger as $$
      DECLARE
      age int;
      BEGIN
        age := (select extract(epoch from current_timestamp - t.created_at)
        from transits t
        where t.tag = NEW.tag
        and t.id in (select id from transits u
           where u.id = t.id
           and u.tag = t.tag
           and u.created_at = (select max(v.created_at) from transits v where v.tag = u.tag)
        ));
        IF (age < 20) THEN
          RAISE EXCEPTION 'created_at too early: %', NEW.created_at;
        END IF;
        RETURN NEW;
      END;
      $$ language plpgsql;

      CREATE TRIGGER validate_transit_create_trigger BEFORE INSERT OR UPDATE ON transits
      FOR EACH ROW EXECUTE PROCEDURE validate_transit_create_time();
    CODE
  end

  def down
    execute <<-CODE
    drop function validate_transit_create_time() cascade;
    CODE
  end
end


user1@debian8 /home/user1/rails/dup_test > ../transit_test.rb ; sleep 20; ../transit_test.rb 

dup_test_development=> select * from transits;
 id  |   tag    |         created_at         |         updated_at         
-----+----------+----------------------------+----------------------------
 158 | test_tag | 2019-01-31 18:38:10.115891 | 2019-01-31 18:38:10.115891
 159 | test_tag | 2019-01-31 18:38:30.609125 | 2019-01-31 18:38:30.609125
(2 rows)

这是我们查询中提供带有标签的最新公交条目的部分

dup_test_development=> select * from transits t
where t.tag = 'test_tag' and t.id in
(select id from transits u where u.id = t.id and u.tag = t.tag and u.created_at =
(select max(v.created_at) from transits v where v.tag = u.tag));

 id  |   tag    |         created_at         |         updated_at         
-----+----------+----------------------------+----------------------------
 159 | test_tag | 2019-01-31 18:38:30.609125 | 2019-01-31 18:38:30.609125
(1 row)

使用我们的标签进行修改,以给出current_timestamp(现在)与最新的运输条目之间的差异。这种差异是PostgreSQL中的间隔。使用UTC匹配Rails:

dup_test_development=> select current_timestamp at time zone 'utc' - created_at
from transits t  where t.tag = 'test_tag' and t.id in
(select id from transits u where u.id = t.id and u.tag = t.tag and u.created_at =
(select max(v.created_at) from transits v where v.tag = u.tag));
    ?column?     
-----------------
 00:12:34.146536
(1 row)

添加Extract(epoch)将其转换为秒:

dup_test_development=> select extract(epoch from current_timestamp at time zone 'utc' - created_at)
from transits t  where t.tag = 'test_tag' and t.id in
(select id from transits u where u.id = t.id and u.tag = t.tag and u.created_at =
(select max(v.created_at) from transits v where v.tag = u.tag));
 date_part  
------------
 868.783503
(1 row)

我们将秒存储为年龄,如果年龄小于20,则会引发数据库异常

运行2次插入,其第二次延迟小于20:

user1@debian8 /home/user1/rails/dup_test > ../transit_test.rb ; sleep 5; ../transit_test.rb 
#<ActiveRecord::StatementInvalid: PG::RaiseException: ERROR:  created_at too early: 2019-01-31 18:54:48.95695
: INSERT INTO "transits" ("tag", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id">
"ERROR:  created_at too early: 2019-01-31 18:54:48.95695\n"

在铁轨外进行短路测试:

#!/usr/bin/env ruby

require 'active_record'
require 'action_view'

path = "/home/user1/rails/dup_test/app/models"
require "#{path}/application_record.rb"
Dir.glob(path + "/*.rb").sort.each do | file |
  require file
end

ActiveRecord::Base.establish_connection(
  :adapter => "postgresql",
  :database  => 'dup_test_development',
  encoding: "unicode",
  username: "user1",
  password: nil
)
class Test
  def initialize()
  end
  def go()
    begin
      t = Transit.new(tag: 'test_tag')
      t.save
    rescue ActiveRecord::StatementInvalid => e
      p e
      p e.cause.message
    end
  end
end

def main
  begin
    t = Test.new()
    t.go()
  rescue Exception => e
    puts e.message
  end
end

main

已经提到使用Redis之类的东西-可能对性能更好

答案 1 :(得分:0)

我认为这是一个并发问题。

ActiveRecord返回后,Rails事务将异步继续进行。只要提交需要15秒才能应用,就会导致此问题。这很长,不太可能,但是可能。

我无法证明这是事实,但这似乎是唯一的解释。要防止这种情况,将需要一个dB存储过程,或者像@PhilipWright建议的那样,或者像您和@kwerle这样的分布式锁。

答案 2 :(得分:-1)

这就是测试的目的。

class Transit <  ActiveRecord::Base
  def new_transit(tag: tag)
  <your code>
  end
end

您测试代码:

  test 'it saves once' do
    <save it once.  check the count, etc>
  end

  test 'it does not save within 10 seconds' do
    <save it once.  Set the created at to 10 seconds ago.  try to save again.  check the count, etc>
  end

p.s。考虑使用redis或类似的东西。否则,您想要执行诸如表锁之类的操作以确保您不会踩踏自己。而且您可能不想做表锁。