在rails中使用insert查询循环遍历大数据的最佳方法是什么？

时间：2016-06-17 14:41:04

标签： ruby-on-rails ruby ruby-on-rails-3 ruby-on-rails-4 activerecord

我必须插入大数据，比方说20k我怀疑我写了一个优化的查询。

它的作用：

  1. 使用sql获取用户的记录，满足一些条件，它在merge_user_records中的活动记录数组中获取超过1k-20k的用户
  2. 从1k-20k用户的活动记录数组中批量处理100个用户
  3. 遍历merge_user记录并在merge_user_records中找到用户模型用户user_id的用户
  4. 仍在循环中，用户调用方法construct_user_notifications为每个用户插入user_notifications。
  5. 仍然在循环中找到用户的设备。
  6. 在设备中运行循环以在每台设备上发送推送通知。
  7. 循环结束

    8. 这是代码。

    merge_users = MergeField.get_user_field_values(notification_template.merge_field, scope_users) #=> **returns users 1k - 20k**
if merge_users.present?
  merge_users.each_slice(100) do |record|
    record.each do |user_record|
      user = User.find_by(id: user_record.user_id)
      text = notification_template.title
      notification_template.description = MustacheDescription.render(notification_template, user_record)
      text += " " + notification_template.description
      Rails.logger.info "Merge field message: #{notification_template.description}"
      construct_user_notifications(notification_template, user_record.user_id) #=> **this calls another method below which actually create notifications for every user.**
      badge = (notification_template.display_screen == "suggestion") ? user.unread_suggestion_notifications.count : user.unread_option_notifications.count
      devices = user.devices.with_notification_token
      if devices.present?
        devices.each do |device|
          PushNotification.notify_ios(text, device.notification_token, badge, {screen: notification_template.display_screen})
          Rails.logger.info "Sending push to user_id #{user_record.user_id} token #{device.notification_token}"
        end
      end
    end
  end
end

def self.construct_user_notifications(notification_template, user_id)
  notification_template.user_notifications.build.tap do |user_notification|
    user_notification.title = notification_template.title
    user_notification.subtitle = notification_template.subtitle
    user_notification.description = notification_template.description
    user_notification.merge_field = notification_template.merge_field
    user_notification.cta = notification_template.cta
    user_notification.cta_key = notification_template.cta_key
    user_notification.secondary_cta = notification_template.secondary_cta
    user_notification.secondary_cta_key = notification_template.secondary_cta_key
    user_notification.show_useful = notification_template.show_useful
    user_notification.category = notification_template.category
    user_notification.display_screen = notification_template.display_screen
    user_notification.sent_at = Time.current
    user_notification.user_id = user_id
    user_notification.filter_preferences = notification_template.filter_preferences
    user_notification.save
  end
end

    我已经为100位用户测试了这个，需要30-40秒。上帝知道生产中的20k用户需要多少钱。

2 个答案:

答案 0 :(得分：3)

我建议将循环的内部内容包装在一个事务块中，该事务块将在最后一次运行所有查询，而不是零碎。这会将每个用户的所有查询分组到一个同时运行的事务中：

merge_users.each_slice(100) do |record|
  ActiveRecord::Base.transaction do
  // code
  end

  if devices.present?
    devices.each do |device|
      PushNotification.notify_ios(text,device.notification_token,badge,{screen: notification_template.display_screen})
      Rails.logger.info  "Sending push to user_id #{user_record.user_id} token #{device.notification_token}"
    end
  end
end

您可以在此处找到有关交易的更多信息：

http://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html

最后我建议您不要直接在块中执行PushNotification.notifify_ios，而应使用DelayedJob或类似功能在后台运行作业。这意味着所有方法调用将在代码本身运行后在后台处理，而不是在循环本身内。

这看起来像是：

if devices.present?
    devices.each do |device|
      PushNotification.delay.notify_ios(text,device.notification_token,badge,{screen: notification_template.display_screen})
      Rails.logger.info  "Sending push to user_id #{user_record.user_id} token #{device.notification_token}"
    end
  end

https://github.com/collectiveidea/delayed_job

答案 1 :(得分：2)

1。循环中的find_by

user = User.find_by(id: user.user_id)会查询20k次！我们可以通过将其放在each循环中来避免这种情况：

merge_users.each_slice(100) do |users|
  users = User.where(id: users.map(&:user_id))
  users.each do |user|
    # loop
  end
end

2。记录器级别

将info更改为debug。磁盘IO总是很慢。

3。 construct_user_notifications功能

notification_template.user_notifications.build将分配20k个对象。 GC也是一个问题。

请仅构建属性，稍后再保存。

例如：

def self.construct_user_notifications(notification_template, user_id)
  {
      title: notification_template.title,
      subtitle: notification_template.subtitle,
      description: notification_template.description,
      merge_field: notification_template.merge_field,
      cta: notification_template.cta,
      cta_key: notification_template.cta_key,
      secondary_cta: notification_template.secondary_cta,
      secondary_cta_key: notification_template.secondary_cta_key,
      show_useful: notification_template.show_useful,
      category: notification_template.category,
      display_screen: notification_template.display_screen,
      sent_at: Time.current,
      user_id: user_id,
      filter_preferences: notification_template.filter_preferences, 
      # more attributes
  }
end

4。 badge次查询

badge = (notification_template.display_screen == "suggestion") ? user.unread_suggestion_notifications.count : user.unread_option_notifications.count

除非devices存在，否则这是不必要的。

您可以稍后查询徽章。

5。推送通知

PushNotification.notify_ios(text, device.notification_token, badge, {screen: notification_template.display_screen})

这可能有一些http请求，这很慢。

您应该使用sidekiq或resque在后​​台工作中执行此操作。

6。保存user_notifications

看一下activerecord-import gem。批量插入更有效。

实施例

merge_users = MergeField.get_user_field_values(notification_template.merge_field, scope_users) #=> **returns users 1k - 20k**

merge_users.each_slice(500) do |users|
  users = User.where(id: users.map(&:user_id))
  user_notifications = Set.new

  users.each do |user|
    text = notification_template.title
    notification_template.description = MustacheDescription.render(notification_template, user)
    text += " " + notification_template.description
    Rails.logger.debug "Merge field message: #{notification_template.description}"

    user_notifications.add construct_user_notifications(notification_template, user.user_id)

    # do this asynchronously
    push_notification(notification_template, user_id)
  end

  UserNotification.import(user_notifications.first.keys, user_notifications.to_a)
end

def self.push_notification(notification_template, user_id)
  devices = Device.where(user_id: user_id).with_notification_token.pluck(:notification_token)
  if devices.present?
    badge = (notification_template.display_screen == "suggestion") ? UnreadSuggestionNotification.where(user_id: user_id).count : UnreadOptionNotification.where(user_id: user_id).count
    devices.each do |device|
      PushNotification.notify_ios(text, device.notification_token, badge, {screen: notification_template.display_screen})
      Rails.logger.debug "Sending push to user_id #{user_id} token #{device.notification_token}"
    end
  end
end


def self.construct_user_notifications(notification_template, user_id)
  {
      title: notification_template.title,
      subtitle: notification_template.subtitle,
      description: notification_template.description,
      merge_field: notification_template.merge_field,
      cta: notification_template.cta,
      cta_key: notification_template.cta_key,
      secondary_cta: notification_template.secondary_cta,
      secondary_cta_key: notification_template.secondary_cta_key,
      show_useful: notification_template.show_useful,
      category: notification_template.category,
      display_screen: notification_template.display_screen,
      sent_at: Time.current,
      user_id: user_id,
      filter_preferences: notification_template.filter_preferences,
      # more attributes
  }
end
相关问题
最新问题