Question

我目前正在开发一款涉及在rails服务器上同步联系人的应用。我正在使用redis服务器和sidekiq在后台执行联系人同步。我的数据库是mongodb，我使用mongoid gem作为ORM。工作流程如下：

手机上的联系人通过app传递到rails服务器，然后在rails服务器上，它在redis服务器中排队。
现在cron job触发sidekiq连接到redis并完成工作。

sidekiq的一项工作如下：

它有一系列联系人（大小高达3000）。
必须处理这些联系人。通过处理我的意思是对DB进行插入查询。

现在的问题是sidekiq花费了大量的时间来完成这项工作。平均而言，完成这项工作需要50-70秒。

以下是相关文件

sidekiq.yml

# Sample configuration file for Sidekiq.
# Options here can still be overridden by cmd line args.
#   sidekiq -C config.yml

:verbose: true
:concurrency:  5
:logfile: ./log/sidekiq.log
:pidfile: ./tmp/pids/sidekiq.pid
:queues:
  - [new_wall, 1]#6
  - [contact_wall, 1]#7
  - [email, 1]#5
  - [gcm_chat, 1]#5
  - [contact_address, 1]#7
  - [backlog_contact_address, 5]
  - [comment, 7]
  - [default, 5]

mongoid.yml

development:
  # Configure available database sessions. (required)
  sessions:
    # Defines the default session. (required)
    default:
          # Defines the name of the default database that Mongoid can connect to.
          # (required).
          database: "<%= ENV['DB_NAME']%>"
          # Provides the hosts the default session can connect to. Must be an array
          # of host:port pairs. (required)
          hosts:
            - "<%=ENV['MONGOD_URL']%>"
          #username: "<%= ENV['DB_USERNAME']%>"
          #password: "<%= ENV['DB_PASSWORD']%>"
          options:

            #pool: 12
        # Change the default write concern. (default = { w: 1 })
        # write:
        # w: 1

        # Change the default consistency model to primary, secondary.
        # 'secondary' will send reads to secondaries, 'primary' sends everything
        # to master. (default: primary)
        # read: secondary_preferred

        # How many times Moped should attempt to retry an operation after
        # failure. (default: The number of nodes in the cluster)
        # max_retries: 20

        # The time in seconds that Moped should wait before retrying an
        # operation on failure. (default: 0.25)
        # retry_interval: 0.25
  # Configure Mongoid specific options. (optional)
  options:
    # Includes the root model name in json serialization. (default: false)
    # include_root_in_json: false

    # Include the _type field in serializaion. (default: false)
    # include_type_for_serialization: false

    # Preload all models in development, needed when models use
    # inheritance. (default: false)
    # preload_models: false

    # Protect id and type from mass assignment. (default: true)
    # protect_sensitive_fields: true

    # Raise an error when performing a #find and the document is not found.
    # (default: true)
    # raise_not_found_error: true

    # Raise an error when defining a scope with the same name as an
    # existing method. (default: false)
    # scope_overwrite_exception: false

    # Use Active Support's time zone in conversions. (default: true)
    # use_activesupport_time_zone: true

    # Ensure all times are UTC in the app side. (default: false)
    # use_utc: false
test:
  sessions:
    default:
      database: db_test
      hosts:
        - localhost:27017
      options:
        read: primary
        # In the test environment we lower the retries and retry interval to
        # low amounts for fast failures.
        max_retries: 1
        retry_interval: 0


production:
  # Configure available database sessions. (required)
  sessions:
    # Defines the default session. (required)
    default:
      # Defines the name of the default database that Mongoid can connect to.
      # (required).
      database: "<%= ENV['DB_NAME']%>"
      # Provides the hosts the default session can connect to. Must be an array
      # of host:port pairs. (required)
      hosts:
        - "<%=ENV['MONGOD_URL']%>"
      username: "<%= ENV['DB_USERNAME']%>"
      password: "<%= ENV['DB_PASSWORD']%>"
      pool: 10
      options:

  # Configure Mongoid specific options. (optional)
  options:

Model.rb

def retry_save_contact_dump(c_dump_id)
      c_dump = ContactDump.where(_id: c_dump_id, status: ContactDump::CONTACT_DUMP_CONS[:ERROR]).first
      return false if c_dump.blank?
      user = User.where(_id: c_dump.user_id).first
      puts "retry_save_contact_dump"
      user.save_contacts_with_name(c_dump.contacts)
      c_dump.status = ContactDump::CONTACT_DUMP_CONS[:PROCESSED]
      c_dump.error_msg = ""
      c_dump.save
    rescue => e
      c_dump.status = ContactDump::CONTACT_DUMP_CONS[:CANTSYNC]
      c_dump.error_msg = e.message
      c_dump.save
   end


def save_contacts_with_name(c_array)
    m_num = Person.get_number_digest(self.mobile_number.to_s)
    c_array.each do |n|
      next if m_num == n["hash_mobile_number"]
      p = Person.where(h_m_num: n["hash_mobile_number"]).first_or_create
      save_friend(p) #if p.persisted?
      p.c_names.create(name: n["name"], user_id: self.id)
    end
  end

ContactDump.rb

class ContactDump
  include Mongoid::Document
  include Mongoid::Timestamps::Created
  include Mongoid::Timestamps::Updated

  field :contacts,   type: Array
  field :status,     type: Integer, default: 0
  field :user_id,    type: BSON::ObjectId
  field :error_msg,  type: String

  CONTACT_DUMP_CONS = {FRESH: 0,  PROCESSED: 1, ERROR: 2, CANTSYNC: 3}
end

如何加快处理工作？我尝试在sidekiq.yml和mongoid.yml池中增加sidekiq的并发性排列，但没有帮助。

whatsApp和其他消息传递应用如何处理联系人同步？

如果需要其他信息，请询问。谢谢。

编辑：如果无法回答这个问题，有人可以建议我在轨道服务器上同步联系人的其他方法。

Answer 1

索引救援。

class ContactDump
  index({status: 1})
end

class Person
  index({h_m_num: 1})
end

Person可能需要更多索引，具体取决于Person.get_number_digest的作用。

添加索引后运行 rake db:mongoid:create_indexes

此外，请删除puts，即使您看不到输出，也不需要在工作人员身上使用，而且即使您看不到输出，也会严重影响您的表现！

减少sidekiq的工作执行时间

1 个答案: