使用Symfony2和Doctrine 2从数据库加载大数据

时间:2012-11-23 10:08:45

标签: database symfony doctrine-orm

我是 Symfony2 的初学者。

我有一个Regions-Countries-States-Cities数据库,结果超过2,000,000。我有8个实体:

Region (recursive with itself) - RegionTranslation

Country - CountryTranslation

State (recursive with itself) - StateTranslation

City - CityTranslation

问题在于,当我想加载国家/地区列表(例如,下拉列表中只有250个寄存器)时,Symfony + Doctrine会加载所有实体结构(所有国家/地区的所有州,以及所有州的所有城市,各自的翻译)。

我认为它花了很多记忆。

这样做的正确方法是什么?我可以只加载具有此结构的国家/地区(和翻译)吗?有什么想法吗?

2 个答案:

答案 0 :(得分:2)

对于没有关联的异议,我遇到了同样的问题。您最好的选择是使用select2的ajax加载(http://ivaynberg.github.com/select2/),这将在搜索框中提供有限数量的项目,并且还会根据框中输入的内容缩小搜索范围。

有些事情需要编码:

javascript文件:

      $(document).ready(function(){
        $('.select2thing').select2({
          minimumInputLength:1
        ,width: "100%"
        ,ajax: {
          url: <<path>> + "entity/json"
         ,dataType: 'jsonp'
         ,quitMillis: 100
         ,data: function (term, page) {
         return {
            q: term, // search term
            limit: 20,
            page: page
         };
        }
    ,results: function (data, page) {
      var more = (page * 20) < data.total;
      return { results: data.objects, more: more };
    }
    }
    });

    }

控制器中的jsonAction:

    /**
    * Lists all Thing entities return in json format
    *
    */
    public function jsonAction(Request $request)
    {
      $em = $this->getDoctrine()->getManager();
      $rep = $em->getRepository('yourBundle:Thing');
      $qb = $rep->createQueryBuilder('e');

      $limit = $request->query->get('limit');
      $current = $request->query->get('current');
      $page=$request->query->get('page');
      $queries=$request->query->get('q');
      $qarray=explode(",", $queries);

      $entities=$rep->getJSON($qarray, $page, $limit);
      $total=$rep->getJSONCount($qarray);
      $callback=$request->query->get('callback');

      return $this->render('yourBundle:Thing:json.html.twig'
         , array(
             'entities'  => $entities
            ,'callback'  => $callback
            ,'total'     => $total
         )
      );
    }

twig模板(json.html.twig,可能会自定义以显示更多内容)

    {{callback}}(
    { "objects" :
    [
    {% for entity in entities %}
    { "id": "{{entity.id}}", "text": "{{entity}}""}
    {% if not loop.last %},{% endif %}
    {% endfor %}
    ],
     "total": {{total}}
    }
    )

变压器:

    use Symfony\Component\Form\DataTransformerInterface;
    use Symfony\Component\Form\Exception\TransformationFailedException;
    use Doctrine\Common\Persistence\ObjectManager;
    use yourBundle\Entity\Thing;

    class ThingTransformer implements DataTransformerInterface
    {
        /**
         * @var ObjectManager
         */
        private $em;

        /**
         * @param ObjectManager $em
         */
        public function __construct(ObjectManager $em)
        {
            $this->em = $em;
        }

        /**
         * Transforms an object (thing) to a string (id).
         *
         * @param  Issue|null $thing
         * @return string
         */
        public function transform($thing)
        {
            if (null === $thing) {return "";}
            if (is_object($thing) && "Doctrine\ORM\PersistentCollection"==get_class($thing)){
              $entity->map(function ($ob){return $ob->getId();});
              return implode(",",$thing->toArray());
            }
            return $thing;
        }

        /**
         * Transforms a string (id) to an object (thing).
         *
         * @param  string $id
         * @return Issue|null
         * @throws TransformationFailedException if object (thing) is not found.
         */
        public function reverseTransform($id)
        {
            if (!$id) {
                return null;
            }

            //if (is_array($id)){
              $qb=$this->em
                ->getRepository('yourBundle:Thing')
                ->createQueryBuilder('t');
              $thing=$qb->andWhere($qb->expr()->in('t.id', $id))->getQuery()->getResult();

              if (null === $entity) {
                throw new TransformationFailedException(sprintf(
                    'A thing with id "%s" does not exist!',
                    $id
                ));
            }

            return $thing;
        }
    }

使用select2控件的Controller必须将'em'传递给表单构建器:

  $editForm = $this->createForm(new ThingType()
     ,$entity
     ,array(
        'attr' => array(
            'securitycontext' => $sc
           ,'em'              => $this->getDoctrine()
                                      ->getEntityManager()
        )
     )
  );

并在表单中输入:

  if (isset($options['attr']['em'])){ $em = $options['attr']['em'];} else {$em=null;}

  $transformer = new ThingTransformer($em);
  $builder->add(
      $builder->create('thing'
         ,'hidden'
         ,array(
             'by_reference' => false
            ,'required' => false
            ,'attr' => array(
                'class' => 'select2thing'
            )
         )
      )
      ->prependNormTransformer($transformer)
  );

答案 1 :(得分:1)

您可以尝试更改hydration mode,使用数组比创建对象消耗更少的内存。

您可以通过iterations使用其他方式来避免内存问题:

最后我认为你不能在不花费大量时间和内存的情况下加载所有内容,那么,为什么不进行多次查询来加载整个数据呢?