我可以同时从同一个Kafka主题中读取多个消费者吗?

时间:2017-03-10 10:38:24

标签: apache-kafka pykafka

目前我有一个Kafka主题。

现在我需要运行多个使用者,以便可以并行读取和处理消息。

这可能。

我正在使用python和pykafka库。

                            <div class="form-group">
                              <label for="in-15" class="control-label">Listing Type</label>
                              <select id="in-15" required name="listing_type" data-placeholder="---" class="form-control">
                                <option label=" "></option>
                                <option value="Sale">Sale</option>
                                <option value="Rent">Rent</option>

                              </select>
                            </div>


                           <div class="form-group">
                              <label for="in-2" class="control-label">Property type</label>
                              <select id="in-2" required name="property_type" data-placeholder="---" class="form-control">
                                <option label=" "></option>
                                <option value="Residential">House</option>
                                <option value="SOHO">Flate</option>

                                <option value="Land">Land</option>
                              </select>
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">Price</label>
                              <input id="in-11" name="price" type="number" min="0" placeholder="" required class="form-control">
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">Area(In Marla)</label>
                              <input id="in-11" name="area" type="number" min="0" placeholder="" required class="form-control">
                            </div>



                            <div class="form-group">
                              <label for="in-11" class="control-label">No of Bedrooms</label>
                              <input id="in-11" name="beds" type="number" min="0" placeholder="" required class="form-control">
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">No of Bethrooms</label>
                              <input id="in-11" name="bathroom" type="number" min="0" placeholder="" required class="form-control">
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">No of Girages</label>
                              <input id="in-11" name="girages" type="number" min="0" placeholder="" required class="form-control">
                            </div>



                             <div id='form-block-3' class='form__block js-form-block'>
                                                  <div class='row'>
                                                    <div class='form-group form-group--description'>
                                                      <label for='in-13' class='control-label'>address</label>
                                                      <textarea name='address' id='in-13' required data-parsley-trigger='keyup' data-parsley-minlength='200' data-parsley-validation-threshold='10' data-parsley-minlength-message='You need to enter at least a 200 caracters long comment..' class='form-control form-control--description'>

                                                      </textarea>
                                                    </div>
                                                  </div>
                                                </div>

                             <div id='form-block-3' class='form__block js-form-block'>
                                                  <div class='row'>
                                                    <div class='form-group form-group--description'>
                                                      <label for='in-13' class='control-label'>Description</label>
                                                      <textarea name='desc' id='in-13' required data-parsley-trigger='keyup' data-parsley-minlength='200' data-parsley-validation-threshold='10' data-parsley-minlength-message='You need to enter at least a 200 caracters long comment..' class='form-control form-control--description'>

                                                      </textarea>
                                                    </div>
                                                  </div>
                                                </div>




                            <div class="form-group">
                              <label for="in-11" class="control-label">Image 1</label>
                              <input id="in-11" name="file1" type="file"  placeholder="" required class="form-control">
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">Image 2</label>
                              <input id="in-11" name="file2" type="file"  placeholder="" required class="form-control">
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">Image 3</label>
                              <input id="in-11" name="file3" type="file"  placeholder="" required class="form-control">
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">Image 4</label>
                              <input id="in-11" name="file4" type="file"  placeholder="" required class="form-control">
                            </div>

                            <div class="form-group">
                              <label for="in-11" class="control-label">Image 5</label>
                              <input id="in-11" name="file5" type="file"  placeholder="" required class="form-control">
                            </div>                      
                          </div>
                          <div class="row">
                            <button type="submit" name="add_adv" class="form__submit">Submit</button>
                          </div>
                        </form>



<?php

                        if(isset($_POST['add_adv']))

                            {

                              $listing_type = $_POST['listing_type'];
                              $property_type = $_POST['property_type'];
                              $beds = $_POST['beds'];
                              $bathroom = $_POST['bathroom'];
                              $price = $_POST['price'];
                              $girages = $_POST['girages'];
                              $address = $_POST['address'];
                              $area = $_POST['area'];
                              $desc = $_POST['desc'];

                              $file1 = $_FILES['file1']['name'];
                              $tmp_file1 = $_FILES['file1']['tmp_name'];

                              $file2 = $_FILES['file2']['name'];
                              $tmp_file2 = $_FILES['file2']['tmp_name'];

                              $file3 = $_FILES['file3']['name'];
                              $tmp_file3 = $_FILES['file3']['tmp_name'];

                              $file4 = $_FILES['file4']['name'];
                              $tmp_file4 = $_FILES['file4']['tmp_name'];

                              $file5 = $_FILES['file5']['name'];
                              $tmp_file5 = $_FILES['file5']['tmp_name'];

                              $loc1 = '../assets/img/'.$file1;
                              $loc2 = '../assets/img/'.$file2;
                              $loc3 = '../assets/img/'.$file3;
                              $loc4 = '../assets/img/'.$file4;
                              $loc5 = '../assets/img/'.$file5;

                              $move1 = move_uploaded_file( $tmp_file1 , $loc1 );
                              $move2 = move_uploaded_file( $tmp_file2 , $loc2 );
                              $move3 = move_uploaded_file( $tmp_file3 , $loc3 );
                              $move4 = move_uploaded_file( $tmp_file4 , $loc4 );
                              $move5 = move_uploaded_file( $tmp_file5 , $loc5 );


                              if( $move1 && $move2  && $move3  && $move4  && $move5  )


                                {
                                          echo $insert_adver = "

                                             INSERT INTO property
                                             ( propertytype , datetime , price , area , desc ,  address , bedroom , bathroom
                                                ,gerages , listingtype , seller_id , smimage , status , smimage2 , smimage3
                                                , smimage4 , smimage5 )

                                            values ( '$property_type' , NOW() ,  '$price' ,  
                                            '$area' , '$desc' , '$address' ,'$beds' 
                                            ,'$bathroom' , '$girages' ,'$listing_type' , '$_SESSION[id]' 
                                            ,'$file1' ,'0' ,'$file2' ,'$file3' ,'$file4' , '$file5')    



                                              ";

                                              $exec_this_query = mysqli_query( $con , $insert_adver );

                                              if($exec_this_query)
                                              {

                                                echo"<script>alert('successfully uploaded ! Please wait for Approval from Admin')</script>";  
                                              }

                                              else
                                              {

                                                  echo"<script>alert('Inner FAILED !')</script>";


                                              }

                                }

                                else

                                    {


                                        echo"<script>alert('FAILED !')</script>";


                                    }





                            }








                    ?>

在两个消费者中都采用相同的信息。我只需要处理一次消息。

2 个答案:

答案 0 :(得分:2)

您需要使用BalancedConsumer代替SimpleConsumer

consumer = topic.get_balanced_consumer(consumer_group=b"charlie",
                                       auto_commit_enable=True)

您还应确保您所使用的主题至少与您要实例化的消费者数量一样多。

答案 1 :(得分:-1)

是的,如果您使用相同的使用者组ID并且主题的分区数应大于使用者,则可以让多个使用者并行读取同一主题,否则将不会为某些使用者分配任何分区和那些消费者不会获取任何数据