使用python 3标准化elasticsearch json响应

时间:2018-06-28 09:27:03

标签: python json pandas elasticsearch

我正在尝试使用python标准化来自elasticsearch的json响应。这是一个文档示例:

    [{'_index': 'prod-sdm-profile-v2',
  '_type': 'profile',
  '_id': 'd7de8fd6-5f02-4013-87e8-7a42edf4463c',
  '_score': None,
  '_source': {'profile': {'addresses': [{'zipCode': '40134',
      'country': 'ITA',
      'locality': 'BOLOGNA',
      'addressLine1': 'VIA IRMA BANDIERA',
      'location': {'lon': 11.3387, 'lat': 44.4938},
      'addressLine2': None,
      'addressLine3': None,
      'type': 'ResidentialAddress',
      'region': 'EMILIA ROMAGNA'}],
    'deceased': None,
    'gender': 'Female',
    'lastModifiedDate': '2018-06-26T12:32:22+02:00',
    'citizenship': [{'country': 'IT',
      'nationalId': None,
      'passport': None,
      'drivingLicense': None,
      'taxCode': 'DMBMRM77T49Z114T',
      'vatNumber': None}],
    'dataResponsibleDealers': [{'name': None, 'id': 'Dealers/000244'}],
    'givenName': 'mariam',
    'honorificPrefix': None,
    'type': 'Lead',
    'birthDate': None,
    'phoneNumbers': None,
    'authPrivacy': False,
    'emails': [{'address': 'm.deambrogio@gmail.com', 'type': 'Email1'}],
    'honorificSuffix': None,
    'estimatedAnnualTrip': None,
    'isBusiness': False,
    'familyName': 'deambrogio',
    'middleName': None,
    'createDate': '2018-06-26T07:12:24.4443921+00:00'},
   'salesOpportunities': [{'firstAppointmentDate': None,
     'campaignSources': [{'sourceTimeStamp': '2018-06-26T09:12:51.2263824+02:00',
       'clusterTags': None,
       'eventTags': {'campaignID': 'walk_in', 'campaignName': 'Walk In'}}],
     'lastModifiedDate': '2018-06-28T09:12:56+02:00',
     'currentStatus': 'Closed',
     'dueDate': '2018-06-28T09:12:51+02:00',
     'lostProspectCampaignData': None,
     'negotiationDate': '2018-06-26T09:12:51.2263824+02:00',
     'isActive': False,
     'closingDate': '2018-06-26T09:17:41+02:00',
     'engageStartDate': '2018-06-26T09:12:51.3045236+02:00',
     'dealerBranchOpportunities': [{'firstAppointmentDate': None,
       'branchLocation': None,
       'dealerCode': '000244',
       'dueDate': '2018-06-28T09:12:51+02:00',
       'engageTimeWorkingHours_Seconds': None,
       'dealerBranchOpportunityId': 'DealerOpportunities/afdbb86d-8b5e-4927-8344-3936ec2f48c7',
       'contracts': [{'totalNetPaid': 17078.0,
         'predeliveredDate': None,
         'tradeIn': [{'extendedWarranty': None,
           'accessories': None,
           'id': 'Vehicles/b11e1fa2-27bc-4a00-bd70-21f7fe42a172',
           'packs': None,
           'maintenancePrograms': None,
           'vehicle': {'kilowatt': 0.0,
            'seriesRevisionDateFrom': None,
            'upholsteryCode': None,
            'engineType': 'Gasoline',
            'subCarLineCode': None,
            'versionName': 'NISSAN Micra 1.2 16V 65CV 3 porte Visia',
            'fullDescription': 'NISSAN Micra 3ª serie NISSAN Micra 1.2 16V 65CV 3 porte Visia',
            'licensePlate': 'cy814vf',
            'modelCode': '3370',
            'brand': 'NISSAN',
            'fullKatashikis': None,
            'mileage': 110000,
            'gradeName': None,
            'firstMatriculationDate': '2005-11-09T00:00:00+01:00',
            'optionals': None,
            'subCarLineName': None,
            'versionCode': '66240',
            'engineCapacity': None,
            'doors': 3,
            'seriesRevision': None,
            'modelName': 'Micra 3ª serie',
            'series': None,
            'gradeCode': None,
            'upholsteryName': None,
            'seriesStartingDate': None},
           'mileages': {'currentMileage': 110000,
            'detections': [{'timeStamp': '2018-06-26T09:16:08+02:00',
              'mileage': 110000}]}}],
         'contractNumber': '000244180600074',
         'totalNetPrice': 16918.87,
         'lastPrintDate': None,
         'type': 'NewCar',
         'versionNumber': 1,
         'products': {'financialPlan': None,
          'serviceExtendedWarranty': None,
          'accessories': [{'code': 'KITINS000000536',
            'name': 'TOUCH 2 WITH GO YARIS LISTINO 750 PROMO 550 EURO',
            'description': 'Toshiko.Web.ViewModels.CarConfigurator.AccessoryViewModel',
            'netPrice': 450.819672}],
          'dealerCustoms': None,
          'serviceDealerCustoms': None,
          'insurancePublicPlan': None,
          'packs': [],
          'serviceMaintenancePackage': None,
          'vehicle': {'kilowatt': 54.0,
           'seriesRevisionDateFrom': '2018-05-01T00:00:00+02:00',
           'upholsteryCode': '20',
           'engineType': 'Hybrid',
           'subCarLineCode': None,
           'versionName': 'YARIS 1.5H ECVT 5P ACTIVE MY18',
           'fullDescription': 'Yaris 5p 1.5 HSD (100 CV) E-CVT Hybrid Active',
           'licensePlate': None,
           'modelCode': 'YARIS',
           'brand': 'T',
           'fullKatashikis': ['NHP130L-CHXNBW'],
           'mileage': 0,
           'gradeName': 'Hybrid Active',
           'firstMatriculationDate': None,
           'optionals': None,
           'subCarLineName': None,
           'versionCode': 'Y15HSTACT4',
           'engineCapacity': '1.5',
           'doors': 5,
           'seriesRevision': 'MY18',
           'modelName': 'Yaris',
           'series': 3,
           'gradeCode': 'HYBRID-ACTIVE',
           'upholsteryName': None,
           'seriesStartingDate': None}},
         'totalGrossPaid': 20800.0,
         'lastContractSigningDate': '2018-06-25T00:00:00+02:00',
         'isDeleted': False,
         'firstContractSigningDate': '2018-06-25T00:00:00+02:00',
         'verifiedDate': '2018-06-26T12:39:17+02:00',
         'contractId': 'Contracts/c80bece5-177e-493b-819c-75450c113ca5',
         'totalDiscount': 0.0,
         'signedDate': '2018-06-25T00:00:00+02:00',
         'settledDate': None}],
       'brand': 'T',
       'outcome': 'Sold',
       'createDate': '2018-06-26T09:12:51.3045236+02:00',
       'branchCommercialName': None,
       'campaignSources': None,
       'dealerName': 'Toy Motor S.r.l.',
       'lastModifiedDate': '2018-06-26T12:32:22+02:00',
       'currentStatus': 'Closed',
       'branchName': None,
       'engageEndDate': '2018-06-26T09:12:51.3045236+02:00',
       'negotiations': [{'negotiationId': 'Offers/ccc32e34-f6dd-4f7a-937f-7de3afe865fd',
         'negotiationDescription': None,
         'salesQuotes': [{'totalNetPaid': 17077.848852,
           'totalGrossPaid': 20800.0,
           'isDeleted': False,
           'lastModifiedDate': '2018-06-26T09:17:41+02:00',
           'tradeIn': [{'extendedWarranty': None,
             'accessories': None,
             'id': 'Vehicles/b11e1fa2-27bc-4a00-bd70-21f7fe42a172',
             'packs': None,
             'maintenancePrograms': None,
             'vehicle': {'kilowatt': 0.0,
              'seriesRevisionDateFrom': None,
              'upholsteryCode': None,
              'engineType': 'Gasoline',
              'subCarLineCode': None,
              'versionName': 'NISSAN Micra 1.2 16V 65CV 3 porte Visia',
              'fullDescription': 'NISSAN Micra 3ª serie NISSAN Micra 1.2 16V 65CV 3 porte Visia',
              'licensePlate': 'cy814vf',
              'modelCode': '3370',
              'brand': 'NISSAN',
              'fullKatashikis': None,
              'mileage': 110000,
              'gradeName': None,
              'firstMatriculationDate': '2005-11-09T00:00:00+01:00',
              'optionals': None,
              'subCarLineName': None,
              'versionCode': '66240',
              'engineCapacity': None,
              'doors': 3,
              'seriesRevision': None,
              'modelName': 'Micra 3ª serie',
              'series': None,
              'gradeCode': None,
              'upholsteryName': None,
              'seriesStartingDate': None},
             'mileages': {'currentMileage': 110000,
              'detections': [{'timeStamp': '2018-06-26T09:16:08+02:00',
                'mileage': 110000}]}}],
           'totalNetPrice': 16918.868852,
           'totalDiscount': None,
           'lastPrintDate': None,
           'type': None,
           'quoteId': 'Quotes/e09c66fa-2497-4902-8dac-aba01796ca9b',
           'products': {'financialPlan': None,
            'serviceExtendedWarranty': None,
            'accessories': [{'code': 'KITINS000000536',
              'name': 'TOUCH 2 WITH GO YARIS LISTINO 750 PROMO 550 EURO',
              'description': 'Installation Kit',
              'netPrice': 450.819672}],
            'serviceDealerCustoms': None,
            'insurancePublicPlan': None,
            'packs': None,
            'serviceMaintenancePackage': None,
            'vehicle': {'kilowatt': 54.0,
             'seriesRevisionDateFrom': '2018-05-01T00:00:00+02:00',
             'upholsteryCode': '20',
             'engineType': 'Hybrid',
             'subCarLineCode': None,
             'versionName': 'YARIS 1.5H ECVT 5P ACTIVE MY18',
             'fullDescription': 'Yaris 5p 1.5 HSD (100 CV) E-CVT Hybrid Active',
             'licensePlate': None,
             'modelCode': 'YARIS',
             'brand': 'T',
             'fullKatashikis': ['NHP130L-CHXNBW'],
             'mileage': 0,
             'gradeName': 'Hybrid Active',
             'firstMatriculationDate': None,
             'optionals': None,
             'subCarLineName': None,
             'versionCode': 'Y15HSTACT4',
             'engineCapacity': '1.5',
             'doors': 5,
             'seriesRevision': 'MY18',
             'modelName': 'Yaris',
             'series': 3,
             'gradeCode': 'HYBRID-ACTIVE',
             'upholsteryName': None,
             'seriesStartingDate': None}},
           'createDate': '2018-06-26T09:12:51.2263824+02:00'}],
         'createDate': None}],
       'closingDate': '2018-06-26T09:17:41+02:00',
       'engageStartDate': None,
       'branchCode': '000244',
       'engageTime_Seconds': None,
       'currentPhase': None,
       'todos': [{'lastModifiedDate': '2018-06-26T09:17:41+02:00',
         'endDate': '2018-06-28T09:12:51.3826338+02:00',
         'firstActionWorkingHours_Seconds': None,
         'dueDate': '2018-06-28T23:59:59+02:00',
         'description': 'Follow up Offer N. 2018-000244-002065',
         'workflowLoopInstance': None,
         'firstAction_Seconds': None,
         'type': 'CloseOffer',
         'hasDueDateReachedLostProspect': False,
         'originalDueDate': None,
         'firstActionEvaluationDescription': None,
         'numberOfPostpositions': 0,
         'id': '928cba2d-28cc-4e23-ab6c-6a01ca4dcb08',
         'status': 'Canceled',
         'createDate': '2018-06-26T09:12:51.3826338+02:00'}],
       'negotiationStartDate': '2018-06-26T09:12:51.2263824+02:00'}],
     'lostProspectInfo': None,
     'currentPhase': None,
     'outcome': 'Sold',
     'salesOpportunityId': 'Opportunities/6933f863-4a30-45d3-957a-fd287a6e8363',
     'createDate': '2018-06-26T09:12:51.3045236+02:00'}]},
  'sort': [7672066]}

我感兴趣的只是“ _source”字段的一部分。 我只需要得到这样的输出:

 profile.addresses.zipcde|profile.addresses.country|profile.addresses.location....|...salesOpportunities.firstAppointmentDate|salesOpportunities.campaignSources.sourceTimeStamp|
0                   40134                      ITA                     BOLOGNA                                           None                  2018-06-26T09:12:51.2263824+02:00                                       
1                    ....                      ....                    ....                                              ....                                               ....
2                    ....                      ....                    ....                                              ....                                               .... 

对于嵌套字段,必须对所有出现的嵌套字段重复父列的值。

我已经读了几天有关该主题的主题,但是我找不到任何适合我的东西(或者也许我无法应用阅读建议)。

可能我必须在循环中使用json_normalize,但是我无法编写正确的代码

请问是否有人可以向我展示逐步实现我的目标的程序。

对不起,我的英语不好。

预先感谢您的回答

0 个答案:

没有答案