Question

我有用户档案的JSON数据，我希望最终使用SPSS进行分析。目前我在Google Refine中导入数据，以运行一些数据清理。然而，我的问题是原始JSON由嵌套对象组成，即例如带有“公司”的“professional_experience”部分，其中包括多个子对象/数组（参见示例）。 Google精简版通过使用该信息创建其他行来处理此问题。然而，这与“关系”（就SQL而言）视图/表结构无关，我需要使用SPSS或Excel或其他任何方式分析数据，因为还有其他子对象（学校，奖项，在高级“主”记录下面的行中也“愚蠢地”填充，但是没有直接（行/列明智）关系（考虑分析）。

正如我所看到的，我需要将这些（子对象）列和行提取到自己的表并创建一些n：m关系，或者至少将其规范化为一个表（当然接受冗余的当然其他未通过的属性）。

我想要最终得到的是一个一致的表来运行某些属性的统计分析/聚类。我认为map reduce实际上不是一个选项。

您是否有人知道如何处理此问题，或者是否可以更直接地处理JSON数据？

{ "users": [
{
  "id": "123456_abcdef",
  "first_name": "Max",
  "last_name": "Mustermann",
  "display_name": "Max Mustermann",
  "page_name": "Max_Mustermann",
  "permalink": "https://www.xing.com/profile/Max_Mustermann",
  "employment_status": "EMPLOYEE",
  "gender": "m",
  "birth_date": {
    "day": 12,
    "month": 8,
    "year": 1963
  },
  "active_email": "max.mustermann@xing.com",
  "time_zone": {
    "name": "Europe/Copenhagen",
    "utc_offset": 2.0
  },
  "premium_services": [
    "SEARCH",
    "PRIVATEMESSAGES"
  ],
  "badges": [
    "PREMIUM",
    "MODERATOR"
  ],
  "wants": "einen neuen Job",
  "haves": "viele tolle Skills",
  "interests": "Flitzebogen schießen and so on",
  "organisation_member": "ACM, GI",
  "languages": {
    "de": "NATIVE",
    "en": "FLUENT",
    "fr": null,
    "zh": "BASIC"
  },
  "private_address": {
    "city": "Hamburg",
    "country": "DE",
    "zip_code": "20357",
    "street": "Privatstraße 1",
    "phone": "49|40|1234560",
    "fax": "||",
    "province": "Hamburg",
    "email": "max@mustermann.de",
    "mobile_phone": "49|0155|1234567"
  },
  "business_address": {
    "city": "Hamburg",
    "country": "DE",
    "zip_code": "20357",
    "street": "Geschäftsstraße 1a",
    "phone": "49|40|1234569",
    "fax": "49|40|1234561",
    "province": "Hamburg",
    "email": "max.mustermann@xing.com",
    "mobile_phone": "49|160|66666661"
  },
  "web_profiles": {
    "qype": [
      "http://qype.de/users/foo"
    ],
    "google+": [
      "http://plus.google.com/foo"
    ],
    "other": [
      "http://blog.example.org"
    ],
    "homepage": [
      "http://example.org",
      "http://other-example.org"
    ]
  },
  "instant_messaging_accounts": {
    "skype": "1122334455",
    "googletalk": "max.mustermann"
  },
  "professional_experience": {
    "primary_company": {
      "id": "1_abcdef",
      "name": "XING AG",
      "title": "Softwareentwickler",
      "company_size": "201-500",
      "tag": null,
      "url": "http://www.xing.com",
      "career_level": "PROFESSIONAL_EXPERIENCED",
      "begin_date": "2010-01",
      "description": null,
      "end_date": null,
      "industry": "AEROSPACE",
      "form_of_employment": "FULL_TIME_EMPLOYEE",
      "until_now": true
    },
    "companies": [
      {
        "id": "1_abcdef",
        "name": "XING AG",
        "title": "Softwareentwickler",
        "company_size": "201-500",
        "tag": null,
        "url": "http://www.xing.com",
        "career_level": "PROFESSIONAL_EXPERIENCED",
        "begin_date": "2010-01",
        "description": null,
        "end_date": null,
        "industry": "AEROSPACE",
        "form_of_employment": "FULL_TIME_EMPLOYEE",
        "until_now": true
      },
      {
        "id": "24_abcdef",
        "name": "Ninja Ltd.",
        "title": "DevOps",
        "company_size": null,
        "tag": "NINJA",
        "url": "http://www.ninja-ltd.co.uk",
        "career_level": null,
        "begin_date": "2009-04",
        "description": null,
        "end_date": "2010-07",
        "industry": "ALTERNATIVE_MEDICINE",
        "form_of_employment": "OWNER",
        "until_now": false
      },
      {
        "id": "45_abcdef",
        "name": null,
        "title": "Wiss. Mitarbeiter",
        "company_size": null,
        "tag": "OFFIS",
        "url": "http://www.uni.de",
        "career_level": null,
        "begin_date": "2007",
        "description": null,
        "end_date": "2008",
        "industry": "APPAREL_AND_FASHION",
        "form_of_employment": "PART_TIME_EMPLOYEE",
        "until_now": false
      },
      {
        "id": "176_abcdef",
        "name": null,
        "title": "TEST NINJA",
        "company_size": "201-500",
        "tag": "TESTCOMPANY",
        "url": null,
        "career_level": "ENTRY_LEVEL",
        "begin_date": "1998-12",
        "description": null,
        "end_date": "1999-05",
        "industry": "ARTS_AND_CRAFTS",
        "form_of_employment": "INTERN",
        "until_now": false
      }
    ],
    "awards": [
      {
        "name": "Awesome Dude Of The Year",
        "date_awarded": 2007,
        "url": null
      }
    ]
  },
  "educational_background": {
    "degree": "MSc CE/CS",
    "primary_school": {
      "id": "42_abcdef",
      "name": "Carl-von-Ossietzky Universtät Schellenburg",
      "degree": "MSc CE/CS",
      "notes": null,
      "subject": null,
      "begin_date": "1998-08",
      "end_date": "2005-02"
    },
    "schools": [
      {
        "id": "42_abcdef",
        "name": "Carl-von-Ossietzky Universtät Schellenburg",
        "degree": "MSc CE/CS",
        "notes": null,
        "subject": null,
        "begin_date": "1998-08",
        "end_date": "2005-02"
      }
    ],
    "qualifications": [
      "TOEFLS",
      "PADI AOWD"
    ]
  }
}

] }

Answer 1

我不确定是否正确理解您要创建的格式，但我可以提供两个您可以开始探索的潜在客户：

使用填充

OpenRefine让您轻松fill down within records，因此存储在顶部主行中的信息会随每条记录一起传输多行。填写所需的字段后，例如，用户（id）返回行模式并使用构面，只导出数据库每个表中所需行的子集。

模板导出

template option允许您以XML / JSON格式导出数据，以便嵌套元素。但是，此功能不支持一对多关系。

如果有帮助，请告诉我们。

Answer 2

你应该能够使用gui导入JSON，在user {}对象下面选择（不在用户对象本身下），这样你就有了一些相当干净的“记录”行，它们也保存了头信息对于每个记录行，例如“__professional experience”等。

使用示例数据下载并导入此项目...确保将网格视图更改为RECORD模式。（网格的左上角）

示例OpenRefine项目显示导入为RECORD行的JSON： https://drive.google.com/file/d/0B533WzlrxWraQnF0NHN4anpFNHM/view?usp=sharing

从那里，您可以按照自己的意愿导出和更改模板，如果需要，甚至只选择“专业体验”行，以及用户ID。

如果您需要处理批量导入JSON记录行或从数据端点或Web服务导入，那么我建议下载并使用社区版本的Pentaho或Talend ETL来为您处理....但您当然可以使用OpenRefine进行初始检查和校准。（我的偏好是Pentaho，它在转换中具有出色的实时右键单击预览记录行，而Martin更喜欢Talend）。

嵌套对象为关系格式

2 个答案: