Question

我目前正在处理一项数据处理任务。

我有两个python脚本，每个脚本都实现了一个单独的功能，但它们对相同的数据进行操作，我认为它们可以合并为一个工作流程，但我无法想到最合乎逻辑的实现方式这个。

数据文件是here，它是JSON，但它有两个不同的组件。

第一部分看起来像这样：

{
    "links": {
        "self": "http://localhost:2510/api/v2/jobs?skills=data%20science"
    },
    "data": [
        {
            "id": 121,
            "type": "job",
            "attributes": {
                "title": "Data Scientist",
                "date": "2014-01-22T15:25:00.000Z",
                "description": "Data scientists are in increasingly high demand amongst tech companies in London. Generally a combination of business acumen and technical skills are sought. Big data experience ..."
            },
            "relationships": {
                "location": {
                    "links": {
                        "self": "http://localhost:2510/api/v2/jobs/121/location"
                    },
                    "data": {
                        "type": "location",
                        "id": 3
                    }
                },
                "country": {
                    "links": {
                        "self": "http://localhost:2510/api/v2/jobs/121/country"
                    },
                    "data": {
                        "type": "country",
                        "id": 1
                    }
                },

它是由第一个python脚本处理的，在这里：

import json
from collections import defaultdict
from pprint import pprint

with open('data-science.txt') as data_file:
    data = json.load(data_file)

locations = defaultdict(int)

for item in data['data']:
    location = item['relationships']['location']['data']['id']
    locations[location] += 1

pprint(locations)

呈现此形式的数据：

这些是位置"id"以及分配给该位置的记录数。

JSON对象的另一部分如下所示：

"included": [
    {
        "id": 3,
        "type": "location",
        "attributes": {
            "name": "Victoria",
            "coord": [
                51.503378,
                -0.139134
            ]
        }
    },

并由此python文件处理：

import json
from collections import defaultdict
from pprint import pprint

with open('data-science.txt') as data_file:
    data = json.load(data_file)

locations = defaultdict(int)

for record in data['included']:
    id = record.get('id', None)
    name = record.get('attributes', {}).get('name', None)
    coord = record.get('attributes', {}).get('coord', None)
    print(id, name, coord)

它以这种格式输出数据：

3 Victoria [51.503378, -0.139134]
1 United Kingdom None
71 data science None
32 None None
3 Victoria [51.503378, -0.139134]
1 United Kingdom None
1 data mining None
22 data analysis None
33 sdlc None
38 artificial intelligence None
39 machine learning None
40 software development None
71 data science None
93 devops None
63 None None
52 Cubitt Town [51.505199, -0.018848]

我真正想要的是最终输出看起来像这样：

3, Victoria, [51.503378, -0.139134], 2673

2673从第一个脚本引用作业计数。

如果它没有任何坐标，例如[51.503378, -0.139134]我可以扔掉它。

我确信可以将这些脚本组合在一起并获得输出，但我不是一个全面的思考者，我无法弄清楚如何做到这一点。

所有真实的项目文件live here。

Answer 1

使用functions是组合这两个脚本的一种方法，毕竟它们处理相同的数据。因此，您应该为每个处理逻辑块创建一个函数，然后在最后组合结果：

import json
from collections import defaultdict
from pprint import pprint

def process_locations_data(data):
    # processes the 'data' block
    locations = defaultdict(int)
    for item in data['data']:
        location = item['relationships']['location']['data']['id']
        locations[location] += 1
    return locations

def process_locations_included(data):
    # processes the 'included' block
    return_list = []
    for record in data['included']:
        id = record.get('id', None)
        name = record.get('attributes', {}).get('name', None)
        coord = record.get('attributes', {}).get('coord', None)
        return_list.append((id, name, coord))
    return return_list    # return list of tuples

# load the data from file once
with open('data-science.txt') as data_file:
    data = json.load(data_file)

# use the two functions on same data
locations = process_locations_data(data)
records = process_locations_included(data)

# combine the data for printing
for record in records:
    id, name, coord = record
    references = locations[id]   # lookup the references in the dict
    print id, name, coord, references

该功能可以有更好的名称，但这应该实现您正在寻找的统一。

将两个python数据处理脚本组合到一个工作流程中

1 个答案: