Question

我正在尝试从pypi中提取pip包的许可证信息，然后加载到pandas数据帧中。之前我做了一个例子，将列表推导加载到PD。但我无法弄清楚这一个...

到目前为止，我写过。

from requests import get

import pandas as pd

import pip

url = 'https://pypi.python.org/pypi'

# packages_list = ['numpy','twisted']

installed_packages = pip.get_installed_distributions()
installed_packages_list = sorted(["%s==%s" % (i.key, i.version)
     for i in installed_packages])

packages = []
licenses = []
summarys = []

for index, package in enumerate(installed_packages_list):
    package = package.split("==")[0]
    full_url = url+'/'+ package +'/json'
    #print 'url is ' + full_url
    page = get(url+'/'+package+'/json').json()


    #print 'Package: ' + package + ', license is:' + page['info']['license'] + '. ' + page['info']['summary']
    packages.append(package)
    licenses.append(page['info']['license'])
    summarys.append(page['info']['summary'])


print packages


pd_packages = pd.DataFrame(
    {
    "packages":[packages],
    "licenses":[licenses],
    "summarys":[summarys]
    })

print pd_packages

Answer 1

试试这个：

def get_pkg_info(pkg, url_pat='https://pypi.python.org/pypi/{}/json'):
    r = requests.get(url_pat.format(pkg))
    if r.status_code != requests.codes.ok:
         return [pkg, None, None]
    d = r.json()
    if d and 'info' in d:
        return [pkg, d['info'].get('license'), d['info'].get('summary')]
    else:
         return [pkg, None, None]

data = [get_pkg_info(x.split('==')[0]) for x in installed_packages_list]

df = pd.DataFrame(data, columns=['package','license','summary'])

演示：

In [166]: pd.options.display.max_rows = 15

In [167]: df = pd.DataFrame(data, columns=['package','license','summary'])

In [168]: df
Out[168]:
                package       license                                            summary
0             alabaster          None        A configurable sidebar-enabled Sphinx theme
1       anaconda-client       UNKNOWN         Anaconda Cloud command line client library
2    anaconda-navigator   Proprietary
3      anaconda-project          None                                               None
4            asn1crypto           MIT  Fast ASN.1 parser and serializer with definiti...
5               astroid          LGPL  A abstract syntax tree for Python with inferen...
6               astropy           BSD         Community-developed python astronomy tools
..                  ...           ...                                                ...
216              xarray        Apache          N-D labeled arrays and datasets in Python
217                xlrd           BSD  Library for developers to extract data from Mi...
218          xlsxwriter           BSD     A Python module for creating Excel XLSX files.
219             xlwings  BSD 3-clause  Make Excel fly: Interact with Excel from Pytho...
220                xlwt           BSD  Library to create spreadsheet files compatible...
221           xmltodict           MIT  Makes working with XML feel like you are worki...
222               yapsy           BSD                          Yet another plugin system

[223 rows x 3 columns]

Answer 2

我认为问题源于您的DataFrame（pd_packages）的创建。软件包，许可证和汇总已经是列表，所以这样做[packages]会使它成为一个列表，列出了下面评论中的输出。

所以不是这个

pd_packages = pd.DataFrame(
    {
    "packages":[packages],
    "licenses":[licenses],
    "summarys":[summarys]
    })

试试这个

pd.DataFrame(
    {
    "packages":packages,
    "licenses":licenses,
    "summarys":summarys
    })

将数据加载到pandas中

2 个答案: