Question

我正在尝试创建一个包含大量股票的数据框，我最终会将其发送到MySQL数据库。我需要获取所有单独的数据帧并将它们连接在一起，使它们的名称和日期保持唯一我目前遇到的问题是代码的连接部分引发了错误，我尝试了合并但是这样做会丢失每个数据框的名称值因此不符合我的需要。我也研究过使用面板，但我读到.to_sql函数仅用于数据帧。任何帮助将不胜感激。

exchList =['A','AA','AAL','AAP','AAPL','ABBV','ABC','ABT','ACN','ADBE','ADI','ADM','ADP','ADS','ADSK','AEE','AEP']
main_df = pd.DataFrame()
start = datetime.datetime(2000,1,1)
end =  datetime.date.today()



for ticker in exchList:
   df = web.DataReader(ticker, "yahoo",start, end)
   df.reset_index(level=df.index.names, inplace=True)
   if main_df.empty:
       main_df = df
   else:
       main_df = main_df.join(df)

错误如下。

ValueError: columns overlap but no suffix specified: Index(['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'], dtype='object')

Answer 1

有一种更优雅的方式 - 将所有代码的数据一步到<

In [126]: p = web.DataReader(exchList, "yahoo",start,

In [129]: p.to_frame() Out[129]: Open        High         Low       Close Date       minor 2000-01-03 A       78.749999   78.937500   67.374999   72.000003 AAPL   104.874997  112.499998  101.687501  111.937502 ABC     15.500000   15.750000   15.250000   15.562500 ABT     35.249948   35.999945   34.749947   34.999948 ADBE    67.250000   67.500000   64.250000   65.562500 ADI     93.500000   93.875000   88.000000   90.187500 ADM     11.999999   12.062499   11.875000   11.999999 ADP     53.499906   53.937406   51.937409   51.999911 ADSK    34.000000   34.625000   32.125000   33.375000 AEE     32.562500   32.625000   31.562500   32.312500 ...                      ...         ...         ...         ... 2017-02-23 ABT     45.029999   45.509998   44.849998   45.400002 ACN    122.589996  122.709999  121.730003  122.480003 ADBE   120.099998  120.150002  118.029999  118.830002 ADI     82.150002   82.160004   81.029999   81.610001 ADM     44.799999   45.270000   44.490002   45.090000 ADP    100.790001  101.779999  100.489998  101.639999 ADS    240.589996  243.520004  239.279999  242.419998 ADSK    86.690002   87.370003   85.919998   87.099998 AEE     54.230000   54.270000   53.689999   54.070000 AEP     65.550003   66.089996   65.309998   66.010002

[63153 rows x 6 columns]

您可能还想重置多索引： code>Pandas.Panel，然后flatten，Panel到DataFrame：< / p> end) Volume Adj Close 4674300.0 46.106304 133949200.0 3.625643 2784800.0 3.297376 10635000.0 9.517434 7384400.0 16.274673 3655600.0 32.584012 984600.0 7.798824 2698800.0 28.858381 2845600.0 8.052905 700800.0 13.102718 ... ... 9389100.0 45.400002 1428000.0 122.480003 2381700.0 118.830002 2277500.0 81.610001 3256200.0 45.090000 1459300.0 101.639999 650800.0 242.419998 1368000.0 87.099998 1438100.0 54.070000 2272900.0 66.010002

In [130]: p.to_frame().reset_index()
Out[130]:
            Date minor        Open        High         Low       Close       Volume   Adj Close
0     2000-01-03     A   78.749999   78.937500   67.374999   72.000003    4674300.0   46.106304
1     2000-01-03  AAPL  104.874997  112.499998  101.687501  111.937502  133949200.0    3.625643
2     2000-01-03   ABC   15.500000   15.750000   15.250000   15.562500    2784800.0    3.297376
3     2000-01-03   ABT   35.249948   35.999945   34.749947   34.999948   10635000.0    9.517434
4     2000-01-03  ADBE   67.250000   67.500000   64.250000   65.562500    7384400.0   16.274673
5     2000-01-03   ADI   93.500000   93.875000   88.000000   90.187500    3655600.0   32.584012
6     2000-01-03   ADM   11.999999   12.062499   11.875000   11.999999     984600.0    7.798824
7     2000-01-03   ADP   53.499906   53.937406   51.937409   51.999911    2698800.0   28.858381
8     2000-01-03  ADSK   34.000000   34.625000   32.125000   33.375000    2845600.0    8.052905
9     2000-01-03   AEE   32.562500   32.625000   31.562500   32.312500     700800.0   13.102718
...          ...   ...         ...         ...         ...         ...          ...         ...
63143 2017-02-23   ABT   45.029999   45.509998   44.849998   45.400002    9389100.0   45.400002
63144 2017-02-23   ACN  122.589996  122.709999  121.730003  122.480003    1428000.0  122.480003
63145 2017-02-23  ADBE  120.099998  120.150002  118.029999  118.830002    2381700.0  118.830002
63146 2017-02-23   ADI   82.150002   82.160004   81.029999   81.610001    2277500.0   81.610001
63147 2017-02-23   ADM   44.799999   45.270000   44.490002   45.090000    3256200.0   45.090000
63148 2017-02-23   ADP  100.790001  101.779999  100.489998  101.639999    1459300.0  101.639999
63149 2017-02-23   ADS  240.589996  243.520004  239.279999  242.419998     650800.0  242.419998
63150 2017-02-23  ADSK   86.690002   87.370003   85.919998   87.099998    1368000.0   87.099998
63151 2017-02-23   AEE   54.230000   54.270000   53.689999   54.070000    1438100.0   54.070000
63152 2017-02-23   AEP   65.550003   66.089996   65.309998   66.010002    2272900.0   66.010002

[63153 rows x 8 columns]

Answer 2

此错误表示您尝试加入的两个数据帧具有共同的列名。要正确连接两者，您需要指定完整的连接方法DataFrame.join(other, on=None, how='left', lsuffix='', rsuffix='', sort=False)

因此您需要指定左后缀或右后缀。例如

df1.columns = ['A','B']
df2.columns = ['B','C']

当你将两者连接在一起时，如果你没有加入'B'，那么你需要指定一个后缀来添加到任一数据帧的列名中，以便在连接的数据框中没有重复的列名

See here for the join method documentation

Answer 3

我认为你想要将表连接在一起，而不是加入它们。为此，请将代码更改为以下内容。

   if main_df.empty:
       main_df = df
   else:
       main_df = pd.concate([main_df,df])

更好的方法是获得所有帧的列表，然后在最后将它们连接起来。

import pandas as pd
l_dfs = list()
for ticker in exchList:
   df = web.DataReader(ticker, "yahoo",start, end)
   df.reset_index(level=df.index.names, inplace=True)
   l_dfs.append(df)
df = pd.concate(l_dfs)

连接pandas数据帧时出错

3 个答案: