Pandas MemoryError在相对较小的excel文件上使用concat

时间:2017-03-28 16:37:35

标签: python excel pandas concat

我正在合并6个xlsx文件,每个文件几MB。它们通常具有相同的数据,但有些具有一些额外的变量(我想保留它们),有些还具有相同变量的唯一名称。下面是我为尝试完成任务而编写的python代码,但是我得到了一个MemoryError(我在任务管理器中看到,当时它占用了大约2GB的内存)。考虑到文件大小,这个程序对我来说花费不到几百MB的RAM是没有意义的,所以我希望有人可以指出我的错误在哪里。

import pandas as pd
import os.path

fpath = 'K:\\DATA\\python\\Mergingtest\\tess\\'

#Specify file locations and sheet names
data_headers = pd.read_excel(os.path.join(fpath, 'TESS_Headers_2006-2015.xlsx'), sheetname='Sheet1', header=1)
data_07 = pd.read_excel(os.path.join(fpath, 'TESS 2007.xlsx'), sheetname='Errors', header=0)
data_08 = pd.read_excel(os.path.join(fpath, 'TESS 2008.xlsx'), sheetname='Errors', header=0)
data_09 = pd.read_excel(os.path.join(fpath, 'TESS 2009.xlsx'), sheetname='Errors', header=0)
data_10 = pd.read_excel(os.path.join(fpath, 'TESS 2010.xlsx'), sheetname='Sheet1', header=0)
data_11 = pd.read_excel(os.path.join(fpath, 'TESS 2011.xlsx'), sheetname='Sheet1', header=0)
data_12_15 = pd.read_excel(os.path.join(fpath, 'COPY_TESS 2012-2015 Master File.xlsx'), sheetname='Sheet1', header=0)

#Standardizing naming for equivalent variables from each file
data_headers.columns = ["fldErrorID", "fdSiteCaseID", "fdSiteID", "fdYear", "fdProvince", "fdAccessionNum", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "event1", "event2", "fdDiscoveryConsequenceEvent", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDiscoveryTime", "fdPersonInvolved", "fdTimeEventOccurred", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdDiscoverersJobSpecify", "fdProductRecordAction", "fdNoAction", "fdClosed", "fldShareWithPHAC", "fdReviewedWithPersonInvolved", "fdBloodSupplierManufacurerNotifi", "cptr_delay2", "clin_lab_err", "sitegraded", "cptr_delay", "gen_code1", "disc_cnsq1", "cnsqnc_gen", "capacity1", "capacity", "severity1", "SECTION A - DISCOVERY INFORMATION: Describe the event", "SECTION A - DISCOVERY INFORMATION: Describe how the event was discovered", "SECTION A - DISCOVERY INFORMATION: Sub-location", "SECTION B - OCCURRENCE INFORMATION: Sub-location", "SECTION B - OCCURRENCE INFORMATION: Did the actual event harm the patient?", "SECTION B - OCCURRENCE INFORMATION: If event caused harm to patient, describe", "SECTION B - OCCURRENCE INFORMATION: Was the near-miss event discovered because of a planned/unplanned detection procedure/process?", "SECTION B - OCCURRENCE INFORMATION: Discovery/Consequence code: Specify", "SECTION B - OCCURRENCE INFORMATION: Actual Severity", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 4", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 8", "SECTION B - OCCURRENCE INFORMATION: What was the outcome of the event to the patient up to 30 days after it was discovered?", "SECTION B - OCCURRENCE INFORMATION: If patient died, did the consequence of the event play a role in death?", "Date Created", "Case State", "Site ID grading", "dvcq"]
data_07.columns = ["fldErrorID", "fdSiteCaseID", "fdProvince", "fdAccessionNum", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "fdYear", "event1", "event2", "fdDiscoveryConsequenceEvent", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDiscoveryTime", "fdPersonInvolved", "fdTimeEventOccurred", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSiteID", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdDiscoverersJobSpecify", "fdProductRecordAction", "fdNoAction", "fdClosed", "fldShareWithPHAC"]
data_08.columns = ["fldErrorID", "fdSiteCaseID", "fdProvince", "fdAccessionNum", "sitegraded", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "fdYear", "event1", "gen_code1", "event2", "fdDiscoveryConsequenceEvent", "cnsqnc_gen", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDiscoveryTime", "fdPersonInvolved", "fdTimeEventOccurred", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSiteID", "Site ID grading", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdDiscoverersJobSpecify", "fdProductRecordAction", "fdNoAction", "fdClosed", "fldShareWithPHAC"]
data_09.columns = ["fldErrorID", "fdSiteCaseID", "fdProvince", "sitegraded", "fdAccessionNum", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "fdYear", "event1", "gen_code1", "event2", "fdDiscoveryConsequenceEvent", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDiscoveryTime", "fdPersonInvolved", "fdTimeEventOccurred", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSiteID", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdDiscoverersJobSpecify", "fdProductRecordAction", "fdNoAction", "fdClosed", "fldShareWithPHAC"]
data_10.columns = ["fldErrorID", "fdSiteCaseID", "fdProvince", "fdAccessionNum", "sitegraded", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "cptr_delay", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "fdYear", "event1", "gen_code1", "dvcq", "event2", "fdDiscoveryConsequenceEvent", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSiteID", "capacity1", "capacity", "severity1", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdNoAction", "fdClosed", "fdDiscoveryTime", "fdReviewedWithPersonInvolved", "fdTimeEventOccurred", "cptr_delay2", "clin_lab_err"]
data_11.columns = ["fldErrorID", "fdSiteCaseID", "fdProvince", "fdAccessionNum", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "fdYear", "event1", "event2", "fdDiscoveryConsequenceEvent", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDiscoveryTime", "fdPersonInvolved", "fdTimeEventOccurred", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSiteID", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdDiscoverersJobSpecify", "fdProductRecordAction", "fdNoAction", "fdClosed", "fldShareWithPHAC"]
data_12_15.columns = ["fldErrorID", "fdSiteCaseID", "fdSiteID", "fdYear", "fdProvince", "fdAccessionNum", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "event1", "event2", "fdDiscoveryConsequenceEvent", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDiscoveryTime", "fdPersonInvolved", "fdTimeEventOccurred", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdDiscoverersJobSpecify", "fdProductRecordAction", "fdNoAction", "fdClosed", "fldShareWithPHAC", "fdReviewedWithPersonInvolved", "fdBloodSupplierManufacurerNotifi", "cptr_delay2", "clin_lab_err", "sitegraded", "cptr_delay", "gen_code1", "disc_cnsq1", "cnsqnc_gen", "capacity1", "capacity", "severity1", "SECTION A - DISCOVERY INFORMATION: Describe the event", "SECTION A - DISCOVERY INFORMATION: Describe how the event was discovered", "SECTION A - DISCOVERY INFORMATION: Sub-location", "SECTION B - OCCURRENCE INFORMATION: Sub-location", "SECTION B - OCCURRENCE INFORMATION: Did the actual event harm the patient?", "SECTION B - OCCURRENCE INFORMATION: If event caused harm to patient, describe", "SECTION B - OCCURRENCE INFORMATION: Was the near-miss event discovered because of a planned/unplanned detection procedure/process?", "SECTION B - OCCURRENCE INFORMATION: Discovery/Consequence code: Specify", "SECTION B - OCCURRENCE INFORMATION: Actual Severity", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 4", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 8", "SECTION B - OCCURRENCE INFORMATION: What was the outcome of the event to the patient up to 30 days after it was discovered?", "SECTION B - OCCURRENCE INFORMATION: If patient died, did the consequence of the event play a role in death?", "Date Created", "Case State"]

concat = (pd.concat([data_headers,data_07,data_08,data_09,data_10,data_11,data_12_15]))
#specify column order to prevent default alphabetical-ordering
concat = concat[["fldErrorID", "fdSiteCaseID", "fdSiteID", "fdYear", "fdProvince", "fdAccessionNum", "fdDateEventOccurred", "fdDateStamp", "fdDiscoveredDuringSpecify", "fdDiscoveryDate", "fdPersonInvolvedSpecify", "fdProductRecordActionSpecify", "event1", "event2", "fdDiscoveryConsequenceEvent", "fdLocationCodeDiscovery", "fdLocationCodeOccurence", "fdDayOfWeekOccurred", "fdDiscoveredDuring", "fdDiscoveryTime", "fdPersonInvolved", "fdTimeEventOccurred", "fdDayOfWeekOccurredHoliday", "fdAdditionalTesting", "fdFloorClinicNotified", "fdOther", "fdProvincialID", "fdPatientSampleRecollected", "fdProductDestroyed", "fdProductRetrieved", "fdRecordCorrected", "fdSeverity", "fdNumberOfUnits", "fdProductType", "fdDiscovery", "fdProductDenied", "fdDiscoverersJob", "fdDiscoverersJobSpecify", "fdProductRecordAction", "fdNoAction", "fdClosed", "fldShareWithPHAC", "fdReviewedWithPersonInvolved", "fdBloodSupplierManufacurerNotifi", "cptr_delay2", "clin_lab_err", "sitegraded", "cptr_delay", "gen_code1", "disc_cnsq1", "cnsqnc_gen", "capacity1", "capacity", "severity1", "SECTION A - DISCOVERY INFORMATION: Describe the event", "SECTION A - DISCOVERY INFORMATION: Describe how the event was discovered", "SECTION A - DISCOVERY INFORMATION: Sub-location", "SECTION B - OCCURRENCE INFORMATION: Sub-location", "SECTION B - OCCURRENCE INFORMATION: Did the actual event harm the patient?", "SECTION B - OCCURRENCE INFORMATION: If event caused harm to patient, describe", "SECTION B - OCCURRENCE INFORMATION: Was the near-miss event discovered because of a planned/unplanned detection procedure/process?", "SECTION B - OCCURRENCE INFORMATION: Discovery/Consequence code: Specify", "SECTION B - OCCURRENCE INFORMATION: Actual Severity", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 1", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 2", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 3", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Product type - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Name of product: Specify - 4", "SECTION B - OCCURRENCE INFORMATION: If product destroyed, provide products: Number of units - 4", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 5", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 6", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 7", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Product type - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Name of product: Specify - 8", "SECTION B - OCCURRENCE INFORMATION: If product denied, provide products: Number of units - 8", "SECTION B - OCCURRENCE INFORMATION: What was the outcome of the event to the patient up to 30 days after it was discovered?", "SECTION B - OCCURRENCE INFORMATION: If patient died, did the consequence of the event play a role in death?", "Date Created", "Case State", "Site ID grading", "dvcq"]]

concat.to_excel(os.path.join(fpath, 'Merge_Output.xlsx'))

1 个答案:

答案 0 :(得分:0)

尝试改为

data_07 = pd.read_excel(os.path.join(fpath, 'TESS 2007.xlsx'), sheetname='Errors') 
data_08 = pd.read_excel(os.path.join(fpath, 'TESS 2008.xlsx'), sheetname='Errors')
#keep headers for all xls files

concat =pd.concat([data_07,data_08], axis = 0) 
#remove data headers and add axis = 0

#get rid of individual files
del data_07
del data_08