Question

我遇到处理超过2700个文件的问题这有用，如果我有一些文件，如几百，我猜它与Windows限制打开文件，如Linux ulimit可以定义系统范围。我确信事情没有被关闭，这就是我收到这个错误的原因。

我有一个通过post发送文件的函数：

def upload_photos(url_photo, dict, timeout):
    photo = dict['photo']
    data_photo = dict['data']
    name = dict['name']
    conn = requests.post(url_photo, data=data_photo, files=photo, timeout=timeout)
    return {'json': conn.json(), 'name': name}

从目录列表的循环中调用：

for photo_path in [p.lower() for p in photos_path]:
        if ('jpg' in photo_path or 'jpeg' in photo_path) and "thumb" not in photo_path:
            nr_photos_upload +=1
    print("Found " + str(nr_photos_upload) + " pictures to upload")
    local_count = 0
    list_to_upload = []
    for photo_path in [p.lower() for p in photos_path]:
        local_count += 1
        if ('jpg' in photo_path or 'jpeg' in photo_path) and "thumb" not in photo_path and local_count > count:
            total_img = nr_photos_upload
            photo_name = os.path.basename(photo_path)
            try :
                photo = {'photo': (photo_name, open(path + photo_path, 'rb'), 'image/jpeg')}
                try:
                    latitude, longitude, compas = get_gps_lat_long_compass(path + photo_path)
                except ValueError as e:
                    if e != None:
                        try:
                            tags = exifread.process_file(open(path + photo_path, 'rb'))
                            latitude, longitude = get_exif_location(tags)
                            compas = -1
                        except Exception:
                            continue
                if compas == -1:
                    data_photo = {'coordinate'    : str(latitude) + "," + str(longitude),
                               'sequenceId'       : id_sequence,
                               'sequenceIndex'    : count
                               }
                else :
                    data_photo = {'coordinate'    : str(latitude) + "," + str(longitude),
                               'sequenceId'       : id_sequence,
                               'sequenceIndex'    : count,
                               'headers'          : compas
                               }
                info_to_upload = {'data': data_photo, 'photo':photo, 'name': photo_name}
                list_to_upload.append(info_to_upload)
                count += 1
            except Exception as ex:
                print(ex)
    count_uploaded = 0
    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
        # Upload feature called from here
        future_to_url = {executor.submit(upload_photos, url_photo, dict, 100): dict for dict in list_to_upload}
        for future in concurrent.futures.as_completed(future_to_url):
            try:
                data = future.result()['json']
                name = future.result()['name']
                print("processing {}".format(name))
                if data['status']['apiCode'] == "600":

                    percentage = float((float(count_uploaded) * 100) / float(total_img))
                    print(("Uploaded - " + str(count_uploaded) + ' of total :' + str(
                        total_img) + ", percentage: " + str(round(percentage, 2)) + "%"))
                elif data['status']['apiCode'] == "610":
                    print("skipping - a requirement arguments is missing for upload")
                elif data['status']['apiCode'] == "611":
                    print("skipping - image does not have GPS location metadata")
                elif data['status']['apiCode'] == "660":
                    print("skipping - duplicate image")
                else :
                    print("skipping - bad image")
                count_uploaded += 1
                with open(path + "count_file.txt", "w") as fis:
                    fis.write((str(count_uploaded)))
            except Exception as exc:
                print('%generated an exception: %s' % (exc))

Answer 1

您可以在C中设置var a = "{a,b} xyz {v}"; a = a.split(" "); a[0] //outupt {a,b} a[2] //output {v}以更改一次可以打开的文件数。

对于python，您必须使用_setmaxstdio中的win32file作为：

pywin32

默认值为import win32file win32file._setmaxstdio(1024) #set max number of files to 1024。并确保您检查平台支持的最大值。

参考：https://msdn.microsoft.com/en-us/library/6e3b887c.aspx

打开文件urllib太多了

1 个答案: