Question

Initial situation

Lets say we have a dictionary storing time-series data in form of:

dic = {'M15': 
        { 
            '100001': { 0: [0,1,2,...],
                        1: [0,1,2,...]
                    },
            '100002': { 0: [0,1,2,...],
                        1: [0,1,2,...]
                    },
                    ...
        },
        'H1': {
            '200001': { 0: [0,1,2,...],
                        1: [0,1,2,...]
                    },
            ...
        },
        ...
}

Now, lets assume this dictionary is stored within a class called data like this:

class data:

    def __init__(self, input: dict):
        self.data = input

newData = data(dic)

As may be obvious, this class shall store the time-series data and return it within an iteratotion for further processing at some point.

My questions

I want to make the class iterable, meaning __next__ shall iterate through all the data within the dictionary (the upcoming question is not about how to iterate over a nested dictionary, so please do not answer this). Data means I only need the arrays at the lowest level within the dictionary, e.g. [0,1,2,...].

Lets assume the data within the dictionary is quite huge - it can fit within memory, but it shall not be duplicated. Therefore, list comprehension is not an option as far as I know because the data would also be stored within this new list besides the dictionary (the dictionary is still needed and an array is not an option in this example). For the sake of completeness, this would look like:

class data:
    def __init__(self, input: dict):
        self.dictionary = input
        self.data  = [series_array for series_key, series_array in series.items() for ... in self.dictionary.items()]
        self.index = 0
    def __iter__(self):
        return self
    def __next__(self):
        self.index += 1
        return self.data[self.index - 1]

Question 1:

Would the list comprehension just point to the data within the dictionary or would it really copy the data?

This means I would have to use a normal iteration over the dictionary, but I cannot think of a way to implement this in __iter__and __next__.

Question 2:

How would I implement this nested dictionary-loop within __iter__and __next__?

Please note that I am seeking an answer for this concrete question and not on "why don't use generators" or "why don't do it this/that way".

Answer 1

Question 1:

Would the list comprehension just point to the data within the dictionary or would it really copy the data?

It will hold a reference to the lists in the dictionary

Question 2:

How would I implement this nested dictionary-loop within __iter__and __next__?

You just need to return an iterator in __iter__ (instead of having the list for example), in this case a the generator expression within your list should be enough:

class Data:
    def __init__(self, input: dict):
        self.dictionary = input
    def __iter__(self):
        return (series_array for series_key, series_array in series.items() for ... in self.dictionary.items())

Iterator-class for nested dictionaries

1 个答案: