我正在尝试找出问题，但到目前为止，我找不到任何解决方案，希望您可以提供帮助。
我有一个DataFrame，我想将str转换为datatime，但是有一些无效的行要过滤掉。这是两个示例：
Out[6]:
# name date
0 aa 2012-11-30T14:00:00+01:00
1 bb 2012-12-01T08:16:00+01:00
2 cc 2012-12-01T10:14:00+01:00
3 ee 2012-12-01T11:05:00+01:00
4 gg 2012-12-01T11:05:00+01:00
In [7]: df2
Out[7]:
# name date
0 aa 2012-11-30T14:00:00+01:00
1 bb 2012-12-01T08:16:00+01:00
2 cc 2012-12-01T10:14:00+01:00
3 ee 2012-12-01T11:05:00+01:00
4 ff fsadfi2 2ih3ro
5 gg 2012-12-01T11:05:00+01:00
In [11]: df.dtypes
Out[11]:
name <class 'str'>
date <class 'str'>
dtype: object
In [12]: df2.dtypes
Out[12]:
name <class 'str'>
date <class 'str'>
dtype: object
df我很好，它在date列中只有有效日期。但是df2有一些无效的行。
让我们首先来看df，并把下面的行转换成datetime：
df['pdate']=df.date.values.astype('datetime64[ns]')
效果很好：
In [16]: df
Out[16]:
# name date pdate
0 aa 2012-11-30T14:00:00+01:00 2012-11-30 13:00:00.000000000
1 bb 2012-12-01T08:16:00+01:00 2012-12-01 07:16:00.000000000
2 cc 2012-12-01T10:14:00+01:00 2012-12-01 09:14:00.000000000
3 ee 2012-12-01T11:05:00+01:00 2012-12-01 10:05:00.000000000
4 gg 2012-12-01T11:05:00+01:00 2012-12-01 10:05:00.000000000
In [17]: df.dtypes
Out[17]:
name <class 'str'>
date <class 'str'>
pdate datetime64[ns]
dtype: object
现在，我尝试使用非常简单的str.contains ::
In [18]: df2_filtered=df2[df2['date'].str.contains(':00')]
In [19]: df2_filtered
Out[19]:
# name date
0 aa 2012-11-30T14:00:00+01:00
1 bb 2012-12-01T08:16:00+01:00
2 cc 2012-12-01T10:14:00+01:00
3 ee 2012-12-01T11:05:00+01:00
4 gg 2012-12-01T11:05:00+01:00
In [20]: df2_filtered.dtypes
Out[20]:
name <class 'str'>
date <class 'str'>
dtype: object
只有5 Rows。
现在，我尝试进行转换，并收到一条不错的错误消息：
In [21]: df2_filtered['pdate']=df2_filtered.date.values.astype('datetime64[ns]')
...:
/usr/local/bin/ipython:1: DeprecationWarning: parsing timezone aware datetimes is deprecated; this will raise an error in the future
#!/opt/local/Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-21-563087d6f949> in <module>
----> 1 df2_filtered['pdate']=df2_filtered.date.values.astype('datetime64[ns]')
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py in __setitem__(self, name, value)
4370 if isinstance(name, six.string_types):
4371 if isinstance(value, (np.ndarray, Column)):
-> 4372 self.add_column(name, value)
4373 else:
4374 self.add_virtual_column(name, value)
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py in add_column(self, name, data, dtype)
5743 # self._length_original = len(data)
5744 # self._index_end = self._length_unfiltered
-> 5745 super(DataFrameArrays, self).add_column(name, data, dtype=dtype)
5746 self._length_unfiltered = int(round(self._length_original * self._active_fraction))
5747 # self.set_active_fraction(self._active_fraction)
/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py in add_column(self, name, f_or_array, dtype)
2872 # give a better warning to avoid confusion
2873 if len(self) == len(ar):
-> 2874 raise ValueError("Array is of length %s, while the length of the DataFrame is %s due to the filtering, the (unfiltered) length is %s." % (len(ar), len(self), self.length_unfiltered()))
2875 raise ValueError("array is of length %s, while the length of the DataFrame is %s" % (len(ar), self.length_original()))
2876 # assert self.length_unfiltered() == len(data), "columns should be of equal length, length should be %d, while it is %d" % ( self.length_unfiltered(), len(data))
ValueError: Array is of length 5, while the length of the DataFrame is 5 due to the filtering, the (unfiltered) length is 6.
说： ValueError：数组的长度为5，而由于过滤，DataFrame的长度为5，（未过滤的）长度为6。
但是据我在df2_filtered中所了解的，我只有5行。我不知道为什么df2中有多少行很重要。
基本上我的问题是如何过滤掉不必要的数据并将列转换为Datetime？
更新
基于Maarten Breddels，我尝试使用：
df2_filtered['pdate']=df2_filtered.date.astype('datetime64[ns]')
这似乎可行，但是当我尝试使用df2_filtered时，我得到以下提示。
In [57]: df2_filtered
Out[57]: ERROR:MainThread:vaex:error evaluating: pdate at rows 0-5
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
result = self[expression]
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 141, in __getitem__
raise KeyError("Unknown variables or column: %r" % (variable,))
KeyError: 'Unknown variables or column: "astype(date, \'datetime64[ns]\')"'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 3467, in table_part
values[name] = df.evaluate(name)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5038, in evaluate
dtype = dtypes[expression] = self.dtype(expression, internal=False)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 2005, in dtype
data = self.evaluate(expression, 0, 1, filtered=False, internal=True, parallel=False)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5143, in evaluate
value = scope.evaluate(expression)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
result = self[expression]
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 136, in __getitem__
self.values[variable] = self.evaluate(expression) # , out=self.buffers[variable])
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 100, in evaluate
result = eval(expression, expression_namespace, self)
File "<string>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/functions.py", line 2106, in _astype
return x.astype(dtype)
AttributeError: 'ColumnStringArrow' object has no attribute 'astype'
ERROR:MainThread:vaex:error evaluating: pdate at rows 0-5
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
result = self[expression]
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 141, in __getitem__
raise KeyError("Unknown variables or column: %r" % (variable,))
KeyError: 'Unknown variables or column: "astype(date, \'datetime64[ns]\')"'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 3467, in table_part
values[name] = df.evaluate(name)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5038, in evaluate
dtype = dtypes[expression] = self.dtype(expression, internal=False)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 2005, in dtype
data = self.evaluate(expression, 0, 1, filtered=False, internal=True, parallel=False)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/dataframe.py", line 5143, in evaluate
value = scope.evaluate(expression)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 94, in evaluate
result = self[expression]
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 136, in __getitem__
self.values[variable] = self.evaluate(expression) # , out=self.buffers[variable])
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/scopes.py", line 100, in evaluate
result = eval(expression, expression_namespace, self)
File "<string>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/vaex/functions.py", line 2106, in _astype
return x.astype(dtype)
AttributeError: 'ColumnStringArrow' object has no attribute 'astype'
# name date pdate
0 aa 2012-11-30T14:00:00+01:00 error
1 bb 2012-12-01T08:16:00+01:00 error
2 cc 2012-12-01T10:14:00+01:00 error
3 ee 2012-12-01T11:05:00+01:00 error
4 gg 2012-12-01T11:05:00+01:00 error
答案 0 :(得分：1)
vaex主要作者在这里。很好的问题，什么halumpago是正确的，df2_filtered内部仍然是6行长。您尝试执行以下操作：
# Adds a numpy arrays to the dataframe
df2_filtered['pdate'] = df2_filtered.date.values.astype('datetime64[ns]')
是将长度为5的数组（即.values返回给您）添加到内部具有一堆长度为6的数组的DataFrame中。这是错误消息试图传达的内容。如果错误尚不清楚，请通过https://github.com/vaexio/vaex/issues告诉我们。从理论上讲，我们可以支持这一点，但是我们必须创建一个长度为6的数组，并在其中复制数据。当您处理10亿行时，这将不是理想的选择。
在vaex中，最好不要使用基础数组（好的，有时您需要它们，因此这就是我们支持.values和朋友的原因）。取而代之的是，表达式系统尝试模拟尽可能接近的Pandas系列/ numpy数组。如果删除.values，则将新的虚拟列而不是数组添加到DataFrame：
# Adds a virtual column (backed by an expression) to the dataframe
# at zero memory cost
df2_filtered['pdate']=df2_filtered.date.astype('datetime64[ns]')
Vaex将愉快地存储该表达式，并仅对尚未过滤掉的行进行评估。请注意，如果您在此新列前后打印出DataFrame的大小（以字节为单位），则其内存使用情况是相同的：
print(df2.nbytes)
648
注意：我们忽略了虚拟列簿记的内存使用，这当然是几个字节，但是当您使用100 GB的数据时，可以忽略不计。）
只是为了好玩，要打破vaex DataFrame与Pandas数据帧相同的幻想，您实际上可以删除过滤器：
print(df2_filtered.drop_filter())
# name date pdate
0 aa 2012-11-30T14:00:00+01:00 2012-11-30 13:00:00.000000000
1 bb 2012-12-01T08:16:00+01:00 2012-12-01 07:16:00.000000000
2 cc 2012-12-01T10:14:00+01:00 2012-12-01 09:14:00.000000000
3 ee 2012-12-01T11:05:00+01:00 2012-12-01 10:05:00.000000000
4 ff fsadfi2 2ih3ro NaT
5 gg 2012-12-01T11:05:00+01:00 2012-12-01 10:05:00.000000000
因此数据实际上从未真正消失过，我们只是对您隐藏了它：）。
这使您可以使用庞大的Vaex DataFrame，并添加许多新列，进行大量过滤，并且仍然没有MemoryError。我们一直在引用原始数据，只是保留用于过滤器的掩码，以及用于计算的表达式。
答案 1 :(得分：0)
IIUC，pd.to_datetime，可让您使用某些关键字参数将列转换为DateTime。在这种情况下，您需要errors='coerce'
print(df)
name date
0 aa 2012-11-30T14:00:00+01:00
1 bb 2012-12-01T08:16:00+01:00
2 cc 2012-12-01T10:14:00+01:00
3 ee 2012-12-01T11:05:00+01:00
4 ff fsadfi22ih3ro
5 gg 2012-12-01T11:05:00+01:00
df['date'] = pd.to_datetime(df['date'],errors='coerce')
print(df)
name date
0 aa 2012-11-30 14:00:00+01:00
1 bb 2012-12-01 08:16:00+01:00
2 cc 2012-12-01 10:14:00+01:00
3 ee 2012-12-01 11:05:00+01:00
4 ff NaT
5 gg 2012-12-01 11:05:00+01:00
现在只需在设置日期列的同时用.dropna()删除行即可。
df.dropna(subset=['date'])
print(df)
name date
0 aa 2012-11-30 14:00:00+01:00
1 bb 2012-12-01 08:16:00+01:00
2 cc 2012-12-01 10:14:00+01:00
3 ee 2012-12-01 11:05:00+01:00
5 gg 2012-12-01 11:05:00+01:00
print(df.dtypes)
name object
date datetime64[ns, pytz.FixedOffset(60)]
dtype: object
答案 2 :(得分：0)
不幸的是，我没有一个完整的答案，但是对于您的问题的这一部分，我可能有个主意：
我不知道df2中有多少行为什么很重要。
这很重要，因为据我所知，vaex通过存储定义该列的操作来构造新列（看起来像它们称为“虚拟列”）。相比之下，熊猫通过计算，存储和复制新列的实际值来构造新列。
Pandas确实很难存储，但是在移动数据时具有很大的灵活性。您不会在Virtual Columns中拥有这种灵活性，但是您的程序在内存使用方面可能会更好。
看看正在执行过滤的行：
df2_filtered=df2[df2['date'].str.contains(':00')]
与熊猫不同，df2_filtered不是内存中的实际“事物”。相反，它是对原始df2数据帧的引用以及一些额外的逻辑，这些逻辑告诉vaex忽略不以'：00'结尾的任何内容。
因此，当您运行时：
df2_filtered['pdate']=df2_filtered.date.values.astype('datetime64[ns]')
您实际上是在要求vaex在df2中创建新列，因为df2_filtered实际上只是对df2的过滤后的引用。 vaex不知道如何处理从df2中过滤出来的行，因此会引发您所看到的错误。
因此，要进行转换，您需要某种方式来填充df2的缺失值。不幸的是，我对vaex不够熟悉，无法提供帮助。我尝试了@datanovice的方法，但是vaex抱怨说它不知道如何处理NaT值。