Question

按时间戳字段过滤数据帧的正确方法是什么？

我尝试了不同的日期格式和过滤形式，没有任何帮助：pyspark返回0个对象，或者抛出一个它不理解日期时间格式的错误

这是我到目前为止所得到的：

from pyspark import SparkContext
from pyspark.sql import SQLContext

from django.utils import timezone
from django.conf import settings

from myapp.models import Collection

sc = SparkContext("local", "DjangoApp")
sqlc = SQLContext(sc)
url = "jdbc:postgresql://%(HOST)s/%(NAME)s?user=%(USER)s&password=%(PASSWORD)s" % settings.DATABASES['default']
sf = sqlc.load(source="jdbc", url=url, dbtable='myapp_collection')

时间戳字段的范围：

system_tz = timezone.pytz.timezone(settings.TIME_ZONE)
date_from = datetime.datetime(2014, 4, 16, 18, 30, 0, 0, tzinfo=system_tz)
date_to = datetime.datetime(2015, 6, 15, 18, 11, 59, 999999, tzinfo=system_tz)

尝试1

date_filter = "my_col >= '%s' AND my_col <= '%s'" % (
    date_from.isoformat(), date_to.isoformat()
)
sf = sf.filter(date_filter)
sf.count()

Out[12]: 0

尝试2

sf = sf.filter(sf.my_col >= date_from).filter(sf.my_col <= date_to)
sf.count()

---------------------------------------------------------------------------
Py4JJavaError: An error occurred while calling o63.count.
: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 4.0 failed 1 times, most recent failure: 
Lost task 0.0 in stage 4.0 (TID 3, localhost): org.postgresql.util.PSQLException: 
ERROR: syntax error at or near "18"
# 
# ups.. JDBC doesn't understand 24h time format??

尝试3

sf = sf.filter("my_col BETWEEN '%s' AND '%s'" % \
     (date_from.isoformat(), date_to.isoformat())
     )
---------------------------------------------------------------------------
Py4JJavaError: An error occurred while calling o97.count.
: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 17.0 failed 1 times, most recent failure:
Lost task 0.0 in stage 17.0 (TID 13, localhost): org.postgresql.util.PSQLException:
ERROR: syntax error at or near "18"

表格中确实存在数据：

django_filters = {
    'my_col__gte': date_from,
    'my_col__lte': date_to
    }
Collection.objects.filter(**django_filters).count()

Out[17]: 1093436

或者这样

django_range_filter = {'my_col__range': (date_from, date_to)}
Collection.objects.filter(**django_range_filter).count()

Out[19]: 1093436

Answer 1

让我们假设您的数据框如下所示：

sf = sqlContext.createDataFrame([
    [datetime.datetime(2013, 6, 29, 11, 34, 29)],
    [datetime.datetime(2015, 7, 14, 11, 34, 27)],
    [datetime.datetime(2012, 3, 10, 19, 00, 11)],
    [datetime.datetime(2016, 2, 8, 12, 21)],
    [datetime.datetime(2014, 4, 4, 11, 28, 29)]
], ('my_col', ))

with schema：

root
 |-- my_col: timestamp (nullable = true)

您希望在以下范围内找到日期：

import datetime, time 
dates = ("2013-01-01 00:00:00",  "2015-07-01 00:00:00")

timestamps = (
    time.mktime(datetime.datetime.strptime(s, "%Y-%m-%d %H:%M:%S").timetuple())
    for s in dates)

可以使用在驱动程序端计算的时间戳进行查询：

q1 = "CAST(my_col AS INT) BETWEEN {0} AND {1}".format(*timestamps)
sf.where(q1).show()

或使用unix_timestamp函数：

q2 = """CAST(my_col AS INT)
        BETWEEN unix_timestamp('{0}', 'yyyy-MM-dd HH:mm:ss')
        AND unix_timestamp('{1}', 'yyyy-MM-dd HH:mm:ss')""".format(*dates)

sf.where(q2).show()

也可以像我在another answer中描述的那样使用udf。

如果您使用原始SQL，则可以使用year，date等提取时间戳的不同元素。

sqlContext.sql("""SELECT * FROM sf
    WHERE YEAR(my_col) BETWEEN 2014 AND 2015").show()

修改：

从Spark 1.5开始，您可以使用内置函数：

dates = ("2013-01-01", "2015-07-01") date_from, date_to = [to_date(lit(s)).cast(TimestampType()) for s in dates] sf.where((sf.my_col > date_from) & (sf.my_col < date_to))

您还可以使用pyspark.sql.Column.between，其中包含范围：

from pyspark.sql.functions import col sf.where(col('my_col').between(*dates)).show(truncate=False) #+---------------------+ #|my_col | #+---------------------+ #|2013-06-29 11:34:29.0| #|2014-04-04 11:28:29.0| #+---------------------+

Answer 2

这样的事情怎么样：

$scope.reset = function () {
        $scope.riskData.cuit = angular.copy($scope.defaultCuit);
        $window.alert('Hey! You clicked!');
    };
    $scope.reset(); // <--- You're calling it here
    $scope.comboData = {

Answer 3

以下内容似乎对我有用（不过有人告诉我这是格式错误还是不正确）...

首先，为窗口的每一端创建一个新列（在此示例中，此列在column_name列中的日期之后100天到200天。

from pyspark.sql import functions as F
new_df = new_df.withColumn('After100Days', F.lit(F.date_add(new_df['column_name'], 100)))
new_df = new_df.withColumn('After200Days', F.lit(F.date_add(new_df['column_name'], 200)))

过滤如下...

要过滤特定范围内的日期：

result= df.where((df.col1> df.col2) & (df.col1 < df.col3))

对于过滤特定范围之外的日期：

result= df.where((df.col1 < df.col2) | (df.col1 > df.col3))

PySpark SQL中的日期时间范围过滤器

3 个答案: