使用Streamz从熊猫DataFrame流式传输

时间:2019-08-05 09:51:19

标签: pandas hvplot holoviz

streamzrcpp.cpp:8:24: error: use of overloaded operator '[]' is ambiguous (with operand types 'Rcpp::Vector<19, PreserveStorage>::const_Proxy' (aka 'const_generic_proxy<19, PreserveStorage>') and 'int') Rcout << some_list[1][0] << "\n"; ~~~~~~~~~~~~^~ rcpp.cpp:8:24: note: built-in candidate operator[](struct SEXPREC *, long) rcpp.cpp:8:24: note: built-in candidate operator[](const struct SEXPREC *, long) rcpp.cpp:8:24: note: built-in candidate operator[](volatile struct SEXPREC *, long) rcpp.cpp:8:24: note: built-in candidate operator[](const volatile struct SEXPREC *, long) 1 error generated. 软件包一起工作,为使用pandas数据帧绘制流数据提供支持。

例如,hvplot软件包具有用于创建随机流数据帧的便捷实用程序:

streamz

可以使用import hvplot.streamz from streamz.dataframe import Random sdf = Random(interval='200ms', freq='50ms') sdf # Stop the streaming with: sdf.stop() 在流式图表中简单地绘制它:

hvplot

是否有一种简单的方法来从预先存在的sdf.hvplot() 数据帧中流式传输数据?

例如,我想说些类似的话:

pandas

然后,我可以使用先前存在的import pandas as pd df=pd.DataFrame({'a':range(0,100),'b':range(5,105)}) sdf = StreamingDataFrame(df, interval='200ms', freq='50ms') 数据帧中的示例数据,而不必使用随机的示例数据。

1 个答案:

答案 0 :(得分:0)

据我所知...

from streamz import Stream
from streamz.dataframe import DataFrame

from time import sleep

from datetime import datetime

#Set up a source stream
source = Stream()

#Create a sample pandas dataframe
samples = pd.DataFrame({'x':[0],'y':[0]})

#The streaming dataframe takes the source stream and sample pandas dataframe
#The sample defines the dataframe schema, maybe?
sdf = DataFrame(source, example=samples)

def stest(r):
    print(datetime.now())
    print(r)

#I don't recall what this does
#I think what I was looking to do was display the last 3 items...?
# ...which this doesn't appear to do!
df = sdf.window(3).full()

#This seems to set a callback on stest when a stream element appears
df.stream.sink(stest)


for i in range(5):

    #pull the next item in the streaming dataframe into the stream
    #We could iloc on an existing dataframe?
    source.emit(pd.DataFrame({'x': [i,i,i], 'y' :[i,i,i]}))

    #Pause for a short while...
    sleep(2)


--------
2020-01-27 19:13:06.816315
   x  y
0  0  0
1  0  0
2  0  0
2020-01-27 19:13:08.824016
   x  y
0  1  1
1  1  1
2  1  1
2020-01-27 19:13:10.829178
   x  y
0  2  2
1  2  2
2  2  2
2020-01-27 19:13:12.835948
   x  y
0  3  3
1  3  3
2  3  3
2020-01-27 19:13:14.843432
   x  y
0  4  4
1  4  4
2  4  4

....

Ah ..找到了一个看起来效果更好的示例。

设置我们要流式传输的数据框:

import pandas as pd
from time import sleep
from datetime import datetime

from streamz import Stream
from streamz.dataframe import DataFrame


#Set up a dummy dataframe
ddf=pd.DataFrame({'a':range(1,5),'b':range(11,15)})
print(ddf)

---
   a   b
0  1  11
1  2  12
2  3  13
3  4  14

并流式传输...

#Create a stream source
source = Stream()

#Create a dataframe model for the stream
samples = pd.DataFrame({'a':[0],'b':[0]})

#Creating the streaming dataframe
sdf = DataFrame(source, example=samples)

#That window thing again...
df = sdf.window(4).full()

#FUnction to run when an item appears on the stream
def stest(r):
    print(datetime.now())
    print(r)

#Add the callback function to streamer
df.stream.sink(stest)

#This does stream from the dataframe
for i in range(len(ddf)):
    source.emit(ddf.iloc[i])
    sleep(2)

df
---
2020-01-27 19:28:05.503123
a     1
b    11
Name: 0, dtype: int64
2020-01-27 19:28:07.505536
a     1
b    11
a     2
b    12
dtype: int64
2020-01-27 19:28:09.508925
a     2
b    12
a     3
b    13
dtype: int64
2020-01-27 19:28:11.514117
a     3
b    13
a     4
b    14
dtype: int64

我还发现了另一种获取数据以流向全息图的方式:

from tornado import gen
from tornado.ioloop import PeriodicCallback

from holoviews.streams import Buffer

import holoviews as hv
hv.extension('bokeh')


import numpy as np
import pandas as pd

df = pd.DataFrame({'x':range(1000), 'y':np.sin(range(1000))})

rowcount = 0
maxrows = 1000

dfbuffer = Buffer(np.zeros((0, 2)), length=20)

@gen.coroutine
def g():
    global rowcount
    item = df[['x','y']].iloc[rowcount].values.tolist()
    dfbuffer.send(np.array([item]))
    rowcount += 1

    if rowcount>=maxrows:
        cbdf.stop()



#How can we get the thing to stop?

cbdf = PeriodicCallback(g, 500)
cbdf.start()
hv.DynamicMap(hv.Curve, streams=[dfbuffer]).opts(padding=0.1, width=600, color = 'green',)

然后使用cbdf.stop()停止它(只有当我尝试使用它时,这似乎才对我不起作用...)

我似乎还没有一个将streamz组件连接到图表的示例(除非holoviews在`下面使用streamz?)