我如何实时将数据从arduino传感器存储到hadoop hdfs

时间:2019-05-22 17:47:56

标签: python hadoop arduino hdfs bigdata

我正在一个涉及使用hdfs的项目中,我想将arduino数据每3s存入一个csv文件中的hadoop hdfs中。

csv文件示例:

“ temp1”,“ datetime1”,“ location1”

“ temp2”,“ datetime2”,“ location2”

“ temp3”,“ datetime3”,“ location3”

我想每3秒向此csv文件添加一行。

我已经尝试了从arduino的串行端口读取并写入nosql数据库的python代码,我尝试执行相同的操作,但是在hdfs路径上发现了一些问题。

# Creating a simple Pandas DataFrame
liste_temp = [temp_string,datetime.datetime.now(),temperature_location]
df = pd.DataFrame(data = {'temp' : liste_temp})
# Writing Dataframe to hdfs
with client_hdfs.write('/test/temp.csv', encoding = 'utf-8') as writer:
                df.to_csv(writer)

错误:

File "templog.py", line 33, in <module> with client_hdfs.write('/test/temp.csv', encoding = 'utf-8') as writer: File "C:\Users\nouhl\AppData\Local\Programs\Python\Python37-32\lib\site-packages\hdfs\client.py", line 460, in write raise
InvalidSchema("No connection adapters were found for '%s'" % url) requests.exceptions.InvalidSchema: No connection adapters were found for 'hdfs://localhost:9870/webhdfs/v1/test/temp.csv

0 个答案:

没有答案