Oauth2 Cygnus和Cosmos Sink似乎不起作用

时间：2015-07-17 14:21:13

标签： oauth fiware fiware-cygnus fiware-cosmos

自上次更新以来，我无法使用Cygnus将数据上传到Cosmos。我知道我们现在需要使用Oauth2令牌来完成它。所以我做了令牌的请求。 

curl -k -X POST "https://cosmos.lab.fiware.org:13000/cosmos-auth/v1/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=password&username=guillaume.jourdain@4planet.eu&password=XXXXX"

我得到一个令牌，但后来​​我尝试检查令牌：

curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/guillaume.jourdain/hostabee?op=liststatus&user.name=guillaume.jourdain@4planet.eu" -H "X-Auth-Token: TheToken"

甚至这个：

curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/guillaume.jourdain/hostabee?op=liststatus&user.name=guillaume.jourdain" -H "X-Auth-Token: TheToken"

每次，对于这个命令中的每一个以及我试过的所有令牌，我得到了这个：

  

用户令牌未经授权

接下来我试图将oauth参数放入我的cygnus conf文件中，每次都会出现这种情况：

2015-07-17 16:17:17,797 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
2015-07-17 16:17:17,798 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The  directory could not be created in HDFS. HttpFS response: 401 Unauthorized

所以是的，目前我有点卡住了。您有什么信息可以帮我解决这个问题吗？

编辑：

这是我的Cygnus配置文件，可能问题就在这里

APACHE_FLUME_HOME/conf/cygnus.conf
orionagent.sources = http-source
orionagent.sinks = hdfs-sink
orionagent.channels = notifications

# Flume source, must not be changed
orionagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# channel name where to write the notification events
orionagent.sources.http-source.channels = notifications
# listening port the Flume source will use for receiving incoming notifications
orionagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
orionagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
# regular expression for the orion version the notifications will have in their headers
orionagent.sources.http-source.handler.orion_version = 0\.23\.*
# URL target
orionagent.sources.http-source.handler.notification_target = /notify

# channel name from where to read notification events
orionagent.sinks.hdfs-sink.channel = notifications
# Flume sink that will process and persist in HDFS the notification events, must not be changed
orionagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.OrionHDFSSink
# IP address of the Cosmos deployment where the notification events will be persisted
orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
orionagent.sinks.hdfs-sink.cosmos_port = 14000
# username allowed to write in HDFS (/user/myusername)
orionagent.sinks.hdfs-sink.cosmos_username = guillaume.jourdain
# dataset where to persist the data (/user/myusername/mydataset)

orionagent.sinks.hdfs-sink.cosmos_password = XXXXX
orionagent.sinks.hdfs-sink.cosmos_dataset = hostABee
orionagent.sinks.hdfs-sink.attr_persistence = column

orionagent.sinks.hdfs-sink.hive_host = 130.206.80.46
orionagent.sinks.hdfs-sink.hive_port = 10000
orionagent.sinks.hdfs-sink.oauth2_token = TheTOKEN


# HDFS backend type (webhdfs, httpfs or infinity)
orionagent.sinks.hdfs-sink.hdfs_api = webhdfs

# channel name
orionagent.channels.notifications.type = memory
# capacity of the channel
orionagent.channels.notifications.capacity = 1000
# amount of bytes that can be sent per transaction
orionagent.channels.notifications.transactionCapacity = 100

现在我收到此错误（和其他人）。水槽和处理程序似乎没有找到

    2015-07-27 14:27:10,562 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: hdfs-sink, type: com.telefonica.iot.cygnus.sinks.OrionHDFSSink
2015-07-27 14:27:10,562 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)] Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load sink type: com.telefonica.iot.cygnus.sinks.OrionHDFSSink, class: com.telefonica.iot.cygnus.sinks.OrionHDFSSink
    at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
    at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.sinks.OrionHDFSSink
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:190)
    at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
    ... 12 more

感谢您的阅读。

1 个答案:

答案 0 :(得分：0)

关于列出HDFS文件夹的WebHDFS命令：

curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/guillaume.jourdain/hostabee?op=liststatus&user.name=guillaume.jourdain@4planet.eu" -H "X-Auth-Token: TheToken"

user.name应为user.name=guillaume.jourdain（不包含@4planet.eu部分）。

关于Cygnus，你升级到0.8.2吗？它是唯一支持OAuth2的Cygnus版本。我猜你没有升级因为es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink日志（这些软件包以前是0.8.0）。您拥有升级here的所有详细信息。

相关问题
最新问题