Question

我是新手来激发流媒体。我想在线流式传输网址以便从某个网址检索信息，我使用了JavaCustomReceiver来传输网址。

这是我使用的代码（source）

public class JavaCustomReceiver extends Receiver<String> {

    private static final Pattern SPACE = Pattern.compile(" ");

    public static void main(String[] args) throws Exception {

        SparkConf sparkConf = new SparkConf().setAppName("JavaCustomReceiver");
        JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, new Duration(1000));

        JavaReceiverInputDStream<String> lines = ssc.receiverStream(
            new JavaCustomReceiver("http://stream.meetup.com/2/rsvps", 80));

        JavaDStream<String> words = lines.flatMap(new 

              FlatMapFunction<String, String>() {

                 @Override
                 public Iterator<String> call(String x) {
                     return Arrays.asList(SPACE.split(x)).iterator();
                 }
              });

        JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
              new PairFunction<String, String, Integer>() {

                 @Override
                 public Tuple2<String, Integer> call(String s) {
                        return new Tuple2<>(s, 1);
                 }
              }).reduceByKey(new Function2<Integer, Integer, Integer>() {
                @Override
                public Integer call(Integer i1, Integer i2) {
                    return i1 + i2;
                }
            });

    wordCounts.print();
    ssc.start();
    ssc.awaitTermination();
}

String host = null;
int port = -1;

public JavaCustomReceiver(String host_, int port_) {
    super(StorageLevel.MEMORY_AND_DISK_2());
    host = host_;
    port = port_;
}

public void onStart() {

    new Thread() {
        @Override
        public void run() {
            receive();
        }
    }.start();
}

public void onStop() {

}


private void receive() {
    try {
        Socket socket = null;
        BufferedReader reader = null;
        String userInput = null;
        try {
            // connect to the server
            socket = new Socket(host, port);
            reader = new BufferedReader(
                    new InputStreamReader(socket.getInputStream(), StandardCharsets.UTF_8));
            // Until stopped or connection broken continue reading
            while (!isStopped() && (userInput = reader.readLine()) != null) {
                System.out.println("Received data '" + userInput + "'");
                store(userInput);
            }
        } finally {
            Closeables.close(reader, /* swallowIOException = */ true);
            Closeables.close(socket, /* swallowIOException = */ true);
        }

        restart("Trying to connect again");
    } catch (ConnectException ce) {
        // restart if could not connect to server
        restart("Could not connect", ce);
    } catch (Throwable t) {
        restart("Error receiving data", t);
    }
}
  }

但是，我不断收到java.net.UnknownHostException

我该如何解决这个问题？我使用的代码有什么问题？

Answer 1

在读取所引用的自定义接收器的代码之后，很明显它是连接到host:port的TCP接收器，而不是可以接收URL的HTTP接收器。您必须将代码更改为从HTTP端点读取。

Spark-Streaming CustomReceiver未知主机异常

1 个答案: