我正在抓取网站https://www.recode.net/，并在抓取工作期间显示日志：
robots.txt我没有得到任何数据来索引。
我查看了网站public class TrafficLightSimulator extends Application implements Runnable{
Circle red = new Circle();
Circle green = new Circle();
Circle yellow = new Circle();
Button b1 = new Button();
@Override
public void start(Stage stage) {
//Drawing a Rectangle
Rectangle rectangle = new Rectangle();
//grid layout
GridPane grid = new GridPane();
grid.setHgap(20);
grid.setVgap(5);
//buttons
HBox hbButtons = new HBox();
Button buttonStart = new Button("Start");
Button buttonStop = new Button("Stop");
buttonStart.setOnAction(new EventHandler<ActionEvent>() {
public void handle(ActionEvent event) {
green.setFill(Color.YELLOW);
}
});
buttonStop.setOnAction(new EventHandler<ActionEvent>() {
public void handle(ActionEvent event) {
TrafficLightSimulator tl=new TrafficLightSimulator();
Thread t1=new Thread(tl);
t1.start();
}
});
//labels and textfeilds
Label lblGreen = new Label("Green");
TextField tfGreen = new TextField("3");
Label lblYellow = new Label("Yellow");
TextField tfYellow = new TextField("3");
Label lblRed = new Label("Red");
TextField tfRed = new TextField("3");
grid.add(lblGreen, 0, 0);
grid.add(tfGreen, 1, 0);
grid.add(lblYellow, 0, 1);
grid.add(tfYellow, 1, 1);
grid.add(lblRed, 0, 2);
grid.add(tfRed, 1, 2);
grid.setPadding(new Insets(320, 5, 30, 40));
hbButtons.getChildren().addAll(buttonStart, buttonStop);
hbButtons.setAlignment(Pos.BOTTOM_CENTER);
//Setting the properties of the rectangle
rectangle.setX(150);
rectangle.setY(75);
rectangle.setWidth(400);
rectangle.setHeight(200);
rectangle.setArcHeight(50);
rectangle.setArcWidth(50);
Color c = Color.web("#404040");
Color color1 = Color.web("#404040");
Color color2 = Color.web("#808080");
Color greenColor = Color.web("#00FF00");
rectangle.setFill(c);
//setting circle properties
green.setCenterX(230);
green.setCenterY(170);
green.setRadius(50);
green.setFill(greenColor);
yellow.setCenterX(345);
yellow.setCenterY(170);
yellow.setRadius(50);
yellow.setFill(color2);
red.setCenterX(465);
red.setCenterY(170);
red.setRadius(50);
red.setFill(color2);
hbButtons.setPadding(new Insets(15, 12, 15, 12));
hbButtons.setSpacing(10); // Gap between nodes
//Creating a Group object
StackPane rootPane = new StackPane();
Pane p1 = new Pane(red, green, yellow);
Pane p2 = new Pane(rectangle);
grid.add(hbButtons, 2, 2, 2, 1);
// grid.add(grid, 2, 0, 0, 0);
rootPane.getChildren().addAll(p2, p1, grid);
//Creating a scene object
Scene scene = new Scene(rootPane, 600, 500);
//Setting title to the Stage
stage.setTitle("Drawing a Rectangle");
//Adding scene to the stage
stage.setScene(scene);
//Displaying the contents of the stage
stage.show();
}
并且没有问题，并且没有禁止我的用户代理。另外，我没有使用任何正则表达式来过滤网址。