Question

我正在尝试创建一个Web抓取程序，该程序将从网站中获取表格并将其转换为“ .csv”文件。

我正在使用Jsoup将数据提取到文档中，并从下面的document.html（） doc.html（）中读取。站在原位的读者可以在我的测试站点拿起18张桌子，但没有桌子数据标签。

您知道什么地方可能出问题吗？

ArrayList<Data_Log> container = new ArrayList<Data_Log>();
ArrayList<ListData_Log> containerList = new ArrayList<ListData_Log>();
ArrayList<String> tableNames = new ArrayList<String>();// Stores native names of tables
ArrayList<Double> meanStorage = new ArrayList<Double>();// Stores data mean per table
ArrayList<String> processlog = new ArrayList<String>();// Keeps a record of all actions taken per iteration
ArrayList<Double> modeStorage = new ArrayList<Double>();
Calendar cal;

private static final long serialVersionUID = -8174362940798098542L;

public void takeData() throws IOException {
    if (testModeActive == true) {
        System.out.println("Initializing Data Cruncher with developer logs");
        System.out.println("Taking data from: " + dataSource);      }
    int irow = 0;
    int icolumn = 0;
    int iTable = 0;
    // int iListno = 0;
    // int iListLevel;

    String u = null;
    boolean recording = false;
    boolean duplicate = false;
    Document doc = Jsoup.connect(dataSource).get();
    Webtitle = doc.title();
    Pattern tb = Pattern.compile("<table");
    Matcher tB = tb.matcher(doc.html());
    Pattern ttl = Pattern.compile("<title>(//s+)</title>");
    Matcher ttl2= ttl.matcher(doc.html());
    Pattern tr = Pattern.compile("<tr");
    Matcher tR = tr.matcher(doc.html());
    Pattern td = Pattern.compile("<td(//s+)</td>");
    Matcher tD = td.matcher(doc.html());
    Pattern tdc = Pattern.compile("<td class=(//s+)>(//s+)</td>");
    Matcher tDC = tdc.matcher(doc.html());
    Pattern tb2 = Pattern.compile("</table>");
    Matcher tB2 = tb2.matcher(doc.html());
    Pattern th = Pattern.compile("<th");
    Matcher tH = th.matcher(doc.html());
    while (tB.find()) {
        iTable++;

        while(ttl2.find()) {
        tableNames.add(ttl2.group(1));
        }
        while (tR.find()) {

            while (tD.find()||tH.find()) {
                u = tD.group(1);
                Data_Log v = new Data_Log();
                v.setTable(iTable);
                v.dataSort(u);
                v.setRow(irow);
                v.setColumn(icolumn);
                container.add(v);
                icolumn++;
            }
            while(tDC.find()) {
                u = tDC.group(2);
                Data_Log v = new Data_Log();
                v.setTable(iTable);
                v.dataSort(u);
                v.setRow(irow);
                v.setColumn(icolumn);
                container.add(v);
                icolumn++;
            }
            irow++;
        }

        if (tB2.find()) {
        irow=0;
        icolumn=0;
        }       
    }

预期结果：

table# logged + "td"s logged

实际结果：

table# logged "td"s省略

Answer 1

由于您正在使用jsoup，请使用它

var url = "<your url>";
var doc = Jsoup.connect(url).get();
var tables = doc.body().getElementsByTag("table");
tables.forEach(table -> {
    System.out.println(table.id());
    System.out.println(table.className());  
    System.out.println(table.getElementsByTag("td"));
});

在尝试使用正则表达式解析html时，建议阅读以下内容

Using regular expressions to parse HTML: why not?

Why is it such a bad idea to parse XML with regex?

RegEx match open tags except XHTML self-contained tags

如何从表格内部获取HTML数据？

1 个答案: