我希望有人能帮助我解决这个问题。
我有一个python脚本，可从html文档中的标记中查找文本，并且需要以以下格式将其写入到csv文件中：
Nessus，ID，文本
但是现在我得到一条垂直线：（
我的代码：
from bs4 import BeautifulSoup
import re
import csv
with open('xx01_med_dansk') as fp:
soup = BeautifulSoup(fp, 'html5lib')
# print(soup.prettify())
with open('dk_snip.csv', 'w') as f:
wr = csv.writer(f)
var1 = soup.find('li', text = re.compile('Scan vendor:'), attrs = {'class' : 'property_name'})
var2 = soup.find('li', text = re.compile('Vendor ID:'), attrs = {'class' : 'property_name'})
vendor = var1.find_next('li')
final = vendor.string
vend_id = var2.find_next('li')
final2 = vend_id.text
for dk_desc in soup.find_all(re.compile("textarea")):
final3 = dk_desc.text
to_csv = final+final2+final3
wr.writerows(to_csv)
我不太确定如何格式化数据以正确输入csv文件。
HTML文件：
<!DOCTYPE html>
<html lang="en">
<head>
<li class="property_name">
<label for="id_194-description">
Description:
</label>
</li>
<li class="property_value">
<textarea class="mceNoEditor" cols="40" id="id_194-description" name="194-description" rows="10" style="width:100%">According to its version, the installation of Oracle Database on the remote host is no longer supported.
Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities.</textarea>
</li>
<li class="property_name">
<label for="id_194-consequence">
Consequence:
</label>
</li>
<li class="property_value">
<textarea class="mceNoEditor" cols="40" id="id_194-consequence" name="194-consequence" rows="10" style="width:100%">The remote host is running an unsupported version of a database server.</textarea>
</li>
<li class="property_name">
<label for="id_194-solution">
Solution:
</label>
</li>
<li class="property_value">
<textarea class="mceNoEditor" cols="40" id="id_194-solution" name="194-solution" rows="10" style="width:100%">Upgrade to a version of Oracle Database that is currently supported.</textarea>
</li>
<li class="property_name">
<label for="id_194-cve_id">
Cve id:
</label>
</li>
<li class="property_value">
<textarea class="mceNoEditor" cols="40" id="id_194-cve_id" maxlength="8192" name="194-cve_id" rows="10" style="width:100%; height:80px"></textarea>
</li>
<input id="id_194-override" name="194-override" type="hidden" value="11953"/>
<input id="id_194-priority" name="194-priority" type="hidden"/>
<li class="property_name">
Vulnerability priority
</li>
<li class="property_value">
<select name="prio_194">
<option selected="selected" value="0">
0
</option>
</select>
: Oracle Database Unsupported (Nessus)
<br/>
</li>
<li class="property_name">
Save
</li>
<li class="property_value">
<input type="submit" value="Save vulnerability changes"/>
</li>
</ul>
</form>
<br style="clear:both"/>
</div>
<div class="box">
<h4>
Related vulnerabilities
</h4>
<hr/>
<h5>
Oracle Database Unsupported (Nessus)
</h5>
<ul>
<li class="property_name">
Description
</li>
<li class="property_value">
According to its version, the installation of Oracle Database on the remote host is no longer supported.
<br/>
<br/>
Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities.
</li>
<li class="property_name">
Consequence
</li>
<li class="property_value">
The remote host is running an unsupported version of a database server.
</li>
<li class="property_name">
Solution
</li>
<li class="property_value">
Upgrade to a version of Oracle Database that is currently supported.
</li>
</ul>
<br style="clear:both"/>
</div>
<div class="box">
<h4>
Create new snippet
</h4>
<form action="/report/vulnerabilityEditor/?
action=edit&
id=194&
sid=&
model=snippet" method="POST">
<ul>
<li class="property_name">
<label for="id_language">
Language:
</label>
</li>
<li class="property_value">
<select id="id_language" name="language" style="width:100%">
<option selected="" value="1">
Danish (DK)
</option>
<option value="2">
English (EN)
</option>
<option value="3">
Icelandic (IS)
</option>
</select>
</li>
<input id="id_vulnerability" name="vulnerability" type="hidden" value="194"/>
<li class="property_name">
<label for="id_title">
Title:
</label>
</li>
<li class="property_value">
<input id="id_title" maxlength="100" name="title" style="width:100%" type="text"/>
</li>
<li class="property_name">
<label for="id_recommendation">
Recommendation:
</label>
</li>
<li class="property_value">
<input id="id_recommendation" maxlength="255" name="recommendation" style="width:100%" type="text"/>
</li>
<li class="property_name">
<label for="id_snippet">
Snippet:
</label>
</li>
<li class="property_value">
<textarea cols="40" id="id_snippet" name="snippet" rows="10" style="width:100%"></textarea>
</li>
<li class="property_name">
Scan type
</li>
<li class="property_value">
<select multiple="multiple" name="scan_type" size="6" style="width:100%">
<option selected="selected" value="5">
COMPANY PCI
</option>
<option selected="selected" value="7">
Other
</option>
<option selected="selected" value="8">
Firewall Audit
</option>
<option selected="selected" value="6">
Penetration Test
</option>
<option selected="selected" value="9">
WIFI Test
</option>
<option selected="selected" value="10">
APP Test
</option>
<option selected="selected" value="1">
External Security Analysis
</option>
<option selected="selected" value="2">
Internal Security Analysis
</option>
<option selected="selected" value="3">
Web Application Test
</option>
<option selected="selected" value="4">
Host Discovery Analysis
</option>
</select>
-- Use ctrl to mark multiple types
</li>
<li class="property_name">
Save
</li>
<li class="property_value">
<input type="submit" value="Save new snippet"/>
</li>
</ul>
<br style="clear:both;"/>
</form>
</div>
<div class="box">
<h4>
Edit snippets
</h4>
<input id="property_vulnerability_id" type="hidden" value="194"/>
<input id="property_url_filter_snippets" type="hidden" value="/report/filterSnippets/"/>
<ul>
<li class="property_name">
Language
</li>
<li class="property_value">
<select id="language" name="language">
<option value="0">
All
</option>
<option value="1">
Danish
</option>
<option value="2">
English
</option>
<option value="3">
Icelandic
</option>
</select>
</li>
<li class="property_name">
Scan Type
</li>
<li class="property_value">
<select id="scantype" name="scantype">
<option value="0">
All
</option>
<option value="5">
COMPANY PCI
</option>
<option value="7">
Other
</option>
<option value="8">
Firewall Audit
</option>
<option value="6">
Penetration Test
</option>
<option value="9">
WIFI Test
</option>
<option value="10">
APP Test
</option>
<option value="1">
External Security Analysis
</option>
<option value="2">
Internal Security Analysis
</option>
<option value="3">
Web Application Test
</option>
<option value="4">
Host Discovery Analysis
</option>
</select>
</li>
</ul>
<br style="clear:both;"/>
<div class="snippet">
<form action="/report/vulnerabilityEditor/?action=edit&id=194&sid=1290&model=snippet" method="POST">
<input id="id_1290-vulnerability" name="1290-vulnerability" type="hidden" value="194"/>
<hr/>
<ul>
<li class="property_name">
<label for="id_1290-language">
Language:
</label>
</li>
<li class="property_value">
<select id="id_1290-language" name="1290-language" style="width:100%">
<option value="1">
Danish (DK)
</option>
<option selected="" value="2">
English (EN)
</option>
<option value="3">
Icelandic (IS)
</option>
</select>
</li>
<li class="property_name">
<label for="id_1290-title">
Title:
</label>
</li>
<li class="property_value">
<input id="id_1290-title" maxlength="100" name="1290-title" style="width:100%" type="text" value="Oracle Database Unsupported"/>
</li>
<li class="property_name">
<label for="id_1290-recommendation">
Recommendation:
</label>
</li>
<li class="property_value">
<input id="id_1290-recommendation" maxlength="255" name="1290-recommendation" style="width:100%" type="text" value="Upgrade to a version of Oracle Database that is currently supported."/>
</li>
<li class="property_name">
<label for="id_1290-snippet">
Snippet:
</label>
</li>
<li class="property_value">
<a href="https://cyberopswiki/index.php/How_to:_Add_figure_number_in_snippet" target="_blank">
How to: Add figure number in snippet.
</a>
</li>
<li class="property_value">
<textarea cols="40" id="id_1290-snippet" name="1290-snippet" rows="10" style="width:100%"><p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">It has been detected, that the installed version of Oracle Application Server is&nbsp;<strong>XXXX.&nbsp;</strong>This version is known to be vulnerable to a number of unspecified vulnerabilities, categorized as 'urgent'.</span></p>
<p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">As this version is no longer supported for this platform, updates or patches may no longer be released, which have the consequence that vulnerabilities can not be patched, leaving the system vulnerable.</span></p>
<p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">In version 10.1.2.0.2 there are, according to http://www.cvedetails.com more than 54 vulnerabilities which affects the installed version.</span></p>
<p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: center; line-height: normal;" align="center"><strong><em><span lang="EN-US" style="font-size: 8pt;">Figure 1: </span></em></strong><em><span lang="EN-US" style="font-size: 8pt;">Oracle Application Server version.</span></em></p>
<p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">More information on these vulnerabilities can be found at:&nbsp;</span><span style="font-size: 10pt;"><a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"><span lang="EN-US" style="color: blue; mso-ansi-language: EN-US;">http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html</span></a><a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"><span lang="EN-US" style="color: blue; mso-ansi-language: EN-US;">&nbsp;</span></a></span><span lang="EN-US" style="font-size: 10pt;">.</span></p>
<p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;">&nbsp;</p>
<p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"><span lang="EN-US" style="font-size: 10pt;">It is recommended that the installed version is updated as soon as possible to the latest version.</span></p></textarea>
</li>
<li class="property_name">
Scan type
</li>
<li class="property_value">
<select multiple="multiple" name="scan_type" size="6" style="width:100%">
<option selected="selected" value="5">
COMPANY PCI
</option>
<option selected="selected" value="7">
Other
</option>
<option selected="selected" value="8">
Firewall Audit
</option>
<option selected="selected" value="6">
Penetration Test
</option>
<option selected="selected" value="9">
WIFI Test
</option>
<option selected="selected" value="10">
APP Test
</option>
<option selected="selected" value="1">
External Security Analysis
</option>
<option selected="selected" value="2">
Internal Security Analysis
</option>
<option selected="selected" value="3">
Web Application Test
</option>
<option selected="selected" value="4">
Host Discovery Analysis
</option>
</select>
-- Use ctrl to mark multiple types
</li>
<li class="property_name">
Update
</li>
<li class="property_value">
<input type="submit" value="Update snippet"/>
</li>
</ul>
</form>
<br style="clear:both;"/>
</div>
<div class="snippet">
<form action="/report/vulnerabilityEditor/?action=edit&id=194&sid=172&model=snippet" method="POST">
<input id="id_172-vulnerability" name="172-vulnerability" type="hidden" value="194"/>
<hr/>
<ul>
<li class="property_name">
<label for="id_172-language">
Language:
</label>
</li>
<li class="property_value">
<select id="id_172-language" name="172-language" style="width:100%">
<option selected="" value="1">
Danish (DK)
</option>
<option value="2">
English (EN)
</option>
<option value="3">
Icelandic (IS)
</option>
</select>
</li>
<li class="property_name">
<label for="id_172-title">
Title:
</label>
</li>
<li class="property_value">
<input id="id_172-title" maxlength="100" name="172-title" style="width:100%" type="text" value="Forældet Oracle Application Server 10g"/>
</li>
<li class="property_name">
<label for="id_172-recommendation">
Recommendation:
</label>
</li>
<li class="property_value">
<input id="id_172-recommendation" maxlength="255" name="172-recommendation" style="width:100%" type="text"/>
</li>
<li class="property_name">
<label for="id_172-snippet">
Snippet:
</label>
</li>
<li class="property_value">
<a href="https://cyberopswiki/index.php/How_to:_Add_figure_number_in_snippet" target="_blank">
How to: Add figure number in snippet.
</a>
</li>
<li class="property_value">
<textarea cols="40" id="id_172-snippet" name="172-snippet" rows="10" style="width:100%"><p style="font-size: 13px;">Det konstateret, at den installerede version af Oracle Application Server er&nbsp;<strong>XXXX.&nbsp;</strong>Denne version indeholder flere kendte samt uspecificeret s&aring;rbarheder, der kategoriseres som v&aelig;rende 'yderst kritiske' og 'kritiske'.</p>
<p style="font-size: 13px;">Da der ikke l&aelig;ngere komme opdateringer til denne platform, vil disse s&aring;rbarheder ikke blive udbedret, hvorfor systemet er meget udsat.</p>
<p style="font-size: 13px;">I version 10.1.2.0.2 findes der if&oslash;lge http://www.cvedetails.com ikke mindre end 54 s&aring;rbarheder, der ber&oslash;rer denne version. Mere information om disse findes p&aring; adressen&nbsp;<a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html">http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html</a><a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html">&nbsp;</a>.</p>
<p style="font-size: 13px;">Det anbefales leverand&oslash;ren af software l&oslash;sningen kontakts, s&aring; der hurtigst muligt kan opgraderes til en nyere, supporteret version.&nbsp;</p></textarea>
</li>
根据马丁斯的建议，我将代码修改如下：
from bs4 import BeautifulSoup
import re
import csv
import glob
def get_danish(text):
return re.compile(r'\b({0})\b'.format(text), flags=re.IGNORECASE).search
with open('dk_snip.csv', 'w', newline='') as f_out:
csv_out = csv.writer(f_out)
# csv_out.writerow(["Nessus", "ID", "Descrip"])
for filename in glob.glob('/home/rj/Documents/snip/snips/*'):
print("Processing:", filename)
with open(filename) as f_in:
soup = BeautifulSoup(f_in, 'html5lib')
var1 = soup.find('li', text = re.compile('Scan vendor:'), attrs = {'class' : 'property_name'})
var2 = soup.find('li', text = re.compile('Vendor ID:'), attrs = {'class' : 'property_name'})
vendor = var1.find_next('li').get_text(strip=True)
vend_id = var2.find_next('li').get_text(strip=True)
# rows = [[vendor, vend_id, dk_desc.get_text(strip=True)] for dk_desc in soup.find_all("textarea")[:3]]
for textarea in soup.find_all("textarea"):
desc = textarea.get_text(strip=True)
if get_danish('dette'):
csv_out.writerows([vendor, vend_id, desc])
答案 0 :(得分：0)
您需要剥离的标签。一种方法是使用BeautifulSoup .get_text(strip=True)函数。
我假设您想为每个TAG重复{{#sec:hasRole('ADMIN')}}<li>ADMIN CONTENT</li>{{/sec:hasRole('ADMIN')}}
{{#sec:hasRole('ADMIN') and hasRole('USER')}}<li>ADMIN & USER CONTENT</li>{{/sec:hasRole('ADMIN') and hasRole('USER')}}
{{#sec:hasAnyRole('ADMIN', 'USER')}}<li>ADMIN OR USER CONTENT</li>{{/sec:hasAnyRole('ADMIN', 'USER')}}
{{#sec:hasRole('USER')}}<li>USER CONTENT</li>{{/sec:hasRole('USER')}}
{{#sec:isAnonymous()}}<li>isAnonymous</li>{{/sec:isAnonymous()}}{{^sec:isAnonymous()}}<li>isAnonymous=false</li>{{/sec:isAnonymous()}}
{{#sec:isRememberMe()}}<li>isRememberMe</li>{{/sec:isRememberMe()}}{{^sec:isRememberMe()}}<li>isRememberMe=false</li>{{/sec:isRememberMe()}}
{{#sec:isAuthenticated()}}<li>isAuthenticated</li>{{/sec:isAuthenticated()}}
{{^sec:isAuthenticated()}}<li>isAuthenticated=false</li>{{/sec:isAuthenticated()}}
{{#sec:isFullyAuthenticated()}}<li>isFullyAuthenticated</li>{{/sec:isFullyAuthenticated()}}
{{^sec:isFullyAuthenticated()}}<li>isFullyAuthenticated=false</li>{{/sec:isFullyAuthenticated()}}
{{#sec:principal}}<li>principal={{username}}{{/sec:principal}}
{{#sec:authentication}}<li>authentication={{.}}{{/sec:authentication}}
{{#sec:permitAll}}<li>permitAll</li>{{/sec:permitAll}}
{{#sec:denyAll}}<li>denyAll</li>{{/sec:denyAll}}
{{^sec:denyAll}}<li>denyAll=false</li>{{/sec:denyAll}}
{{^sec:hasIpAddress('192.168.2.1')}}<li>hasIpAddress('192.168.2.1')=false</li>{{/sec:hasIpAddress('192.168.2.1')}}
{{#sec:isMember(3)}}<li>isMember(3){{/sec:isMember(3)}}
{{#sec:@webSecurity.check(authentication,request)}}<li>@webSecurity.check(authentication,request){{/sec:@webSecurity.check(authentication,request)}}
和textarea值。下面显示了如何完成此操作：
Nessus哪个会给您以下输出CSV文件：
ID注意：由于文本包含换行符，因此CSV格式会自动将这些单元格用双引号引起来。它将正确加载到另一个软件包中。
该脚本搜索当前文件夹中的所有匹配文件。对于每个from bs4 import BeautifulSoup
import csv
import re
import glob
import random
def get_language(text):
# This will need to be added using another library - currently random
return random.choice(["en", "dk"])
with open('dk_snip.csv', 'w', newline='') as f_out:
csv_out = csv.writer(f_out)
csv_out.writerow(["Nessus", "ID", "Text"])
for filename in glob.glob('*.html'): # search all HTML files in the current folder
print("Processing:", filename)
with open(filename) as f_in:
soup = BeautifulSoup(f_in, 'html5lib')
var1 = soup.find('li', text=re.compile('Scan vendor:'), attrs = {'class' : 'property_name'})
var2 = soup.find('li', text=re.compile('Vendor ID:'), attrs = {'class' : 'property_name'})
nessus = var1.find_next('li').get_text(strip=True)
id = var2.find_next('li').get_text(strip=True)
for textarea in soup.find_all("textarea"):
desc = textarea.get_text(strip=True)
if get_language(desc) == 'dk':
csv_out.writerow([nessus, id, desc])
，都会调用Nessus,ID,Text
Nessus,55786,"According to its version, the installation of Oracle Database on the remote host is no longer supported.
Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities."
Nessus,55786,The remote host is running an unsupported version of a database server.
Nessus,55786,Upgrade to a version of Oracle Database that is currently supported.
，这需要使用另一个库进行编码。如果检测到textarea（或所需的其他条件），则该行将添加到CSV文件中。
如果您的文本区域包含HTML，则可能需要使用对BeautifulSoup的另一个调用来进一步处理它：
get_language()