我有10GB的XML文件需要解析。 XML的示例结构是
<?xml version="1.0" encoding="UTF-8"?>
<proteinAtlas xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://v18.proteinatlas.org/download/proteinatlas.xsd" schemaVersion="2.5">
<entry version="18" url="http://v18.proteinatlas.org/ENSG00000000003">
<name>TSPAN6</name>
<synonym>T245</synonym>
<synonym>TM4SF6</synonym>
<synonym>TSPAN-6</synonym>
<identifier id="ENSG00000000003" db="Ensembl" version="88.38">
<xref id="O43657" db="Uniprot/SWISSPROT"/>
</identifier>
<proteinClasses>
<proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
<proteinClass source="MDM" id="Md" parent_id="" name="Membrane proteins predicted by MDM"/>
<proteinClass source="MEMSAT3" id="Me" parent_id="" name="MEMSAT3 predicted membrane proteins"/>
<proteinClass source="MEMSAT-SVM" id="Mf" parent_id="" name="MEMSAT-SVM predicted membrane proteins"/>
<proteinClass source="Phobius" id="Mg" parent_id="" name="Phobius predicted membrane proteins"/>
<proteinClass source="SCAMPI" id="Mh" parent_id="" name="SCAMPI predicted membrane proteins"/>
<proteinClass source="SPOCTOPUS" id="Mi" parent_id="" name="SPOCTOPUS predicted membrane proteins"/>
<proteinClass source="THUMBUP" id="Mj" parent_id="" name="THUMBUP predicted membrane proteins"/>
<proteinClass source="TMHMM" id="Mk" parent_id="" name="TMHMM predicted membrane proteins"/>
<proteinClass source="MDM" id="M1" parent_id="" name="1TM proteins predicted by MDM"/>
<proteinClass source="MDM" id="M4" parent_id="" name="4TM proteins predicted by MDM"/>
<proteinClass source="SignalP" id="Sb" parent_id="Se" name="SignalP predicted secreted proteins"/>
<proteinClass source="HPA" id="Za" parent_id="" name="Predicted intracellular proteins"/>
<proteinClass source="UniProt" id="Ua" parent_id="" name="UniProt - Evidence at protein level"/>
<proteinClass source="Kim et al 2014" id="Ea" parent_id="" name="Protein evidence (Kim et al 2014)"/>
<proteinClass source="Ezkurdia et al 2014" id="Eb" parent_id="" name="Protein evidence (Ezkurdia et al 2014)"/>
</proteinClasses>
<proteinEvidence evidence="Evidence at protein level">
<" source="HPA" evidence="Evidence at transcript level"/>
<evidence source="MS" evidence="Evidence at protein level"/>
<evidence source="UniProt" evidence="Evidence at protein level"/>
</proteinEvidence>
<tissueExpression source="HPA" technology="IHC" assayType="tissue">
<summary type="tissue"><![CDATA[Cytoplasmic and membranous expression in most tissues.]]></summary>
<verification type="reliability" description="Antibody staining mainly consistent with RNA expression data. Pending external verification. ">approved</verification>
<image imageType="selected">
</tissueExpression>
</entry>
因此,我需要解析的每个节点都是“ entry”,并且在整个文件中都将遵循相同的结构
我在网上找到了一个示例,该示例如何按节点解析节点,效果很好
branchFunction <- function() {
store <- new.env()
func <- function(x, ...) {
ns <- getNodeSet(x, path = "//name")
proteinEviden = getNodeSet(x, path = "proteinEvidence")
tissueExpression = getNodeSet(x, path = "tissueExpression/summary")
tissueExpression1 = getNodeSet(x, path = "tissueExpression/verification")
value <- xmlValue(ns[[1]])
value2 <- xmlGetAttr(proteinEviden[[1]], "evidence")
print(value)
print(value2)
# if storing something ...
# store[[some_key]] <- some_value
}
getStore <- function() { as.list(store) }
list(entry = func, getStore=getStore)
}
myfunctions <- branchFunction()
xmlEventParse(
file = "proteinatlas.xml",
handlers = NULL,
branches = myfunctions
)
这很好,但是随着它的进行,它速度变慢并且内存开始累积。知道如何释放内存,或者是否有其他方法可以解析大型XML文件。
答案 0 :(得分:0)
不确定它是否具有更高的内存效率,但是尝试一下不会无济于事:
use omp_lib
REAL*8 r,summ,sl
parameter (N=10000000)
dimension r(N)
do i=1,N
r(i)=i
enddo
summ=0.0d00
sl=0.0d00
!$OMP PARALLEL FIRSTPRIVATE(sl) SHARED(r,summ)
!$OMP DO
do i=1,N
sl=sl+r(i)
enddo
!$OMP END DO
!$OMP CRITICAL
summ=summ+sl
!$OMP END CRITICAL
!$OMP END PARALLEL
write(*,*)'SUM',summ
end
答案 1 :(得分:0)
我在网上发现的是,R可能不是解析这么大的XML文件的最佳语言。 Python确实有这样的库,并且它们运行良好。我尝试了其中一个,但似乎工作正常。 –