Question

我有10GB的XML文件需要解析。 XML的示例结构是

<?xml version="1.0" encoding="UTF-8"?>
<proteinAtlas xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://v18.proteinatlas.org/download/proteinatlas.xsd" schemaVersion="2.5">
    <entry version="18" url="http://v18.proteinatlas.org/ENSG00000000003">
        <name>TSPAN6</name>
        <synonym>T245</synonym>
        <synonym>TM4SF6</synonym>
        <synonym>TSPAN-6</synonym>
        <identifier id="ENSG00000000003" db="Ensembl" version="88.38">
            <xref id="O43657" db="Uniprot/SWISSPROT"/>
        </identifier>
        <proteinClasses>
            <proteinClass source="MDM" id="Ma" parent_id="" name="Predicted membrane proteins"/>
            <proteinClass source="MDM" id="Md" parent_id="" name="Membrane proteins predicted by MDM"/>
            <proteinClass source="MEMSAT3" id="Me" parent_id="" name="MEMSAT3 predicted membrane proteins"/>
            <proteinClass source="MEMSAT-SVM" id="Mf" parent_id="" name="MEMSAT-SVM predicted membrane proteins"/>
            <proteinClass source="Phobius" id="Mg" parent_id="" name="Phobius predicted membrane proteins"/>
            <proteinClass source="SCAMPI" id="Mh" parent_id="" name="SCAMPI predicted membrane proteins"/>
            <proteinClass source="SPOCTOPUS" id="Mi" parent_id="" name="SPOCTOPUS predicted membrane proteins"/>
            <proteinClass source="THUMBUP" id="Mj" parent_id="" name="THUMBUP predicted membrane proteins"/>
            <proteinClass source="TMHMM" id="Mk" parent_id="" name="TMHMM predicted membrane proteins"/>
            <proteinClass source="MDM" id="M1" parent_id="" name="1TM proteins predicted by MDM"/>
            <proteinClass source="MDM" id="M4" parent_id="" name="4TM proteins predicted by MDM"/>
            <proteinClass source="SignalP" id="Sb" parent_id="Se" name="SignalP predicted secreted proteins"/>
            <proteinClass source="HPA" id="Za" parent_id="" name="Predicted intracellular proteins"/>
            <proteinClass source="UniProt" id="Ua" parent_id="" name="UniProt - Evidence at protein level"/>
            <proteinClass source="Kim et al 2014" id="Ea" parent_id="" name="Protein evidence (Kim et al 2014)"/>
            <proteinClass source="Ezkurdia et al 2014" id="Eb" parent_id="" name="Protein evidence (Ezkurdia et al 2014)"/>
        </proteinClasses>
        <proteinEvidence evidence="Evidence at protein level">
            <" source="HPA" evidence="Evidence at transcript level"/>
            <evidence source="MS" evidence="Evidence at protein level"/>
            <evidence source="UniProt" evidence="Evidence at protein level"/>
        </proteinEvidence>
        <tissueExpression source="HPA" technology="IHC" assayType="tissue">
            <summary type="tissue"><![CDATA[Cytoplasmic and membranous expression in most tissues.]]></summary>
            <verification type="reliability" description="Antibody staining mainly consistent with RNA expression data. Pending external verification. ">approved</verification>
            <image imageType="selected">
         </tissueExpression>
    </entry>

因此，我需要解析的每个节点都是“ entry”，并且在整个文件中都将遵循相同的结构

我在网上找到了一个示例，该示例如何按节点解析节点，效果很好

branchFunction <- function() {
  store <- new.env() 
  func <- function(x, ...) {
    ns <- getNodeSet(x, path = "//name")
    proteinEviden = getNodeSet(x, path = "proteinEvidence")
    tissueExpression = getNodeSet(x, path = "tissueExpression/summary")
    tissueExpression1 = getNodeSet(x, path = "tissueExpression/verification")
    value <- xmlValue(ns[[1]])
    value2 <- xmlGetAttr(proteinEviden[[1]], "evidence")

    print(value)
    print(value2)

    # if storing something ... 
    # store[[some_key]] <- some_value
  }
  getStore <- function() { as.list(store) }
  list(entry = func, getStore=getStore)
}

myfunctions <- branchFunction()

xmlEventParse(
  file = "proteinatlas.xml",
  handlers = NULL, 
  branches = myfunctions
)

这很好，但是随着它的进行，它速度变慢并且内存开始累积。知道如何释放内存，或者是否有其他方法可以解析大型XML文件。

Answer 1

不确定它是否具有更高的内存效率，但是尝试一下不会无济于事：

  use omp_lib
  REAL*8 r,summ,sl
  parameter (N=10000000)
  dimension r(N)

  do i=1,N
  r(i)=i
  enddo

  summ=0.0d00
  sl=0.0d00

  !$OMP PARALLEL FIRSTPRIVATE(sl) SHARED(r,summ)
  !$OMP DO 
  do i=1,N
  sl=sl+r(i)
  enddo
  !$OMP END DO
  !$OMP CRITICAL
  summ=summ+sl
  !$OMP END CRITICAL
  !$OMP END PARALLEL

  write(*,*)'SUM',summ

  end

Answer 2

我在网上发现的是，R可能不是解析这么大的XML文件的最佳语言。 Python确实有这样的库，并且它们运行良好。我尝试了其中一个，但似乎工作正常。 –

解析R

2 个答案: