Question

在以下数据中：

data01 =

contig  start    end    haplotype_block 
2   5207    5867    1856
2   155667    155670    2816
2   67910    68022  2
2   68464    68483  3
2   525    775  132
2   118938    119559    1157

data02 =

contig    start   last    feature gene_id gene_name   transcript_id
2   5262    5496    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   5579    5750    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   5856    6032    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   6115    6198    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   916 1201    exon    scaffold_200001.1   NA  scaffold_200001.1
2   614 789 exon    scaffold_200001.1   NA  scaffold_200001.1
2   171 435 exon    scaffold_200001.1   NA  scaffold_200001.1
2   2677    2806    exon    scaffold_200002.1   NA  scaffold_200002.1
2   2899    3125    exon    scaffold_200002.1   NA  scaffold_200002.1

问题：

我想比较这两个数据帧的范围（开始 - 结束）。
如果范围重叠，我想将数据02中的gene_id和gene_name值传输到data01中的新列。

我尝试过（使用熊猫）：

data01['gene_id'] = ""
data01['gene_name'] = ""

data01['gene_id'] = data01['gene_id'].\
apply(lambda x: data02['gene_id']\
        if range(data01['start'], data01['end'])\
           <= range(data02['start'], data02['last']) else 'NA')

如何改进此代码？我目前正在坚持使用大熊猫，但如果使用字典更好地解决问题，我会对它开放。但是，请解释这个过程，我愿意学习，而不仅仅是得到答案。

谢谢，

期望的输出：

contig  start    end    haplotype_block    gene_id    gene_name
2   5207    5867    1856    scaffold_200003.1,scaffold_200003.1,scaffold_200003.1   CP5,CP5,CP5

# the gene_id and gene_name are repeated 3 times because three intervals (i.e 5262-5496, 5579-5750, 5856-6032) from data02 overlap(or touch) the interval ranges from data01 (5207-5867)

# So, whenever there is overlap of the ranges between two dataframe, copy the gene_id and gene_name.

# and simply NA on gene_id and gene_name for non overlapping ranges

2   155667    155670    2816    NA    NA
2   67910    68022  2    NA    NA
2   68464    68483  3    NA    NA
2   525    775  132    scaffold_200001.1   NA
2   118938    119559    1157    NA    NA

Answer 1

我意识到你正在使用python，但使用经典的生物信息学工具bedtools intersect可以很容易地解决你的问题：http://bedtools.readthedocs.io/en/latest/content/tools/intersect.html

您的两个输入文件都遵循标准的BED格式：http://bedtools.readthedocs.io/en/latest/content/general-usage.html

Bedtools intersect为您提供了如何确定两个区域之间的交叉点或重叠的构成的高级逻辑。我相信它也可以直接在bgzipped输入上运行。

Answer 2

你应该在python中使用区间树函数它们非常高效且对内存友好，我尝试了类似的事情将它运行到一些问题，后来解决了但这是我写的代码， Using Interval tree to find overlapping regions

你可以建立这个代码。

Answer 3

s1 = data01.start.values
e1 = data01.end.values
s2 = data02.start.values
e2 = data02['last'].values

overlap = (
    (s1[:, None] <= s2) & (e1[:, None] >= s2)
) | (
    (s1[:, None] <= e2) & (e1[:, None] >= e2)
)

g = data02.gene_id.values
n = data02.gene_name.values

i, j = np.where(overlap)
idx_map = {i_: data01.index[i_] for i_ in pd.unique(i)}

def make_series(m):
    s = pd.Series(m[j]).fillna('').groupby(i).agg(','.join)
    return s.rename_axis(idx_map).replace('', np.nan)

data01.assign(
    gene_id=make_series(g),
    gene_name=make_series(n),
)

Answer 4

如果您想要比床具更快的东西和/或Python科学堆栈中的本地居民想要的东西，请尝试pyranges：

import pyranges as pr

c1 = """Chromosome  Start    End    haplotype_block
2   5207    5867    1856
2   155667    155670    2816
2   67910    68022  2
2   68464    68483  3
2   525    775  132
2   118938    119559    1157"""

c2 = """Chromosome Start End  feature gene_id gene_name   transcript_id
2   5262    5496    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   5579    5750    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   5856    6032    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   6115    6198    exon    scaffold_200003.1   CP5 scaffold_200003.1
2   916 1201    exon    scaffold_200001.1   NA  scaffold_200001.1
2   614 789 exon    scaffold_200001.1   NA  scaffold_200001.1
2   171 435 exon    scaffold_200001.1   NA  scaffold_200001.1
2   2677    2806    exon    scaffold_200002.1   NA  scaffold_200002.1
2   2899    3125    exon    scaffold_200002.1   NA  scaffold_200002.1"""

gr1, gr2 = pr.from_string(c1), pr.from_string(c2)

j = gr1.join(gr2).sort()

print(j)
# +--------------+-----------+-----------+-------------------+-----------+-----------+------------+-------------------+-------------+-------------------+
# |   Chromosome |     Start |       End |   haplotype_block |   Start_b |     End_b | feature    | gene_id           | gene_name   | transcript_id     |
# |   (category) |   (int32) |   (int32) |           (int64) |   (int32) |   (int32) | (object)   | (object)          | (object)    | (object)          |
# |--------------+-----------+-----------+-------------------+-----------+-----------+------------+-------------------+-------------+-------------------|
# |            2 |       525 |       775 |               132 |       614 |       789 | exon       | scaffold_200001.1 | nan         | scaffold_200001.1 |
# |            2 |      5207 |      5867 |              1856 |      5262 |      5496 | exon       | scaffold_200003.1 | CP5         | scaffold_200003.1 |
# |            2 |      5207 |      5867 |              1856 |      5579 |      5750 | exon       | scaffold_200003.1 | CP5         | scaffold_200003.1 |
# |            2 |      5207 |      5867 |              1856 |      5856 |      6032 | exon       | scaffold_200003.1 | CP5         | scaffold_200003.1 |
# +--------------+-----------+-----------+-------------------+-----------+-----------+------------+-------------------+-------------+-------------------+
# Unstranded PyRanges object has 4 rows and 10 columns from 1 chromosomes.
# For printing, the PyRanges was sorted on Chromosome.

print(j.df)
#   Chromosome  Start   End  haplotype_block  Start_b  End_b feature            gene_id gene_name      transcript_id
# 0          2    525   775              132      614    789    exon  scaffold_200001.1       NaN  scaffold_200001.1
# 1          2   5207  5867             1856     5262   5496    exon  scaffold_200003.1       CP5  scaffold_200003.1
# 2          2   5207  5867             1856     5579   5750    exon  scaffold_200003.1       CP5  scaffold_200003.1
# 3          2   5207  5867             1856     5856   6032    exon  scaffold_200003.1       CP5  scaffold_200003.1

如何通过比较值的范围来合并两个pandas数据帧（或传输值）

4 个答案: