Question

我有一个包含1000行和3个变量的表（一个ID，一个COUNTRY和一个字符串变量“VAR1”）。 VAR1是由空格分隔的单词组成的句子。

按照COUNTRY，我想要所有夫妻（或所有三胞胎）的字数。非常重要的是夫妻（或三胞胎）是所有单词的交叉（不一定是一步一步）。也许，我们可以使用ngrams hql-function来实现它，但是当我使用它时，它会逐步计算单词而不是所有的交叉。

让我们举个例子来了解我想要的东西：

> **"ID" "COUNTRY" "VAR1"**
> "1" "CANADA" "dad mum child"
> "2" "CANADA" "dad mum dog"
> "3" "USA"    "bird lion car"

VAR1不一定是3个字的长度。这只是为了简化。

我希望获得2-ngrams的4个步骤中的结果：

第1步：最重要的步骤：用2

交叉单词

> "1" "CANADA" "dad mum" 1
> "1" "CANADA" "dad child" 1
> "1" "CANADA" "mum dad" 1
> "1" "CANADA" "mum child" 1
> "1" "CANADA" "child dad" 1
> "1" "CANADA" "child mum" 1
> "2" "CANADA" "dad mum" 1
> "2" "CANADA" "dad dog" 1
> "2" "CANADA" "mum dad" 1
> "2" "CANADA" "mum dog" 1
> "2" "CANADA" "dog dad" 1
> "2" "CANADA" "dog mum" 1
> "3" "USA"    "bird lion" 1
> "3" "USA"    "bird car" 1
> "3" "USA"    "lion bird" 1
> "3" "USA"    "lion car" 1
> "3" "USA"    "car bird" 1
> "3" "USA"    "car lion" 1

第2步：订购2克

> "1" "CANADA" "dad mum" 1
> "1" "CANADA" "child dad" 1
> "1" "CANADA" "dad mum" 1
> "1" "CANADA" "child mum" 1
> "1" "CANADA" "child dad" 1
> "1" "CANADA" "child mum" 1
> "2" "CANADA" "dad mum" 1
> "2" "CANADA" "dad dog" 1
> "2" "CANADA" "dad mum" 1
> "2" "CANADA" "dog mum" 1
> "2" "CANADA" "dad dog" 1
> "2" "CANADA" "dog mum" 1
> "3" "USA"    "bird lion" 1
> "3" "USA"    "bird car" 1
> "3" "USA"    "bird lion" 1
> "3" "USA"    "car lion" 1
> "3" "USA"    "bird car" 1
> "3" "USA"    "car lion" 1

第3步：按ID，COUNTRY，2-ngrams

分开

> "1" "CANADA" "dad mum"
> "1" "CANADA" "child dad"
> "1" "CANADA" "child mum"
> "2" "CANADA" "dad mum"
> "2" "CANADA" "dad dog"
> "2" "CANADA" "dog mum"
> "3" "USA"    "bird lion"
> "3" "USA"    "bird car"
> "3" "USA"    "car lion"

第4步：按COUNTRY计算，2-ngrams

> "CANADA" "dad mum" 2 
> "CANADA" "child dad" 1 
> "CANADA" "child mum" 1
> "CANADA" "dad dog" 1 
> "CANADA" "dog mum" 1 
> "USA"    "bird lion" 1 
> "USA"    "bird car" 1 
> "USA"    "car lion" 1

非常感谢你

Answer 1

with        cte as
            (
                select  t.ID
                       ,t.COUNTRY
                       ,pe.pos
                       ,pe.val
                from    mytable t
                        lateral view posexplode (split(VAR1,'\\s+')) pe
            )

select      t1.COUNTRY  
           ,concat_ws(' ',t1.val,t2.val)    as combination
           ,count (*)                       as cnt

from                cte t1

            join    cte t2

            on      t2.id   =
                    t1.id

where       t1.pos < t2.pos

group by    t1.COUNTRY  
           ,t1.val 
           ,t2.val
;

+----------+--------------+------+
| country  | combination  | cnt  |
+----------+--------------+------+
| CANADA   | dad child    | 1    |
| CANADA   | dad dog      | 1    |
| CANADA   | dad mum      | 2    |
| CANADA   | mum child    | 1    |
| CANADA   | mum dog      | 1    |
| USA      | bird car     | 1    |
| USA      | bird lion    | 1    |
| USA      | lion car     | 1    |
+----------+--------------+------+

ngrams单词hive的组合

1 个答案: