如何使用BigQuery找到最常见的二元组？

Question

我想在桌子上找到最常见的二元词（一对单词）。我怎么能用BigQuery做到这一点？

Answer 1

BigQuery现在支持SPLIT（）：

SELECT word, nextword, COUNT(*) c 
FROM (
SELECT pos, title, word, LEAD(word) OVER(PARTITION BY created_utc,title ORDER BY pos) nextword FROM (
SELECT created_utc, title, word, pos FROM FLATTEN(
  (SELECT created_utc, title, word, POSITION(word) pos FROM
   (SELECT created_utc, title, SPLIT(title, ' ') word FROM [bigquery-samples:reddit.full])
  ), word)
))
WHERE nextword IS NOT null
GROUP EACH BY 1, 2
ORDER BY c DESC
LIMIT 100

Answer 2

现在有一个新功能：ML.NGRAMS()：

WITH data AS  (
  SELECT REGEXP_EXTRACT_ALL(LOWER(title), '[a-z]+') title_arr
  FROM `fh-bigquery.reddit_posts.2019_08`  
  WHERE title LIKE '% %'
  AND score>1
)


SELECT APPROX_TOP_COUNT(bigram, 10) top
FROM (
  SELECT ML.NGRAMS(title_arr, [2,2]) x
  FROM data
), UNNEST(x) bigram
WHERE LENGTH(bigram) > 10

文档：

• https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-preprocessing-functions#mlngrams

Answer 3

标准SQL版本：

SELECT word, nextword, COUNT(*) c FROM (
SELECT pos, title, word, LEAD(word) OVER(PARTITION BY created_utc,title ORDER BY pos) nextword FROM (
SELECT created_utc, title, word, pos FROM ( 
    SELECT created_utc, title, SPLIT(title, ' ') word FROM `bigquery-samples.reddit.full`), UNNEST(word) as word WITH OFFSET pos))
WHERE nextword IS NOT null
GROUP BY 1, 2
ORDER BY c DESC
LIMIT 100

取消嵌套数组时，可以使用以下语法检索该元素的位置：

UNNEST(word) as word WITH OFFSET pos