Question

我试图通过Python迭代/分页检索视频的所有评论。我已使用开发人员密钥正确登录

import googleapiclient.discovery as gg
import googleapiclient.errors as gge

yt = gg.build(api_service_name= 'youtube', api_version= 'v3', developerKey = M_KEY)
comments= []
page= ''

while True:
        request = yt.commentThreads().list(
            part= "snippet,replies",
            order= "relevance",
            maxResults= 100,
            pageToken= page,
            textFormat= "plainText",
            videoId= video['id']
            # video is a static dictionary i've saved outside the script
        )

        try:
            response = request.execute()
            page= response['nextPageToken']
            comments.extend(response['items'])
            print('Comments extended')
        except KeyError:
            # there are no more pages
            print('Iteration ended')
            break
        except gge.HttpError as error:
            print('HTTP error:', error.__dict__['resp']['status'])

我期望它要做的是迭代注释页面，直到response['nextPageToken']抛出KeyError为止，这意味着不再有注释页面。取而代之的是，执行会完美地进行了十几次迭代（最好），然后开始抛出processingFailure错误，内容如下所示：

{
  "error": {
    "errors": [
      {
        "domain": "youtube.commentThread",
        "reason": "processingFailure",
        "message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the <code>commentThread</code> resource in the request body to ensure that it is valid.",
        "locationType": "other",
        "location": "body"
      }
    ],
    "code": 400,
    "message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the <code>commentThread</code> resource in the request body to ensure that it is valid."
  }
}

我已尝试同时记录page和videoId，以确保它们没有问题，但它们都是有效的。当该错误发生但没有任何变化时，我也尝试过time.sleep()长达15分钟。
这是发生错误时以json格式发出的请求，使用request.to_json()捕获，感谢@stvar的建议：

{
  "uri": "https://www.googleapis.com/youtube/v3/commentThreads",
  "method": "POST",
  "body": "part=snippet%2Creplies&order=relevance&maxResults=100&pageToken=QURTSl9pM2xlemsyWjAzYlBhWkNRMTFMMWIyMjFsVVNnS2U0WE8zTkwxRzdRdkpsdlVqWHlwWEV2SmxkeUxjdkR1UWk3eVU1OTI1cmJEeUtJZHRGQWVmY21PUGxVOXBER3YtckE2NlhSWlRwQzR0Y2VyY0JDbC1uNVRaSU56RklzejJERmRCc2lLUjV2Rm1LV0Njc3ByMjliRXRMZmNjRFJucFgwNFNBYVhkSFJzYXkzNVpKTXNNSzNfWmVGd3dSRWxYQmwyQmxnWGJwNFZidVpiYjlOWjBabFFsalZFZkdqZFV0SHlrVEJqclppVnRtMjZCTnYtQm9WWjFrQ0dELUlLTnYwWG50cU5BQXJ3Ukh3VE9PZnNaZ0tZaWN1ZTdBakJkWFp5Ymo5M2R5Y0g1aWVsWUUzUVg0TU83Q2JZQ1IxWnRTMXUyTFhpSDdmMU9GTmtiQUE0UjdyVUVBelNnSjhTTDVsLU1TaERwVHdvSVhkX1ktNVBTc2xkX09zcjBOT3E3Z2lVWWRPRFhkVF9NN1JaQTEyUEJmU1hNbUtvM2JzU1NzOFRid29wTEo3Q0hucmJnNHcwNUJzaGtqSE8wa2g2U0FUY3pQbDJ1bGNnaFRKNEJCRm90TVNyWXNSREgyQVFqMU9PNnY3elBGSEhrYXJSMUJYS09yQ0tOVE5Oa3l5V00tdGY3TTlwY3o0VXJsaWRua1BrNXVhWmVLMzV1T2NmOEhqaUNucEdheTRfZjNiM1JkYUJuaGZqQjFMV1c0NFRJNXlzR2trdFpLemV1SU12V0tCTW11b1RMU05PXzA0eVdHM3lRclpZaC1BN3k4RDdhTW1uTHZtbDVsRzBVTVFHdkdkMTN4VHQzNW1tZ3BoY0F6VDJVWTFhUWpxdW8td1M0bnkzQTRtVGc5bGxQNV81ejV1dm1JX1ZDRVZIXzI3eXVnbHJBcC1Lb0NULUhHOGp3ZGNGeVFKbFNXbVh0Y2NQei1UbjBFLVhuZWp0eXh5NzVjOEtjS1FqTUppQWdOSDRmWWtSOUZPRHQtSEpsTnJtNWZVX2t4VDlVTDB6WmxWTHN5dlZzZllNQkFBOEJNMWZkOEtoTk5jMnQ4Y0hydXVScTNILXZLXzJodGFUNmxhQnEtay1PVV9yYzJFNEhKaDZjcUszV3ZGM2VLaUxJZjlwRmViYXRfVGRSOFZ6OF9vU2h6WjVqNkhVU0tqZHduLTNlaFhuTHFXSG1WSk1HUVQ4dkdIdDZvUFdKNkxOeFlhTmJzd1J4dGQzLXBHUmsxaHYtdFc2cTI1VDZsMWJGdE5Pb1RmR2hlRGM3cjZPcDJ2eTljQk1GcTJXaTFtNFhndzlWbDBOby1kNzZhLU5WNTI1VUlzRmpQSkRvSlFFMUQzNllzbi0tU01OYTg1a2poS2ZrWHpQMjQyd1hDb3h4blE5ZlJmN2xIMEstRFR6cUFWcTNDRDFfbjNubXY4Z2ZseGdVY2NjTWk0NzQ3SDFZcWs2eWxZWlB0Vl9iSldlbktOMjVFWUp6UnVRb3dfOXFQdmhBZEN5clJpX1g4aVhmdERnbE5XX1FjVWNXODRtSm1LSUpDVnJHVGlEeUtGb3BPMVYyWU5TbnQzY29NLUY5c3Y2WmpNVTNlVjIxQ2RwSzlKTUZwY0RxY2FlMGFtd2tucFpjeUtDN2xwOERYcDJwSU1RY0dIdXdCTmJIcWdjbTh1Q04wVTh1dktzeVdob09wX25uU19BMFNlRlBrNG1wZFRKVVJFVzVfdGQxbGFYemFqZjJOQTd1R0NCZ0RrMWlTS3BMMy1hY0FMd25KTGFYelJPQjZvRnlYMnBFelhCREgyRDJ3TnJWNldWUllqOVVvdHV2cVRXRXlBbkJpaFJpd3RIc2RaamVaUERldXItT1pkTVVFczBzNi1hZmhDYTFzWVM3SElsYkxtMkoybC03YlZVRkt1NEVSWV8tWHRJTko4d2hqWllWVU04UXlkQV84ZjFzVm01bW83cTd4R3ZOSVNabGRSaXVlTU91MXR6RVFYeTNwNHd3bzNVY1RncHdzY1VKQWw2eWNvcmdER0N5RjZiQkRmNnh0S256MzhreldFTm9XMDhlY1VUeEhnNTM2bHNYVlpKdGJrdHd1Y3VCc0hYOHlEc1EyZXJLTUlMTlVQb0FmU0hpdy1WdS1iT19fTTlMQUVWa3BnWloxSXdUZHotMW5zWWVnTVBzelE3VmQtRlBOajNfcmJJNnlZYkpDdmxKWXoxcjBZYkV4Q2duNGx1MTlrYWVVOXktT0lVX2dfVnc2cW9nSEVHSHZCUzFHRFFPaW1ydUxlY012bVQtaVBnTXQ1VWxOZVI3YW9nTkhFdHlwUGlneFdOM2Jkcm1iWEcxN25pQVI0TUpvVW1hemlrYWl6M0dnSTQ0VWhVWVMxaHEzeS05cnJRSkJ6TEVEZTB4anYySzR6WWhyMEtkZ0ZVMkxDZXlkcHFCSTBfU2Z3ZEVTYkE1YlE1SXI0M3lhUnJGZ1l0QVZFU2ZqRDE1MUhSLU1lU2dxWFpUem04RHVqOVBTUkZhbkFLWUJ2aGZsX0w2SzBabTBoUUxiYVZxT25ydk56U01YdklJZmtPemxtT0Nrc1JGOWVRQnMydk5lZkRjRExEUzRaWXFfTlNRdVNOUTFacFdLcklqRXc1NDg2eGU2NXlid3Y0SnRVeWplVE5CVVF4Qm01ajJfOWY2U2NWcWlVajYwYXJ3eXZ6RGt4cldCMndES0wzZ0xERl91bmlQaDVtUmNXSERXekJCVDAtd2ZnVFBadERnQlJWUEl3cWxCb3FfOEh2NkJCUlZqUThCMUk0OXM3Sjk3Sng3WFBpdUlEUFRnLV9kMnhoa2Z5QVpLRFNLSWl0ck1WUnhKRWFaZ0J0VDZmTjY3MG5SMkZQYUx0YTQ3dmgzYzhpa0Nua0dIS0VzSGYzOERiYkN4ZXM1ZkpkYV9nMEJrMnA2aHgycWFfZ04ySmRGellBaUphdXA2X213WXNxVW1QbUpfa2xZZTUwbVA2azMyRjV0Q2dRcWJVajFuVTFjRHB5QUZUcTZ1X3RwNGVBSmVoNWt6ajFZNTkwS1J2TzZreHhfQ1EzTTdDUGpCb0V1SXdFWmh6YjF0Q1NHUnoxTy11MURZak03ZnNEX0NCSUxPbkVuZGZ5VDlCMkkwNl9lLUw2bk05MUVfU29NTmg4WFBSSGNibnZ4T3h4T25Yal81a3NDMG5veDBERVdLRVBEV0pqRi1CbEpnV01HajhjR29jbXFEdXpIcFdCblcyZ2dsX1ZUNHJuMkxHUFV6ekZiaklFMXpob0w4MXJNOXVhZ2dMX1oxdzNkRi1sV3JDaGpING10b21qYTI0QW00dWFOOVZHSWJlT3lZVy1qSzNnUGxPU0hTZnFTS2VVLW80cHFqWTAtYV9lbHI5WHdnZV9nM25oX2Iwd2lUeXgxQndDcENrczUxb3RlZlZkSnREU20xSjdCM0VBZkZ4X1pkQ2YwQUszcWJxRVdwM25wM0w2WmpNTG1WM0RyWlU3TkpGeElBeGRkUTZlWEttazJuLW1mdFZtMTNVamRKZEthMkFfbS1HSnRtMFRTbFlVYnBqQ2puaUVrY2xieTB0TDF4UDNUWGNMUFo3enFtajB5T0dZbGRYQkxDQzlxdGNXN0d4V093RXUxd2Z4ek9oVEZzVUpabHpxeHNFVlVUbF9ORGk5U1BhVHg3QlJEWjdqM2g1VlFhb184SU1ZMERFLWJnY0htaXk2aXFoTjBMaUg2eTdjVmhHekFCbmdjNXJ2WkpQVE9jY09RRXE1SVJYenhvdVJxVGRDbnd5WGdrZnlyWG96T2FXcVRVRXBiWHBGRGFnZTUybmFJam1HanpPelNSZHBJLW5yaDgybm5BdldNNTRtcGNuaXRGbzJmODhyc2IzYmNUdDNoekNaRG5Oa2k1OUNXU2tuSnd2OG1GVjBGM2xid0lWd3QwWDR3Ukc4TlZZcXdkY2FOdS0xaWlsajAwQTBLVnJTVzljLS1WZF9WWGl0Y21naDdpX2c1djJwRFpuU3pWY1VNRWNOTy1GaVdSTUxDTHZ6VDFNYl9HRmsxVEZlTG5fVFdrNzRYdlpoSVRwMHJjSjdJT21lU09ObEtqTlNiX0FCNGtXRldpSHByY0htSGxOYS1JbWZWbWRCZUlLaFhxUG5haGZCMm1PbklDMGFJS0pmZ2RjUUtVSWpLeVBrTk5DMXo5VUVkeGZKRFRtZDh4OHJkV3BEbi0wUnJXY2x1Rk9XU04xeXpkbnA5U1FnV2huOTFYVlBGRG5Rem9CODdfRUU2d3liNy1LQ0JHbjFsVVRQc3pMS3JudjhDVG1Bb2xjYU1MaUprajZGT2dkS0ktd3IzNXZMSWhFQm9oamdGZW5KcWNOQ0Q5NDZGWXFzTE5peEVyenJHWG9ZaWtNRjVoUXJNVFVxdjhSYUgyRHZKcXpmOWtMRHJmV1dMRklLUTFvLVJZbDY5dzRFcXJxUGJoenF4SFpLYUFoOWpFcnNNWVZmMmsycy1YVXllMkhwNzJrOGl5eUpTTXdQVFhWTjJ4MVA1OC1FbnVlbGZSRDQwOWxkcUwwSkRWTFVHMUdhUi1jVnlBZVJKZmZNY3Z2OTQ4MVdqWDV5WFBvSjJJY25VZDdOc2JyUkRKQ1IybXY2NG5uX3NJUmJubXFvTHQ3dHVoV0pMQmRlb2tQOUU2cU1xN3gzVnFsMzVqVDhjSFNuSmNYZWczSG1veEhGWDlkTDNBRmtXVFhBaVQwSDR2dTJBaU1nRUJVZndMS3gzUjNXNlRRcGJqNnJkbldzT3FjdU1NSkwtOVZ2WEIzMEYtQlFFNW84dUthQjFvZFg4OVota255cWt5Mlo1cXpoWTJKel9vTmpLbHRlQVN0WkVURDh3WG8wSTY3OE0yTTF4OF9KQ3ctc1Uzd3hxejdyRlB0NzVXZkxKU1RvOUw1TWtEMjNCaWVaLTkzaExwTHlwaG9DaC1qcjE4cHgxRHRtMmc3TnZXaTJleHNHVW1WTzJlMkVlWFd6QUxfOENSUEJRU0E5d08xa2hjSkxBSG5kY1BlTzJxUkR0U3ppTG0zR0xTOTRLUU9YSmxGSWFoR0JjRmp4b1pUTmNCTFoxaTdBd0RpYmhJLVJyX19qOFkyLW5UUW4wYjBsbTFpa3MzSThqd3pFa0Q0aGZrVWFBWXhMdS1LaUg0R0xRNlB4YUdxb0xVdEtaMm1ld0ZVbi1uMTlGLVN5bDhNakN3czlnQ2ZYZHhFS2hlQ1ByZzNhNHljdFhiOHpJdDg3ZVdyeGFBVW1jdEkxbnJMMUw2WGpQcUFDc0N6WWQtMzhCNTZ0VWtXWnBlRmRIWnl5VkpkeF9XMzZnZG5MWnpoVFNRcGJhTjAxOUNPWkpZeTh6Zk9QQW9paU5IOXVESHgybkRHeUREY1ZCMFBIaVZ2aFN5dFNuWWJZbWJKX1N5dG9LUXl6LVViMzFIWHZURkVVVU1iTl9NdUNiTEwwem9CQ0EtODNyMGJLaGh1bGpXRkFBRWxXR1dTYlNIa3NEenN1NlBid2gxZTUyUFNhem5yVWN4Y0tF&textFormat=plainText&videoId=CJ_GCPaKywg&key=m_developer_key&alt=json",
  "headers": {
    "accept": "application/json",
    "accept-encoding": "gzip, deflate",
    "user-agent": "google-api-python-client/1.7.9 (gzip)",
    "content-length": "5730",
    "x-http-method-override": "GET",
    "content-type": "application/x-www-form-urlencoded"
  },
  "methodId": "youtube.commentThreads.list",
  "resumable": null,
  "response_callbacks": [],
  "_in_error_state": false,
  "body_size": 0,
  "resumable_uri": null,
  "resumable_progress": 0
}

注意：我需要在我的请求中加入order= "relevance"，因为我主要需要获得投票最多的评论。

无处可寻，我希望你能帮助我

Answer 1

根据Google的Python客户端库sample code和Google的Youtube API sample code，您应该已经对分页循环进行了编码，如下所示：

DECLARE @var varchar(max) = 'ROOT(10) AS TEN, ROOT(50) AS FIFTY, ROOT(70) AS SEVENTY'

;WITH CTE AS (
    SELECT STUFF([var], patIdx, idx - patIdx, '') [var], 1 [rn]
    FROM (
        SELECT @var [var], PATINDEX('%ROOT([0-9]%) AS %', @var) patIdx, CHARINDEX(' AS ', @var) + 4 idx
    ) a
    UNION ALL
    SELECT STUFF([var], patIdx, idx - patIdx, ''), [rn] + 1
    FROM (
        SELECT [var], PATINDEX('%ROOT([0-9]%) AS %', [var]) patIdx, CHARINDEX(' AS ', [var]) + 4 idx, [rn] FROM CTE
    ) a
    WHERE PATINDEX('%ROOT([0-9]%) AS %', [var]) > 0
)

SELECT TOP 1 [var] FROM CTE
ORDER BY [rn] DESC

Answer 2

这不能解决您的问题。它只是表明通过GET查询端点 request方法成功从API获取所需的页面响应。

# comments-wget [-d] VIDEO_ID [PAGE_TOKEN]

$ comments-wget() { 
    local x='eval'
    [ "$1" == '-d' ] && { 
        x='echo'
        shift
    }

    local v="$1"
    quote2 -i v

    local p="$2"
    quote2 -i p

    local O="/tmp/$v-comments%d.json"
    local o
    local k=0
    while :; do
        printf -v o "$O" "$k"
        [ ! -f "$o" ] && break
        (( k++ ))
    done
    quote o

    k="$APP_KEY"
    quote2 -i k
    local a="$AGENT"
    quote2 a

    local c="\
wget \
--debug \
--verbose \
--no-check-certif \
--output-document=$o \
--user-agent=$a \
'https://www.googleapis.com/youtube/v3/commentThreads?key=$k&videoId=$v&part=replies,snippet&order=relevance&maxResults=100&textFormat=plainText&alt=json${p:+&pageToken=$p}'"

    $x "$c"
}

$ PAGE_TOKEN=...

$ AGENT=... APP_KEY=... comments-wget CJ_GCPaKywg "$PAGE_TOKEN"
Setting --verbose (verbose) to 1
Setting --check-certificate (checkcertificate) to 0
Setting --output-document (outputdocument) to /tmp/CJ_GCPaKywg-comments0.json
Setting --user-agent (useragent) to ...
DEBUG output created by Wget 1.14 on linux-gnu.

--2019-06-10 17:41:11--  https://www.googleapis.com/youtube/v3/commentThreads?...
Resolving www.googleapis.com... 172.217.19.106, 216.58.214.202, 216.58.214.234, ...
Caching www.googleapis.com => 172.217.19.106 216.58.214.202 216.58.214.234 172.217.16.106 172.217.20.10 2a00:1450:400d:808::200a
Connecting to www.googleapis.com|172.217.19.106|:443... connected.
Created socket 5.
Releasing 0x0000000000ae57c0 (new refcount 1).

---request begin---
GET /youtube/v3/commentThreads?.../1.1
User-Agent: ...
Accept: */*
Host: www.googleapis.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Expires: Mon, 10 Jun 2019 14:43:39 GMT
Date: Mon, 10 Jun 2019 14:43:39 GMT
Cache-Control: private, max-age=0, must-revalidate, no-transform
ETag: "XpPGQXPnxQJhLgs6enD_n8JR4Qk/OUAqOrEpA9YYqmVx0wqn9en_OrE"
Vary: Origin
Vary: X-Origin
Content-Type: application/json; charset=UTF-8
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Length: 205965
Server: GSE
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"

---response end---
200 OK
Registered socket 5 for persistent reuse.
Length: 205965 (201K) [application/json]
Saving to: ‘/tmp/CJ_GCPaKywg-comments0.json’

100%[==========================================>] 205,965      580KB/s   in 0.3s   

2019-06-10 17:41:18 (580 KB/s) - ‘/tmp/CJ_GCPaKywg-comments0.json’ saved [205965/205965]

请注意，上面的shell函数quote和quote2是youtube-data.sh中的壳函数（实际上并不需要）。从上面发布的JSON请求对象的$PAGE_TOKEN字符串中提取body。

下一个问题是：为什么您的python代码使用POST请求方法？可能是造成问题的原因吗？

Answer 3

问题是，我们无法真正检索每个视频的所有评论。

https://issuetracker.google.com/issues/134912604
我们目前不支持在整个流中进行分页。因此，无法检索该视频的所有1000+条CommentThreads

检索CommentThreads列表时出现processingFailure错误（400）

3 个答案: