Question

我有以下代码，我试图调用api 10000次，但我收到错误：

package main

import (
    "fmt"

    "net/http"
    "runtime"
    "sync"
    "time"
)

func main() {

    nCPU := runtime.NumCPU()
    runtime.GOMAXPROCS(nCPU)

    var wg sync.WaitGroup
    totalRequests := 100000
    wg.Add(totalRequests)

    fmt.Println("Starting Go Routines")

    start := time.Now()
    total := 0

    for i := 0; i < totalRequests; i++ {

        go func(current int) {
            defer wg.Done()

            startFunc := time.Now()
            _, err := http.Get("http://127.0.0.1:8080/event/list")
            // resp, err := http.Get("https://graph.facebook.com/v2.4/me" + "?fields=id%2Cname&access_token=" + "CAACEdEose0cBAEpQvcsvVMQu5oZCyyDjcEPQi9yCdiXimm4F0AYexGHPZAJHgpyrFOJN5X1VMcicNJjlkaCquUqHMZAfRrtxx6K9cRIROrA0OmbqAqCcg8ZA3qJZCHCl68I1n4LtFb5qxPcotlP5ne5PBakK0OUa7sc6FAOWwByOnFtNZBpIe8XDeM4YFa33sDfftVUpZCoBgZDZD")

            if err != nil {
                fmt.Println(err)
            }
            // defer resp.Body.Close()
            elapsedFunc := time.Since(startFunc)
            fmt.Println("The request (", current, ") took", elapsedFunc, "No of requests completed", total)
            total++

        }(i)

    }

    wg.Wait()
    elapsed := time.Since(start)
    fmt.Println("\nThe total time with cores", elapsed)
    fmt.Println("\nTerminating Program")
}

我得到的错误：

Get http://127.0.0.1:8080/event/list: dial tcp 127.0.0.1:8080: socket: too many open files
The request ( 5390 ) took 1.619876633s No of requests completed 2781
Get http://127.0.0.1:8080/event/list: dial tcp 127.0.0.1:8080: socket: too many open files
The request ( 7348 ) took 650.609825ms No of requests completed 1445

Answer 1

正如评论中提到的其他人一样，您的主要问题是您超出了流程的开放文件限制。

您可以使用通道轻松实现信号量来限制并发：

totalRequests := 100000
concurrency := 1024
sem := make(chan bool, concurrency)

start := time.Now()
total := int32(0)

for i := 0; i < totalRequests; i++ {
    sem <- true

    go func(current int) {
        startTime := time.Now()

        // Make request here

        elapsedTime := time.Since(startTime)
        atomic.AddInt32(&total, 1)
        fmt.Printf("Request %d took %s. Requests completed: %d\n", current, elapsedTime, atomic.LoadInt32(&total))

        <-sem
    }(i)
}

for i := 0; i < cap(sem); i++ {
    sem <- true
}
elapsedTotal := time.Since(start)
fmt.Printf("\nTotal time elapsed: %s\n", elapsedTotal)

这会将并行请求的数量限制为concurrency中指定的任何内容。

正如您所看到的，total变量使用atomic包递增，因为我们正在从可能的并行goroutine修改该变量，这可能会在不安全地修改时产生不正确的总数，就像您所做的那样

请参阅此博客文章了解原始示例＆amp;在Go：http://jmoiron.net/blog/limiting-concurrency-in-go

中解释限制并发性

修改

正如下面JimB所提到的，另一种常见方法是让concurrency个goroutines在我们将它们提供给它们的同时完成这项工作。这是一个可用于此的通用do函数：

func do(total, concurrency int, fn func(int)) { workQueue := make(chan int, concurrency) var wg sync.WaitGroup wg.Add(concurrency) for i := 0; i < concurrency; i++ { go func() { for i := range workQueue { fn(i) } wg.Done() }() } for i := 0; i < total; i++ { workQueue <- i } close(workQueue) wg.Wait() }

我们生成了concurrency goroutines，然后开始向workQueue频道发送值，直到发送total为止。通过关闭workQueue通道，我们有效地终止了goroutines中的范围循环。之后我们等到剩下的所有goroutine都运行完毕。

对于有问题的用例，可以像这样使用：

totalRequests := 1000000 concurrency := 1024 do(totalRequests, concurrency, func(i int) { // Make request here fmt.Printf("Request %d done.\n", i) })

如何在每个例程调用api的情况下并行运行10000个goroutine？

1 个答案: