捕获实时窗口数据

Question

我想捕获指定应用程序窗口的图像数据（例如像素的红色，绿色，蓝色值），并定期（例如每100毫秒）在R中捕获此数据，以制造游戏机器人。理想情况下，我希望将数据直接读取到数据框中，以便可以立即将其用作学习算法的输入。

有哪些建议的方法可以做到这一点？我在Ubuntu linux上使用R。

Answer 1

首先，您需要一种方法来捕获屏幕截图并将其保存到（临时）文件中。我发现（使用this和this的最好方法是使用ImageMagick。

如果已安装ImageMagick，则可以使用以下R函数/ skript捕获屏幕，加载数据并在R中绘制数据（您可能希望根据所需的分辨率等调整参数）。 / p>

最后，如果您想对图像进行更多处理，建议使用imager软件包。

跳过并加载图像到R

library(imager) # for loading and plotting of the PNG file
library(glue)   # for string parsing

capture_screen <- function(file = "screenshot.png") {
  system(glue("import -window root -resize 2560x1440 -delay 200 {file}"))
  invisible(file)
}

file <- capture_screen()

shot <- load.image(file)

shot
#> Image. Width: 2560 pix Height: 1440 pix Depth: 1 Colour channels: 3

shot[1:10, 1:10, ]
#> , , 1
#> 
#>            [,1]      [,2]      [,3]      [,4]      [,5]      [,6]
#>  [1,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [2,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [3,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [4,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [5,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [6,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [7,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [8,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>  [9,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#> [10,] 0.3440452 0.3385519 0.3357443 0.3331807 0.3284199 0.3248798
#>            [,7]      [,8]      [,9]     [,10]
#>  [1,] 0.3221790 0.3186389 0.3138781 0.3112993
#>  [2,] 0.3221790 0.3186389 0.3138781 0.3112993
#>  [3,] 0.3221790 0.3186389 0.3138781 0.3112993
#>  [4,] 0.3221790 0.3186389 0.3138781 0.3112993
#>  [5,] 0.3221790 0.3186389 0.3138781 0.3112993
#>  [6,] 0.3221790 0.3186389 0.3138781 0.3112993
#>  [7,] 0.3221790 0.3186389 0.3138781 0.3112993
#>  [8,] 0.3221790 0.3186389 0.3138781 0.3112840
#>  [9,] 0.3221790 0.3186389 0.3138018 0.3109636
#> [10,] 0.3221942 0.3185931 0.3136797 0.3123674
#> 
#> , , 2
#> 
#>            [,1]      [,2]      [,3]      [,4]      [,5]      [,6]
#>  [1,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [2,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [3,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [4,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [5,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [6,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [7,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [8,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>  [9,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#> [10,] 0.3319905 0.3292439 0.3244831 0.3209583 0.3181506 0.3152209
#>            [,7]      [,8]      [,9]     [,10]
#>  [1,] 0.3121691 0.3096513 0.3048905 0.3013504
#>  [2,] 0.3121691 0.3096513 0.3048905 0.3013504
#>  [3,] 0.3121691 0.3096513 0.3048905 0.3013504
#>  [4,] 0.3121691 0.3096513 0.3048905 0.3013504
#>  [5,] 0.3121691 0.3096513 0.3048905 0.3013504
#>  [6,] 0.3121691 0.3096513 0.3048905 0.3013504
#>  [7,] 0.3121691 0.3096513 0.3048905 0.3013504
#>  [8,] 0.3121691 0.3096513 0.3048905 0.3013352
#>  [9,] 0.3121691 0.3096513 0.3048600 0.3008164
#> [10,] 0.3121996 0.3095750 0.3045701 0.3070420
#> 
#> , , 3
#> 
#>           [,1]      [,2]      [,3]      [,4]      [,5]      [,6]      [,7]
#>  [1,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [2,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [3,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [4,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [5,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [6,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [7,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [8,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#>  [9,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2828412
#> [10,] 0.300679 0.2975357 0.2945907 0.2916915 0.2887007 0.2857862 0.2829328
#>            [,8]      [,9]     [,10]
#>  [1,] 0.2799268 0.2769360 0.2740215
#>  [2,] 0.2799268 0.2769360 0.2740215
#>  [3,] 0.2799268 0.2769360 0.2740215
#>  [4,] 0.2799268 0.2769360 0.2740215
#>  [5,] 0.2799268 0.2769360 0.2740215
#>  [6,] 0.2799268 0.2769360 0.2740215
#>  [7,] 0.2799268 0.2769360 0.2740215
#>  [8,] 0.2799268 0.2769360 0.2740215
#>  [9,] 0.2799268 0.2769207 0.2734569
#> [10,] 0.2796063 0.2775616 0.2841840

绘制图像

plot(shot)

# render only a subset
imsub(shot, y <= 200, y >= 100, x <= 500, x >= 200) %>% plot()

^{由reprex package（v0.2.1）于2019-03-25创建}

编辑

如果您只想加载没有screenshot.png文件的数据，也可以这样做

capture_screen2 <- function(xmin = 0, xmax = Inf, ymin = 0, ymax = Inf) {
  tmp <- tempfile(fileext = ".png")
  system(glue("import -window root -resize 2560x1440 -delay 200 {tmp}"))

  img <- load.image(tmp)
  a <- try(unlink(tmp))
  imsub(img, x >= xmin, x <= xmax, y >= ymin, y <= ymax)
}

shot <- capture_screen2()

编辑2：速度问题

我做了benchmarking的代码速度，最快的选择是将scrot（在Ubuntu上为sudo apt install scrot）与png软件包结合使用：< / p>

system(glue::glue("scrot --silent screenshot.png"))
img <- png::readPNG("screenshot.png")
dim(img)