我正在使用项目管道运行Scrapy项目,该项目管道将数据写入PostgreSQL数据库。它是Docker Compose多容器应用程序,具有以下docker-compose.yml
:
version: '3'
services:
privoxy:
build: ./privoxy
links:
- tor
tor:
build:
context: ./tor
args:
password: "foo"
scraper:
build: ./scraper
environment:
- http_proxy=http://privoxy:8118
- PGPASSWORD=iperpassword
links:
- tor
- privoxy
- db
volumes:
- scraper-data:/scraper/apkmirror_scraper/data
- scraper-jobdir:/scraper/apkmirror_scraper/crawls
db:
image: postgres
environment:
- POSTGRES_PASSWORD=iperpassword
volumes:
- db-data:/var/lib/postgresql/data
volumes:
scraper-data:
scraper-jobdir:
db-data:
我正在使用官方存储库中的postgres
图像(https://hub.docker.com/_/postgres/),并将/var/lib/postgresql/data
目录放入https://docs.docker.com/compose/compose-file/#volume-configuration-reference之后的卷中。
问题在于,当我启动应用程序时,db
服务会输出与我之前尝试过的事情有关的日志和错误,而不是我现在尝试做的事情。这就是我所看到的:
apkmirrorscrapercompose_tor_1 is up-to-date
apkmirrorscrapercompose_db_1 is up-to-date
apkmirrorscrapercompose_privoxy_1 is up-to-date
Recreating apkmirrorscrapercompose_scraper_1
Attaching to apkmirrorscrapercompose_tor_1, apkmirrorscrapercompose_db_1, apkmirrorscrapercompose_privoxy_1, apkmirrorscrapercompose_scraper_1
scraper_1 | List of databases
scraper_1 | Name | Owner | Encoding | Collate | Ctype | Access privileges
scraper_1 | -----------+----------+----------+------------+------------+-----------------------
scraper_1 | postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
scraper_1 | template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
scraper_1 | | | | | | postgres=CTc/postgres
scraper_1 | template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
scraper_1 | | | | | | postgres=CTc/postgres
scraper_1 | (3 rows)
scraper_1 |
scraper_1 | Postgres is up - executing command
scraper_1 | Tor appears to be working. Proceeding with command...
scraper_1 | 2017-06-30 12:32:59 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: apkmirror_scraper)
scraper_1 | 2017-06-30 12:32:59 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'apkmirror_scraper', 'NEWSPIDER_MODULE': 'apkmirror_scraper.spiders', 'SPIDER_MODULES': ['apkmirror_scraper.spiders']}
scraper_1 | 2017-06-30 12:32:59 [scrapy.middleware] INFO: Enabled extensions:
scraper_1 | ['scrapy.extensions.corestats.CoreStats',
scraper_1 | 'scrapy.extensions.telnet.TelnetConsole',
scraper_1 | 'scrapy.extensions.memusage.MemoryUsage',
scraper_1 | 'scrapy.extensions.closespider.CloseSpider',
scraper_1 | 'scrapy.extensions.logstats.LogStats']
db_1 | The files belonging to this database system will be owned by user "postgres".
db_1 | This user must also own the server process.
db_1 |
db_1 | The database cluster will be initialized with locale "en_US.utf8".
db_1 | The default database encoding has accordingly been set to "UTF8".
db_1 | The default text search configuration will be set to "english".
db_1 |
db_1 | Data page checksums are disabled.
db_1 |
db_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1 | creating subdirectories ... ok
db_1 | selecting default max_connections ... 100
db_1 | selecting default shared_buffers ... 128MB
db_1 | selecting dynamic shared memory implementation ... posix
db_1 | creating configuration files ... ok
db_1 | running bootstrap script ... ok
db_1 | performing post-bootstrap initialization ... ok
db_1 |
db_1 | WARNING: enabling "trust" authentication for local connections
db_1 | You can change this by editing pg_hba.conf or using the option -A, or
db_1 | --auth-local and --auth-host, the next time you run initdb.
db_1 | syncing data to disk ... ok
db_1 |
db_1 | Success. You can now start the database server using:
db_1 |
db_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1 |
db_1 | waiting for server to start....LOG: database system was shut down at 2017-06-29 13:40:40 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | done
db_1 | server started
db_1 | ALTER ROLE
db_1 |
db_1 |
db_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1 |
db_1 | LOG: received fast shutdown request
db_1 | waiting for server to shut down....LOG: aborting any active transactions
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | done
db_1 | server stopped
db_1 |
db_1 | PostgreSQL init process complete; ready for start up.
db_1 |
db_1 | LOG: database system was shut down at 2017-06-29 13:40:41 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:41:53 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:43:22 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 13:52:02 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
db_1 | LOG: database system is ready to accept connections
db_1 | LOG: autovacuum launcher started
db_1 | ERROR: column "developer" of relation "apks" does not exist at character 31
db_1 | STATEMENT: INSERT INTO apks (url, title, developer, app, version_name, version_code, architectures, package, apk_file_size, android_min_version, android_target_version, supported_dpis, md5_signature, time_uploaded, time_scraped, image_urls, images, file_urls, files, created_on, updated_on) VALUES ('http://www.apkmirror.com/apk/sony-mobile-communications/backup-restore-unknown-developer/backup-restore-unknown-developer-1-2-a-1-23-release/backup-restore-1-2-a-1-23-android-apk-download/', 'Backup & restore 1.2.A.1.23', 'Sony Mobile Communications', 'Backup & restore', '1.2.A.1.23', '2360343', ARRAY['arm'], 'com.sonymobile.synchub', 9541452, '6.0', '7.1', ARRAY['nodpi'], 'eda39fedfc33ea939a5bb4926b3980c1', '2017-04-25T16:30:00+00:00'::timestamptz, '2017-06-29T14:10:39+00:00'::timestamptz, ARRAY['http://www.apkmirror.com/wp-content/themes/APKMirror/ap_resize/ap_resize.php?src=http%3A%2F%2Fwww.apkmirror.com%2Fwp-content%2Fuploads%2F2017%2F04%2F58ff36aea8e0c.png&w=96&h=96&q=100'], '{}', ARRAY['http://www.apkmirror.com/wp-content/themes/APKMirror/download.php?id=203068'], '{}', '2017-06-29T14:19:21.661193'::timestamp, '2017-06-29T14:19:21.661207'::timestamp) RETURNING apks.id
db_1 | LOG: received smart shutdown request
db_1 | LOG: autovacuum launcher shutting down
db_1 | LOG: shutting down
db_1 | LOG: database system is shut down
db_1 | LOG: database system was shut down at 2017-06-29 16:28:54 UTC
db_1 | LOG: MultiXact member wraparound protections are now enabled
虽然我没有显示有关应用程序的详细信息,但我的主要观点是,我肯定错误消息中显示的数据来自之前的运行,而不是来自我下载的APK下载页面#&# 39; m目前正试图刮伤。
简而言之,似乎PostgreSQL正在缓存'以前的陈述,即使在我停止申请后也尝试重新运行它们,请docker-compose build
和docker-compose up
。在我看来,这种行为必须以某种方式由/var/lib/postgresql/data
中的数据驱动,因为这个卷是重新启动容器时唯一的持久性。
确实如此吗?如果是这样,我怎么能禁用这个语句缓存'行为?
答案 0 :(得分:3)
这只是意味着docker-compose正在重新启动先前停止的容器(或重新连接到它,如果它已经在运行),并且因为没有必要重新构建映像:
apkmirrorscrapercompose_db_1 is up-to-date
因此,您在日志中“再次”看到的是该容器的docker logs
,其中包含postgres的所有生命生成日志。
只需删除容器并重新开始(数据库中的数据将保留在卷中,因此它将继续存在):
docker-compose rm db