Question

问题：

我在高峰使用时间遇到性能瓶颈，而系统资源仍然可用。

在高峰使用时间内，我在服务器上遇到响应缓慢2小时的响应。

背景

我正在运行 Revive Adserver PHP + MySQL应用的专用服务器上工作，以便在具有以下规范的专用服务器上每天提供9百万次展示：

CPU: AMD 2.3Ghz 8-cores
RAM: 32GB
SSD: 250GB x 2, RAID-1

下图是高峰使用时间内的数字：

load average: 1.18, 0.83, 0.65
load average: 1.00, 0.81, 0.64
load average: 1.23, 0.86, 0.66

             total       used       free     shared    buffers     cached
Mem:         31820      15843      15977         79        147      13424
-/+ buffers/cache:       2271      29549 
Swap:         2047        405       1642

netstat -an | grep :80 | wc -l给了我：

11232

iotop给我的性能低于10M / s，而大多数统计数据都带有写活动的mysql进程

Total DISK READ: 0.00 B/s | Total DISK WRITE: 6.63 M/s

磁盘性能测试在高峰使用时间内给我168MB/s，这意味着我的SSD性能仍有很大的空间：

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync; unlink test

16384+0 records in
16384+0 records out
1073741824 bytes (1.1 GB) copied, 6.38736 s, 168 MB/s

服务器使用 Nginx 作为服务静态内容的反向代理：

nginx -v

nginx version: nginx/1.9.4

服务器正在使用 Apache 进行FastCGI PHP处理程序：

httpd -V

Server version: Apache/2.2.15 (Unix)
   :
   :
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)

php -v

PHP 5.4.45 (cli) (built: Sep  9 2015 14:53:48) 
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
    with the ionCube PHP Loader v4.6.0, Copyright (c) 2002-2014, by ionCube Ltd.

MRTG图表在高峰使用时间内给出了稳定的结果（在100 Mbit / s连接上）：

44 Mbit/s outgoing throughput

3.5 Mbit/s incoming throughput

据我所知，服务器工作量似乎在容量之下。

我观察到并怀疑的内容：

netstat -an | grep :80 | wc -l 11232的结果很高，我需要对此做点什么。
MySQL需要进行微调，因为广告投放会不断对数据库进行读/写活动。
瓶颈可能是在给定时间运行的php个进程的数量还是设置了任何其他限制或上限？

从我上面提到的，如何识别和微调性能瓶颈？

我不知道从哪里开始。

我只能在高峰使用时间内进行2小时的实时测试。

更新1：

摘录httpd.conf

Timeout 60
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 5

<IfModule prefork.c>
StartServers       8
MinSpareServers    25
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

SHOW VARIABLES;

auto_increment_increment    1
auto_increment_offset   1
autocommit  ON
automatic_sp_privileges ON
back_log    50
basedir /usr
big_tables  OFF
binlog_cache_size   32768
binlog_direct_non_transactional_updates OFF
binlog_format   STATEMENT
binlog_stmt_cache_size  32768
bulk_insert_buffer_size 8388608
character_set_client    utf8
character_set_connection    utf8
character_set_database  latin1
character_set_filesystem    binary
character_set_results   utf8
character_set_server    latin1
character_set_system    utf8
character_sets_dir  /usr/share/mysql/charsets/
collation_connection    utf8_unicode_ci
collation_database  latin1_swedish_ci
collation_server    latin1_swedish_ci
completion_type NO_CHAIN
concurrent_insert   AUTO
connect_timeout 10
datadir /var/lib/mysql/
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_storage_engine  InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit    100
delayed_insert_timeout  300
delayed_queue_size  1000
div_precision_increment 4
engine_condition_pushdown   ON
error_count 0
event_scheduler OFF
expire_logs_days    0
external_user   
flush   OFF
flush_time  0
foreign_key_checks  ON
ft_boolean_syntax   + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit    20
ft_stopword_file    (built-in)
general_log OFF
general_log_file    /var/lib/mysql/adsrv336.log
group_concat_max_len    1024
have_compress   YES
have_crypt  YES
have_csv    YES
have_dynamic_loading    YES
have_geometry   YES
have_innodb YES
have_ndbcluster NO
have_openssl    DISABLED
have_partitioning   YES
have_profiling  YES
have_query_cache    YES
have_rtree_keys YES
have_ssl    DISABLED
have_symlink    DISABLED
hostname    adsrv336.dedicatedhost.com
identity    0
ignore_builtin_innodb   OFF
init_connect    
init_file   
init_slave  
innodb_adaptive_flushing    ON
innodb_adaptive_hash_index  ON
innodb_additional_mem_pool_size 8388608
innodb_autoextend_increment 8
innodb_autoinc_lock_mode    1
innodb_buffer_pool_instances    1
innodb_buffer_pool_size 134217728
innodb_change_buffering all
innodb_checksums    ON
innodb_commit_concurrency   0
innodb_concurrency_tickets  500
innodb_data_file_path   ibdata1:10M:autoextend
innodb_data_home_dir    
innodb_doublewrite  ON
innodb_fast_shutdown    1
innodb_file_format  Antelope
innodb_file_format_check    ON
innodb_file_format_max  Antelope
innodb_file_per_table   OFF
innodb_flush_log_at_trx_commit  1
innodb_flush_method 
innodb_force_load_corrupted OFF
innodb_force_recovery   0
innodb_io_capacity  200
innodb_large_prefix OFF
innodb_lock_wait_timeout    50
innodb_locks_unsafe_for_binlog  OFF
innodb_log_buffer_size  8388608
innodb_log_file_size    5242880
innodb_log_files_in_group   2
innodb_log_group_home_dir   ./
innodb_max_dirty_pages_pct  75
innodb_max_purge_lag    0
innodb_mirrored_log_groups  1
innodb_old_blocks_pct   37
innodb_old_blocks_time  0
innodb_open_files   300
innodb_print_all_deadlocks  OFF
innodb_purge_batch_size 20
innodb_purge_threads    0
innodb_random_read_ahead    OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads  4
innodb_replication_delay    0
innodb_rollback_on_timeout  OFF
innodb_rollback_segments    128
innodb_spin_wait_delay  6
innodb_stats_method nulls_equal
innodb_stats_on_metadata    ON
innodb_stats_sample_pages   8
innodb_strict_mode  OFF
innodb_support_xa   ON
innodb_sync_spin_loops  30
innodb_table_locks  ON
innodb_thread_concurrency   0
innodb_thread_sleep_delay   10000
innodb_use_native_aio   ON
innodb_use_sys_malloc   ON
innodb_version  5.5.44
innodb_write_io_threads 4
insert_id   0
interactive_timeout 28800
join_buffer_size    131072
keep_files_on_create    OFF
key_buffer_size 8388608
key_cache_age_threshold 300
key_cache_block_size    1024
key_cache_division_limit    100
large_files_support ON
large_page_size 0
large_pages OFF
last_insert_id  0
lc_messages en_US
lc_messages_dir /usr/share/mysql/
lc_time_names   en_US
license GPL
local_infile    OFF
lock_wait_timeout   31536000
locked_in_memory    OFF
log OFF
log_bin OFF
log_bin_trust_function_creators OFF
log_error   /var/log/mysqld.log
log_output  FILE
log_queries_not_using_indexes   OFF
log_slave_updates   OFF
log_slow_queries    OFF
log_warnings    1
long_query_time 10.000000
low_priority_updates    OFF
lower_case_file_system  OFF
lower_case_table_names  0
max_allowed_packet  1048576
max_binlog_cache_size   18446744073709547520
max_binlog_size 1073741824
max_binlog_stmt_cache_size  18446744073709547520
max_connect_errors  10
max_connections 151
max_delayed_threads 20
max_error_count 64
max_heap_table_size 16777216
max_insert_delayed_threads  20
max_join_size   18446744073709551615
max_length_for_sort_data    1024
max_long_data_size  1048576
max_prepared_stmt_count 16382
max_relay_log_size  0
max_seeks_for_key   18446744073709551615
max_sort_length 1024
max_sp_recursion_depth  0
max_tmp_tables  32
max_user_connections    0
max_write_lock_count    18446744073709551615
metadata_locks_cache_size   1024
min_examined_row_limit  0
multi_range_count   256
myisam_data_pointer_size    6
myisam_max_sort_file_size   9223372036853727232
myisam_mmap_size    18446744073709551615
myisam_recover_options  OFF
myisam_repair_threads   1
myisam_sort_buffer_size 8388608
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
net_buffer_length   16384
net_read_timeout    30
net_retry_count 10
net_write_timeout   60
new OFF
old OFF
old_alter_table OFF
old_passwords   OFF
open_files_limit    1024
optimizer_prune_level   1
optimizer_search_depth  62
optimizer_switch    index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on
performance_schema  OFF
performance_schema_events_waits_history_long_size   10000
performance_schema_events_waits_history_size    10
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances   1000
performance_schema_max_file_classes 50
performance_schema_max_file_handles 32768
performance_schema_max_file_instances   10000
performance_schema_max_mutex_classes    200
performance_schema_max_mutex_instances  1000000
performance_schema_max_rwlock_classes   30
performance_schema_max_rwlock_instances 1000000
performance_schema_max_table_handles    100000
performance_schema_max_table_instances  50000
performance_schema_max_thread_classes   50
performance_schema_max_thread_instances 1000
pid_file    /var/run/mysqld/mysqld.pid
plugin_dir  /usr/lib64/mysql/plugin/
port    3306
preload_buffer_size 32768
profiling   OFF
profiling_history_size  15
protocol_version    10
proxy_user  
pseudo_slave_mode   OFF
pseudo_thread_id    2107455
query_alloc_block_size  8192
query_cache_limit   1048576
query_cache_min_res_unit    4096
query_cache_size    0
query_cache_type    ON
query_cache_wlock_invalidate    OFF
query_prealloc_size 8192
rand_seed1  0
rand_seed2  0
range_alloc_block_size  4096
read_buffer_size    131072
read_only   OFF
read_rnd_buffer_size    262144
relay_log   
relay_log_index 
relay_log_info_file relay-log.info
relay_log_purge ON
relay_log_recovery  OFF
relay_log_space_limit   0
report_host 
report_password 
report_port 3306
report_user 
rpl_recovery_rank   0
secure_auth OFF
secure_file_priv    
server_id   0
skip_external_locking   ON
skip_name_resolve   OFF
skip_networking OFF
skip_show_database  OFF
slave_compressed_protocol   OFF
slave_exec_mode STRICT
slave_load_tmpdir   /tmp
slave_max_allowed_packet    1073741824
slave_net_timeout   3600
slave_skip_errors   OFF
slave_transaction_retries   10
slave_type_conversions  
slow_launch_time    2
slow_query_log  OFF
slow_query_log_file /var/lib/mysql/adsrv336-slow.log
socket  /var/lib/mysql/mysql.sock
sort_buffer_size    2097152
sql_auto_is_null    OFF
sql_big_selects ON
sql_big_tables  OFF
sql_buffer_result   OFF
sql_log_bin ON
sql_log_off OFF
sql_low_priority_updates    OFF
sql_max_join_size   18446744073709551615
sql_mode    
sql_notes   ON
sql_quote_show_create   ON
sql_safe_updates    OFF
sql_select_limit    18446744073709551615
sql_slave_skip_counter  0
sql_warnings    OFF
ssl_ca  
ssl_capath  
ssl_cert    
ssl_cipher  
ssl_key 
storage_engine  InnoDB
stored_program_cache    256
sync_binlog 0
sync_frm    ON
sync_master_info    0
sync_relay_log  0
sync_relay_log_info 0
system_time_zone    UTC
table_definition_cache  400
table_open_cache    400
thread_cache_size   0
thread_concurrency  10
thread_handling one-thread-per-connection
thread_stack    262144
thread_statistics   OFF
time_format %H:%i:%s
time_zone   SYSTEM
timed_mutexes   OFF
timestamp   1486188305
tmp_table_size  16777216
tmpdir  /tmp
transaction_alloc_block_size    8192
transaction_prealloc_size   4096
tx_isolation    REPEATABLE-READ
unique_checks   ON
updatable_views_with_limit  YES
userstat    OFF
version 5.5.44-cll-lve
version_comment MySQL Community Server (GPL) by Atomicorp
version_compile_machine x86_64
version_compile_os  Linux
wait_timeout    28800
warning_count   0

SHOW GLOBAL STATUS;

Aborted_clients 2188
Aborted_connects    30745
Binlog_cache_disk_use   0
Binlog_cache_use    0
Binlog_stmt_cache_disk_use  0
Binlog_stmt_cache_use   0
Bytes_received  336007637149
Bytes_sent  135868200114
Com_admin_commands  2165671522
Com_assign_to_keycache  0
Com_alter_db    0
Com_alter_db_upgrade    0
Com_alter_event 0
Com_alter_function  0
Com_alter_procedure 0
Com_alter_server    0
Com_alter_table 0
Com_alter_tablespace    0
Com_analyze 0
Com_begin   2582
Com_binlog  0
Com_call_procedure  0
Com_change_db   2167726231
Com_change_master   0
Com_check   721
Com_checksum    0
Com_commit  2582
Com_create_db   0
Com_create_event    0
Com_create_function 0
Com_create_index    5174
Com_create_procedure    0
Com_create_server   0
Com_create_table    2587
Com_create_trigger  0
Com_create_udf  0
Com_create_user 0
Com_create_view 0
Com_dealloc_sql 10294
Com_delete  90591
Com_delete_multi    107
Com_do  0
Com_drop_db 0
Com_drop_event  0
Com_drop_function   0
Com_drop_index  0
Com_drop_procedure  0
Com_drop_server 0
Com_drop_table  0
Com_drop_trigger    0
Com_drop_user   0
Com_drop_view   0
Com_empty_query 0
Com_enable_governor 0
Com_enable_governor_reconn  0
Com_enable_governor_lve 0
Com_enable_governor_reconn_lve  0
Com_execute_sql 10294
Com_flush   0
Com_grant   0
Com_ha_close    0
Com_ha_open 0
Com_ha_read 0
Com_help    0
Com_insert  732167877
Com_insert_select   7871
Com_install_plugin  0
Com_kill    0
Com_load    2587
Com_lock_tables 336
Com_lvecmd  0
Com_optimize    0
Com_preload_keys    0
Com_prepare_sql 10294
Com_purge   0
Com_purge_before_date   0
Com_release_savepoint   0
Com_rename_table    0
Com_rename_user 0
Com_repair  0
Com_replace 61
Com_replace_select  0
Com_reset   0
Com_resignal    0
Com_revoke  0
Com_revoke_all  0
Com_rollback    0
Com_rollback_to_savepoint   0
Com_savepoint   0
Com_select  16316658
Com_set_option  4335664028
Com_signal  0
Com_show_authors    0
Com_show_binlog_events  0
Com_show_binlogs    3
Com_show_charsets   0
Com_show_client_statistics  0
Com_show_collations 0
Com_show_contributors   0
Com_show_create_db  321
Com_show_create_event   0
Com_show_create_func    0
Com_show_create_proc    0
Com_show_create_table   39055
Com_show_create_trigger 0
Com_show_databases  965
Com_show_engine_logs    0
Com_show_engine_mutex   0
Com_show_engine_status  0
Com_show_events 0
Com_show_errors 0
Com_show_fields 326814
Com_show_function_status    0
Com_show_grants 3
Com_show_index_statistics   0
Com_show_keys   49105
Com_show_master_status  11
Com_show_open_tables    0
Com_show_plugins    0
Com_show_privileges 0
Com_show_procedure_status   0
Com_show_processlist    5
Com_show_profile    0
Com_show_profiles   0
Com_show_relaylog_events    0
Com_show_slave_hosts    0
Com_show_slave_status   11
Com_show_status 17
Com_show_storage_engines    0
Com_show_table_statistics   0
Com_show_table_status   39590
Com_show_tables 26194
Com_show_thread_statistics  0
Com_show_triggers   38841
Com_show_user_statistics    0
Com_show_variables  2792
Com_show_warnings   0
Com_slave_start 0
Com_slave_stop  0
Com_stmt_close  10294
Com_stmt_execute    10294
Com_stmt_fetch  0
Com_stmt_prepare    10294
Com_stmt_reprepare  0
Com_stmt_reset  0
Com_stmt_send_long_data 0
Com_truncate    2587
Com_uninstall_plugin    0
Com_unlock_tables   336
Com_update  44026
Com_update_multi    5124
Com_xa_commit   0
Com_xa_end  0
Com_xa_prepare  0
Com_xa_recover  0
Com_xa_rollback 0
Com_xa_start    0
Compression OFF
Connections 2107678
Created_tmp_disk_tables 2899422
Created_tmp_files   6
Created_tmp_tables  3078901
Delayed_errors  0
Delayed_insert_threads  0
Delayed_writes  0
Enable_governor 0
Flush_commands  1
Handler_commit  748577022
Handler_delete  5716734
Handler_discover    0
Handler_prepare 0
Handler_read_first  377473
Handler_read_key    1100551034
Handler_read_last   57
Handler_read_next   403408954
Handler_read_prev   437
Handler_read_rnd    11287933
Handler_read_rnd_next   261237848
Handler_rollback    84
Handler_savepoint   0
Handler_savepoint_rollback  0
Handler_update  790681045
Handler_write   781433379
Innodb_buffer_pool_pages_data   7724
Innodb_buffer_pool_bytes_data   126550016
Innodb_buffer_pool_pages_dirty  594
Innodb_buffer_pool_bytes_dirty  9732096
Innodb_buffer_pool_pages_flushed    91473015
Innodb_buffer_pool_pages_free   3
Innodb_buffer_pool_pages_misc   465
Innodb_buffer_pool_pages_total  8192
Innodb_buffer_pool_read_ahead_rnd   0
Innodb_buffer_pool_read_ahead   2767609
Innodb_buffer_pool_read_ahead_evicted   57329
Innodb_buffer_pool_read_requests    12257284809
Innodb_buffer_pool_reads    27826982
Innodb_buffer_pool_wait_free    0
Innodb_buffer_pool_write_requests   4117902862
Innodb_data_fsyncs  570895897
Innodb_data_pending_fsyncs  1
Innodb_data_pending_reads   0
Innodb_data_pending_writes  0
Innodb_data_read    501890338816
Innodb_data_reads   30632828
Innodb_data_writes  655956295
Innodb_data_written 3482707620864
Innodb_dblwr_pages_written  91473015
Innodb_dblwr_writes 7099319
Innodb_have_atomic_builtins ON
Innodb_log_waits    0
Innodb_log_write_requests   396703814
Innodb_log_writes   555866212
Innodb_os_log_fsyncs    556959782
Innodb_os_log_pending_fsyncs    1
Innodb_os_log_pending_writes    0
Innodb_os_log_written   484844787200
Innodb_page_size    16384
Innodb_pages_created    155964
Innodb_pages_read   30632831
Innodb_pages_written    91473015
Innodb_row_lock_current_waits   0
Innodb_row_lock_time    7629691
Innodb_row_lock_time_avg    79
Innodb_row_lock_time_max    141020
Innodb_row_lock_waits   96421
Innodb_rows_deleted 5716233
Innodb_rows_inserted    26794984
Innodb_rows_read    1809037295
Innodb_rows_updated 726405176
Innodb_truncated_status_writes  0
Key_blocks_not_flushed  0
Key_blocks_unused   6679
Key_blocks_used 563
Key_read_requests   7690348
Key_reads   3306
Key_write_requests  5070367
Key_writes  501
Last_query_cost 0.000000
Max_used_connections    113
Not_flushed_delayed_rows    0
Open_files  201
Open_streams    0
Open_table_definitions  400
Open_tables 400
Opened_files    11858607
Opened_table_definitions    78012
Opened_tables   151658
Performance_schema_cond_classes_lost    0
Performance_schema_cond_instances_lost  0
Performance_schema_file_classes_lost    0
Performance_schema_file_handles_lost    0
Performance_schema_file_instances_lost  0
Performance_schema_locker_lost  0
Performance_schema_mutex_classes_lost   0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost  0
Performance_schema_rwlock_instances_lost    0
Performance_schema_table_handles_lost   0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost  0
Performance_schema_thread_instances_lost    0
Prepared_stmt_count 0
Qcache_free_blocks  0
Qcache_free_memory  0
Qcache_hits 0
Qcache_inserts  0
Qcache_lowmem_prunes    0
Qcache_not_cached   0
Qcache_queries_in_cache 0
Qcache_total_blocks 0
Queries 9420404640
Questions   7254733117
Rpl_status  AUTH_MASTER
Select_full_join    23575
Select_full_range_join  1007
Select_range    2816388
Select_range_check  0
Select_scan 3328035
Slave_heartbeat_period  0.000
Slave_open_temp_tables  0
Slave_received_heartbeats   0
Slave_retried_transactions  0
Slave_running   OFF
Slow_launch_threads 0
Slow_queries    36162
Sort_merge_passes   0
Sort_range  13900
Sort_rows   26363706
Sort_scan   44854
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher  
Ssl_cipher_list 
Ssl_client_connects 0
Ssl_connect_renegotiates    0
Ssl_ctx_verify_depth    0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts    0
Ssl_finished_connects   0
Ssl_session_cache_hits  0
Ssl_session_cache_misses    0
Ssl_session_cache_mode  NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size  0
Ssl_session_cache_timeouts  0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries  0
Ssl_verify_depth    0
Ssl_verify_mode 0
Ssl_version 
Table_locks_immediate   788956379
Table_locks_waited  0
Tc_log_max_pages_used   0
Tc_log_page_size    0
Tc_log_page_waits   0
Threads_cached  0
Threads_connected   8
Threads_created 2107677
Threads_running 2
Uptime  9223463
Uptime_since_flush_status   9223463

更新2：

以下是一些显示当前memcached统计信息的附加信息。不确定它是否表明存在任何与瓶颈有关的问题：

STAT pid 2032
STAT uptime 9253771
STAT time 1486219189
STAT version 1.4.24
STAT libevent 1.4.13-stable
STAT pointer_size 64
STAT rusage_user 15108.811111
STAT rusage_system 62027.906324
STAT curr_connections 7
STAT total_connections 1999851
STAT connection_structures 108
STAT reserved_fds 20
STAT cmd_get 2195755075
STAT cmd_set 19730696
STAT cmd_flush 73
STAT cmd_touch 0
STAT get_hits 2185756795
STAT get_misses 9998280
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 206042443758
STAT bytes_written 11178528686414
STAT limit_maxbytes 536870912
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT malloc_fails 0
STAT bytes 3683296
STAT curr_items 766
STAT total_items 9902981
STAT expired_unfetched 226228
STAT evicted_unfetched 0
STAT evictions 0
STAT reclaimed 1201893
STAT crawler_reclaimed 0
STAT crawler_items_checked 0
STAT lrutail_reflocked 16
END

更新3：

在另一天的高峰时段，我得到了：

> netstat -an | grep :80 | wc -l
11281

load average: 2.46, 1.96, 1.83

> free -m
             total       used       free     shared    buffers     cached
Mem:         31820      14713      17107         80        149      12342
-/+ buffers/cache:       2220      29600 
Swap:         2047        403       1644

此外，这里是fcgid.conf

FcgidIPCDir /var/run/mod_fcgid/sock
FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

FcgidIdleTimeout 40
FcgidProcessLifeTime 30
FcgidMaxRequestsPerProcess 100000
FcgidMaxProcesses 16
FcgidMaxProcessesPerClass 4
FcgidMinProcessesPerClass 0
FcgidConnectTimeout 60
FcgidIOTimeout 1000
FcgidInitialEnv RAILS_ENV production
FcgidIdleScanInterval 20

带宽使用时间为：

36 Mbit/s outgoing
3 Mbit/s incoming

同样，似乎还有很多免费资源可供使用。

使用更新的Fcgid*指令，单个页面上的广告需要8到10秒才能加载。

在非高峰时段，广告投放通常需要不到一秒钟到最多1.5秒才能加载。

另外，根据phpMyAdmin，数据库中最大的两个表是3.6GiB和2.8GiB，理论上它们应该适合我服务器上的32GB RAM。 / p>

我还注意到在我的服务器监控统计信息中，CPU进程在500休眠进程中不断修复。有关1周间隔图表，请参阅以下内容。这也可能是一个问题吗？

此处还是高峰使用时的完整服务器运行状况快照。

如何确定是否有任何与操作系统相关的限制会对性能设置上限？

例如。活动连接，文件描述符，mysql相关，php或httpd进程等。

更新4：

以下是过去7天的带宽图表。

上述问题在中午的高峰时段发生了2~3小时。

更新5：

调整MaxClients 64并尝试MaxClients 20后，结果不会发生太大变化，它仍需要5到10秒的时间来投放广告。

> netstat -an | grep :80 | wc -l
11310

重新启动innodb_buffer_pool_size=16G后，我尝试将/etc/my.cnf添加到mysqld，现在SHOW VARIABLES;给了我

innodb_buffer_pool_size 17179869184

top命令显示：

8517 mysql     20   0 18.4g 1.3g 7264 S 31.2  4.2   1:33.18 mysqld

但广告响应仍然需要5-10秒才能加载，当出现问题时，浏览器的状态栏始终显示waiting for domain-name.com...超过10秒。

我注意到了一件事。

每当我重新启动Apache时，一切都响应10秒，在apache重启后的前10秒内，广告服务在0.5秒内即时响应。

之后，再次放慢速度。

这让我想知道当并发连接太多时，Apache或PHP是否会导致阻塞连接。

这是我当前的my.cnf，httpd.conf，fcgid.conf：

的my.cnf

[mysqld]
bind-address=127.0.0.1
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0

innodb_buffer_pool_size=16G
query_cache_type = OFF
thread_cache_size = 20

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

的httpd.conf

Timeout 60
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 5

<IfModule prefork.c>
StartServers       8
MinSpareServers    25
MaxSpareServers   20
ServerLimit  256
MaxClients   64
MaxRequestsPerChild  4000
</IfModule>

fcgid.conf

FcgidIdleTimeout 40
FcgidProcessLifeTime 30
FcgidMaxProcesses 1000
FcgidMaxProcessesPerClass 100
FcgidMinProcessesPerClass 0
FcgidConnectTimeout 60
FcgidIOTimeout 1000
FcgidInitialEnv RAILS_ENV production
FcgidIdleScanInterval 20

更新6

以下是使用更新5

SHOW GLOBAL STATUS

Aborted_clients 4
Aborted_connects    291
Connections 25804
Max_used_connections    17
Opened_files    121295
Opened_table_definitions    1032
Opened_tables   1481
Threads_cached  11
Threads_connected   6
Threads_created 17
Threads_running 1

看Opened_files对我来说似乎很高。

这可能是锁定系统的原因吗？

更新7：

带宽放大图表，5分钟平均

更新8（临时解决方案）：

我在高峰时段过去几天尝试了大部分选项，但实际上没有一个能帮助解决问题。

对我来说，唯一的临时解决方案是设置cronjob以便在有问题的时间内每隔30秒正常重启httpd。

我计划使用Nginx + PHP-FPM 7.1 + Percona 5.7进行另一次服务器设置以测试情况。

如果我想出一个有效的解决方案，我会在这里发布更新。

Answer 1

44Mbs / 100Mbs - 几乎饱和。以太网可以处理的只有~60 / 100。我怀疑你的尖刺很高。

因此，要么增加该机器的带宽，要么添加运行除MySQL之外的所有服务器。

您的广告是否存储在MySQL中？或者您只存储广告的链接，广告服务器是否包含html？我问，因为在前一种情况下，您将需要多个额外的广告服务器，并且您可能需要多个Slave来提供流量。（这是因为庞大的广告将通过网络从MySQL传播到单独的广告服务器。）

在任何情况下，您都应该在现在上扩展Web服务器和MySQL。

什么是MySQL SHOW GLOBAL STATUS LIKE 'Max_used_connections';？如果这不仅仅是说。 30，那么MySQL可能会绊倒自己。当任何服务器（网络或数据库）试图“太多”时＃34;事情，资源均匀分享。这导致吞吐量停滞并严重延长延迟。当这种情况发生时，你最好在实际的最高级别上扼杀东西 - 如果它只是让事情变得更糟，那么就不要再开始另一个过程了。 11232听起来像这样的情况？

负载平均值听起来像CPU不是问题。（它很少用于MySQL。）

分析VARIABLES和STATUS

<强>观察：

版本：5.5.44-cll-lve
31.1 GB的RAM
正常运行时间= 106d 18:04:23
您没有在Windows上运行运行64位版本
您似乎完全（或大部分）运行InnoDB 115计算变量/状态/表达式;有趣的讨论如下

更重要的问题

对于InnoDB的buffer_pool，你使用的RAM不到1％。将innodb_buffer_pool_size提升至16G。这应该会显着减少I / O.但是，由于您的数据集似乎相当小，因此可能没什么用。
innodb_log_file_size = 90M - 目前您已经拥有了“小小的”＆＃39;默认为5M。 90M会减少某些减慢速度的事情。但是......改变那个设置并不容易，特别是5.5。所以，把它作为最后的手段＆＃39;。（您应该计划在某个时候进行升级。）
有一半的tmp表正在溢出到磁盘。 1/5正在进行全表扫描。让我们看看一些缓慢的查询;它们可能会得到改进 - 通过架构更改，查询更改和/或索引更改。这可能是改善延迟的最重要因素。
79次/秒。你在写什么？他们可以用某种方式进行批量处理吗？
将long_query_time降低为2;它可以帮助您找到慢速查询。并设置slow_query_log = ON。
每个连接更改数据库一千次（235次/秒）？发生了什么事？
Thread_cache_size现在为0;改为20.＆＃34; 0＆＃34;导致每个新连接经历一个有点冗长的建立过程的过程，等等。超过0时，避免这个过程。可能是另一项重大改进。
max_connections = 151 - 我已经讨论过在客户端减少对此的需求。（无需更改此号码。）
query_cache_type = OFF - 避免一些开销

详细信息和其他观察结果

( innodb_buffer_pool_size / _ram ) = 128M / 33393370726.4 = 0.40% - 用于InnoDB buffer_pool的RAM的百分比

( open_files_limit ) = 1,024 - ulimit -n - 要允许更多文件，请更改ulimit或/etc/security/limits.conf或sysctl.conf（kern.maxfiles＆amp; kern.maxfilesperproc）或其他内容（取决于操作系统）

( Innodb_log_writes ) = 555,866,212 / 9223463 = 60 /sec

( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 484,844,787,200 / (9223463 / 3600) / 2 / 5M = 18 - 比率

( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 9,223,463 / 60 * 5M / 484844787200 = 1.66 - InnoDB日志轮换之间的分钟数从5.6.8开始，这可以动态更改;一定要改变my.cnf。 - （轮换之间60分钟的建议有些武断。）调整innodb_log_file_size。

( Created_tmp_disk_tables / (Created_tmp_disk_tables + Created_tmp_tables) ) = 2,899,422 / (2899422 + 3078901) = 48.5% - 溢出到磁盘的临时表的百分比 - 也许增加tmp_table_size和max_heap_table_size;避免blob等。

( Select_scan / Com_select ) = 3,328,035 / 16316658 = 20.4% - 选择进行全表扫描的百分比。（可能被Stored Routines愚弄了。） - 添加索引/优化查询

( Com_insert + Com_delete + Com_delete_multi + Com_replace + Com_update + Com_update_multi ) = (732167877 + 90591 + 107 + 61 + 44026 + 5124) / 9223463 = 79 /sec - 写/秒 - 50次写入/秒+日志刷新可能会最大化正常驱动器的I / O写入容量

( long_query_time ) = 10.000000 = 10 - 截止（秒）用于定义＆＃34;慢＆＃34;查询。 - 建议2

( Com_change_db / Connections ) = 2,167,726,231 / 2107678 = 1,028 - 每个连接的数据库切换 - （次要）考虑使用＆＃34; db.table＆＃34;语法

( Com_change_db ) = 2,167,726,231 / 9223463 = 235 /sec - 可能来自USE声明。 - 考虑使用db.tbl语法连接DB，消除虚假的USE语句等。

( Threads_created / Connections ) = 2,107,677 / 2107678 = 100.0% - 流程创建的快速性 - 增加thread_cache_size（非Windows）

Com_create_index = 2 / HR - 发生了什么？每周一次是CREATE INDEX的高频率

have_symlink和local_infile可能是安全漏洞。

您有半个查询缓存。您应该同时设置query_cache_type = OFF和query_cache_size = 0。（根据谣言）有一个＆＃39; bug＆＃39;在QC代码中，除非你关闭这两个设置，否则会留下一些代码。

Answer 2

@KDX一个重要因素可能是您的innodb_log_buffer_size=8388608和您的innodb_log_file_size=5242880较小且不实用。此链接https://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_log_buffer_size显示innodb_log_file_size of over 50M。这可以减轻你的一些麻烦。尽可能更改.ini / .cnf shutdown / restart。应遵循5.5 refman的此操作指南。

如何在专用服务器上识别和微调PHP + MySQL应用程序的性能瓶颈？