Incident description
All sites on wordpress1.geant.org became unreachable at Sun Mar 25 21:25:17 CEST 2018
Incident severity: CRITICAL
Data loss: NO
Monitoring alerted: YES (on nagios.terena.org)
Timeline
| Time (CEST) | |
|---|---|
| 21:25 | Apache server stop accepting incoming requests, nagios.terena.org host reported about this issue by email |
| Mon Mar 26 04:26 | Nicole Harris reported about outage on #it channel in slack and created ticket |
| Mon Mar 26 10:00 | Qaiser Ahmed or Michael Haller rebooted VM wordpress1.geant.org |
| Mon Mar 26 10:05 | nagios.terena.org reported that apache is working again |
Total downtime: 12 hours.
Analysis
According data from Michael Haller there was out of memory message on screen. apache2 process has been killed by kernel due out of memory condition. The same happened with mysqld process.
Logs/screendumps
Mar 25 21:20:44 wordpress1 kernel: [2726661.284027] Mem-Info:
Mar 25 21:20:44 wordpress1 kernel: [2726661.284031] active_anon:338218 inactive_anon:113477 isolated_anon:0
Mar 25 21:20:44 wordpress1 kernel: [2726661.284031] active_file:203 inactive_file:191 isolated_file:46
Mar 25 21:20:44 wordpress1 kernel: [2726661.284031] unevictable:0 dirty:0 writeback:0 unstable:0
Mar 25 21:20:44 wordpress1 kernel: [2726661.284031] slab_reclaimable:7615 slab_unreclaimable:14069
Mar 25 21:20:44 wordpress1 kernel: [2726661.284031] mapped:5426 shmem:5158 pagetables:13559 bounce:0
Mar 25 21:20:44 wordpress1 kernel: [2726661.284031] free:13717 free_pcp:176 free_cma:0
Mar 25 21:20:44 wordpress1 kernel: [2726661.284035] Node 0 DMA free:8208kB min:352kB low:440kB high:528kB active_anon:3368kB inactive_anon:3396kB active_file:0kB inactive_file:24kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB man
aged:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:148kB shmem:148kB slab_reclaimable:32kB slab_unreclaimable:264kB kernel_stack:0kB pagetables:200kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:128
all_unreclaimable? no
Mar 25 21:20:44 wordpress1 kernel: [2726661.284045] lowmem_reserve[]: 0 1968 1968 1968 1968
Mar 25 21:20:44 wordpress1 kernel: [2726661.284048] Node 0 DMA32 free:46660kB min:44700kB low:55872kB high:67048kB active_anon:1349504kB inactive_anon:450512kB active_file:812kB inactive_file:740kB unevictable:0kB isolated(anon):0kB isolated(file):184kB
present:2080704kB managed:2031928kB mlocked:0kB dirty:0kB writeback:0kB mapped:21556kB shmem:20484kB slab_reclaimable:30428kB slab_unreclaimable:56012kB kernel_stack:5488kB pagetables:54036kB unstable:0kB bounce:0kB free_pcp:704kB local_pcp:120kB free_cm
a:0kB writeback_tmp:0kB pages_scanned:11688 all_unreclaimable? yes
Mar 25 21:20:44 wordpress1 kernel: [2726661.284054] lowmem_reserve[]: 0 0 0 0 0
Mar 25 21:20:44 wordpress1 kernel: [2726661.284057] Node 0 DMA: 15*4kB (UME) 24*8kB (UME) 21*16kB (UME) 19*32kB (UME) 10*64kB (UME) 12*128kB (UE) 3*256kB (UE) 4*512kB (E) 2*1024kB (ME) 0*2048kB 0*4096kB = 8236kB
Mar 25 21:20:44 wordpress1 kernel: [2726661.284070] Node 0 DMA32: 878*4kB (UMEH) 484*8kB (UME) 354*16kB (UME) 303*32kB (ME) 153*64kB (UE) 44*128kB (EH) 24*256kB (MEH) 3*512kB (MH) 1*1024kB (H) 0*2048kB 0*4096kB = 46872kB
Mar 25 21:20:44 wordpress1 kernel: [2726661.284082] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Mar 25 21:20:44 wordpress1 kernel: [2726661.284083] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 25 21:20:44 wordpress1 kernel: [2726661.284084] 6184 total pagecache pages
Mar 25 21:20:44 wordpress1 kernel: [2726661.284086] 563 pages in swap cache
Mar 25 21:20:44 wordpress1 kernel: [2726661.284087] Swap cache stats: add 2612000, delete 2611437, find 81016618/81522083
Mar 25 21:20:44 wordpress1 kernel: [2726661.284088] Free swap = 0kB
Mar 25 21:20:44 wordpress1 kernel: [2726661.284089] Total swap = 1003516kB
Mar 25 21:20:44 wordpress1 kernel: [2726661.284090] 524174 pages RAM
Mar 25 21:20:44 wordpress1 kernel: [2726661.284091] 0 pages HighMem/MovableOnly
Mar 25 21:20:44 wordpress1 kernel: [2726661.284092] 12215 pages reserved
Mar 25 21:20:44 wordpress1 kernel: [2726661.284093] 0 pages cma reserved
Mar 25 21:20:44 wordpress1 kernel: [2726661.284094] 0 pages hwpoisoned
...
Mar 25 21:20:44 wordpress1 kernel: [2726661.284320] Out of memory: Kill process 25796 (mysqld) score 97 or sacrifice child
Mar 25 21:20:44 wordpress1 kernel: [2726661.284609] Killed process 25796 (mysqld) total-vm:828704kB, anon-rss:218468kB, file-rss:0kB
...
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] mysqld: Out of memory (Needed 128909312 bytes)
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] mysqld: Out of memory (Needed 54362112 bytes)
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
Mar 25 21:20:46 wordpress1 mysqld:
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: The InnoDB memory heap is disabled
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: Compressed tables use zlib 1.2.8
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: Using Linux native AIO
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: Using CPU crc32 instructions
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] InnoDB: Initializing buffer pool, size = 128.0M
Mar 25 21:20:46 wordpress1 mysqld: InnoDB: mmap(139722752 bytes) failed; errno 12
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] Plugin 'InnoDB' init function returned error.
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [Note] Plugin 'FEEDBACK' is disabled.
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] Unknown/unsupported storage engine: InnoDB
Mar 25 21:20:46 wordpress1 mysqld: 180325 21:20:46 [ERROR] Aborting