Server down ogni giorno

[furiaceka]

Buona sera a tutti ho un server dedicato su leaseweb con 1gb di ram e un dual core E4550. Ho un forum phpbb 3.0.7 con circa 10000 iscritti e in media circa 100 utenti connessi insieme.

Ultimamente ogno giorno o quasi il sito va down e non riesco ad accedervi più neanche da ssh. Questo l'error log:

[Thu Sep 16 18:09:33 2010] [error] [client 93.152.157.158] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.test0:)
[Thu Sep 16 18:10:56 2010] [error] [client 93.152.157.158] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.test0:)
[Thu Sep 16 18:30:37 2010] [error] [client 91.213.152.18] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Thu Sep 16 18:30:37 2010] [error] [client 91.213.152.18] File does not exist: /var/www/html/400.shtml
[Thu Sep 16 18:36:51 2010] [error] [client 87.230.88.174] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Thu Sep 16 18:39:24 2010] [error] [client 87.230.88.174] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Thu Sep 16 18:40:07 2010] [notice] SIGHUP received.  Attempting to restart
WARNING: MaxClients of 300 exceeds ServerLimit value of 256 servers,
 lowering MaxClients to 256.  To increase, please see the ServerLimit
 directive.
[Thu Sep 16 18:40:08 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 18:40:08 2010] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Thu Sep 16 18:40:08 2010] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8g DAV/2 PHP/5.2.9 configured -- resuming normal operations
[Thu Sep 16 18:40:49 2010] [error] [client 94.34.219.95] Invalid URI in request les/SF_Glacier/theme/images/icon_tb_quote.gif HTTP/1.1
[Thu Sep 16 18:40:49 2010] [error] [client 94.34.219.95] File does not exist: /var/www/html/400.shtml
[Thu Sep 16 18:59:22 2010] [error] [client 93.148.192.109] File does not exist: /var/www/html/favicon.ico
[Thu Sep 16 18:59:22 2010] [error] [client 93.148.192.109] File does not exist: /var/www/html/404.shtml
[Thu Sep 16 18:59:22 2010] [error] [client 93.148.192.109] File does not exist: /var/www/html/favicon.ico
[Thu Sep 16 18:59:22 2010] [error] [client 93.148.192.109] File does not exist: /var/www/html/404.shtml
[Thu Sep 16 18:59:25 2010] [error] [client 93.148.192.109] File does not exist: /var/www/html/favicon.ico
[Thu Sep 16 18:59:25 2010] [error] [client 93.148.192.109] File does not exist: /var/www/html/404.shtml
[Thu Sep 16 19:14:32 2010] [error] [client 91.213.152.18] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Thu Sep 16 19:14:32 2010] [error] [client 91.213.152.18] File does not exist: /var/www/html/400.shtml
[Thu Sep 16 19:46:00 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 19:46:00 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 16 19:46:01 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 19:46:01 2010] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8g DAV/2 PHP/5.2.9 configured -- resuming normal operations

oppure:

[Wed Sep 15 23:41:03 2010] [notice] SIGHUP received.  Attempting to restart
WARNING: MaxClients of 256 exceeds ServerLimit value of 100 servers,
 lowering MaxClients to 100.  To increase, please see the ServerLimit
 directive.
[Wed Sep 15 23:41:04 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Wed Sep 15 23:41:04 2010] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Wed Sep 15 23:41:04 2010] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8g DAV/2 PHP/5.2.9 configured -- resuming normal operations
[Wed Sep 15 23:45:00 2010] [error] [client 91.213.152.18] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Wed Sep 15 23:45:00 2010] [error] [client 91.213.152.18] File does not exist: /var/www/html/400.shtml
[Thu Sep 16 00:07:46 2010] [notice] SIGHUP received.  Attempting to restart
WARNING: MaxClients of 256 exceeds ServerLimit value of 100 servers,
 lowering MaxClients to 100.  To increase, please see the ServerLimit
 directive.
[Thu Sep 16 00:07:47 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 00:07:47 2010] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Thu Sep 16 00:07:47 2010] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8g DAV/2 PHP/5.2.9 configured -- resuming normal operations
[Thu Sep 16 00:10:02 2010] [notice] SIGHUP received.  Attempting to restart
WARNING: MaxClients of 256 exceeds ServerLimit value of 100 servers,
 lowering MaxClients to 100.  To increase, please see the ServerLimit
 directive.
[Thu Sep 16 00:10:03 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 00:10:03 2010] [warn] WARNING: Attempt to change ServerLimit ignored during restart
[Thu Sep 16 00:10:03 2010] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8g DAV/2 PHP/5.2.9 configured -- resuming normal operations
[Thu Sep 16 00:11:01 2010] [notice] caught SIGTERM, shutting down
[Thu Sep 16 00:11:04 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 00:11:04 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 16 00:11:05 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 00:11:05 2010] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8g DAV/2 PHP/5.2.9 configured -- resuming normal operations
[Thu Sep 16 00:16:08 2010] [notice] caught SIGTERM, shutting down
[Thu Sep 16 00:16:11 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 00:16:11 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Sep 16 00:16:12 2010] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Sep 16 00:16:12 2010] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8g DAV/2 PHP/5.2.9 configured -- resuming normal operations
[Thu Sep 16 00:31:48 2010] [error] [client 91.213.152.18] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Thu Sep 16 00:31:48 2010] [error] [client 91.213.152.18] File does not exist: /var/www/html/400.shtml
sh: /cat: No such file or directory

Ho alzato il maxclients a 256 ma niente.Vi allego i file di config di apache se il problema è quello ovviamente.

Se volete altri log o altro chiedete.. Monitorando la cpu sta con 20% di media durante la giornata con picchi di 50% però ogni tanto crash e devo fare il power-cycle da pannello leaseweb perchè non accedo neanche tramite ssh!

[Baldoz]

Ciao furiaceka,

fatti una domanda...... per caso vuoi troppo dal tuo server?

Le configurazioni sono modificabili e di certo possono aiutare a sopportare il carico ma quando esso è veramente eccessivo bisogna mettere mano al portafogli e passare ad una soluzione più performante!

Ad ogni modo venendo alla tua specifica situazione se hai un blocco anche a livello di SSH o potrebbero essere dei piccoli DDoS (che l'azienda dovrebbe segnalarti) oppure il tuo server carica così tanto da non riuscire a rispondere alle richieste....

Per monitorare la cosa ti consiglio di creare un bello script bash che esegue il comando uptime e free -m e lo metti in crontab ogni cinque minuti...

dopo 24/48 ore puoi avere una stima media di carico e di utilizzo della ram.

Se mi dai le caratteristiche della macchina posso essere più preciso!


Ciao!

[CoolNetwork]

sarebbe da "capire" come mai muore il server. .. se per troppo load di apache o mysql o il mix di entrambi.


ti conviene installare un tool di monitor tipo munin e monitorare un po' le risorse del server per avere un visione migliore della situazione.

[infinytech]

Ma quant'è circa il carico nei momenti di picco?

[furiaceka]

ecco più info sul processore:

vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Duo CPU     E4500  @ 2.20GHz
stepping        : 13
cpu MHz         : 2200.306
cache size      : 2048 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes

Uno screen del traffico di settembre 2010:


Allora sinceramente non penso proprio sia problema di capacità del server in quanto 1Gb di Ram di cui vedo picchi di massimo 350Mb in uso mi paiono abbastanza così come il processore dual core più che dignitoso per le visite che ho. Forse c'è qualcosa che non va nelle configurazioni o qualche servizio che spreca risorse.. Il sito è molto reattivo fino al fatidico blocco totale.

Vi posto per completezza il my.cnf usato:

[mysqld]
local-infile=0
skip-innodb

#
# * Fine Tuning
#
query_cache_min_res_unit = 4K 
key_buffer              = 80M
max_allowed_packet      = 16M
thread_stack            = 128K
thread_cache_size       = 64

# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover          = BACKUP
#max_connections        = 100
table_cache            = 512
thread_concurrency     = 8


#
# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 32M


#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
#log            = /var/log/mysql/mysql.log
#
# Error logging goes to syslog. This is a Debian improvement :)
#
# Here you can see queries with especially long duration
#log_slow_queries       = /var/log/mysql/mysql-slow.log
long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id              = 1
#log_bin                        = /var/log/mysql/mysql-bin.log
#expire_logs_days        = 10
#max_binlog_size         = 100M
#binlog_do_db           = include_database_name
#binlog_ignore_db       = include_database_name
#
# * BerkeleyDB
#
# Using BerkeleyDB is now discouraged as its support will cease in 5.1.12.
skip-bdb
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
# You might want to disable InnoDB to shrink the mysqld process by circa 100MB.
#skip-innodb
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem



[mysqldump]
quick
quote-names
max_allowed_packet      = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer              = 16M

#
# * NDB Cluster
#
# See /usr/share/doc/mysql-server-*/README.Debian for more information.
#
# The following configuration is read by the NDB Data Nodes (ndbd processes)
# not from the NDB Management Nodes (ndb_mgmd processes).
#
# [MYSQL_CLUSTER]
# ndb-connectstring=127.0.0.1


#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

Ho usato mysqltuner per l'ottimizzazione.

Mi dite come aggiungere al cron il monitoraggio automatico delle risorse, non sono molto pratico di cron!

Grazie mille per i vostri aiuti gentilissimi come sempre!
P.S. anche a me era venuto il dubbio del ddoss... questa è stata la risposta di leaseweb:

It does not look like your server has been under a DoS attack. I see the traffic pick up during the day and go down during the night. If the server was under a DoS attack it would be a flat line of high traffic.

[furiaceka]

dimenticavo mi date due dritte per installare munin.
L'ho installato poi in munin.conf ho messo

[stat.miodominio.com]
address localhost
use_node_name yes

mentre in munin-node.conf in allow devo mettere l'indirizo ip del mio server?

Grazie

[EvolutionCrazy]

che sistema operativo è?

puoi cominciare con sysstat intnto......

[Baldoz]

Io scommetto che si pianta semplicemente perchè ha 1GB di ram, nei momenti di carico la utilizza completamente swappa e di conseguenza si pianta!

Apro le puntante

[furiaceka]

che sistema operativo è?

puoi cominciare con sysstat intnto......

Linux 32bit

systat l'avevo installato qualche giorno fa ecco un outpunt del comando sar:

Linux 2.6.18-6-686 (OBB001.local)       09/17/2010      _i686_

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:05:01 AM     all     31.95      0.00      2.86      0.15      0.00     65.03
12:15:01 AM     all     21.57      0.00      2.12      0.38      0.00     75.93
12:25:01 AM     all     31.49      0.00      2.75      0.19      0.00     65.56
12:35:01 AM     all     19.02      0.00      1.73      0.70      0.00     78.55
12:45:01 AM     all     17.79      0.47      1.73      0.33      0.00     79.68
12:55:01 AM     all     19.70      0.03      1.75      0.10      0.00     78.42
01:05:01 AM     all     21.85      0.03      1.75      0.26      0.00     76.11
01:15:01 AM     all     20.46      0.02      1.57      0.11      0.00     77.84
01:25:01 AM     all      8.37      0.03      0.89      0.08      0.00     90.64
01:35:01 AM     all     10.37      0.03      1.01      0.10      0.00     88.50
01:45:01 AM     all      8.19      0.02      0.83      0.08      0.00     90.87
01:55:01 AM     all      7.68      0.02      0.85      0.05      0.00     91.40
02:05:01 AM     all      7.87      0.03      0.82      0.02      0.00     91.26
02:15:01 AM     all      3.75      0.02      0.51      0.02      0.00     95.69
02:25:01 AM     all      7.35      0.02      0.74      0.02      0.00     91.87
02:35:02 AM     all      4.91      0.03      0.53      0.01      0.00     94.52
02:45:01 AM     all      6.42      0.03      0.63      0.01      0.00     92.91
02:55:01 AM     all      4.38      0.03      0.47      0.01      0.00     95.12
03:05:01 AM     all      4.03      0.03      0.48      0.01      0.00     95.45
03:15:01 AM     all      4.50      0.03      0.47      0.01      0.00     94.98
03:25:01 AM     all      4.92      0.03      0.53      0.01      0.00     94.52
03:35:01 AM     all      2.42      0.02      0.36      0.01      0.00     97.19

03:35:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:45:01 AM     all      2.25      0.03      0.33      0.00      0.00     97.39
03:55:01 AM     all      4.26      0.02      0.48      0.02      0.00     95.22
04:05:01 AM     all      2.29      0.02      0.35      0.01      0.00     97.33
04:15:01 AM     all      4.17      0.03      0.42      0.01      0.00     95.38
04:25:01 AM     all      2.59      0.02      0.36      0.01      0.00     97.02
04:35:01 AM     all      3.86      0.03      0.43      0.01      0.00     95.68
04:45:01 AM     all      3.49      0.03      0.39      0.01      0.00     96.09
04:55:01 AM     all      3.03      0.02      0.39      0.00      0.00     96.56
05:05:01 AM     all      2.59      0.03      0.35      0.01      0.00     97.03
05:15:01 AM     all      5.54      0.03      0.53      0.01      0.00     93.89
05:25:01 AM     all      2.05      0.02      0.33      0.02      0.00     97.58
05:35:01 AM     all      2.17      0.02      0.37      0.06      0.00     97.38
05:45:01 AM     all      2.04      0.03      0.34      0.07      0.00     97.52
05:55:01 AM     all      2.70      0.02      0.38      0.06      0.00     96.85
06:05:01 AM     all      2.88      0.03      0.42      0.07      0.00     96.60
06:15:01 AM     all      3.23      0.02      0.46      0.06      0.00     96.23
06:25:01 AM     all      5.91      0.03      0.72      0.08      0.00     93.27
06:35:01 AM     all      2.81      0.03      0.48      0.06      0.00     96.63
06:45:01 AM     all      5.32      0.03      0.64      0.07      0.00     93.94
06:55:01 AM     all      6.16      0.02      0.66      0.07      0.00     93.08
07:05:01 AM     all      9.79      0.02      0.98      0.04      0.00     89.16
07:15:01 AM     all      9.44      0.02      0.88      0.01      0.00     89.65

07:15:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:25:01 AM     all      7.53      0.03      0.70      0.01      0.00     91.73
07:35:01 AM     all      6.21      0.03      0.65      0.02      0.00     93.09
07:45:01 AM     all     13.20      0.03      1.02      0.02      0.00     85.73
07:55:02 AM     all     15.15      0.03      1.07      0.02      0.00     83.74
08:05:01 AM     all     11.16      0.03      1.00      0.03      0.00     87.80
08:15:02 AM     all     13.29      0.03      1.10      0.02      0.00     85.57
08:25:01 AM     all      9.37      0.03      0.97      0.03      0.00     89.61
08:35:01 AM     all     12.90      0.03      1.08      0.03      0.00     85.96
08:45:01 AM     all     15.24      0.03      1.33      0.03      0.00     83.37
08:55:01 AM     all     24.62      0.03      1.90      0.02      0.00     73.43
09:05:01 AM     all     22.04      0.02      1.74      0.05      0.00     76.14
09:15:01 AM     all     12.50      0.03      1.26      0.09      0.00     86.13
09:25:01 AM     all     18.07      0.02      1.76      0.11      0.00     80.03
09:35:01 AM     all     24.84      0.03      2.05      0.08      0.00     73.00
09:45:01 AM     all     25.92      0.03      2.11      0.07      0.00     71.89
09:55:03 AM     all     30.91      0.02      2.55      0.09      0.00     66.42
10:05:01 AM     all     34.15      0.03      3.02      0.14      0.00     62.67
10:15:01 AM     all     31.95      0.02      2.84      0.23      0.00     64.96
10:25:01 AM     all     35.06      0.03      2.77      0.26      0.00     61.89
10:35:01 AM     all     33.83      0.03      2.65      0.20      0.00     63.30
10:45:04 AM     all     28.73      0.02      2.36      0.16      0.00     68.72
10:55:01 AM     all     41.84      0.03      3.17      1.38      0.00     53.58

10:55:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:05:01 AM     all     28.40      0.02      2.54      1.16      0.00     67.88
11:15:01 AM     all     24.92      0.02      2.35      0.88      0.00     71.83
11:25:01 AM     all     22.10      0.03      2.04      0.61      0.00     75.23
11:35:01 AM     all     29.33      0.02      2.57      0.57      0.00     67.51
11:45:01 AM     all     27.00      0.03      2.53      0.42      0.00     70.02
11:55:01 AM     all     25.90      0.03      2.31      0.49      0.00     71.28
12:05:01 PM     all     19.44      0.02      2.01      0.56      0.00     77.97
Average:        all     13.62      0.03      1.24      0.15      0.00     84.95

e di iostat

OBB001:~# iostat
Linux 2.6.18-6-686 (OBB001.local)       09/17/2010      _i686_

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          17.97    0.02    1.59    0.17    0.00   80.25

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               7.87        33.41       223.73    1949045   13052666

Io scommetto che si pianta semplicemente perchè ha 1GB di ram, nei momenti di carico la utilizza completamente swappa e di conseguenza si pianta!

Apro le puntante

Io mi informo sull'update però perchè 2/3/4 mesi fa con lo stesso numero di utenti pressappoco la stessa mole di traffico i problemi si presentavano di rado?

[FlameNetworks]

Inizia ad installare Munin e poi il resto si vedrà.

[furiaceka]

Inizia ad installare Munin e poi il resto si vedrà.

Ciao flame ci ho provato ma come ho scritto sopra ho questi dubbi.

dimenticavo mi date due dritte per installare munin.
L'ho installato poi in munin.conf ho messo
Citazione:
[stat.miodominio.com]
address localhost
use_node_name yes
mentre in munin-node.conf in allow devo mettere l'indirizo ip del mio server?

Grazie

in pratico accedo a munin ma i grafici rimangono vuoti sicuramente sbaglio qualcosa nella configurazione!

[FlameNetworks]

Ma che distribuzione linux è?

[furiaceka]

Ma che distribuzione linux è?

Debian GNU/Linux 5.0
Versione kernel 2.6.18-6-686
con directadmin come pannello se ti puo servire saperlo..
Grazie mille

[FlameNetworks]

Se i grafici rimangono vuoti è perchè probabilmente c'è qualcosa che non va nel crontab.

Posso sapere com'è che hai installato munin?

[furiaceka]

Se i grafici rimangono vuoti è perchè probabilmente c'è qualcosa che non va nel crontab.

Posso sapere com'è che hai installato munin?

Con i classici:
apt-get install munin
apt-get install munin-node

poi ho fatto:
munin-node-configure
ln -s /usr/share/munin/plugins/apt /etc/munin/plugins/apt
ln -s /usr/share/munin/plugins/pat_all /etc/munin/plugins/apt_all


Dopo di che ho modificato i file /etc/munin/munin-node.conf
e /etc/munin/munin.conf come da allegato anche se non ho capito cosa devo mettere sinceramente.
dove 62.212.74.214 è l'ip del mio server.

Per completezza ti allego i due file cron (munin.cron.txt e munin-node.cron.txt) che sono all'interno della cartella /etc/cron.d

ovviamente l'estensione .cron.txt e .txt le ho aggiunte se no non me li faceva allegare