python readline

# sudo aptitude install build-essential libreadline6-dev

#cd Python-..

# sed 's/#readline/readline/' Modules/Setup.dist > file.tmp && mv file.tmp Modules/Setup.dist

#./configure
#make
#make install

Nagios Indicator

Gnome indicator for nagios

It adds icon indicator to gnome panel.
This applet automatically check your nagios server and notify about all problems using popup messages.


https://github.com/volodymyrko/nagios_indicator

mysql: find InnoDB table size

How to find Innodb tables size?

"show table status" command doesn't show separated Innodb tables size, it showes total InnoDB data size. So we can use INFORMATION_SCHEMA for finding size of each InnoDB table.

for example, try  find 10 biggest InnoDB tables:

#mysql -A


mysql> use INFORMATION_SCHEMA
mysql> SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE, TABLE_ROWS, DATA_LENGTH, INDEX_LENGTH, CREATE_TIME, UPDATE_TIME FROM TABLES WHERE ENGINE='InnoDB' ORDER BY DATA_LENGTH DESC limit 10;



using  INFORMATION_SCHEMA you can find data_length, index_length and other useful information

mysql: How find fragmented MySQL tables

#mysql

> use information_schema

> select ENGINE, TABLE_NAME,Round( DATA_LENGTH/1024/1024) as data_length , round(INDEX_LENGTH/1024/1024) as index_length, round(DATA_FREE/ 1024/1024) as data_free from information_schema.tables where DATA_FREE > 0;




The "You must to know" answer

first at all you must to understand that Mysql tables get fragmented when update a row, so it's a normal situation. When a table is created, lets say imported using a dump with data, all rows are stored with no fragmentation in many fixed size pages. When you update a variable length row, the page containing this row is divided in two or more pages to store the changes, and these new two(or more) pages contains blank spaces filling the unused space.

This does not impact in your performance, unless of course the fragmentation growth too much. What is too much fragmentation, well let's see the query you're looking for:

select ENGINE, TABLE_NAME,Round( DATA_LENGTH/1024/1024) as data_length , round(INDEX_LENGTH/1024/1024) as index_length, round(DATA_FREE/ 1024/1024) as data_free from information_schema.tables where DATA_FREE > 0;

The DATA_LENGTH and INDEX_LENGTH are the space your data and indexes are using, and DATA_FREE is the total amount of bytes unused in all the table pages (fragmentation).

Here's an example of real production table

| ENGINE | TABLE_NAME | data_length | index_length | data_free |
| InnoDB | comments | 896 | 316 | 5 |

In this case we have a Table using (896 + 316) = 1212 MB, and have data a free space of 5 MB. This means a "ratio of fragmentation" of:

5/1212 = 0.0041

...Which is a really low "fragmentation ratio".

I've been working with tables with a ratio near 0.2 (meaning 20% of blank spaces) and never notice a slow down on queries, even if I optimize the table, the performance is the same. But apply a optimize table on a 800MB table takes a lot of time and blocks the table for several minutes, impracticable on production.

So, if you consider what you win in performance and the time wasted in optimize a table, I prefer NOT OPTIMIZE.

If you think it's better for storage, see your ratio and see how much space can you save when optimize. It's usually not too much, so I prefer NOT OPTIMIZE.

And if you optimize, the next update will create blank spaces by splitting a page in two or more. But it's faster to update a fragmented table than a not fragmented one, because if the table is fragmented an update on a row not necessarily will split a page.

I hope this help you.



http://serverfault.com/questions/202000/how-find-and-fix-fragmented-mysql-tables
http://dba.stackexchange.com/questions/16341/how-do-you-remove-fragmentation-from-innodb-tables

Add virtualhost name to apache logfile

If you have many ServerAlias in some apache VirtualHost and You want to see name of virtualhost in lofgile add  %{host}i to logformat direcive,

for example,

LogFormat "%h %{host}i %l %u %D %T %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

php performance

strace huge lstat try:
php.ini


; ...
; Determines the size of the realpath cache to be used by PHP. This value should
; be increased on systems where PHP opens many files to reflect the quantity of
; the file operations performed.
realpath_cache_size=1M

; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.
realpath_cache_ttl=300
; ...

bash: як замінити стрічку у файлі

маємо файл file.txt

# cat file.txt
this is simple text file
here is 1-st line
4-th line
some 3-rd line
end of file



Наприклад, треба замінити стрічку яка містить  "4-th line" . Зробити це можна за допомогою sed-у:


# cat file.txt | sed '/4-th line/c 2-nd line'
this is simple text file
here is 1-st line
2-nd line
some 3-rd line
end of file


Але треба памятати, що sed міняє сам файл у такому випадку. Тому, якщо треба змнити файл, можна зробити шось таке:


# cat file.txt | sed '/4-th line/c 2-nd line' > tmp.txt && mv tmp.txt file.txt


таким вийшов  новий файл


# cat file.txt
this is simple text file
here is 1-st line
2-nd line
some 3-rd line
end of file

Запуск django/flask аплікацій за допомгою supervisord

Існує декілька варіантів запуску django як fastcgi-серверу на продакшин-серверах. Типовим варіантом є написання startup-скрпту і розміщення його в /etc/init.d/ (детальніше https://code.djangoproject.com/wiki/InitdScriptForLinux).

Доволі зручним для запуску і контролю над django-програмами є supervisord. Supervisord дозволяє користувачам здійснювати моніторинг і контроль ряд процесів в UNIX-таких операційних систем. Його можна використовувати для запуску та конторлю різноманітних сервісів, зокрема django, celery, tornado, flask

Встанолюємо (на debain) supervisord:

aptitude install supervisor

supervisor напсианий на python-і, тому його можна і встановлювати за допомгою easy_install, pip ... Але якщо у вас немає потреб встановлювати певно кокретної версії, то краще скористатися штатним пакетним менеджером (aptitude, apt-get в debian-і)


конфігураціїні файли supervisor-а знаходяться в директорії /etc/supervisor:
/etc/supervisor/supervisord.conf - конф. файл самого supervisor-а,
а файли з лписом наших сервісів (зокрема django), які ми будемо запускати, повинні знаходитися в /etc/supervisor/conf.d/

ось, як виглядає конфігураційний файл (/etc/supervisor/conf.d/project.conf) для запуску django :


[program:project]  # тут project - це унікальне імє проекту
command = /home/user/www/project/.env/bin/python manage.py runfcgi host=127.0.0.1 port=10889 daemonize=false
directory = /home/user/www/project/
user = user
autostart=true
autorestart=true
startsecs = 3
redirect_stderr=true
stdout_logfile=/var/log/supervisor/project.log

отже створили файл, далі запускаємо:

supervisorctl reread
supervisorctl update

і дивимося статус:

supervisorctl status
project RUNNING pid 2637, uptime 0:25:13


і ще приклад для flask-аплікації, яка запускається за допомогою gunicorn

[program:flask_app]
command = /home/user/flask/flask_app/.env/bin/gunicorn flask_app:app -b 127.0.0.1:10081
directory = /home/user/flask/flask_app/
user = user
autostart=true
autorestart=true
startsecs = 3
redirect_stderr=true
stdout_logfile=/var/log/supervisor/flask_app.gunicorn.log

Встановлення svn серверу на debian з підтримкою cyrus-sasl2

Встановлємо сам svn

aptitude install subversion
aptitude install sasl2-bin libsasl2-modules libsasl2-2

додаємо користувача

useradd -d /var/lib/svn/ -r -s /bin/false svn

створюємо домашній каталог, де  і будуть розміщені наші svn-репозиторії

mkdir /var/lib/svn
chmod 750 /var/lib/svn/

створюємо сам репозиторій з іменем project
sudo -u svn svnadmin create  project


редагуємо  /var/lib/svn/project/conf/svnserve.conf:

[general]
anon-access = none
auth-access = write

realm = Project

[sasl]
use-sasl = true
min-encryption = 128
max-encryption = 256



створємо файл  /usr/lib/sasl2/svn.conf з таким вмістом

pwcheck_method: auxprop
auxprop_plugin: sasldb
sasldb_path: /var/lib/svn/project/passwd.db
mech_list: DIGEST-MD5

/var/lib/svn/project/passwd.db - це файл з паролями

додати користувача можна командою:
saslpasswd2 -f /var/lib/svn/project/passwd.db -u Project  -c



wget wget http://odyniec.net/articles/ubuntu-subversion-server/svnserve

додаємо до start-stop-daemon опцію " --chuid svn:svn "


chmod u+x svnserve
mv svnserve /etc/init.d/svnserve
update-rc.d svnserve defaults

/etc/init.d/svnserve start 

django startup script

#!/bin/sh -u


ROOT_DIR="/data/www"

PIDFILE="${ROOT_DIR}/django.pid"

SITE_DIR="${ROOT_DIR}/site"

USER="vova"


MANAGE="$SITE_DIR/wmf/manage.py"

COMMAND="${SITE_DIR}/.env/bin/python"

PORT="10080"

HOST="127.0.0.1"

OPTIONS="$MANAGE runfcgi method=threaded host=${HOST} port=${PORT} pidfile=${PIDFILE}"


case "$1" in

      start)

            echo "Starting django site ...";

              /sbin/start-stop-daemon -S -p ${PIDFILE} -c ${USER}  --exec  $COMMAND -- $OPTIONS

      ;;

      stop)

           echo "Stoping django site  ...";

            /sbin/start-stop-daemon -Kp ${PIDFILE}

            test -f ${PIDFILE} && rm ${PIDFILE}

      ;;

      restart|reload)

            echo "Restarting api2 server ...";

            $0 stop;

            sleep 1;

            $0 start;

      ;;

      *)

      echo "use $0: stop|start"

      ;;

esac



where (django-1.5):

# ls -1a /data/www/site
.env/
.git/
.gitignore
README.md
requirements.txt
wmf/


# ls -1a /data/www/site/wmf/
manage.py
media/
static/
wmf/

[alert] [client 192.168.1.2] /home/www/domains/domain.com/htdocs/.htaccess: AuthType not allowed here


to fix add to config AuthConfig option to AllowOverride directive:




    AllowOverride ....... AuthConfig     ......
     ......

gentoo portage archive

Index of /gentoo/portage-archive/2011/portage-20111215/app-emulation/lxc/

gentoo pecl-memcache problem

memcache-2.2.6 # make
/bin/sh /home/vova/memcache-2.2.6/libtool --mode=compile cc -I/usr/lib64/php5.2/include/php -I. -I/home/vova/memcache-2.2.6 -DPHP_ATOM_INC -I/home/vova/memcache-2.2.6/include -I/home/vova/memcache-2.2.6/main -I/home/vova/memcache-2.2.6 -I/usr/lib64/php5.2/include/php -I/usr/lib64/php5.2/include/php/main -I/usr/lib64/php5.2/include/php/TSRM -I/usr/lib64/php5.2/include/php/Zend -I/usr/lib64/php5.2/include/php/ext -I/usr/lib64/php5.2/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /home/vova/memcache-2.2.6/memcache.c -o memcache.lo
/home/vova/memcache-2.2.6/libtool: line 481: CDPATH: command not found
libtool: Version mismatch error. This is libtool 2.4, but the
libtool: definition of this LT_INIT comes from an older release.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4
libtool: and run autoconf again.
make: *** [memcache.lo] Error 63



If you have error with libtool version after make - way to install:
phpize --clean && phpize
rm aclocal.m4
aclocal
autoconf
./configure
make
make install
[memcache-2.2.6, gentoo.x86-64]

fail2ban: Fix for iptables actions failing when run concurrently

Actions that call iptables fail when they are run at the same time by separate threads. I have added a lock to disallow prevent the problem. The patch should be applied to server/action.py

download there:
http://sourceforge.net/tracker/?func=detail&aid=3413485&group_id=121032&atid=689046

How to show the Grub 2 menu at boot time

 

To show the Grub 2 menu at boot time it is only the right hand shift key that works

directadmin: How to disable count directory size in file manager

if you want to disable count summary directory size in directadmin filemanager add filemanager_du=0 to directadmin.conf


cat /usr/local/directadmin/conf/directadmin.conf
...
filemanager_du=0
...


than restart directadmin:


/etc/init.d/directadmin restart

nginx: Too many open files

if you have error:

socket() failed (24: Too many open files)

you need change  worker_rlimit_nofile in nginx.conf . (Default: 1024)


#cat  /etc/nginx/nginx.conf
user www-data;
worker_processes 1;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;


worker_rlimit_nofile 4096; ### default is 1024


events {
worker_connections 1024;
# multi_accept on;
}





than restart (reload) nginx  /etc/init.d/nginx restart



and check :


#ps waux|grep nginx
   root 2604 0.0 0.0 9252 908 pts/0 S+ 09:53 0:00 grep --color=auto nginx
   root 3330 0.0 0.2 113516 40324 ? Ss Jul26 0:00 nginx: master process /usr/sbin/nginx
   www-data 18098 0.7 0.2 113516 40168 ? S Jul30 10:35 nginx: worker process


#cat /proc/18098/limits |grep files
  Max open files 4096 4096 files  

install mod_rpaf with apache-2.4

Apache-2.4 has own module mod_remoteip

"This module is used to treat the useragent which initiated the request as the originating useragent as identified by httpd for the purposes of authorization and logging, even where that useragent is behind a load balancer, front end server, or proxy server."

you may use it with nginx such:


....
    RemoteIPHeader X-Forwarded-For
....

but I found 2 issues for me:

1. I must change LogFormat ( replace %h with %a) to show original IP in logs
2.  /server-status/ page show nginx ip address instead original

So, i deside to use mod_rpaf. Defaut veriosn does not compile with aapche-2.4.
To solve the issue:


 replace  “remote_” with “client_” in “mod_rpaf-2.0.c” 


make it:

apxs -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c

or install

apxs -i  -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c

and use it with apache



    RPAFenable On
    RPAFproxy_ips 127.0.0.1 
    RPAFsethostname On
    RPAFheader X-Forwarded-For

libtool: link: cannot find the library `/usr/local/apr/lib/libexpat.la'

I got this error, when install apache on gentoo linux, rather emere app-admin/apache-tools ends with error:

libtool: link: cannot find the library `/usr/lib64/libexpat.la' or unhandled argument `/usr/lib64/libexpat.la'
make[1]: *** [htpasswd] Помилка 1
make[1]: *** Очікування завершення завдань...
libtool: link: cannot find the library `/usr/lib64/libexpat.la' or unhandled argument `/usr/lib64/libexpat.la'
make[1]: *** [htdigest] Помилка 1
libtool: link: cannot find the library `/usr/lib64/libexpat.la' or unhandled argument `/usr/lib64/libexpat.la'
make[1]: *** [rotatelogs] Помилка 1
make[1]: Залишаю каталог "/var/tmp/portage/app-admin/apache-tools-2.2.22/work/httpd-2.2.22/support"
make: *** [all-recursive] Помилка 1

I found such way to solve this issue:

cd /var/tmp/portage/www-servers/apache-2.2.22-r1/work/httpd-2.2.22/srclib/apr-util/xml/expat/
./configure
# make
# cp libexpat.la /usr/lib/

than, I can emerge apache without problem and errors.

bash: remove file extension

To get filename without extention try this:

Code:

file='file.txt'
name=${f%\.*}

or

# echo file_name.jpg  | sed 's/\(.*\)\..*/\1/'

file_name

#