quinta-feira, 30 de abril de 2015

Mysql Configuration and Master Slave Replication

Mysql Configuration
key_buffer = 4048M
key-buffer-size = 32M
max_connect_errors = 1000000
max_connections = 50
table_cache = 500
thread_concurrency = 0
thread_cache_size = 50

innodb_flush_log_at_trx_commit=1
sync_binlog=1
innodb_thread_concurrency = 0
innodb_read_io_threads=64
innodb_write_io_threads=64
innodb_buffer_pool_size=26GB
innodb_additional_mem_pool_size=4GB

query_cache_type = 1
query_cache_limit = 56M

join_buffer_size = 128M
read_buffer_size = 128M

Master Slave Replication

Edit the file /etc/mysql/my.cnf:

bind-address = 0.0.0.0

server-id = 20
log_bin = /var/log/mysql/mysql-bin.log
binlog_do_db = yourDB_name # Change this to the Database you want to keep synchronized with the Master
replicate_do_db = yourDB_name # Change this to the Database you want to replicate to the Slave
slave-skip-errors = 1062 # Ignore errors on duplicate inserts.. You can change this to any type of error you want to skip (the replication / synchornization keeps going, no need to restart it)
log_slave_updates = 1

Drupal 7

Update & Upgrade

apt-get update 
apt-get upgrade

Timezone change (if you do not need to change timezone, ignore this step)

dpkg-reconfigure tzdata

Adding repository (edit /etc/apt/sources.list and add links to dotdeb repository)

vi /etc/apt/sources.list
deb http://packages.dotdeb.org wheezy all
deb-src http://packages.dotdeb.org wheezy all

Adding keys for the repository

wget http://www.dotdeb.org/dotdeb.gpg
cat dotdeb.gpg | sudo apt-key add -

Update & Upgrade

apt-get update 
apt-get upgrade

Nginx & Mysql & Memcached installation

apt-get install nginx-extras mysql-server mysql-client memcached

You probably noticed that I use nginx-extras package, you can see comparison between available Nginx packages here.
(all configuration files later in this setup are made for nginx-extras, so if you install other nginx package, some stuff will not work, but you can remove them from suggested nginx configuration)

PHP-FPM & APC installation

apt-get install php5-fpm php5-gd php5-mysql php-apc php-pear php5-cli php5-common php5-curl php5-mcrypt php5-cgi php5-memcached

Drush installation (v5.9)

pear channel-discover pear.drush.org
pear install drush/drush

(if you want double check which drush version is installed, enter in terminal this line: drush version)

Nginx & PHP-FPM restart
While now is installed all necessary, it's not yet needed to restart, but is not bad to check if default stuff are installed properly without errors.

/etc/init.d/php5-fpm restart
/etc/init.d/nginx restart

PHP-FPM configuration & multiple pool
This PHP5-FPM pool configuration uses three pools, www1, www2 and www3 and unix sockets (defined in upstream_phpcgi_unix.conf). You need to create three pools or alternatively, change upstream_phpcgi_unix.conf to match your existing PHP5-FPM configuration.

cd  /etc/php5/fpm/pool.d
mv  www www1
cp  www1 www2
cp  www1 www3

Now we have three pools, and is needed to change in each of this files reference to sockets (each pool require its own socket). Or you can download all three files below and upload them to folder /etc/php5/fpm/pool.d

vi /www1

line 4 - rename www to www1
line 33 - put listen path to: /var/run/php5-fpm-www1.sock

vi /www2

line 4 - rename www to www2
line 33 - put listen path to: /var/run/php5-fpm-www2.sock

vi /www3

line 4 - rename www to www3
line 33 - put listen path to: /var/run/php5-fpm-www3.sock

Nginx configuration for Drupal

1. Remove default nginx configuration (backup)

mv /etc/nginx /etc/nginx.old

2. Getting troubleshooter configuration from github
This configuration has several changes from original perusio configuration, and is only for Drupal 7.
Perusio configuration you can use on both 6 and 7 version of Drupal.

cd /etc
git clone https://github.com/troubleshooter/drupal7-with-nginx.git
mv drupal7-with-nginx nginx

3. Adjust nginx for your domain
In folder /etc/nginx/sites-available you will find configuration for example.com domain.Rename this file to yourdomain.com.conf

cd nginx/sites-available
mv example.com.conf  yourdomain.com.conf

After that you need to change domain name in this file according to your needs. Replace all reference for example.com to yourdomain.com.

vi yourdomain.com.conf

Additional, path to root of your site is then /var/www/sites/yourdomain.com, also adjust for your needs. (e.g in my setup all sites are /var/www/mydomain.com)

If you want that server do not listen for IPv6, comment all reference to them. Or if you want that server listens on exact IPv6 address, insert them in nginx config. You can see which IPv6 address have the server with this command:

ip -f inet6 addr show eth0

3. Nginx microcache
Microcaching requires the presence of /var/cache/nginx/microcache directory which don't exist by default. You must create it and grant the appropriate permissions to the Nginx user (in Debian 7 it's www-data)

mkdir /var/cache/nginx/
mkdir /var/cache/nginx/microcache/
chown www-data:www-data /var/cache/nginx/microcache

4. Nginx SSL
This Nginx configuration requires SSL (if you do not want SSL, skip this step, but you need then adjust Nginx configuration). Below are instruction for generating self-signed certificate, and location for them are following current Nginx configuration. You can adjust all this for your needs.

Create directory

mkdir /etc/nginx/ssl

Generate key & certificate

openssl genrsa -des3 -out ssl.key 1024
(enter your key)

openssl req -new -key ssl.key -out server.csr
(enter your information for certificate)

cp ssl.key ssl.key.org
openssl rsa -in ssl.key.org -out ssl.key

openssl x509 -req -days 365 -in server.csr -signkey ssl.key -out ssl-unified.crt

Moving key and certificate according to current nginx configuration (or edit location in yourdomain.com.conf)

mkdir /etc/nginx/ssl/certs/
mv ssl-unified.crt /etc/nginx/ssl/certs/ssl-unified.crt

mkdir /etc/nginx/ssl/private
mv ssl.key /etc/nginx/ssl/private/ssl.key

5. Enabling your site configuration
Final step for Nginx is to create sites-enabled folder, and then create symlinks to sites which you want to enable

mkdir /etc/nginx/sites-enabled
ln -s /etc/nginx/sites-available/000-default.conf /etc/nginx/sites-enabled/000-default.conf
ln -s /etc/nginx/sites-available/youbeq.com.conf /etc/nginx/sites-enabled/youbeq.com.conf

Nginx & PHP-FPM restart
This should be last step, restart PHP-FPM and Nginx.

/etc/init.d/php5-fpm restart
/etc/init.d/nginx restart

If Nginx throw error regarding types_hash, server_names or variables try to uncomment following lines and change values in /etc/nginx/nginx.conf :

variables_hash_max_size 1024; # default 512
variables_hash_bucket_size 64; # default is 64

or add these two line

types_hash_bucket_size 2048;
server_names_hash_bucket_size 64;

Now you should have configured Nginx & PHP-FPM & APC & Memcached and Drush on Debian 7.
Final step is clean Drupal installation or migration of existing website.

WWW directory
By default /var/www directory do not exist, so we need to create them and assign ownership to www-data group
(if you work with www-data group as you should, drush need read/write permission)

mkdir /var/www
chown www-data:www-data /var/www

Drupal installation (switching to www-data)

su www-data
cd /var/www
wget http://ftp.drupal.org/files/projects/drupal-7.23.tar.gz
tar -xzvf drupal-7.23.tar.gz
mv drupal-7.23 yourdomain.com

Additional configuration

APC - tweak your APC settings with following or adjust as you wish

vi /etc/php5/mods-available/apc.ini

add this lines:
apc.enabled=1
apc.shm_segments=1
apc.shm_size=64M
apc.ttl=7200
apc.write_lock = 1
apc.slam_defense = 0

Memcached
Configuration file is located in etc folder: /etc/memcached.conf.
Probably is good to increase memory pool at least to 128MB (default is 64MB)
Change line 23 from -m 64 to -m 128

PHP-FPM & Nginx & Mysql

All further tweaking depends also on your server and Drupal sites. High traffic sites need a lot more tweaking from this. You have some example of configuration for mysql, php-fpm, etc.., but use them with caution.

Install PHP Redis

https://github.com/nicolasff/phpredis

wget https://github.com/nicolasff/phpredis/archive/master.zip

unzip master.zip

cd phpredis-master

phpize

./configure

make && make install

nano /etc/php5/fpm/conf.d/redis.ini

extension=redis.so

/etc/init.d/php5-fpm restart

Mail Server

Installation

In order to install Postfix with SMTP-AUTH and TLS, first install the postfix package from the Main repository using your favorite package manager. For example:

sudo apt-get install postfix

Simply accept the defaults when the installation process asks questions. The configuration will be done in greater detail in the next stage.

Configuration

From a terminal prompt:

sudo dpkg-reconfigure postfix

Insert the following details when asked (replacing server1.example.com with your domain name if you have one):

General type of mail configuration: Internet Site
NONE doesn't appear to be requested in current config
System mail name: server1.example.com
Root and postmaster mail recipient: <admin_user_name>
Other destinations for mail: server1.example.com, example.com, localhost.example.com, localhost
Force synchronous updates on mail queue?: No
Local networks: 127.0.0.0/8
Yes doesn't appear to be requested in current config
Mailbox size limit (bytes): 0
Local address extension character: +
Internet protocols to use: all

Now is a good time to decide which mailbox format you want to use. By default Postifx will use mbox for the mailbox format. Rather than editing the configuration file directly, you can use the postconf command to configure all postfix parameters. The configuration parameters will be stored in /etc/postfix/main.cf file. Later if you wish to re-configure a particular parameter, you can either run the command or change it manually in the file.

To configure the mailbox format for Maildir:

sudo postconf -e 'home_mailbox = Maildir/'

You may need to issue this as well:

sudo postconf -e 'mailbox_command ='

Note: This will place new mail in /home/username/Maildir so you will need to configure your Mail Delivery Agent to use the same path.

Configure Postfix to do SMTP AUTH using SASL (saslauthd):

sudo postconf -e 'smtpd_sasl_local_domain ='
sudo postconf -e 'smtpd_sasl_auth_enable = yes'
sudo postconf -e 'smtpd_sasl_security_options = noanonymous'
sudo postconf -e 'broken_sasl_auth_clients = yes'
sudo postconf -e 'smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination'
sudo postconf -e 'inet_interfaces = all'

Next edit /etc/postfix/sasl/smtpd.conf and add the following lines:

pwcheck_method: saslauthd
mech_list: plain login

Generate certificates to be used for TLS encryption and/or certificate Authentication:

touch smtpd.key
chmod 600 smtpd.key
openssl genrsa 1024 > smtpd.key
openssl req -new -key smtpd.key -x509 -days 3650 -out smtpd.crt # has prompts
openssl req -new -x509 -extensions v3_ca -keyout cakey.pem -out cacert.pem -days 3650 # has prompts
sudo mv smtpd.key /etc/ssl/private/
sudo mv smtpd.crt /etc/ssl/certs/
sudo mv cakey.pem /etc/ssl/private/
sudo mv cacert.pem /etc/ssl/certs/

Configure Postfix to do TLS encryption for both incoming and outgoing mail:

sudo postconf -e 'smtp_tls_security_level = may'
sudo postconf -e 'smtpd_tls_security_level = may'
sudo postconf -e 'smtpd_tls_auth_only = no'
sudo postconf -e 'smtp_tls_note_starttls_offer = yes'
sudo postconf -e 'smtpd_tls_key_file = /etc/ssl/private/smtpd.key'
sudo postconf -e 'smtpd_tls_cert_file = /etc/ssl/certs/smtpd.crt'
sudo postconf -e 'smtpd_tls_CAfile = /etc/ssl/certs/cacert.pem'
sudo postconf -e 'smtpd_tls_loglevel = 1'
sudo postconf -e 'smtpd_tls_received_header = yes'
sudo postconf -e 'smtpd_tls_session_cache_timeout = 3600s'
sudo postconf -e 'tls_random_source = dev:/dev/urandom'
sudo postconf -e 'myhostname = server1.example.com' # remember to change this to yours

The file /etc/postfix/main.cf should now look like this:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

myhostname = server1.example.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = server1.example.com, example.com, localhost.example.com, localhost
relayhost =
mynetworks = 127.0.0.0/8
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
smtpd_sasl_local_domain =
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination
smtpd_tls_auth_only = no
#Use these on Postfix 2.2.x only
#smtp_use_tls = yes
#smtpd_use_tls = yes
#For Postfix 2.3 or above use:
smtp_tls_security_level = may
smtpd_tls_security_level = may
smtp_tls_note_starttls_offer = yes
smtpd_tls_key_file = /etc/ssl/private/smtpd.key
smtpd_tls_cert_file = /etc/ssl/certs/smtpd.crt
smtpd_tls_CAfile = /etc/ssl/certs/cacert.pem
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

Restart the postfix daemon like this:

sudo /etc/init.d/postfix restart

Authentication

The next steps are to configure Postfix to use SASL for SMTP AUTH.

First you will need to install the libsasl2-2, sasl2-bin and libsasl2-modules from the Main repository [i.e. sudo apt-get install them all].

Note: if you are using Ubuntu 6.06 (Dapper Drake) the package name is libsasl2.

We have to change a few things to make it work properly. Because Postfix runs chrooted in /var/spool/postfix we have change a couple paths to live in the false root. (ie. /var/run/saslauthd becomes /var/spool/postfix/var/run/saslauthd):

Note: by changing the saslauthd path other applications that use saslauthd may be affected.

First we edit /etc/default/saslauthd in order to activate saslauthd. Remove # in front of START=yes, add the PWDIR, PARAMS, and PIDFILE lines and edit the OPTIONS line at the end:

# This needs to be uncommented before saslauthd will be run automatically
START=yes

PWDIR="/var/spool/postfix/var/run/saslauthd"
PARAMS="-m ${PWDIR}"
PIDFILE="${PWDIR}/saslauthd.pid"

# You must specify the authentication mechanisms you wish to use.
# This defaults to "pam" for PAM support, but may also include
# "shadow" or "sasldb", like this:
# MECHANISMS="pam shadow"

MECHANISMS="pam"

# Other options (default: -c)
# See the saslauthd man page for information about these options.
#
# Example for postfix users: "-c -m /var/spool/postfix/var/run/saslauthd"
# Note: See /usr/share/doc/sasl2-bin/README.Debian
#OPTIONS="-c"

#make sure you set the options here otherwise it ignores params above and will not work
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"

Note: If you prefer, you can use "shadow" instead of "pam". This will use MD5 hashed password transfer and is perfectly secure. The username and password needed to authenticate will be those of the users on the system you are using on the server.

Next, we update the dpkg "state" of /var/spool/postfix/var/run/saslauthd. The saslauthd init script uses this setting to create the missing directory with the appropriate permissions and ownership:

dpkg-statoverride --force --update --add root sasl 755 /var/spool/postfix/var/run/saslauthd

This may report an error that "--update given" and the "/var/spool/postfix/var/run/saslauthd" directory does not exist. You can ignore this because when you start saslauthd next it will be created.

Finally, start saslauthd:

sudo /etc/init.d/saslauthd start

Testing

To see if SMTP-AUTH and TLS work properly now run the following command:

telnet localhost 25

After you have established the connection to your postfix mail server type

ehlo localhost

If you see the lines

250-STARTTLS
250-AUTH

among others, everything is working.

Type quit to return to the system's shell.

Troubleshooting

Remove Postfix from chroot

If you run into issues while running Postfix you may be asked to remove Postfix from chroot to better diagnose the problem. In order to do that you will need to edit /etc/postfix/master.cf locate the following line:

smtp      inet  n       -       -       -       -       smtpd

and modify it as follows:

smtp      inet  n       -       n       -       -       smtpd

Then restart Postfix:

sudo /etc/init.d/postfix restart

Configuring saslauthd to Default

If you don't want to run Postfix in a chroot, or you'd like to not use chroot for troubleshooting purposes you will probably also want to returnsaslauthd back to its default configuration.

The first step in accomplishing this is to edit /etc/default/saslauthd comment the following lines we added above:

#PWDIR="/var/spool/postfix/var/run/saslauthd"
#PARAMS="-m ${PWDIR}"
#PIDFILE="${PWDIR}/saslauthd.pid"

Then return the saslauthd dpkg "state" to its default location:

dpkg-statoverride --force --update --add root sasl 755 /var/run/saslauthd

And restart saslauthd:

sudo /etc/init.d/saslauthd restart

Using Port 587 for Secure Submission

If you want to use port 587 as the submission port for SMTP mail rather than 25 (many ISPs block port 25), you will need to edit /etc/postfix/master.cf and uncomment the line

submission inet n      -       n       -       -       smtpd

Installation

The installation is extremely simple, just install the following packages:

dovecot-imapd
dovecot-pop3d

For example, using apt-get:

sudo apt-get install dovecot-imapd dovecot-pop3d

Configuration

To configure dovecot, you edit the file /etc/dovecot/dovecot.conf. There are a couple of choices which you need to make.

Choice of Protocols

The choice is broadly between two protocols: IMAP and POP3. POP3 is useful when e-mail is checked from only one computer. It is best for people who download their email and then work off-line. IMAP is the better choice when you would like to check your mail from multiple computers (at work and home, for example). IMAP has the added benefit of accessing folders on the server, allowing you to organize your e-mail, and access it from anywhere. For more information, see the wikipedia articles on POP3 and IMAP. IMAPS and POP3S are more secure than the simple IMAP and POP3 because they use TLS encryption to connect.

Once you have chosen, amend the following line in the file /etc/dovecot/dovecot.conf:

protocols = pop3 pop3s imap imaps

This enables those protocols when dovecot is started. Note: if you like, you can enable all the above protocols, or you can choose to enable just one or any number of them. In addition, add the following line in the "protocol pop3" section in the /etc/dovecot/dovecot.conf:

pop3_uidl_format = %08Xu%08Xv

Choice of Mailboxes

Recent Ubuntu releases (any since 2008)

Dovecot currently supports maildir and mbox formats. They are the most commonly used mailbox formats. They both have their own benefits, discussed here.

If you have trouble figuring out what you are using, it's most likely mbox. Maildir mails are almost always stored in ~/Maildir/ directory, which contains cur/, new/ and tmp/ subdirectories. In maildir each mail is stored in a separate file, while with the mbox format one file contains all mails in the mailbox. Files in /var/mail/ are nearly always mbox files, one for each user.

Maildir is better overall because it is more scalable and can't get corrupted so easily. So, if you have trouble figuring out what you should be using and have a choice, choose maildir.

To configure Dovecot for your mailbox format use (for maildir):

mail_location = maildir:~/Maildir

or, for mbox;

mail_location = mbox:~/mail:INBOX=/var/mail/%u

Note: You will also need to configure your MTA to transfer the incoming mail to this type of mailbox. (If you installed Postfix with instructions from the previous wiki, then choose mbox.)

Setting up Maildir

Recent Ubuntu releases (any since 2008)

Do these steps ONLY if you want Maildir. This setup will put the Maildir in each user's home directory.

If you are using Postfix as your MTA, then add the following line to your /etc/postfix/main.cf file:

home_mailbox = Maildir/

Edit /etc/dovecot/dovecot.conf:

mail_location = maildir:/home/%u/Maildir

It's a good idea to pre-create the Maildir for future users:

sudo maildirmake.dovecot /etc/skel/Maildir
sudo maildirmake.dovecot /etc/skel/Maildir/.Drafts
sudo maildirmake.dovecot /etc/skel/Maildir/.Sent
sudo maildirmake.dovecot /etc/skel/Maildir/.Trash
sudo maildirmake.dovecot /etc/skel/Maildir/.Templates

Then, for an existing user:

sudo cp -r /etc/skel/Maildir /home/myuser/
sudo chown -R myuser:usergroup /home/myuser/Maildir
sudo chmod -R 700 /home/myuser/Maildir

Test

Start dovecot:

/etc/init.d/dovecot start

To check that it is running, type the command

ps -A | grep dovecot

You should see the dovecot service running. If you have enabled imap, or pop3, you can also try to log in with the commands

telnet localhost pop3

telnet localhost imap2

If you see something like the following, the installation has been successful.

matt@kalliope:~$ telnet localhost pop3
Trying localhost...
Connected to localhost.
Escape character is '^]'.
+OK dovecot ready.

Authentication

If you've gotten this far, dovecot is working but may still have to be configured so that users can login to check their mail. The simplest login method is sending the user login info in plain text and allowing access only to users that already have UNIX access (ie. in /etc/passwd) to the machine dovecot is running on. Allowing access only to users that already have UNIX access is the dovecot default, nothing has to be done to configure this. Also, plain text logins are enabled by default when using secure protocols. To enable plain text logins when not using a secure protocol, put the following line in /etc/dovecot/dovecot.conf:

disable_plaintext_auth = no

If you are using the secure protocols imaps or pop3s, plain text logins is not a problem, because the transfer is done via an encrypted connection. If you are using simply imap or pop3, and will be connecting to the server from outside your local computer or home network, it is a good idea to configure more secure authentication. For further details about how to do this, see this page on the dovecot website.

SSL

To configure Dovecot to use SSL, edit the file /etc/dovecot/dovecot.conf and amend the following lines (in some cases you may simply have to remove the # symbol from the beginning of the line):

ssl = yes

ssl_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
ssl_key_file = /etc/ssl/private/ssl-cert-snakeoil.key

The cert and key files are created automatically by Dovecot when you install it. (The keys mentioned above are actually the ones created by Postfix, however, but are suitable for use by Dovecot as well). Please note, that these keys are not signed and will give "bad signature" errors when connecting from a client. To avoid this, you can use commercial certificates, or even better, you can use your own SSL certificates. Hopefully, a guide will appear soon on this wiki on how to do this. In the meantime, here are some good ones: this guide on the Linux howto database, and this guide on the Debian Administration website.

Accessing from Outside

In order to access your mail server from another computer, you'll have to configure your firewall or router to allow connections to the server on the necessary ports:

IMAP - 143
IMAPS - 993
POP3 - 110
POP3S - 995

You'll also need to uncomment following line in /etc/dovecot/dovecot.conf:

listen = *

However, this method may cause conflicts with other servers already listening on other ports. The alternative (and probably more desirable) method, then, is to enable the specific listening ports for the protocols that are intended to be used. For example, for IMAP/IMAPS and POP3/POP3S, add to the correct protocol imap and protocol pop3 sections:

protocol imap {
     listen = *:143
     ssl_listen = *:993
     ...
     }

protocol pop3 {
     listen = *:110
     ssl_listen = *:995
     ...
     }

Rmmod-a

quinta-feira, 30 de abril de 2015

Mysql Configuration and Master Slave Replication

Drupal 7

Nginx configuration for Drupal

Additional configuration

Mail Server

Installation

Configuration

Authentication

Testing

Troubleshooting

Remove Postfix from chroot

Configuring saslauthd to Default

Using Port 587 for Secure Submission

Other Postfix Guides

Postfix Basic Setup

Postfix Virtual Mailbox and Antivirus Filtering

Postfix Setup For Sender Policy Framework (SPF) Checking

Postfix Setup For DKIM email signing and verification

Add Dspam to Postfix

Postfix Complete Solution

Dovecot LDAP

Dovecot SASL

Installation

Configuration

Choice of Protocols

Choice of Mailboxes

Setting up Maildir

Test

Authentication

SSL

Accessing from Outside