Phine Solutions web work notes

Here is the background of the problem:

Recently I added SSL certificate (purchased from a CA) to my Apache server. After making configuration changes, opening the port and restarting the server, the https connection still won’t work. From Chrome, I got “Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error.”; and from FireFox, simply “connection interrupted”. Neither search term really offer much help. However one tips really helped is to check http on port 443 directly.

That worked. So in my case, it’s almost like Apache does not recognize SSL request. A new round of checking configuration and certificate files ensued.

I did find the fix in the end. In my Apache virtual host files, I have virtual host entries defined like this:

<VirtualHost SERVER_IP>

I had to change it to <VirtualHost SERVER_IP:80>

It’s probably because Apache matched the SSL request to a non-SSL virtual host and tried to serve it. So if you are adding SSL to the mix, remember to check the previously defined virtual hosts on port 80 and tighten up the rule.

5 years ago if you ask me to build my own Linux VPS for my websites I would’ve shaken my head and said it was too much for a non-sysadmin like me. Now I’m pretty comfortable of doing it. I want to share my thoughts in this post and hopefully it can be useful to other web builders.

When I first started I put my websites on shared hosting servers. While it was cheap and easy to setup you often don’t get the best performance on your dollar. This is especially true when your site gets more traffic and you are anal about the downtime like me. Until I discovered VPS. VPS is a great solution to move up from shared hosting. Having a VPS server means you have full control over a virtual host so you can install and configure the way you like; also, because using VPS means “renting” a slice of a physical server, someone else takes care about the racking, networking and hardware maintenance.

There are generally two types of VPS. One is “managed” and the other is obviously, “un-managed”. “managed” in this case means the service provider will help you install apps, trouble shoot, and in some cases, walk you step by step to help you resolve an issue, as long as you ask. And often times you have a full web based system control panel, like “CPanel” installed for you. The later case, apparently, don’t have this kind of service. You are basically given a barebone server and you are on your own. As you can see a managed service will be more costly.

I started with managed VPS. But as my Linux skill gets better the “managed part” of the service becomes less and less necessary. CPanel is a great tool but it also a big resource consumer itself. Sometimes you might find most of your system resource is consumed by addons, not the main apps like web server or the database.

To grow out of a managed service the key is to try and learn. for example, if you choose to stay on CPanel forever (not that there is anything wrong with it:)), you’ll stay on it for ever. To take the leap you need to be ready for it. It took me sometime to get to the comfort level that I’m at now. During which, I found a number of guidelines that I begin to follow.

Keep a good and updated server diary

I think this is the first thing to do when you start handle your own server. A good server diary can not only help you trouble shoot, it’s also a good reference when you need to re-install, upgrade, or in some less frequent cases, move to new service provider.

Like any other service you buy, hosting service provider’s service quality can go down too. You pretty much don’t have any other choices except voting with your feet. A good service log can make changing host a lot easier. Recently when I switched my VPS provide, it only took me a few hours to stand up full services on a brand new VPS host. The process also forced me to refresh and update my server diary.

Utilize the external services

To host a website on VPS, we have to install pretty much everything ourselves. They include at least Apache web server, PHP and MySQL. However, besides the basic LAMP stack, we also need to take care of services like DNS, email service. To keep your admin work as simple as possible, I strongly recommend outsourcing DNS or email to external service provider. For DNS, there are lots of options. I use dnsmadeeasy.com. For only a few dallors a month, you are completely shielded from managing your DNS app. Although you still need to understand what a “cname” is and how to change the DNS record to point your web site to the correct ip, you have a much much smaller learning curve.

Email is another service that can get quite complex. I found using Google App’s email service makes a lot of sense. Since their email service is based on Gmail, the IMAP , effective Spam filter and web access are all included naturally. Without the email server and SmapAssasin taking up resource on your server, your server is also better optimized. Google Apps has both free and paid version. Another benefit of using reputable external email service is the trust level you gain for your emails. A lot of my users who use Yahoo mail couldn’t receive message because they were marked spam. But since I started using Gmail service it has been a lot better.

If your web site has a lot of user generated content like photos, you may also consider using the cloud storage like Amazon AWS. I’m generally against building your site completely depending on the cloud but it another subject.

Install from source

I know this is quite a debatable subject. Installing application from source doesn’t always give you the type of control that you can have using the packaging tools. However, there are several benefit that can’t be overlooked.

First, you have the full control on the binary and you can build the exact binary that you want. For example, when building your own PHP binary, you can specify the features you want to enable to have a small footprint. Same can be applied to Apache httpd server. This will directly impact the memory usage of your web server.

Secondly if you are accustomed to source installation you will not need to hunt around for the latest RPM or whatever installation package that built by someone else. You can stay updated with the latest version of software. Since the same procedure can be universally applied to all the Linux distros you are less likely to be affected by the different packaging tool that different Linux distro offers.

And lastly, it’s really not that hard to do.

Some basic steps

With a brand new VPS, there are some basic setups that have to be run to ensure the security and basic usability. Your VPS service provider will configure your VPS to a certain degree before handing over so you might need to look into the system configuration like partitioning before proceeding to the steps below.

Update system information

echo “mynewserver.com” > /etc/hostname

hostname -F /etc/hostname

ln -sf /usr/share/zoneinfo/US/Eastern /etc/localtime

Create users

Adding users is the second must.

groupadd johndoe

useradd -d /home/johndoe -g johndoe -p johndoespassword

To create user “apache” for your web server you’ll need the following command:

groupadd apache

useradd apache -c “Apache Server” -d /dev/null -g apache -s /sbin/nologin

Turn off the unnecessary services

chkconfig –list | grep 3:on

chkconfig <service name> off

netstat -an

Secure SSH

You want to turn off the root access:

/etc/ssh/sshd_config file, set this: PermitRootLogin no

Also you want to set up public/private key authentication.

I would also recommend changing the port from 22 to something else.

Set up a firewall using iptables

If you are reading this article you probably know what iptables is. The tricky part is how to configure it. A few years ago I went a great length to learn what tables and chains are and how to set up a shell script to configure an iptables firewall. The problem was I soon forgot what I had learned since configuring iptables is not something I do on a daily basis for a developer like myself. And guess what, I locked myself out on my first try in a new server.

Luckily there are tools today which wraps around iptables and expose an easy to use configuration interface. This makes the life a lot easier for me. APF is what I use and the project page can be found here: http://www.rfxn.com/projects/advanced-policy-firewall/.

Install some utilities, compiler and libraries

I only use CentOS/Redhat system as example and yum is my command of choice for packaging tool. Again they are just basic tools and libraries for installing Apache, PHP so others might need to be installed as well. But the key again, is to keep a good log of what has been installed so you have a good reference when you build your next server.

yum install man

yum install vixie-cron

yum install wget

yum install rsync

yum groupinstall ‘Development Tools’

yum install mailx

yum install zlib-devel

yum install openssl-devel

yum install libxml2-devel

yum install curl

yum install curl-devel

yum install libjpeg-devel

yum install libpng-devel

yum install mysql-devel

yum install libxslt-devel

yum install libmcrypt

yum install libmcrypt-devel

yum install libevent

yum install libevent-devel

Install applications

Now it comes the time to install your beloved apps. One thing to remember if you install from source is to create script in /etc/init.d and add the service entry. For example after installing Apache http server, you need to add the httpd startup/shutdown script to /etc/init.d and add it to your service list:

chkconfig –add httpd

chkconfig –levels 235 httpd on

Install a MTA

Ok this one was really a thinker. I spent at least 5 hours to get this to work and finally I was able to use Sendmail to relay through my Gmail account.

A little background:

I have a Linux VPS with CentOS installed. The only email MTA is the default Sendmail. Everything else is pretty much the standard CentOS 4 installation. I don’t intend to use this box as a mail server or any other type of email processor. What I was trying to do is to add some basic capability to send out email from the box using my existing email accounts hosted in Gmail. And I didn’t want to install any additional software such as Postfix for this.

That being said, let me continue to take you down the path that I have gone through, without the stumbling blocks.

My approach was basically: problem -> Google for solutions -> trouble shoot -> Google again. So I found a lot of useful content on the web during the process.

1. Check sendmail

Since gmail uses TLS, you will need to make sure your sendmail is compiled with TLS (for encryption) and SASL (for authentication). This is the command to use to check it:

/usr/sbin/sendmail -d0.1 -bv root

In my case, sendmail does have the necessary compilation flags so I was good. If yours doesn’t, you’ll need to re-compile sendmail and update the binary that is used to start the sendmail service, which is not covered here.

2. Upgrading Cyrus SASL

If your SASL installation doesn’t have the “plain” and “login” lib you will have authentication problem with Gmail. You can see why when you get to the sendmail configuration in the later steps. The common error in the /var/log/maillog is this:

AUTH=client, available mechanisms do not fulfill requirements

It was a vague error and I was so frustrated with it at one point I was ready to give up. However, this article about setting up Postfix with Gmail casted some light and helped me figured out the cause.

The problem is that SASL doesn’t have all the necessary plugins. The “login” and “plain” are the plugins necessary to talk to Gmail smtp. So I had to upgrade SASL to fix the problem. Here is what I did:

$ wget http://ftp.andrew.cmu.edu/pub/cyrus-mail/cyrus-sasl-2.1.21.tar.gz
$ tar -xzf cyrus-sasl-2.1.21.tar.gz
$ cd cyrus-sasl-2.1.21
$ ./configure
$ make
$ make install

$ mv /usr/lib/sasl2 /usr/lib/sasl2.orig
$ ln -s /usr/local/lib/sasl2 /usr/lib/sasl2

Note: if you have issue installing Cyrus SASL around compiling digestmd5.c, it’s because your compiler is too new. Read here to find out how to patch it.

Since I just switched out the old sasl2 lib without recompiling sendmail, I was concerned sendmail would poop during runtime. Luckily that didn’t happen. Dynamic lib rocks!

3. Generate SSL certificate

I made a directory called certs under /etc/mail. Here are the commands that I used to generate the SSL certificates.

openssl req -new -x509 -keyout cakey.pem -out cacert.pem -days 3650
openssl req -nodes -new -x509 -keyout sendmail.pem -out sendmail.pem -days 3650

Notice I made the certificates good for almost 10 years. I didn’t needed the cacert.pem to be exact.

I also copied /usr/share/ssl/ca-bundle.crt to /ect/mail/certs and included it in the sendmail configuration file. Other wise you’ll see some error like this:

unable to get local issuer certificate

The reason is that the ca bundle file has the Gmail certificate issuer. Although I read it somewhere that email still goes out with this error. Nonetheless, we don’t need to see this if we can fix it.

4. Configure sendmail

With the preparations above we are ready to configure sendmail. I found this tutorial very useful in terms of getting the correct sendmail configurations.

In summary, I have the /etc/mail/auth/client-info looks like this:

AuthInfo:smtp.gmail.com “U:root” “I:username@gmail.com” “P:password” “M:PLAIN”
AuthInfo:smtp.gmail.com:587 “U:root” “I:username@gmail.com” “P:password” “M:PLAIN”

If you use Gmail hosted email with your own domain name, you will have username@hostname.tld in there.

Make sure run:

$ makemap -r hash client-info.db < client-info

and chmod 600 on client info files.

Essential lines in my sendmail.mc

FEATURE(`authinfo’,`hash /etc/mail/auth/client-info.db’)dnl
define(`SMART_HOST’,`smtp.gmail.com’)dnl
define(`RELAY_MAILER_ARGS’, `TCP $h 587′)
define(`ESMTP_MAILER_ARGS’, `TCP $h 587′)

define(`CERT_DIR’, `/etc/mail/certs’)
define(`confCACERT_PATH’, `CERT_DIR’)
define(`confCACERT’, `CERT_DIR/ca-bundle.crt’)
define(`confCRL’, `CERT_DIR/ca-bundle.crt’)
define(`confSERVER_CERT’, `CERT_DIR/sendmail.pem’)
define(`confSERVER_KEY’, `CERT_DIR/sendmail.pem’)
define(`confCLIENT_CERT’, `CERT_DIR/sendmail.pem’)
define(`confCLIENT_KEY’, `CERT_DIR/sendmail.pem’)

define(`confAUTH_MECHANISMS’, `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN’)
TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN’)

NOTE: Be aware that smart-quotes used in the code examples will not be recognised if pasted into your files! Ensure replacing smart-quotes by regular quotes (see comments below for further detail). – thanks Johnny for the suggestion.

The certificate files are generated/copied from the previous step. I’m no sendmail expert so the configuration lines may not be perfect. But it works. Let me know if you have better settings.

One tip I found very useful is to use the debugging feature. You can set a high log level in the sendmail.mc to see at which step sendmail choked and for what reason.

Also, make sure run “make” or m4 every time you touch the sendmail.mc.

m4 sendmail.mc > sendmail.cf

So that’s pretty much it. I restart the sendmail service and out goes my email.

UPDATE:

Recently I installed Sendmail on a brand new VPS and had hard time get the authentication working. It turned out saslauthd was not running. So a note for new system is to make sure saslauthd has to be up and running (better use chkconfig to make sure it starts up at run level 3) in order to get Sendmail authentication working. This may help resolve some issues in the comments.

Using a third party DNS have some great benefit:

1. By outsourcing the DNS look up the load is reduced on your own server, especially like a VPS or dedicated box which runs everything: Apache, MySQL, Named.
2. Well know DNS operates on a backbone and provide each DNS request with complete redundancy.
3. Taking the Named service off from your server not only reduces load but also reduces the open port and maintenance work.
4. An external DNS server will provide better control. This really applies to the domain or sites on shared hosting service and the capability of updating the DNS zone info such as TTL is limited. When it comes to changing the hosting provider, the delay of switching hosting DNS can be unpredictable.

I tried out dnsmadeeasy.com for one of my site and the result was great. It was amazingly simple to create domain name record in dnsmadeeasy account and the change was propagated in a matter of minutes. Another site dyndns.com also provides this kind of service with the similar fee structure. Using the free DNS check service from intodns.com (dnsstuff.com requires fee now) is also recommended to make sure everything is in order.

Just some iptables cheatsheet :

show the current iptables:

iptables -L

check firewall service:

service iptables status

enable iptables service:

chkconfig –level 345 iptables on

save iptables

iptables-save

save iptables to /etc/sysconfig/iptables

service iptables save

Ban an IP:

iptables -A INPUT -s a.b.c.d -j DROP
# using a netmask:
iptables -A INPUT -s x.y.z.0/24 -j DROP

To delete a rule:

iptables -D INPUT -s a.b.c.d -j DROP

APC stands for “Alternative PHP Cache”. It’s one of the 3 PHP accelerators out there (the other 2 are Zend and eAccelerator).

The installation package can be found here: http://pecl.php.net/package/APC

Follow the installtion guide in the package, the apc.so is installed under:

/usr/local/lib/php/extensions/no-debug-non-zts-20060613/

This path will probably varies in different systems.

Now modify the php.ini:

1. extension_dir needs to be modified to the path above.

2. add the extension=apc.so to activate it.

3. tweak a few settings:

apc.shm_size=30 <– 30m is the default value. If up it to something like 128m, I would think the Linux system shared memory setting will also need to be increased more than that. It is 32 by default, which can be found in this file: /proc/sys/kernel/shmmax

I leave the shm_size to 30 for now.

apc.ttl=7200
apc.user_ttl=7200

4. restart the httpd service. Copy apc.php to the webserver. Check phpinfo(). Watch the Apache error_log.

To increase the APC shm_size, the kernel’s max shared memory size will also need to be increased since it is set to a very low value by default.

Add kernel.shmmax=134217728 to /etc/sysctl.conf, and run sysctl -p to make the setting take effect. This will increase the max shared memory size to 128MB.

It is quite easy to create web server access restriction using the Cpanel. There is a configuration setting for "Password Protect Directories" in Cpanel for setting up a user name and password for directory access. What this really does is to create a password file and refer it in the .htaccess.

In stead of using Cpanel, one can always run the process through the command line and it may actually be easier.

Create the passwd file

htpasswd is an Apache utility command to create and update the flat-files used to store usernames and password for basic authentication of HTTP users. Since it will create name and password pair(s) in a flat file, the password is encrypted either using a MD5 version from Apache or crypt() system call.

The following command line can be used to generate a file name passwdfile:

htpasswd -c /home/user/etc/passwdfile admin

his will create a NEW passwdfile and add user "admin" in it. The command will also prompt for the password that you wish to give to this user.

To add a new user, the "-c" option cannot be used.

To remove a user, simply open the htpasswd file and delete the line.

Modify .htaccess

To turn the password into effect, you can add  the following lines in the .htacess file:

AuthType Basic
AuthName "Restricted Area"
AuthUserFile "/home/user/etc/passwdfile"
require valid-user

This will allow anyone in the .htpasswd file to have access.

Besides the flat text password file, one can also use alternate password storage such as DBM or DBD format according to Apache document.

Group the users

There maybe situations that there are a lot of users and they are granted access to different resources. It maybe easier to group the users instead of creating a bunch of .htpasswd files.

A group file is simply a flat file that contains some user names. An admin group file (let's call it admingroup) may look like this:

Admin: admin jdoe mjones

And the .htaccess should include the following:

AuthType Basic
AuthName "Admin Group Only"
AuthUserFile "/home/user/etc/htpasswd" "/home/user/etc/.dmingroup"
Require group Admin

As specified in .htaccess, this is really just a "basic" way for authentication. It is most suitable for a small group of users to access some resource and there is no need to create more sophisticated authenticated method.

If you own a Linux box and use ssh to access it over the internet, chances that it will be under unauthorized login attempt or even brute-force attack. Even you have a strong password for your account, the constant poking from people or evil-bot is some kind of a nuisance to say the least.

Messages like below in /var/log/secure shows how annoying they can be:

Nov 25 23:13:21 —- sshd[21529]: input_userauth_request: invalid user test7

Nov 25 18:13:21 —- sshd[21523]: reverse mapping checking getaddrinfo for h63-210-66-233.seed.net.tw failed – POSSIBLE BREAKIN ATTEMPT!

Nov 25 23:13:30 —- sshd[21607]: input_userauth_request: invalid user test8

Nov 25 18:13:30 —- sshd[21602]: reverse mapping checking getaddrinfo for h63-210-66-233.seed.net.tw failed – POSSIBLE BREAKIN ATTEMPT!

To fully utilize the capability that ssh offers, we should always use public/private key access to a *nix box that is running OpenSSH. Below are some simple steps I used to implement this methodology.

Since I am using PuTTY, the setup and testing are done using putty.exe and puttygen.exe that are downloaded from here.

1. Create public and private key pair.

This can be accomplished using PuTTYgen. Once the program is started, click on the "Generate" button and keep moving your mouse. You can't be lazy here because the it will not proceed until you make your move.

2. save the public and private keys

Once the keys are generated, you need to create a key comment and your private passphrase. The passphrase is tied to your keys so without it your keys are useless. The public key is basically plain text that shows in the box. The private key is in binary form and should be stored with a .ppk extension.

3. place the public key

The public key needs to be stored in the Linux server as $HOME/.ssh/authorized_key2. Since it is plain text you can copy the key from the previous screen and paste them in a Linux editor and save it. An IMPORTANT step is to set the right permission on $HOME, $HOME/.ssh or $HOME/.ssh/authorized_keys so they aren't more permissive than sshd allows by default, which means they can only be read and write by the current account.

The following command can be used to achieve this: $ chmod go-w $HOME $HOME/.ssh $ chmod 600 $HOME/.ssh/authorized_keys

4. place the private key

In PuTTY, you will need to load the private key to your PuTTY session and save the session:

After this step, yu should be able to try the newly configured ssh access. You should be asked to enter the passphase this time, instead of the password. Once this is verified, you can proceed to next step.

 
5. turn off the password authentication on OpenSSH

In the /etc/ssh/sshd_config, there is an option called "PasswordAuthentication", just set it to "no".

Restart sshd and you should be running more secured ssh now.

Even you are running more securely after these measures, you still can't stop people from scanning port 22 and trying to get authenticated repeatedly using a list of user name and password. To reduce this kind of noise, you can also change the running port of sshd. The port configuration is the first parameter in the /etc/ssh/ssh_config file.

There are several parameters in httpd.conf file that I pay attention to. Although the default values normally work out for me, sometimes they need to be tweaked for better performance.

HostnameLookups off 

Apache can look up the visitor's ip and come up with the hostname. And this process will likely slow down the server a bit. 

MaxClients

The max number of child process to spawn. Each Apache child process will take up some memory so this count should be determined on the available memory (total memory – allocation for other processes) and normal Apache child process size.

KeepAlive On

Whether to re-use the connection for multiple HTTP requests. Opening socket is an expensive process. The KeepAlive option can help reduce the overhead of frequent connections.

KeepAliveTimeout = 15

The number of seconds to keep a socket alive waiting on other request. Setting this number to high can cause the too many connections linger and evetually no more connections are available when the MaxClients is reached.

Timeout 300

Disconnect when idle time reaches this value. Depending on the average page loading time it can be lowered to reduce the hanging request.

SendBufferSize

The size of the ouput buffer. Normally I don't mess with this setting as far as it's a reasonable value and big enough for most of my web page sizes. 

Directory settings – The <Directory> tag for the web server root 

AllowOverRide All – tells Apache to pick up the .htaccess setting from each individual directory if available. For better performance this needs to be set to none since Apache will attempt to open .htaccess for each file name request.

Options Index – tells Apache not to perform directory listing when there is no default pages like index.htm(l)… 

Options FollowSymLinks - Apache sever will follow the symbolic links.

Options SymLinksIfOwnerMatch – do not set this. 

Allow directive – If there is need to control which hosts can access the server, use the ip address as much as possible. For example, "Allow from 10.1.2.3" is better than "Allow from allowedhost.com". When the server sees a hostname, it will perform a reverse DNS lookup on the ip to get the hostname, and then do a forward lookup on the hostname to assure the ip addresses match. And this is an expensive process.

Modules 

Only enable the necessary ones to keep the server lean and mean.

This is a great post about writing secure PHP code and part 2. The articles pretty much cover all the points we need to look at to write secure PHP code.

Additionally this post talks about how hackers can use Google code search and a simple sitemap to gain access to your system.