Phine Solutions web work notes

This past weekend my PC was infected by malware and basically it keeps opening the Firefox windows with different websites. After spending hours on how to remove it I gave up. Frankly I think a lot of those sypwares or malwares are created by people who sells the spyware removal software but it is another subject.

So I  upgraded to Vista hoping for a more secured OS.

I have read a lot of negative reviews about Vista and I was always hesitant to upgrade but so far the experience has been quite positive.

The PC I am running Vista on has Intel Dual core processor and 3G ram on it so I didn’t feel a slight slowdown while running applications on Vista. As a matter of fact, I feel everything is a bit faster than XP.

The graphic interface is quite nice and there are certainly a lot of improvements. It seems to me Microsoft also borrowed some ideas from Max OS X, which is not necessarily a bad thing.

A lot of people are complaining about the Vista’s User Account Control (UAC) security prompt. Basically you get a lot of popups when you try to install something or run something that needs to open a port or changes system files. Yes it is annoying but I am not too much bothered by it, as long as it really helps to protect my system (fingers crossed ).

The areas that I think can be improved:

Why so many versions? “Home”, “Business”, “Premium”, “Ultimate”…they are just confusing. The pricing ladder can probably help Microsoft targeting different users and make more money but too many version also create more work on marketing and support.

The UAC is there for a reason but it can be designed better. I know it is not Unix so “sudo” command is not an option but maybe something cleverer so less users will choose to disable it after installation?

Sometimes Vista feels clunky to use. Possiblly because it is designed to incorporate the new features and the old ones so there are many ways to get to the same destination. Simplicity is the ultimate sophistication and Mac OS shows. Maybe WIndows 7 addresses this issue better?

I have only used it for a week so I can’t say for sure Vista’s improved security really delivers. Hopefully I didn’t waste my time and money to upgrade.

Update:

One big hassle that I ran into using Vista is the “safely remove the USB drive” feature. I have a Seagate portable hard drive and I used to use the “safely remove hardware” icon on the toolbar in XP to make sure the data doesn’t get corrupted when unplugging it from USB port. But the feature doesn’t work well in Vista. The icon is still available but Vista always tells me “it is in use and close any program or window that might be using the device…”. I tried some suggestions such as disabling the caching but it didn’t work. Finally I found this Drive Tool page that really saved my time. Thank you Uwe Sieber!

Basically this is what I am trying to do here:

SELECT * FROM Table1 WHERE Table1.col1 = ‘bob’ AND Table1.col3 IN (SELECT Table2.col2 FROM Table2 WHERE Table2.col1 = ‘alice’)

Both Table1 and Table2 have the same column with the name “col1″. And the value of Table1.col3 should match values coming from Table2.col2.

However the query always returns empty set, even there are definitely data matches via the IN where clause. As a matter of fact, if I use the values directly from the subquery the whole query would work.

At a first glance, this looks like a correlated subquery issue but it’s not. Even Table1 and Table2 share the same column name, the subquery is not referring to any Table1 column in the where clause, so it shouldn’t be an issue explained here.

Anyone has some insight on this?

Ok this one was really a thinker. I spent at least 5 hours to get this to work and finally I was able to use Sendmail to relay through my Gmail account.

A little background:

I have a Linux VPS with CentOS installed. The only email MTA is the default Sendmail. Everything else is pretty much the standard CentOS 4 installation. I don’t intend to use this box as a mail server or any other type of email processor. What I was trying to do is to add some basic capability to send out email from the box using my existing email accounts hosted in Gmail. And I didn’t want to install any additional software such as Postfix for this.

That being said, let me continue to take you down the path that I have gone through, without the stumbling blocks.

My approach was basically: problem -> Google for solutions -> trouble shoot -> Google again. So I found a lot of useful content on the web during the process.

1. Check sendmail

Since gmail uses TLS, you will need to make sure your sendmail is compiled with TLS (for encryption) and SASL (for authentication). This is the command to use to check it:

/usr/sbin/sendmail -d0.1 -bv root

In my case, sendmail does have the necessary compilation flags so I was good. If yours doesn’t, you’ll need to re-compile sendmail and update the binary that is used to start the sendmail service, which is not covered here.

2. Upgrading Cyrus SASL

If your SASL installation doesn’t have the “plain” and “login” lib you will have authentication problem with Gmail. You can see why when you get to the sendmail configuration in the later steps. The common error in the /var/log/maillog is this:

AUTH=client, available mechanisms do not fulfill requirements

It was a vague error and I was so frustrated with it at one point I was ready to give up. However, this article about setting up Postfix with Gmail casted some light and helped me figured out the cause.

The problem is that SASL doesn’t have all the necessary plugins. The “login” and “plain” are the plugins necessary to talk to Gmail smtp. So I had to upgrade SASL to fix the problem. Here is what I did:

$ wget http://ftp.andrew.cmu.edu/pub/cyrus-mail/cyrus-sasl-2.1.21.tar.gz
$ tar -xzf cyrus-sasl-2.1.21.tar.gz
$ cd cyrus-sasl-2.1.21
$ ./configure
$ make
$ make install

$ mv /usr/lib/sasl2 /usr/lib/sasl2.orig
$ ln -s /usr/local/lib/sasl2 /usr/lib/sasl2

Note: if you have issue installing Cyrus SASL around compiling digestmd5.c, it’s because your compiler is too new. Read here to find out how to patch it.

Since I just switched out the old sasl2 lib without recompiling sendmail, I was concerned sendmail would poop during runtime. Luckily that didn’t happen. Dynamic lib rocks!

3. Generate SSL certificate

I made a directory called certs under /etc/mail. Here are the commands that I used to generate the SSL certificates.

openssl req -new -x509 -keyout cakey.pem -out cacert.pem -days 3650
openssl req -nodes -new -x509 -keyout sendmail.pem -out sendmail.pem -days 3650

Notice I made the certificates good for almost 10 years. I didn’t needed the cacert.pem to be exact.

I also copied /usr/share/ssl/ca-bundle.crt to /ect/mail/certs and included it in the sendmail configuration file. Other wise you’ll see some error like this:

unable to get local issuer certificate

The reason is that the ca bundle file has the Gmail certificate issuer. Although I read it somewhere that email still goes out with this error. Nonetheless, we don’t need to see this if we can fix it.

4. Configure sendmail

With the preparations above we are ready to configure sendmail. I found this tutorial very useful in terms of getting the correct sendmail configurations.

In summary, I have the /etc/mail/auth/client-info looks like this:

AuthInfo:smtp.gmail.com “U:root” “I:username@gmail.com” “P:password” “M:PLAIN”
AuthInfo:smtp.gmail.com:587 “U:root” “I:username@gmail.com” “P:password” “M:PLAIN”

If you use Gmail hosted email with your own domain name, you will have username@hostname.tld in there.

Make sure run:

$ makemap -r hash client-info.db < client-info

and chmod 600 on client info files.

Essential lines in my sendmail.mc

FEATURE(`authinfo’,`hash /etc/mail/auth/client-info.db’)dnl
define(`SMART_HOST’,`smtp.gmail.com’)dnl
define(`RELAY_MAILER_ARGS’, `TCP $h 587′)
define(`ESMTP_MAILER_ARGS’, `TCP $h 587′)

define(`CERT_DIR’, `/etc/mail/certs’)
define(`confCACERT_PATH’, `CERT_DIR’)
define(`confCACERT’, `CERT_DIR/ca-bundle.crt’)
define(`confCRL’, `CERT_DIR/ca-bundle.crt’)
define(`confSERVER_CERT’, `CERT_DIR/sendmail.pem’)
define(`confSERVER_KEY’, `CERT_DIR/sendmail.pem’)
define(`confCLIENT_CERT’, `CERT_DIR/sendmail.pem’)
define(`confCLIENT_KEY’, `CERT_DIR/sendmail.pem’)

define(`confAUTH_MECHANISMS’, `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN’)
TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN’)

NOTE: Be aware that smart-quotes used in the code examples will not be recognised if pasted into your files! Ensure replacing smart-quotes by regular quotes (see comments below for further detail). – thanks Johnny for the suggestion.

The certificate files are generated/copied from the previous step. I’m no sendmail expert so the configuration lines may not be perfect. But it works. Let me know if you have better settings.

One tip I found very useful is to use the debugging feature. You can set a high log level in the sendmail.mc to see at which step sendmail choked and for what reason.

Also, make sure run “make” or m4 every time you touch the sendmail.mc.

m4 sendmail.mc > sendmail.cf

So that’s pretty much it. I restart the sendmail service and out goes my email.

UPDATE:

Recently I installed Sendmail on a brand new VPS and had hard time get the authentication working. It turned out saslauthd was not running. So a note for new system is to make sure saslauthd has to be up and running (better use chkconfig to make sure it starts up at run level 3) in order to get Sendmail authentication working. This may help resolve some issues in the comments.

Depending on the type of the technology you use to build your web sites, there are different ways to put your code out there. Since a lot of sites today are developed in PHP, updating probably means upload a bunch of scripts to the server.

One way to do this is uploading the whole directory from the site root and switch the Apache site directory using symbolic link. But if you only have a handful of files updated it is really not necessary to upload everything every time. And often times there are user uploaded content and log files generated by the web server in the file system, which you don’t want to lose during the process.

Another way that I prefer is using the “sync” methodology. Basically I use a tool to compare the differences between my local development drive and the remote directory, and let the tool handle the remote copying and deleting.

“rsync” it is a great tool from *nix family which can sync two locations using ssh protocol. Although it is command line based you can always write a simple script to automate it. But if you are developing from a Windows PC it might be a bit difficult. I used to run rsync under Cygwin on XP and it does very well backing up files from the remote server. However because of the different ways of handling file permission under Cygwin and Windows I had a lot of problem to commit to rsync as a deployment tool.

For a long time I also used a tool called “Site Publisher” from helexis.com. It is a small FTP based tool which you can use to set up different site profiles and sync the code from your local drive to remote directory. It has worked very well for me but there are a couple of issues that made me to look for new ones:

• No sftp support
• Since I have firewall installed I have to use “Active” mode for FTP. But Site Publisher seems to have some problem with this. It would hang during a transfer session, which is not acceptable for a production release.

Recently I found “InstantSync(TM)” from sitedesigner.com and have been very happy about it. It support sftp so I can probably shutdown the FTP server on my host. It also supports multiple site profiles and the file transmission has been rock solid. Although it cost $99 I think it is definitely worth the money.

During the search process I also evaulated TurboFTP and SynchronEX which didn’t fit my needs. In my opinion, TurboFTP is primarily a FTP tool so there are a lot of features that are great for FTP but not necessarily in my case. I just want something simple to use and does this one thing good. SynchronEX looks promising but its user interface to set up site profile is hard for me to comprehend.

Disclaimer: The tools reviews mentioned above are solely based on personal experience and I have no affiliation with the companies mentioned.

Akismet is a great comment spam catching service provided by wordpress.com. I use an external PHP client to test the comment or contact form spam in my sites and it had been working great. Only recently, I noticed that sometimes I receive several same message from the contact form from the same user. Originally I thought it must have been some “annoying” users who sent multiple identical messages to get noticed, but when the number of cases increased, it became an issue to invest.

After some tryouts myself it turned out the submission of a contact form was painfully slow. The users must had been refreshing the page in frustration of the slow response. And the cause was the Akismet client, which took about 40-60 seconds to return. Naturally I checked the web to see if this had been an issue to this other people. Although the information was sparse, there are a couple of solutions, and both works.

1. Upgrde the Akismet client

I use PHP5 Akismet client by Alex, and sure enough there is a new version (0.4) that addresses this issue. A more detailed test by Jan De Poorter can be found here. So by changing HTTP version back to 1.0 fixed the issue.

2. Modify the HTTP request

But why the newer and better HTTP 1.1 had problem? One of the big improvement in HTTP/1.1 is the capability to handle the persistent connection. In a nutshell, a client can keep the TCP connection open in HTTP/1.1 and use the same connection for the subsequent requests to save the resource by not opening new connection each time. Sounds promising right? However, this makes the client to be responsible to close the connection since the server would not have any idea when it’s done. If a client does not wish to handle it, it must specify “Connection: close” in the http header and tells the server that this request SHOULD NOT be considered persistent after the current request/response is complete.

If you use PHP5 Akismet client and want to stick with HTTP/1.1, you can add this in the http_post function:

$http_request .= “Connection: Close\r\n”;

Which will also fix the performance problem.

Start from yesterday I noticed some changes with my iGoogle page.

iGoogle is my personalized home page that has all my gadgets on one page for quick access, for example, RSS reader, Gmail, Calendar, etc. One noticeable changes is that the “Home” tab is moved to the left of the screen, with a gadget menu drop down. From the new “Home menu”, I’ll be able to open the reader or Gmail on the same iGoogle page without having to deal with a new browser window (or tab).

It takes me a little getting used to at the beginning. The new menu on the left takes up some space and the home view is definitely squeezed in this design. Plus, some of the functionality changes the behavior so I had to adjust. After a little while I started to feel the convenience.

I guess this is another step toward a browser desktop from Google. The Home menu definitely reminds me of the Windows start menu or the dashboard on Mac OS. With more and more gadgets (application) released to the wild, iGoogle can become very useful if you like Google’s product.

Alias module in Apache http server provides an easy way to map a URL path to a local file directory. For example, the popular MySQL web access tool phpMyAdmin is often installed as a alias in the Apache web server. Usually a line of code like this will do the trick:

alias /phpMyAdmin “/my/directory/phpMyAdmin”

Recently I tried to setup a new server in the same way but kept getting 403 forbidden (You don’t have permission to access /phpMyAdmin on this server) message from the web server. Thinking that this must have been some trivial issue like a typo, it turned out to be a long troubleshooting process. Finally I figured out the couple of things that needed to be fixed:

First, the directory access permission needs to be set up correctly.

The default directory access permission is usually set pretty strict. It looks like this:

<Directory />

Options FollowSymLinks

AllowOverride None

Order deny,allow

Deny from all

</Directory>

It provides good security but also prevents the access to the others. So we’ll have to re-configure the directory that is mapped to an alias (Same should be done for the virtual server as well). So adding this into the httpd.conf will make it better:

<Directory “/my/directory/phpMyAdmin”>

Options All -Indexes

AllowOverride all

Order allow,deny

Allow from all

</Directory>

However, I was still getting the error: You don’t have permission to access /phpMyAdmin on this server, which was quite puzzling since I was pretty sure I was on the right track.

Now this time even the error message was the same it was a different cause. Since I disallowed file listing on the directory (-Indexes) Apache refused to server the URI …/phpMyAdmin. All I needed to do was to add index.php to the DirectoryIndex directive so Apache knows what to display when a directory name is requested. Of course …/phpMyAdmin/index.php would also work without the change.

A seemingly small problem caused me a lot of time to fix.

As a long time PC/Windows user I finally got my first Apple computer: a shiny white Macbook.

Macbook is not a complicated machine. It doesn’t overwhelm you with bells and whistles. But everything is there to work. The hardware design is truly the best.

The Mac OS X is a great OS. I didn’t have much learning curve to get familiar with it except to learn all the keyboard shortcuts since I want to stick with the touchpad without a mouse. I found out that you can put two fingers on the touchpad and scroll down a page, which is an awesome feature. One great thing about Mac OS X is that it is based on the Free BSD so I can always use a terminal to do the work. It is essentially a Unix machine with a nice GUI.

Mac app did crash once. My iTunes crashed and a message box popped up so I can end the program. However it didn’t bring down the whole machine and everything else was business as usual. There is another time that iTunes appeared to be in a runaway mode and I had to kill it manually. Seems that Apple has some work to do on the iTunes. But overall it synced with my iPod without any problems. The mac sleep mode also works flawlessly, which I always have problems with my Windows machine.

So here is my short review. I remember some famous people once said “simplicity is the ultimate sophistication”, and a Mac reminds me a lot of that.

We often check if a variable is defined or not in javascript before proceeding the next line of code. Usually the code somewhat looks like this:

var myUndefinedVar = 1;

if (myUndefinedVar) {

// do some stuff

}

Even the browser will throw out an Javascript error, the code works since there is no else statement or other intended logics beneath the if block in this case. Otherwise, we won’t be so lucky since the browser will stop executing the rest of the statements. A better way is using the typeof operator, which will not result in an browser error even the variable is undefined. Here is a code example:

if (typeof(myUndefinedVar) == ‘undefined’) {

// do this

} else {

// do that

}

Notice that, the following code will work whether the document element is there or not.

if (document.getElementById(‘blah’)) {

}

The reason is that document is always a valid object and getElementById will either return an object or a null value, which in either case, will not cause the browser to throw up.

Google released Chrome yesterday, a new web browser, currently available for Windows XP only.

It is very simple to install and It is very fast based on my experience. I haven’t tried many sites though. So there are probably sites that don’t play well with Chrome. Chrome will not have a bookmark if you don’t import one from Firefox or IE. And also, I can’t use a drop down to see the sites I used to visit from the address bar (or so called omnibox in Chrome), which I think is a big inconvenience. However, upon opening a new tab, a bunch of sites that I have visited will be there to provide some kind shortcuts to the sites that I used to visit.

It is probably still early to say how dominant (or how irrelevant) Chrome will be in the future but it will certainly change the way people access the Internet. I am already using Chrome for my GMail, Google Reader and some other Google apps with Firefox opened for other web browsing and development work. When there are more and more browser based apps available, Chrome may be the perfect choice to run those heavy JavaScript sites.

I think besides the fresh look and some usability innovations of this new browser, the key is its open source root. Looks like the desktop app itself is on a good start, now the next thing is to extend it into other territories like mobile and smart devices, which is quite an open field to get into.

In the Desktop environment I think it is still too early to tell how long it will take to be adopted by the mainstream, considering 20-25% of the web users are still using IE6 today. Google might have to bundle Chrome with some really great browser apps to get it replace IE on people’s Windows desktop. In the meantime Microsoft will keep improving IE so it is still a long way to go for Chrome. But I think Chrome is a real threat to the browsers like Opera today and if when the development of 3rd party add-ons starts to catch on for Chrome, it will most definitely cut into Firefox’s market share.