Phine Solutions web work notes

My MySQL replication slave stopped replicating the master. Checked the log file it has this error:

Error reading packet from server: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master (server_errno=1236)

Got fatal error 1236: ‘log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master’ from master when reading data from binary log

Naturally I started increasing the max_allowed_packet in my.cnf. No help. So I looked up the web and found this bug documented. Bingo I thought…oh wait, the bug was discovered a few years back and my version of MySQL should have included the fix already. Ok back to square one.

Just to poke around I ran “show master status” on the master server and compare bin file name and position with what’s in the logs on slave server. The files do not match. It may or may not cause the error 1236 but it’s definitely not right. So I went ahead ran the following command to re-sync:

On the slave:

mysql -u root -pxxxxx

SLAVE STOP

LOAD DATA FROM MASTER

CHANGE MASTER TO MASTER_LOG_FILE=’the-right-bin.xxx’, MASTER_LOG_POS=the-right-position;

SLAVE START

Now step two to load data from mater created a lot of error logs since my slave DB had had data so I really should have clean up the DB before run the commands.

So the replication is back to work now. Not sure what had really caused error 1236. Get my fingers crossed.

CVS has always been my choice of source control tool. It does what is does and I have always been happy about it. However recently I moved my source code repository from an online server to a home machine, and had an opportunity to look into Subversion, the newer source control system that has been adopted by more and more developers.

I consider Subversion has these major advantages over CVS:

• Since it is the new kid on the block there are more interests to maintain and develop it, and also there are a lot more 3rd party client tools.
• Completely designed to be suitable for all file types. CVS was originally designed for text files only so there are some limitations or hurdles when you store binary or unicode files.
• Renaming a file or directory is as easy as just “renaming” it. In CVS, you have to delete, add with new name and commit. For a developer like me who can’t make up my mind when naming files and directories, this one is a keeper.
• More ways to access repository. From native svn server to ssh and http(s), subversion has more ways to connect to, and therefore, more ways to bypass a firewall that blocks certain ports.
• More clients available. Have I mentioned this in the first point? During my years of using CVS I only managed to find two legit GUI client: TortoiseCVS and WinCVS, other than the command line and IDE plugin. Subversion has a lot more in this department.
• Some other pros like atomic commit and directory versioning are also good but kind of trivial for me.

Now one biggest “con”, considered by some, is the absence of “tagging” capability in Subversion. CVS lets you “tag” a label on a file, so you can create a “snapshot” on a repository. If  you use tag to mark the latest production release (and move the tag accordingly) in CVS you’ll miss this functionality. But in my opinion this is just one way to use source control and this shouldn’t be considered a disadvantage in Subversion. In this example, a production release branch can always be created with the release number, which I consider a cleaner way than using a tag. Ultimately everyone has a different way to use a source/version control tool and I think it is a good design point for Subversion to abandon tag. Although I have used “tagging” before in CVS it is not a heart breaker for me to lose it.

There is one thing I do miss is that not being able to hack into local working copy and change repository locations. I used to write a script which search for CVS/Root and CVS/Repository folder and modify the repository name and server/path when I move my CVS repository from one server to another. Now it seems I have to do a fresh checkout in similar situation in Subversion. Which is probably not a bid deal.

All in all I’m pretty happy with the move. My Subversion client is TortoiseSVN, using cygwin ssh to connect to repository through svn+ssh.

During the research I also found Git and plenty of Git vs. Subversion comparison discussion. From what I understand Git facilitates a heavy local work copy so you don’t have to connect to a central repository server, a so-called “distributed source control system”. I can see this feature is extremely useful for projects with a lot of developer and active branches, because the collaborations can happen among the “local copies”.

Any tips on using Subversion? Please share them here. As a new Subversion converter anything will be helpful.

Amazon has released its Relational Database service today. It has the full capability of MySQL and the service lives in the Cloud, as its other services like S3 or SQS, etc do.

Also similar to other services, it is charged by usage.

Why it is a great news? A couple of things came to my mind when I heard it.

RDS can be your single DB server. If you are a web developer who manages a small/medium site on shared hosting server you won’t have the luxury to build a standalone DB server and you’ll likely end up with Apache and MySQL running on the same box, and you know it is a pain in the butt when moving a site from one host to another (transferring files, re-establish DB, and probably making code changes to point to the new DB). With RDS, you can simply create your DB instance in the cloud and skip moving DB when you need to move your hosting service. This may sound only benefit your hosting company since one less use is competing for the resource, but you definitely get the advantage of flexibility and probably some increment on reliability.

However, moving DB to cloud will increase network latency, and as a developer I have my share of concern on the Cloud-based services. It’s out there and I have very little control of. We have had a few times of S3 going offline in the past and it’s not fun for those services or sites whose architectural diagram live inside the cloud service itself. So for a large site owner with a “mission critical” database, I would recommend using RDS to cut down the cost of setting up DB mirrors. For instance, we often using a second MySQL DB instance to sync with the master; and RDS can replace the slave in this case. It’ll be useful for failover, backup and real time restore, and a little bit of downtime is probably okay. The biggest benefit is that we can save a lot by eliminating the cost of server, software management and system admin work.

It’s certainly a bad news for service like FathomDB, which is a cloud relational DB service built based on Amazon’s EC2. This probably makes a good argument against making a long term commitment on building services which rely on platforms that in other people’s control. When a nice Facebook app get popular, who’s to stop Facebook to build the similar one for their own as the market has been proven? Same goes to iPhone apps. Make some quick bucks and move on is the way to go.

Any other ideas of using RDS? Let me know. Cloud is certainly making things cheaper for us web builders.

We moved into our new home a year ago and back then the builder asked me about the network wiring of the house. Basically the home will be wired with cat5 cables for phone service. If I want a network (data) jack additionally, more cables will be added – and of course the cost. Buying a new home was financially demanding so I went on the cheap side of me and opted for voice only.

I have been using wireless network since then and I was generally pretty happy about it. However I’m moving my office to the basement and if the router follows me the wireless access will be very spotty from the second floor. Adding more access point is not a good option in my opinion so I decided to re-configure the wiring so I can have the wired data access in rooms that I want.

One big pre-condition for this setup is that I DON”T have a landline and don’t intend to get one. Like 20% of American household (and growing) we only use cellphones. I do have a SIP phone (basically a VOIP solution) and it needs a data port. Currently it is connected to my wireless router through a VOIP adapter. My VOIP phone is another good reason to have more data ports around the house. I have a cordless phone set which only needs one line to the base station (all the other handsets are cord free).

So I made the decision to go ahead re-wire cat5 to support data only.

One alternative – if you need voice jack – is to split the cat5 cable to support 2 jacks, one for voice and the other for data. Obviously that requires more work. Since this is not my configuration I’ll briefly mention it at the end of the post.

What is cat 5 cable

Here we need to have a little bit of explanation on cat5. “Category 5″ is a cable specification.

The cable looks familiar? Yep, you see it everyday if you look under the desk, around your ethernet switches.

A cat 5 cable includes 8 wires, every two of which are twisted together, known as twisted pairs. The wires are color coded, they are: orange, white orange, green, white green, blue, white blue, brown and white brown.

These wires carry voice or data signal. As a matter of fact, only 2 pairs (4 wires) are needed to support a 10/100 BASE-T network; only 1 pair is needed for a voice line. That’s why it’s possible to split one cat 5 for both data and voice. Again, in my case, I’m going all data.

Planning

A good planning always pays. Here is a list of things you need to check and prepare before start cutting wires.

• You house is wired with cat 5 or cat 5e. Usually this is done in newer house and it can be confirmed by checking the outlet. It should say cat5 on it. If this is not the case, stop reading and go visit reddit.com.
• You have “star” wiring. Meaning every single wire branches out from the central panel. If they are in a loop, you can stop now and go watch a movie.

• You can access your switch board and it is either inside the house or in your basement. Because you’ll put cable modem and network switch there it needs to be securely located.

• Equipments, tools and parts.

1. The cat 5 jacks. I bought 10 for $29 and didn’t even need one at the end.

2. The Leviton data board.

3. A bunch of ethernet cables.

4. A wired router

5. Some tools

• Internet downtime.

The internet will be broken for a few hours and you’ll need to let the wife know.

Let’s roll

1. Re-wire the patch panels

Note: I chose to do this first – instead of the jack – was because there is a chance the wiring of the jacks will work without any changes.

The following photo shows the patch board where all the cat 5 cables are originally terminated. Notice the third from the left is the new data board that I added. The builder’s contractor did a great job labeling the cable so I didn’t have any problem to locate the one that I want to rewire.

So I basically cut of the wires and punch them in to the new data panel (3rd one on the left). Remember the color codes for those twisted pairs? Now it’s the time to use them. Below is my color code mapping.

Although the color codes seem to be a straight one to one mapping there is somewhat a standard to follow. And it matters because both ends of the cable need to follow the same standard to get the data juice flowing. Just remember T-568A is the standard we are going by here. You may or may not need to rewire the jack to T-568A, depending on what your builder used originally. To learn more about the standards you can visit this page.

The Leviton patch board has instruction on the color mapping on each punch slot so it was super easy to find the correct punch slot. It’s the first time that I work on these punchdowns and I have to say it was pretty easy to get a hang of it. Just keep a steady hand and enjoy that little clicks when you push the wire in. The plastic tool that comes with the patch panel was good enough for me and I don’t think you need to invest a $30 punch tool unless you are planning to do this to make a living. Again, I’m cheap.

This photo shows the two wires are terminated into data panel. The cables coming out of the black ports go into the router.

2. Connect the patch panel to a router

Now is the time to notify your family that the internet will be down. I took the cable modem and connect it directly to cable company’s video feed. Did I mention I don’t have cable TV? If you do, you probably need a splitter. Then, connect the cable modem to a wired Linksys router (I left the wireless router upstairs and it will be my access point). Now it’s also a good time to configure/check the router if it’s a new one. Once you verified it is online, connect it to your patch board and make sure you use the right port where the wires were terminated.

3. Check and re-wire the jack, if needed.

If you are as lucky as me your jack is a data port already. If not, it is possible it’s not wired using T568A standard so you’ll need to rewire it that way. The cat 5 jack should have a label on it that clearly indicates which color goes where for A or B standard so it should be pretty easy to punch and attach. Here is the color code mapping:

As I mentioned I didn’t even need to rewire the jack but I did one anyway. Here is a picture:

At this point you should have one room wired. So plug in a your computer you should have the internet. If not, check the wires, connections.

4. Take a break and finish up

Now it’s a good time to take a break and give yourself a pad on the back. If you have more outlets to convert, repeat the same steps. Once you are done, tidy up the wires and outlets.

The old wireless router can now be used as a simple access point. Just make sure to re-configure it to a different static IP (192.168.1.2 for example) and turn off DHCP (The wired router is doing that now). Connect one of the LAN port to a network switch or a data jack you are done.

Voice + Data over one cat 5

As I mentioned I don’t use this kind of setup but since I did some research on it I might as well write it down.

Remember there are 4 pairs of wires in a cat 5 cable? We just used 2 pair. The 2 pairs left can be used for phone lines, and as matter of fact, two phone lines if you wish.

1. Make a phone jack

You can use a cheaper voice jack for this one. Here is the color code mapping:

2. Terminate phone jacks

Since I don’t have a land line of my own so I don’t get a chance to practice this. Basically one twisted pair can support one voice line so you can play with it and see what’s the best fit for you.

One suggestion is that you should use a regular voice jack, which is smaller than cat 5 jack so you can’t accidentally plug in data cable in a voice jack. Since there is always a chance the wires are not done correctly, the high voltage generated from the phone rings can toast your network adapter if the cable is plugged into the wrong jack.

Some think there will be data transfer degradation when the voice line is in use. Based on what I read and talking to friend who actually did this setup, it seems more like just a theory.

Document what you did

Once you get it working please make sure finishing the last mile before start surfing the web: documenting what you did. Label the wires, mark the jacks. Or write a blog post, maybe.

So that’s how I rewired my home to version 2.

As landline becomes less and less popular these days I really think this is a setup that may become more and more popular. As we are going to see more and more “network intelligent” electronic devices in homes in the future. Devices to stream internet video to TV is a good example and is already making the front run. With more bandwidth hungry devices in your home wireless access point can become a bottleneck. In that case, the more data port, the merrier. Of course if you are building a new home and have the chance and means, you can pimp the wiring up by doubling cat 5 or even adding cat 6. But for most of the existing homes, it’s not that easy to drag wires behind the walls and this might be a more feasible solution.

Google launched Fast Flip yesterday. It’s an image browser for the snapshot images taken from web pages (they are categorized of course). It is very quick and it’s supposed to make online news reading fast and furious.

On the revenue side, there will be ads, and there will be revenue sharing with the publishers who want to flip with Google.

Google takes the usual product launching route. Put it in “lab”, collect user feedback and go from there.

If I’m a big time publisher like New York Times I would develop my own version of “Fast Flip”, NOW.

This is the exact kind of innovation that the publishers should take. You have the content, use it; you have the original digital copy, you can do better job converting them to images; the technology and development? a small team should be able to crank it out in a few months.

Pre-ordered Snow Leopard and received it on Friday (the official release day) afternoon via Fedex. For a total order of $29 this is some treatment.

The upgrade was smooth for me. Just pop in the DVD and let it run the course. There are some nice interface tweaks, and not to mention the saved disk space. But a few enhancements that I like the best are:

• Access over local network are much much faster. I have a shared drive in the LAN and it used to be a pain to view large images and QuickTime movie files over the network. After the upgrade, the delay is gone and the video plays smoothly.
• The new QuickTime Player has the video, audio and screen recording built in now. Since my MacBook has the camera and microphone built in, making an video is as conveniently as making a few clicks and start talking.
• The Microsoft Exchange support. Works very well. Enough said.

I’m no Apple fanboy but sometimes I love using Apple product because it provides an “environment”. An environment that has a nice mixture and balance of great design and usability. For us web builders, this is both an influence and inspiration. It helps us, in a subtle way, to make product with good taste and also works.

So the new Yahoo home page has been rolled out for a few weeks. For the most part, I like the new home page. It provides more information with some seemingly moderate changes. However, there are a couple of design points that I think can be made better.

The first is the big flyout from the “MY FAVORITES” menu. Basically when you move your mouse over an item, a flyout will pop out. I know it has its own merit, and the feature gives visitors shortcuts under the main category. But a big flyout box that fills out most part of the web page can be quite annoying, especially sometimes it was triggered by an accidental hover over the links. Any most of time I just want to click on the menu and go to my favorite subject – with this flyout action, I have to deal with some “delay” before getting where I want to go.

My suggestion: make the flyout smaller, simpler and more importantly, to load quicker.

Same place, “MY FAVORITES” list can get longer and there is not enough slots to display all of them,  to find the rest of the favorites, I have to click on a little “>” button to sort of go to the second page of the favorites.

Why not make this one a flyout then? Definitely the right feature for the right place.

Update:

Looks like Yahoo just updated their page design. Now when you hover over the left menu, a much smaller flyout will show, which you can click on to open the quick view. Much better design than kicking open the annoying page view blocker directly.

Uploading files through a web page is supported in HTML, but with some serious limitations. Usually we set up a Multi-part html form which has an input field that automatically comes with a “browse” button for end users to select a file from local file system to upload, but only one at a time. During which the page will pretty much hang until the web server complete the uploading process and returns the next page. Certainly you can provide multiple input fields to add a little bit of convenience, but there is really no way to upload multiple files at once using just html, and it is quite a tedious process if there are a lot of photos to upload.

Recently I wanted to make uploading photos easier one my site. I want my user to be able to upload any number of photos all at once, and be able to view the uploading progress as well. Since the basic html does not provide this kind of capability and nor does the client side script like JavaScript, “plugins” have to be used. I basically have two options to use: Java Applet or Adobe Flash. Microsoft Silverlight is another tool which can create rich client on the browser, but since it is still a relatively new player in the field I didn’t put a lot of considerations in it.

Two real life examples are Flickr’s photo uploading tool and Facebook’s photo application. Flickr uses Flash and Facebook uses Java Applet respectively. Both of the uploaders are integrated nicely to match the style and flavor of their sites and most importantly are easy to use. In a quick comparison, Flickr’s uploader is light and straightforward. You can select multiple files and see the actual progress during uploading. But the Facebook counterpart lets you to view your computer’s file system from the tool so making file selection is easier. Plus it has some simple photo manipulation functionality like rotating the photos in place.

Flickr’s photo uploader

Facebook’s Photo Uploader

To choose which technology to use we have to look at pros and cons, which probably varies in individual situations. I’ll try to provide some key areas that I found useful to take into consideration when making the decision. Although I ended up choosing the Applet, the decision was largely based on the fact that I’m more familiar with Java and there is really no time for me to pick up Flash and go from there. For some, Flash might be a better choice.

Browser compatibility Plugin technology

Almost all the modern browsers today support Java and Flash. And there is no need to write different versions of application to deal with the browser issues. However, due to the fact that Java and Flash themselves are constantly being upgraded it is necessary to watch the code and make sure it is supported by different Java or Flash versions out there. I think when Adobe released Flash 10 there was some security implementation that broke a lot of Flash app. Sun has been very good on making the newer version of Java backward compatible. Although the newest Java technology is at version 1.6, it is backward compatible to 1.4 and 1.5, which pretty much covers all Java versions out there on user’s machines.

Both Java Applet and Flash requires the plugins to be installed so browsers can display them correctly. Sometimes they come with the browser installation by default. I don’t have any data but it seems both Java plugin and Adobe Flash plugin have good penetration on the popular browsers today (Firefox, IE etc.).

The slowness when loading Applet, at least the very first time.

Applet is slow to load because the browser has to start Java JVM first then start the Applet. A user’s machine may be hung for a few seconds to get over this. So if an applet is a beast and a user has a slow internet connection it won’t look to well. I would say this was a more serious problem a couple of years ago than now. Sun has been improving the Java browser plugin technology and the loading time has been much better with the modern browsers today. I noticed a significant difference between Firefox 3.5 and IE6 on the same machine, with FF beating IE hands down.

Flash, on the other hand feels much lighter, probably due to the fact that Flash is a smaller tool. But the smaller size also comes with limitations in terms of the application capabilities.

Nonetheless I’ll give Flash one point here.

Application capability

Since Applet is written in Java language it pretty much has everything that Java provides — minus those restricted by the security constraints of course. The rich GUI interface and user interactions and programming APIs are extensive. So it definitely has more capabilities than Flash can provide. Using the file uploader as an example, with Java I can develop GUI interface that user can only view and select multiple files from harddrive, but also drag and drop files to a certain destination, which is a pretty regular task on a desktop. With the rich image libraries available Applet can also have a lot more “pre-processing” power like reducing image size, rotating image to a certain orientation and converting images or adding watermarks, etc. Although it is debatable whether whether we should put this kind of the processing functionality on the client side, because it will most certainly make the client heavy to load and expensive to run, the capability is there. Choosing wisely, they are great additions to your web site.

Since Flash is lacking the capability on interacting with desktop things listed above are impossible to be implemented. So I give Applet a point here.

Code development and IDE

Java is a mature programming language and the development support reflects this. Learning the language aside, it is quite easy to set up a development environment and start coding. Flash on the other hand, has a smaller developer base and there is less information on the web that I can use to pick up this tool quickly. I might have a little bias here since I’m more familiar with Java. Flash to me, is a lot like a designer’s tools.

So I give my bias point to Java.

Available products

There are products already developed in both camps. Some of them are free and some of them require a fee. Nothing can beat free, but you get what you paid for. Especially for web apps, a nice demo is far from production. The paid product usually goes through better testing by the developer than Joe’s little free app.

I listed the ones that I found with good quality and support tracking record below.

- Applet

The free jUploader

The jUploader is developed by William JinHua Kwong and it can be downloaded from SourceForge. It seems to be actively maintained. The jar file for basic version is about 370k in size. I’m sure it can be reduced with Java bytecode obfuscator and shrinker software.

The app itself is quite good and documentation is sufficient. However the GUI interface is not as polished as the next candidate: JumpLoader.

JumpLoader – can be free, but will cost if you want the source

JumpLoader is developed by jmaster and it is the best I have seen so far. Depending on your requirement you can have a just simple file uploader or quite sophisticated pre-uploading image processor, you can see the demos here. The app itself it very customizable from the functionality to user interface. But it comes with a price. The “small” version is over 500k in size, which is still too large in my opinion.

- Flash

SWFUpload

This is the free Flash uploader that I like. And it looks similar to the one in Flickr. If you have the Flash development and don’t mind putting some work you might want to seriously consider this one.

Element-IT  MultiPowUpload

I didn’t download and try the product out myself. But based on the demo I like the features that MultiPowUpload provides. Notice that with the base code was under 100k in size. With some plugin, the Flash code is still in the 100k to 200k range.

Like I mentioned I generally prefer the paid version unless the software has enough open source developer support.

As web evolves the difference between a browser and a desktop application is less and less noticeable. Hopefully you find this post helpful to make a decision if you are in a similar situation. Also feel free to make a comment if I missed something here.

Bing is the new search engine from Microsoft, although they call it “Decision Engine” we all know it is just a branding trick. I used it a little bit and I was pleasantly surprised.

The feature I really like is the preview feature. For example if you search for “PHP tutorial” you get this page:

http://www.bing.com/search?q=php+tutorial&go=&form=QBRE

A page from php.net is listed as the first entry, which is in my opinion, beats the result from Yahoo. What’s more, you can hover your mouse on each entry of the result page, a little link will appear on the side and you can use your mouse to activate it to show a preview of the page. I tested several queries and the previews are very accurate. The preview also shows other links on the page that are relevant to the subject, which help to decide whether this page is “click through worthy”.

I don’t know if Bing can dethrone Google on web search, and as long as Google continues delivering the relevant results it is hard for anyone to make a legit threat on its dominance. But MS seems to have some nice things going in Bing just off the bat. Since Google relies on its search engine for most if not all its revenue, MS has an advantage here to make some bolder move, and be more innovative, like a real decision engine that was just released: hunch.com.

After watching some Google IO keynote videos I got interested in Google Web Toolkit and played with it a little bit. It is quite an interesting tool. Basically the tool helps Java developers to develop JavaScript using the familiar Java programming language on familiar development IDE such as Eclipse. And probably more importantly, it takes care of the nittygritty details such as browser compatibility and code optimization. So one can write it once and be pretty comfortable about running the code on different type of browsers.

Unlike other Java to JavaScript frameworks (jMaki for example) that require actually to be hosted in a Java VM container, GWT produces the client side JavaScript code as a final product. So Java, in this case, is just a tool, not part of the end result. Of course, if there is server in the backend to handle the RPC calls,  Java Servlets can be used to facilitate that, but it is not mandatory.

GWT is not just a tool for Java developers who don’t like coding JavaScript. It also put AJAX into better perspective by separating the client and service code in the code base. Using JSON as the messaging protocol between the client and server, a GWT client application can talk to any web server that speaks the language, which doesn’t have to be implemented in Java. GWT also provides a way to work around SOP – the Same Origin Policy.

SOP is the security policy that is in the web browser to stop a client script to communicate with web server that the script is not originally from, in another word, any arbitrary web server. This is a necessary security measurement, but it makes the client less distributive. For example, a web service provider would like to distribute a small version of client script, which can be embedded in any web sites, and get the dynamic information from the service host. GWT handles this by requiring server to return JSON output in a “<script>” tag, and a callback function will also need to be defined to handle the output.

With the advance of web technology, JavaScript will play much bigger part in rich web applications in the future. I think the biggest strength of GWT is its capability to scale up the size of a JavaScript project. Following the Java’s footprint a project created using GWT can be better managed, tested and collaborated among developers.

One drawback is that one has to be familiar with Java. And obviously there is a little bit of learning curve on those GWT GUI APIs. Nonetheless it is a great tool for Java developer to natually adopt and use to create some great JavaScript applications.