Romen's Techno-Babble

A Simple Property Viewer with Reflection and ZK

2012-01-04T16:44:00.001+11:00

Sometimes I need to display a POJO’s properties/attributes. These POJOs may contain dozens of properties making it tedious to invoke all the getXxx() methods. So Java Reflection is a perfect candidate for such a use case.
In the following example, the LoginResponse class contains dozens of fields that I want to display. The formatMethodName() method beautifies the getXxx() method names into more human user friendly names.

import java.lang.reflect.Method;
import com.laws.acc.ws.*;
import com.laws.acc.model.*;

LoginResponse loginResponse=AccSoapClientProxy.getLoginResponse();
void populateRows() {  
	if(loginResponse!=null) {
		Method[] methods = LoginResponse.class.getDeclaredMethods();
		for(int i=0; i<methods.length; i++) {
			
			
			Method method=methods[i];
			//Object arglist[]={};
			if(method.getName().startsWith("get")) {
				Row row=new Row();
				Label label=new Label(formatMethodName(method.getName()));
				label.setStyle("font-style: italic;");
				label.setParent(row);
				new Label(method.invoke(loginResponse, null).toString()).setParent(row);
				row.setParent(rows);
			}
		}
	}
}
/**
 * turn strings such as 'getUserName' into 'User Name'
 */
private String formatMethodName(String name) {
	// the method name should start with 'get' followed by the property name
	// otherwise, ignore the method.
	if(name==null || name.length()<4) return null;
	StringBuffer sb=new StringBuffer();
	
	// skip the first 3 characters, they are 'get'
	for(int i=3; i<name.length(); i++) {
		char ch = name.charAt(i);
		if(i>3 && ch>='A' && ch<='Z') {
			sb.append(" ");
		}
		sb.append(ch);
	}
	return sb.toString().replace("Id", "ID").replace("Dt", "Timestamp").replace("Lre", "LRE").replace("Mvno", "MVNO");
	
}

The GUI component I am using is a ZK Grid with each row showing a property value.



	
// put the above java code here.

	


	
	
		
		
		populateRows();

Server Error 500 on Saturday Lotto!

2012-01-04T16:29:00.001+11:00

I tried to check my New Year’s eve NSW Saturday Lotto results on their official website, and one group of my numbers consistently resulted in server error 500.

Try this: http://www.nswlotteries.com.au/cgi-bin/results.pl?numlookup=1&game=satlotto&draw=3177&num1=4&num2=37&num3=1&num4=2&num5=3&num6=6

It turns out that as long as the group of numbers only match the supplementary numbers and nothing else, then it will result in errors. In the above example, the actual result of the Lotto draw was: 7, 8, 28, 33, 38, 41 with supplementary numbers 4 and 37.

On RSA Hack …

2011-10-30T11:55:00.001+11:00

Recently our IT system admin surveyed the company about some changes on how we would use the RSA SecureID. As I was totally oblivious about the RSA hacking incident, I did not pay much attention to the survey. Then earlier this month when I was doing a product demo at a client site, I found that my company VPN login stopped working half way during the demo, and it remained so for the next couple of days until I rang up IT support and got it reset. Obviously, as a precautionary action, the sys admin had changed the RSA RADIUS policy …

Although the RSA hacking incident and the related APT attacks did not affect my company, it is enough to get the IT department worried. In the Open Letter to RSA Customers, RSA called the attack ‘extremely sophisticated’. However, knowing that it comes from the Chairman of RSA, you have to take it with a grain of salt. In the blog Anatomy of an Attack by Uri Rivner from RSA, we gain some insight into what had happened at a very high level. I find nothing new in the method of the attack though. However, there are a couple of points that bring a smirk or two.

First is the use of reverse-connect mode of the backdoor software to circumvent the firewall at the remote end. This reminds me of the bad/good old days before the proliferation of the internet. My job was to develop IVRs for customers across many industries. To maintain the system, we would need to connect to them remotely. In those days, the connection method of choice was via good old PSTN line modems, which are plugged into the IVR system. The ‘normal’ mode of connection would be to dial up to the modem and login (using the UNIX cu command). However, to save on long-distance call cost, we usually buried hidden option in the IVR menu. Upon entering a code, the IVR would invoke some shell script to get the remote modem to dial back to the office modem and give a login prompt upon connection – using the UNIX ct command. This use of reverse connection is totally legit, by the way.

The second and more obvious point is the use of social engineering as the starting point. We have seen from spy movies time and time again that human is the weakest link in the line of defence. This social engineering is made extremely easy and cheap today, thanks to the massive adoption of social network services. This is part of the reason I doubt the words of ‘extremely sophisticated’ coming from RSA Chairman. It is extremely easy to find out someone’s personal details from their social network web pages, and find a list of friends that they trust. This makes it extremely easy to spoof an email with backdoor attached and have a high probability of it getting viewed and opened. Once the backdoor is in, all bets are off. You don’t have to be ‘extremely sophisticated’ to snoop around and gain more data – e.g. most people use their mailbox as a file repository, when someone has access to the Outlook PST file, then they can see a history of everything. This is partly the reason I never use thick mail clients and use the webmail instead – after all, it is the era of the cloud man!

Another interesting statistics is by looking at Who Else Was Hit by the RSA Attackers. Apparently, similar attacks have happened to hundreds of other companies recently and close to 90% of the attacks came from Beijing. No wonder people conclude that the attacks were state sponsored! China is a country where censorship is absolutely in every medium. The mass majority of the netizens are subjected to the GFW of China, which dynamically blocks destinations based on myriad of rules, including a government issued list of sensitive keywords. So in a country where vast number of web addresses are inaccessible to the normal citizens (including many social websites, and even this blogger site), only the privileged and well resourced can carry out such ‘extremely sophisticated’ attacks.

Oracle vs. Google Lawsuit

2011-09-16T14:38:00.002+10:00

I have read news about Oracle’s lawsuit against Google regarding its Android platform, alleging that Google is copying from Java. Some had even concluded that Android is in jeopardy. I had not paid much attention to the details of the lawsuit, until today when I came across Google’s rebuttal by a technical expert Prof. Owen Astrachan of Duke University. The report is published on Groklaw - it’s really fun to read.

The central issue of the lawsuit seems to be whether the Java API (or API of any language, library, even C-header files) are copyrightable. Google’s argument is that APIs are not creative expression, hence not copyrightable; but the implementation of the APIs are. Prof. Owen gave strong arguments backed by facts and through analogies.

Surely, anyone who has done software programming would not consider interfaces copyrightable. What do you think? – click your preference in the poll on the upper right hand corner of this page to express your view.

Updated 2011-10-30: the verdict is out. Total votes: 6; Yes vote: 5 (83%); No vote: 1 (16%).

The Power of Openness

2011-07-31T11:22:00.001+10:00

I was going to name this blog “why Samsung TVs are much better than Sony’s”. However, it may sound out of place in a technical blog focusing on computing and telco technologies.

As I have blogged recently about the issues of openness vs. closeness – e.g. Android vs. iOS, the same issue manifests itself in the broader consumer world, such as flat screen TVs.

With Australian government’s decision of moving from analogue TV to digital, the new flat screen TVs are selling like hot cakes. The available choices are dazzling – each boasts their own technical advantages in terms of picture quality, energy saving, sound quality etc. Your average consumers cannot care about all the technical mumbo jumbos, all they want is a TV! But is it?

As the display technology advances and matures, all the major manufacturers from all the major countries can produce decent display units/screens. So there is no point trying to analyse which technology is better in this regard – at the end of the day, what ever looks and sounds pleasing to you should win. So do those manufactures differ in any way? Yes they do – they differ in how open they are in terms of embracing different technologies.

Nowadays, it is common to record and view your favourite HD TV programs and movies from USB memory sticks or hard disks. A HD video can take any where from 1GB to 10GB of space. As we know, the major file systems supported by the TV manufactures are FAT/FAT32 and NTFS. With FAT32, there is an upper limit on the file size, being 4GB-1Byte. This basically rules out FAT or FAT32 as a valid format for your USB storage. This is where it matters – manufactures such as Sony and Panasonic do not support anything other than FAT/FAT32 on their TV and BluRay players (or USB 1 devices); whereas Samsung and most Chinese brands do.

There is good reason for Sony to be restrictive. It owns media companies too – Sony Pictures. It’s in their interest to stop you from recording or backing up your movies into one place to make things more convenient. So it is the same theme being played out in the computing industry for decades. Consumers can cast their vote with their hard-earned currency – the classical supply-demand interplay.

Getting Cloudy

2011-07-22T11:21:00.001+10:00

Last week I attended the Amazon Web Services APAC Tour in Sydney. It was quite helpful to see the AWS in action and their customer testimonials in real life.

There is no doubt that the cloud environment can help many IT departments and internet-oriented businesses. The main benefits are cost reduction and improved scalability and reliability. Unlike many IT fads in previous years, there is real momentum and benefits of adopting the cloud approach for many businesses and government organisations alike.

When it comes to telco BSS and OSS applications, the use case is quite different from many other industries. At the bottom level, many BSS and OSS applications need to be connected to the network in real-time. This can be real-time charging, service control for BSS and real-time network performance management for OSS. These real-time interfaces are usually dedicated links using telco standards (e.g. SS7 based), which conventional cloud providers do not support. So you will have no choice but to host these applications locally.

For the non-real-time (i.e. offline) telco applications, many of them need to access the vast repository of usage records, transaction records and network events. A telco may choose to host only the front-office web applications in the cloud and have them retrieve the necessary data stored locally over the internet; or take advantage of the cloud storage and ship all the transaction data to the cloud over the internet in a batch mode frequently. Either way, the requirement on the WAN link between the telco and the cloud data centers is very high. Let’s assume 1 million subscribers with 0.5BHCA and the CDR size at 750 bytes (note that CDRs from the 3GPP Bi interface may be much larger). This produces 375MB of CDR data a day. When it comes to OSS network management, the alarm repository can easily go into gigabytes and terabytes.

There may be a compelling business case for small telcos who do not have large amount of data and a slow growth model. Still, reliable WAN link with high bandwidth is required. These links may not be cheap and are charged on an ongoing basis – unlike the local computer servers which are one-off expenditure.

On the contrary, large telcos do not have problem with network links because in many cases, they own the network. Yet their data volume can also be orders of magnitude higher than the small ones. If you are not physically located in the same country or continent with their cloud provider’s data centers, then having a reliable WAN link becomes even more expensive if it’s possible at all – because the submarine cables are owned by consortiums, not any single telcos.

Also, beware of the pricing model of the cloud providers. They charge for each virtual machine instance, each I/O, storage, etc. The other day, I played with AWS EC2 by creating a supposedly free Linux based micro-instance, and snooping around by changing directories and doing ‘ls’, etc. before terminating the instance. To my surprise, there were total of 13,473 I/Os, which I could not account for. I got charged for a total of 5 cents for less than an hour of activities (most idle any way). So, take a close look at the pricing model of the cloud provider, it may turn out to be much cheaper to buy your own Windows servers to host your Outlook than doing it in the cloud.

So a middle ground solution is for a telco to host their own cloud environment and use this for their own BSS/OSS systems as well as providing cloud services to their customers. Quite a few telcos have already taken this route. Of course, hosting a cloud is expensive in terms of CAPEX and OPEX. There must be a good business case to do so.

Related Articles:

Apple Development

2011-06-22T10:14:00.001+10:00

When it comes to software development, I am absolutely an omnivore. I embrace any platform, computing languages and frameworks. I use them to develop my pet projects to form my basis of evaluating the technology.

However, I had never touched any Apple technology until last month when my wife twisted my arm to buy an iPhone 4. Not because I dislike Apple’s philosophy and its snobbish ways, but because it was so irrelevant for me. But after I bought the iPhone 4 I got the urge of developing on the iOS and the iPhone so that my $1000+ investment on the iPhone was not a total waste.

So I happily registered to be an Apple Developer. Then I needed to get the SDK, an iOS simulator and and IDE. Xcode 4 was about the only tool available. When I logged in and ready to download Xcode 4, I was told:

Hi Romen,     You must be an iOS or Mac Developer Program member to download Xcode 4 or you can purchase Xcode 4 from the Mac App Store.

The membership costs $99 a year. This is not an option for I just want to try it out. The Xcode 4 costs $4.99 from the AppStore – hmm, not bad though the customer review was not very flattering… Then I realised that Xcode can only run on Mac OS X, which means I would have to fork out another $1000+ to buy an Apple computer just for using Xcode. I don’t mind an Apple computer, but there is no justification for its doubled price comparing to other brands with gross margin at 7 times of others. I do have an Mac OS X virtual machine image on VMWare, but it was too slow just to run itself, let alone Xcode. This reminds me of the early 1990s when you had to pay left and right just to learn a computing language, which was especially touch for students like me.

Also, once you finish developing your app, you’d have to go through Apple Store to sell them and get exploited by Apple again. Apple does not do a good job to keep a high quality of what they sell anyway – we only had the iPhone 4 for 3 weeks and already experienced buggy apps – buttons on the default SMS app and Facebook app were greyed out when they shouldn’t be and can only be fixed by bouncing the software or rebooting the phone (by the way, I do not see any rationale in Apple’s decision of not letting the user exit each app.)

The barrier to entry seems too high for a private user. I am used to the open source environments of many innovative technologies (led by Java perhaps). For me personally to buy into the Apple’s closed-door philosophy is just too much. So I have to ditch Apple technology as I always have.

There is plenty to explore in the Android domain.

Must-Haves On A New ₳₱₱£€ iPhone 4

2011-06-03T12:44:00.001+10:00

The whole ₳₱₱£€ experience from start to end, from developer to user, from manufacturer to partner is just a big money grabbing exercise by Apple at every step involved. ₳₱₱£€ not only asks you to pay left and right but also restricts on what you can do and how you do things. To get back some sanity when using the new iPhone 4, here are some things that have to be done:

Jailbreak it – for iPhone 4 running iOS 4.3.3 (8J2) here is a good starter.
Change root password – after Jailbreaking, the phone’s (running BSD Unix) root password is automatically set to ‘alpine’. This must be changed. Note that the MobileTerminal app does not work on iOS 3 and 4 at the time of this post. So to do this, it is better to install OpenSSH and use a client such as PuTTY to connect to the iPhone with root login and use the passwd Unix command to change the root password.
Ditch iTune – iTune forces you to sync apps between iPhone and the computer. This is not only waste of time and the computer disk space, but also creates problems when there are multiple computers. CopyTrans is a good solution to this.
Get a decent GPS app – the default map application assumes that the phone is connected to the internet all the time, which can be extremely costly with many carriers and impractical in many countries. So there is a need to have a GPS app that can store map files on the iPhone locally. xGPS seems to fit the bill, although it cannot be used fully offline (like TomTom or Nokia Maps) – you’ll still need to be connected to Google when searching for addresses and working out routes.

As I am totally new to any ₳₱₱£€ product, any free apps suggestions are welcome.

RIP Symbian, Hello iPhone 4

2011-05-31T22:04:00.001+10:00

The joint announcement from Nokia’s Kai Öistämö and Microsoft’s Andy Lees last month had formally declared the death of Symbian. The new N8’s and E7’s may well be the last emperors of the Symbian dynasty.

In reality, the Symbian applications community has been languishing for a long time, far behind the new comers. Trying to find a decent internet radio software today, the only option I could find was Nokia Internet Radio. I installed in on my N95 and it soon crashed my phone and caused it to reboot. Such problems have been reported back in 2008. Yet, it seems that nothing has been done.

On the other hand there are far more software in the iPhone and Android communities. Take internet radio for example, TuneIn supports just about any mobile OS except Symbian. It has access to over 40,000 stations and runs very smoothly on my new iPhone 4. It’s a far cry from the pathetic Nokia Internet Radio.

I had never owned any Apple product until yesterday – after some nagging I bought a white iPhone 4 for Rose. I always thought Apple products were for women and children for two reasons:

they look cool
they are easy to use

But after playing with the iPhone 4, I changed my mind – it’s not easy to use at all! It’s just as bad as any other products, if not worse.

First of all, you cannot even start using the iPhone without a MicroSIM card. At least, my trusted N95 could work without SIM card and function normally except for telco services. So the first hurdle for me was to cut my normal SIM card with scissors and make it into a MicroSIM. I was amazed that it actually worked!.

After that, I had to register on AppStore by providing my private information. Even if you want to install a free software, you are still required to put in your credit card detail - what the heck?!

Then I found that the iPhone refused to load any of my music files that were accumulated over the years – not even the polyphonic ring tone that I have been using for years. That was the final straw. I had no option but to jailbreak, after which the whole user experience became much more acceptable. At least the process was much easier and quicker than cracking my N95.

Now the women and children in my household can better enjoy the over hyped iPhone 4 before they get bored with it, just like the old XM5800.

Throttling

2011-05-17T12:33:00.001+10:00

A while ago, I blogged about the NBN Australia and how much benefits that city folks like myself will (not) receive. There is another little quirk that NBN Co. never mentions – throttling.

Throttling refers to the traffic control measure applied by all ISPs and telcos to limit the speed/bandwidth available to the end users – squeezing the pipe, if you like.

Have you ever wondered why your Limewire/Frostwire bandwidth is so low (as low as a few Kbps – lower than dial-up speed) although you have allowed the highest settings in the software? That is because ISPs throttle many P2P traffic including Limewire/Frostwire. Such traffic policy control has been standardised by the telco industry body 3GPP under the topics of PCRF and PCEF (TS 23.203), where DPI is usually used to detect the type of traffic (e.g. Limewire, IMAP, FTP, Skype, etc.) and apply the service policies and charging to the individual traffic sessions. These standards are implemented by all the major network equipment vendors.

Throttling is applied because such P2P traffic can amount to more than 80% of the total traffic of an ISP, which is not surprising considering a HD movie can be as big as 10s of Gigabytes. So it is pretty certain that even if you have an unlimited internet plan with 1Gbps burst rate, it can still take days/weeks to download that movie you are dying to see.

Cheating Red Alert 3

2011-05-08T10:40:00.002+10:00

I recently started to play Red Alert 3 (v1.0) just for the amusement. I am not a serious game player, so naturally I needed to cheap my way through some chapters.

Unlike games such as War Craft 3, which have built-in cheat codes, RA3 has none (or at least I could not find any on the net). Fortunately, there is CheatEngine – an very handy open source bundle of tools that have been especially built for cheating in games! Since CE has been built by hacker for hackers, documentation is scarce and not easy to find.

From a quick search, there are a couple of ways to cheat in RA3:

Modify money - use the CE’s memory scanner to search for the memory addresses that store the money amount. Once found, modify them to give yourself virtually unlimited money. This approach has been demonstrated on YouTube. The video is blurry, but the method is exactly the same as the first tutorial of CE, which is bundled with the CE installation.
God Mode, Unlimited Resource and Quick Research – this is done by code injection. The assembly source code is available on the CE Forum. However, the forum does not say how to apply the hack. Here, I will show a step by step guide on applying the hack.

Once both RA3 and CE are running (doesn’t matter which is first),

go to CE and select the process named ‘…ra3_1.0.game’ and open it.
Click the ‘Memory view’ button on CE to open the memory view window.
From the Memory Viewer window, select the menu Tools –> Auto Assemble (Ctrl-A). This will pop up the Auto assemble window.
Paste the assembly code found in the CE Forum (also listed below) into the Auto assemble window.
Press Execute button. Click Yes on the confirmation message to inject the code. You should get another message saying successful.

That’s it. When you switch back to RA3, you will find that your money is about 100,000, researches are very fast and the units can take a constant pounding for several minutes without losing health.

The assembly code is shown below. Notice the highlighted lines – to turn the particular cheat off, change the value to 0. Also note that the code for later versions of RA3 is different and they can be found on the CE Forum as well.

// Command and Conquer - Red Alert 3 
// Game Version  : 1.0.3174.697 
// Script Version: 1.0 
// CE Version    : 5.4 
// Resource, Research and GodMode 
// 08-Nov-2008 

[ENABLE] 
alloc(MyCode,1024) 

label(_MonResource) 
label(_GodMode) 
label(_MonRPoints) 
label(_MonPlayerID) 
label(_GodM0) 
label(_BackMR) 
label(_BackGM) 
label(_BackMRP) 
label(_BackMPI) 
label(_ExitMR) 
label(_ExitGM) 
label(_ExitMRP) 
label(pResource) 
label(pLastOne) 
label(iPlayerID) 
label(iEnableGM) 
label(iEnableMRP) 
label(iEnableMR) 

registersymbol(MyCode) 
registersymbol(pLastOne) 
registersymbol(iEnableGM) 
registersymbol(iEnableMRP) 
registersymbol(iEnableMR) 
//============================= 
// Hacking Points 
ra3_1.0.game+3e4acc: 
 jmp _MonResource 
 nop 
_BackMR: 

ra3_1.0.game+30d4ae: 
 jmp _GodMode 
_BackGM: 

ra3_1.0.game+4004e5: 
 jmp _MonRPoints 
_BackMRP: 

ra3_1.0.game+58bb45: 
 jmp _MonPlayerID 
 nop 
_BackMPI: 

MyCode: 
//========================================= 
_MonResource: 
 push eax 
 mov eax,[iPlayerID] 
 cmp eax,[ecx+20]           // Player's?... 
 pop eax 
 mov ecx,[ecx+000000e4]     // Original code 
 mov [pResource],ecx        // Save ptr for debugging 
 jne _ExitMR                // ...Jump if false 

 cmp dword ptr [iEnableMR],0 
 je _ExitMR                // Jump if Monitor Resource is disabled 

 mov ecx,[ecx] 

 cmp dword ptr [ecx+04],#100000 
 jge _ExitMR                // Jump if greater then 100000 

 mov dword ptr [ecx+04],#100000 

_ExitMR: 
 mov ecx,[pResource] 
 jmp _BackMR                // back to main code 

//========================================= 
_GodMode: 
 push eax 

 mov eax,[esi-08]           // Get ptr to Unit 
 or eax,eax                 // Null Ptr? 
 jz _ExitGM                 // Jump if true 

 mov eax,[eax+00000418]     // Get ptr to Player 
 mov eax,[eax+20]           // Get ID 

 cmp eax,[iPlayerID]        // Player's?... 
 jne _ExitGM                // Jump if false 

 mov [pLastOne],esi        // Save ptr for debugging 

 mov eax,[esi+30] 
 cmp eax,00000070          // Is it an effect? 
 je _ExitGM                // Jump if true 

_GodM0: 
 cmp dword ptr [iEnableGM],0 
 je _ExitGM                // Jump if God Mode is disabled 

 movss xmm0,[esi+0c]       // Get Maximum HP 

_ExitGM: 
 movss [esi+04],xmm0       // Original code 

 pop eax 
 test eax,eax              // Restore EFLAGS 
 jmp _BackGM               // Back to main code 

//========================================= 
// Quick Research 
_MonRPoints: 
 push edx 

 movss [esi+2c],xmm0       // Original code 

 cmp dword ptr [iEnableMRP],0 
 je _ExitMRP               // Jump if Quick Research is disabled 

 mov edx,[esi+28] 
 mov edx,[edx+20] 
 cmp edx,[iPlayerID]       // Player´s research? 
 jne _ExitMRP              // Jump if false 

 mov edx,43af0000          // 350.0 
 cmp edx,[esi+2c] 
 jle _ExitMRP 

 mov [esi+2c],edx          

_ExitMRP: 
 pop edx 
 jmp _BackMRP              // Back to main code 

//========================================= 
_MonPlayerID: 
 mov eax,[edi+00000080] 
 mov [iPlayerID],eax       // Save Player ID for further use 
 jmp _BackMPI              // Back to main code 

//========================================= 
// Variables 
iPlayerID: 
 dd 0 
pResource: 
 dd 0 
pLastOne: 
 dd 0 
iEnableGM: 
 dd 1 
iEnableMRP: 
 dd 1 
iEnableMR: 
 dd 1 

//========================================= 
// Original Codes 

[DISABLE] 
ra3_1.0.game+3e4acc: 
 mov ecx,[ecx+000000e4] 

ra3_1.0.game+30d4ae: 
 movss [esi+04],xmm0 

ra3_1.0.game+4004e5: 
 movss [esi+2c],xmm0 

ra3_1.0.game+58bb45: 
 mov eax,[edi+00000080] 

unregistersymbol(MyCode) 
unregistersymbol(pLastOne) 
unregistersymbol(iEnableGM) 
unregistersymbol(iEnableMRP) 
unregistersymbol(iEnableMR) 

dealloc(MyCode)

IE9 and MIME Type Handling

2011-04-18T14:48:00.001+10:00

I upgraded my Internet Explorer from v8 to v9 a couple of weeks ago. I immediately discovered that the SyntaxHighlighterI use on my Blog is no longer working. I turned on the console view (by pressing F12) of IE9 and it showed that the Javascript file of SyntaxHighlighter loaded from my Google Site is actually named syntaxhighlighter2.txt; however, the type of the link was text/javascript.

This mismatch between the file name extension and the specified MIME type upsets IE9, so it refuses to load my SyntaxHighlighter script. I tried to turn off the MIME handling feature in IE9 by changing the Windows registry following instructions in the following articles:

None worked.

So I took the simple approach – change my SyntaxHighlighter file extension from .txt to .js – and it worked.

Hacking LinkedIn API – Take 2

2011-03-24T16:49:00.003+11:00

I had taken a long holiday from work and from internet. So I hadn’t touched this blog till now. I discovered that many visits to my blog was to one of my post on Hacking LinkedIn API.

When the LinkedIn APIs first came out I played with it and bypassed the convoluted and stupid OAuth manual process. I documented the approach in my blog post. However, soon after I posted it, LinkedIn had changed the login page and the access code page so that the HTML posting and scraping code no longer worked. I promised to update my Java code, so here it is.

This time, I am using a later version of the Java wrapper for the LinkedIn APIs – LinkedIn-J 1.0.361. The Java API has totally changed from the previous one I used.

The main difference of my 2nd attempt of hacking the APIs are as following:

Using HTMLEditorKit

Once you go to the authorisation URL returned by LinkedIn, it displays a login form (you must clear your browser’s cookie to disable the auto-login). In this login form, there are a number of hidden fields just like before. However, this time LinkedIn had added a few more fields and a dynamic one – named csrfToken. When we submit the form, we must include all the hidden field values as well. So we need to parse this HTML string to retrieve the dynamic field values. I used HTMLEditorKit library because it’s part of Java Swing so no external JARs are required. The login form looks something like this.

...

So to retrieve the field values, I added a HTML parser callback class.

class ReportAttributes extends HTMLEditorKit.ParserCallback {
 public String csrfToken, sourceAlias;

 public void handleStartTag(HTML.Tag tag, MutableAttributeSet attributes, int position) {
  this.listAttributes(attributes);
 }
 public void handleSimpleTag(HTML.Tag tag, MutableAttributeSet attributes, int position) { 
  this.listAttributes(attributes); 
 } 
 private void listAttributes(AttributeSet attributes) {
  if (attributes.containsAttribute(HTML.Attribute.ID, "csrfToken-oauthAuthorizeForm")) {
   csrfToken=attributes.getAttribute(HTML.Attribute.VALUE).toString();
   System.out.println("csrfToken="+csrfToken);
  } else if (attributes.containsAttribute(HTML.Attribute.ID, "sourceAlias-oauthAuthorizeForm")) {
   sourceAlias=attributes.getAttribute(HTML.Attribute.VALUE).toString();
   System.out.println("sourceAlias="+sourceAlias);
  }
 }
}

Enabling Cookies

It turned out that you must enable cookies otherwise LinkedIn will complain when you try to submit the login form. So here is the snippet for enabling cookie.

CookieManager manager = new CookieManager();
 manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
 CookieHandler.setDefault(manager);

The overall structure of the code is pretty similar to before. Here is the full source code. Just modify the highlighted lines and it should just work for you.

package com.laws.LinkedIn;

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.CookieHandler;
import java.net.CookieManager;
import java.net.CookiePolicy;
import java.net.HttpURLConnection;
import java.net.URL;

import javax.swing.text.AttributeSet;
import javax.swing.text.MutableAttributeSet;
import javax.swing.text.html.HTML;
import javax.swing.text.html.HTMLEditorKit;

import com.google.code.linkedinapi.client.LinkedInApiClient;
import com.google.code.linkedinapi.client.LinkedInApiClientFactory;
import com.google.code.linkedinapi.client.oauth.LinkedInAccessToken;
import com.google.code.linkedinapi.client.oauth.LinkedInOAuthService;
import com.google.code.linkedinapi.client.oauth.LinkedInOAuthServiceFactory;
import com.google.code.linkedinapi.client.oauth.LinkedInRequestToken;
import com.google.code.linkedinapi.schema.Person;

class ParserGetter extends HTMLEditorKit {
   public HTMLEditorKit.Parser getParser() {
     return super.getParser();
   }
 }
class ReportAttributes extends HTMLEditorKit.ParserCallback {
 public String csrfToken, sourceAlias;

  public void handleStartTag(HTML.Tag tag, MutableAttributeSet attributes, int position) {
   this.listAttributes(attributes);
  }
  public void handleSimpleTag(HTML.Tag tag, MutableAttributeSet attributes, int position) { 
   this.listAttributes(attributes); 
  } 
  private void listAttributes(AttributeSet attributes) {
   if (attributes.containsAttribute(HTML.Attribute.ID, "csrfToken-oauthAuthorizeForm")) {
  csrfToken=attributes.getAttribute(HTML.Attribute.VALUE).toString();
  System.out.println("csrfToken="+csrfToken);
  } else if (attributes.containsAttribute(HTML.Attribute.ID, "sourceAlias-oauthAuthorizeForm")) {
  sourceAlias=attributes.getAttribute(HTML.Attribute.VALUE).toString();
  System.out.println("sourceAlias="+sourceAlias);
   }
  }
}


public class Main {
 static final String apiKey="your api key";
 static final String secretKey="your secret key";
 static final String login="name%40company.com";
 static final String password="password";
 
 static public String getPin(String authUrl, String token) {
  DataOutputStream dataOut;
  ParserGetter kit = new ParserGetter();
     HTMLEditorKit.Parser parser = kit.getParser();
     ReportAttributes callback = new ReportAttributes();
     // must enable cookie, otherwise LinkedIn will not give you the access code
     CookieManager manager = new CookieManager();
     manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
     CookieHandler.setDefault(manager);


        try {
         // this section gets the LinkedIn login form.
            URL url = new URL(authUrl);
            HttpURLConnection con = (HttpURLConnection)url.openConnection();
            con.setRequestMethod("POST");
            con.setUseCaches(false);
            con.setDoInput(true);
            con.setDoOutput(true);
            con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
            //SSLException thrown here if server certificate is invalid
            InputStreamReader reader = new InputStreamReader(con.getInputStream());
            parser.parse(reader, callback, true);
            System.out.println("-------------------------------------");
            
            // POST the login form and get the access/verification code.
            url = new URL("https://www.linkedin.com/uas/oauth/authorize/submit");
            con = (HttpURLConnection)url.openConnection();
            con.setRequestMethod("POST");
            con.setUseCaches(false);
            con.setDoInput(true);
            con.setDoOutput(true);
            con.setRequestProperty("User-Agent", "Mozilla/4.0");
            con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
            
            dataOut = new DataOutputStream(con.getOutputStream());

            String s="session_login="+login
             +"&session_password=" + password
             + "&duration=0&authorize=Ok%2C%20I'll%20Allow%20It&extra=&access=-3&agree=true&oauth_token="
          +token+"&appId=&csrfToken="+callback.csrfToken+"&sourceAlias="+callback.sourceAlias;
            System.out.println("writing bytes: "+s);
            dataOut.writeBytes(s);
            dataOut.flush();
            dataOut.close();

            //SSLException thrown here if server certificate is invalid
            String returnedHtml=convertStreamToString(con.getInputStream());
            //System.out.println(returnedHtml);
            
            /* extract the pin from the html string. the block looks like this       
          73336* It turns out that the whole html string only contains one 'div with class="access-code"'
             * also it seems that the pin is always 5-digit long,
             * so we will just crudely detect that string and get the pin out.
             * A proper HTML parser should be used in a real application.  
             */
            int i=returnedHtml.indexOf("access-code\">");
            String pin = returnedHtml.substring(i+13, i+13+5);
            System.out.println("pin="+pin);
            return pin;         
        } catch (Exception e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
   return null;
  }
    }

 /**
  * @param args
  */
 public static void main(String[] args) {
  final LinkedInOAuthService oauthService = LinkedInOAuthServiceFactory.getInstance().createLinkedInOAuthService(
    apiKey, secretKey);
  LinkedInRequestToken requestToken = oauthService.getOAuthRequestToken();
  
        System.out.println("request token: ");
        System.out.println("  auth URL: "+requestToken.getAuthorizationUrl());
        System.out.println("  token: "+requestToken.getToken());
        System.out.println("  token secret: "+requestToken.getTokenSecret());
        System.out.println("  expiration time: "+requestToken.getExpirationTime());
        
        // get the access code
        String pin=getPin(requestToken.getAuthorizationUrl(), requestToken.getToken());
        
        LinkedInAccessToken accessToken = oauthService.getOAuthAccessToken(requestToken, pin);
        final LinkedInApiClientFactory factory = LinkedInApiClientFactory.newInstance(apiKey, secretKey); 
        final LinkedInApiClient client = factory.createLinkedInApiClient(accessToken);  
        
        // now we can call the LinkedIn APIs.
        Person profile = client.getProfileForCurrentUser();
        System.out.println("I am "+profile.getFirstName()+" "+profile.getLastName());
 }
 // Stolen liberally from http://www.kodejava.org/examples/266.html
    public static String convertStreamToString(InputStream is) {
        /*
         * To convert the InputStream to String we use the BufferedReader.readLine()
         * method. We iterate until the BufferedReader return null which means
         * there's no more data to read. Each line will appended to a StringBuilder
         * and returned as String.
         */
        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        StringBuilder sb = new StringBuilder();

        String line = null;
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "\n");
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        return sb.toString();
    }

}

Month without Internet

2011-01-09T22:30:00.001+11:00

I had a wonderful month-long holiday in the Philippines. In the province (as they call the countryside in Pinoy lingo) home, I did not have any fixed line phone service, hence no wired (ADSL or cable) internet access. So I purchased wireless broadband USB dongles from both Smart and Globe (both made by Huawei). They claim to be 3G broadband. But there were no 3G coverage from Smart at where I lived; and there was 3G signal from Globe – sometimes 3G, sometimes HSPDA and sometimes just GPRS. Even with 3G coverage, the data speeds were pathetic – the overall speed was worse than dial up modems in the old days – definitely narrowband (speeds ranging from 0 to about 40kbps most of the time).

So instead of feeling frustrated with Philippines telcos (as I usually do), I decided to stay away from the internet altogether. Without internet and piano, I found that I had so much time and got to do what I had always wanted to do so badly for so long. I managed to read a Cisco CCNA book and practiced the labs using Cisco Packet Tracer and GNS3.

More rewardingly, I managed to start doing water colour paintings. I had always wanted to learn and practise water color for many years – I bought several books, some cheap brushes, colors and a pallet over the years. But I never had a chance to start painting until then. Once I started, I did not want to stop. I was painting at home, on tour, on other islands. I ended up with over a dozen nature and still life paintings. It was the first time that I used the proper gear – water color papers, brushes (they are absolute bargains from National Book Store in Tacloban – the same brushes would cost 10 times more in Sydney). Now that I am back in Sydney, I will keep painting and keep the online time to a minimum.

SN 1979C

2010-11-19T10:31:00.001+11:00

I have been travelling for the last week. Everywhere I went there were news about the youngest black hole ever found. I could not understand how people kept saying the black hole (if it is one) is 30 years old yet it's 50 million light years away. I found the answer in the NASA's CHANDRA X-Ray Observatory site where the original news was released.

The 30-year number is the observed value meaning that the actual age of the object is well over that and we will not know its current status until 50 million years later. It also means that the Earth has been safe from it for the last 50 million years and there is no need for a panic attack.

What the article does not reveal is in what state the object was when it was first observed in 1979 – whether it had already turned into a supernova / black hole or still a star.

JAAS with Tomcat

2010-09-26T09:32:00.002+10:00

As JAAS has been included as part of JRE core since Java 1.4, it should be quite prevalent by now. However, I cannot seem to find many online resources that detail the steps involved to set up JAAS with Tomcat. Here, I outline these steps from a beginner’s perspective.

Step 1 – Implement the JASS interfaces.

The basic interfaces to be implemented are

Principal – there should be two implementations of this interface: One representing the user; another one (or more) represent the roles. The important method of this interface is getName() which returns the name of the user or role that it represents.
LoginModule – this is the main class that usually serves as the wrapper for customised authentication logic. The login() method is where the customised login logic should be invoked.

Once implemented, JAR up these classes including the customised login classes and put it in $CATALINA_BASE/lib.

See the code segments at the end for example implementations.

Step 2 – Configure Tomcat Server

Edit $CATALINA_BASE/conf/server.xml to put in the JAASRealm configuration. As instructed by Tomcat’s Realm-HOW-TO documentation, this can be done at various levels - <Engine>, <Host> or <Context>. I added it inside the <Host> tag:

...

Notice the userClassNames and roleClassNames are the classes that were created in step 1.

Step 3 – The JRE login configuration

Create a JAAS Login Configuration file. I called it jaas.config and put it under $CATALINA_BASE/conf:

acczk {
  com.laws.acc.jaas.AccLoginModule required;
};

There are two ways to configure the JRE to load the login ocnfiguration file. I prefer to specify it in JAVA_OPTS. For Tomcat, this means to create a setenv.sh or setenv.bat in the $CATALINA_BASE/bin directory. Here is my setenv.bat:

set JAVA_OPTS=-Djava.security.auth.login.config==C:/Tools/apache-tomcat-7.0.2/conf/jaas.config

Step 4 – The Login Form

The login method can be BASIC or FORM-based. I prefer FORM. A vanilla HTML form should do. I happen to be using ZK so my HTML login form is in the file login.zul:






Welcome to ${c:l('client_name')}!


     
	 
	    import com.laws.acc.ws.*;
	    
		String errMsg = null; 
		
		LoginResponse loginResponse=AccSoapClientProxy.getLoginResponse();
		if(loginResponse!=null) {
			switch(loginResponse.getResultCode()) {
			case ResultCode.NOT_LOGGED_IN:
				errMsg="Not logged in.";
				break;
			case ResultCode.CONNECTION_ERROR:
				errMsg="Cannot login to server:\n"+loginResponse.getErrorDescription();
				break;
			default:
				errMsg="Login failure:\n"+loginResponse.getErrorDescription();
			}
		}
	     
	         
	        
		
			
			
		     
		                 
			                     
				Username :                     
				                 
			                 
			                     
				Password :                     
				                 
			             
		         
	         
	    
		         
		         
	     
	     
	${errMsg}     
	         
		j_username.focus();

The important part here is the HTML form employing j_username, j_password and j_security_check.

Step 5 - Configure web.xml

This is standard configuration to enable security constraints.


      ACC Security Constraint
      
         Protected Area
         
         /index.zul
	 /secure/*
         
         DELETE
         GET
         POST
         PUT
      
      
         
         Acc User
      
    
	
	  Generic authenticated ACC user
	  Acc User
	
    
    
      FORM
      Form-Based Authentication Area
      
        /login.zul
        /login.zul
      
    
  	
  	120

My JAAS implementation files:

AccLoginModule.java:

package com.laws.acc.jaas;

import java.io.IOException;
import java.util.Map;

import javax.security.auth.Subject;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.auth.login.LoginException;
import javax.security.auth.spi.LoginModule;

import com.laws.acc.ws.AccSoapClientProxy;
import com.laws.acc.ws.LoginResponse;

public class AccLoginModule implements LoginModule {
	// initial state
    private Subject subject;
    private CallbackHandler callbackHandler;
    private Map sharedState;
    private Map options;
    
    private AccPrincipal principal;
    private boolean committed = false;

	@Override
	public boolean abort() throws LoginException {
		if(!committed)
			return false;
		if(principal!=null) {
			logout();
			principal=null;
		}
		return true;
	}

	@Override
	public boolean commit() throws LoginException {
		if (principal == null){
            return false;
        }
        if (!subject.getPrincipals().contains(principal))
            subject.getPrincipals().add(principal);
	// hardcoded role assignment.
        AccRole role=new AccRole();
        if(!subject.getPrincipals().contains(role))
        	subject.getPrincipals().add(role);
        committed = true;
        return (true);
	}

	@Override
	public void initialize(Subject subject, CallbackHandler callbackHandler,
			Map sharedState, Map options) {
		this.subject = subject;
		this.callbackHandler = callbackHandler;
		this.sharedState = sharedState;
		this.options = options;
	}

	@Override
	public boolean login() throws LoginException {
		if (callbackHandler == null)
            throw new LoginException("No CallbackHandler specified");
        Callback callbacks[] = new Callback[2];
        callbacks[0] = new NameCallback("Username: ");
        callbacks[1] = new PasswordCallback("Password: ", false);
 
        // Interact with the user to retrieve the username and password
        String username = null;
        String password = null;
        try {
            callbackHandler.handle(callbacks);
            username = ((NameCallback) callbacks[0]).getName();
            password = new String(((PasswordCallback) callbacks[1]).getPassword());
            LoginResponse response = AccSoapClientProxy.login(username,password);
            if (response.getResultCode()!=0)
                return false;
            principal  = new AccPrincipal(response);
            return true;
        } catch (IOException e) {
            throw new LoginException(e.toString());
        } catch (UnsupportedCallbackException e) {
            throw new LoginException(e.toString());
        }
	}

	@Override
	public boolean logout() throws LoginException {
		committed=false;
		subject.getPrincipals().remove(principal);

		return false;
	}

}

AccPrincipal.java:

package com.laws.acc.jaas;

import java.io.Serializable;
import java.security.Principal;
import com.laws.acc.ws.*;

public class AccPrincipal implements Principal, Serializable {

	/**
	 * 
	 */
	private static final long serialVersionUID = 5002820876845306935L;
	private LoginResponse loginResponse=null;
	
	public AccPrincipal(LoginResponse lr) {
		this.loginResponse=lr;
	}
	
	@Override
	public String getName() {
		return loginResponse.getUserName();
	}

	@Override
	public boolean equals(Object other) {
		if(other==null) return false;
		if(other==this) return true;
		if(other instanceof AccPrincipal) {
			AccPrincipal that=(AccPrincipal) other;
			return loginResponse.getUserId()== that.getLoginResponse().getUserId();
		} else
			return false;
	}
	
	public LoginResponse getLoginResponse() {
		return loginResponse;
	}

	public void setLoginResponse(LoginResponse loginResponse) {
		this.loginResponse = loginResponse;
	}

	@Override
	public int hashCode() {
		return loginResponse.getUserId();
	}
}

AccRole.java:

package com.laws.acc.jaas;

import java.security.Principal;

public class AccRole implements Principal {

	@Override
	public String getName() {
		return "Acc User";
	}
	
	@Override
	public boolean equals(Object other) {
		if (other==null) return false;
		if(other==this) return true;
		if(other instanceof AccRole) {
			return this.getName().equals(((AccRole) other).getName());
		} else
			return false;
	}

	@Override
	public int hashCode() {
		return getName().hashCode();
	}
}

Java Logging & Properties File

2010-09-21T12:02:00.002+10:00

I prefer to have the properties file at the application level or JAR level if I am writing a library, with as many configuration entries in the same file as logically possible so that I end up with as few files as possible. So my properties file is at the root level of the JAR file.

To load the config.properties file at the root of the JAR:

// ResourceBundle expectes the file extension to be .properties by default.
ResourceBundle bundle = ResourceBundle.getBundle("config",
    Locale.getDefault(),
    Thread.currentThread().getContextClassLoader());
String dmsAddress = bundle.getString("dmsAddress");

Similarly when I use Java Logging I put the logging.properties file at the same location and load it upon application start up using LogManager so that I don't have to tamper with the properties file at JRE/lib.

The logging.properties file:

handlers = java.util.logging.ConsoleHandler
.level=INFO

java.util.logging.ConsoleHandler.level = INFO
com.laws.acc.level = ALL
com.laws.acc.ws.level = ALL

It is better to copy the JRE's logging.properties file and modify it. To load the logging configuration:

LogManager logMan=LogManager.getLogManager();
logMan.readConfiguration(Thread.currentThread().getClass().getResourceAsStream("/logging.properties"));

Resources: Smartly Load Your Properties

NBN Australia

2010-08-16T12:26:00.001+10:00

As the federal election campaign comes to its final week, the issue of national broadband network (NBN) has become one of the most important policy differentiators between Labor and the Coalition.

I have not been paying much attention to NBN until a couple of months ago when a friend of mine was trying to search for jobs in NBN yet found very few. As the government claims that NBN will directly employs tens of thousands of people, you’d expect many job ads from them. But that’s not the case. I did a search on one of the job sites back then and only found a handful.

I posted a question on what real benefits average Australians will expect from the NBN on the NBN Australia discussion group. There has been no answers so far. As I live in a big city and there are many broadband options to choose from – FTTH, cable/HFC, xDSL, 3G or non-standard wireless ones (unWired) offered by many communications service providers (CSPs). I don’t see the need of spending my hard-earned tax dollars trying to improve on something that I cannot experience.

What I do want to see is to spend my hard-earned tax dollars on those who really need it – regional Australia. There is a big gap between big cities and regional area in terms of infrastructure. Some people have to drive for hours just to go to a hospital – that’s if they have a car. For those who don’t, they will have to rely on community buses, which takes even longer.

Dick Smith has raised the issue about population and sustainability in Australia. It was hotly debated on Q&A. One of the issues was population distribution and the infrastructure to support the population. There is no doubt that regional towns want to attract more population and therefore economic growth. You cannot achieve this without improving the infrastructure. So I believe the government should spend less money overall, and focus on narrowing the gap, which will provide real long-term benefits to the Australian people.

Last week, NBN triumphantly announced that the speed of the fibre network can go to 1Gbps – “10 times what they originally envisaged”. At least that is what most of the TV news reports have told you. To non-tech heads like Tony Abbot, that might sound astounding. But when you go into the details, it’s not that exciting but could be puzzling. Apparently the 1Gbps is burst rate, not committed rate (note that the committed rate is what users really experience). Does that mean NBN and the government were originally envisaging 100Mbps burst rate on a FTTH that costs 46 billion dollars, which is downright laughable?

Communication networks are hierarchical and each link of the transmission network matters (by the way, can NBN tell us what the bandwidth between Tasmania and mainland is/are?). The overall user experience also depends on all the links from point A to B. The network speed we experience is equal to the weakest link – see Ultra-fast broadband will be slow on overseas links. Just a few days ago, I downloaded VMWare image of Mac OS X Leopard (2GB) using bit torrent. It took less than 30 minutes. One day later I tried to download Zorin OS – a Ubuntu Linux distro (1GB) using FTP, it took over 8 hours at which point the connection to the website was broken. So I had to redo it the next day using a mirror site which still took a couple of hours, but it was worth it. The OS is awesome!

Also, in a wired home environment (or should I say wireless), people typically use WiFi and laptops. With the latest WiFi standard (802.11n), you can expect 40Mbps (and 200Mbps burst). So again, there is a very weak link right at home.

Google WAVES Good-Bye

2010-08-06T09:46:00.001+10:00

I was shocked when I saw the above heading on one of the news items yesterday. I remember I saw an online video of a demo of Wave mid last year. I was very curious and anxious to try it. However, I could not find out how to join! It was apparently by invitation only. Then I forgot about it – until yesterday. So that online video turned out to be my first and last encounter with the great Wave.

Someone sited top 11 reasons why Wave failed. I believe the top reason for me is the missing marketing of the product. Every now and then I would receive promotional emails on my Gmail account about Google Ads. But there has been nothing about Wave. I have resisted the temptation to put it on my blog thus far – mainly because I am too lazy and the traffic to my blog is very low (maybe 60 per day and half of them are from crawlers).

If you take a closer look to how Google works internally, it does not seem so surprising. It is part of Google’s corporate policy to have their employees spend up to 20% of their time to work on any thing that they think interesting. If the management thinks the project has potential, the company will resource and fund it. No doubt Wave was born out of such conditions. So it’s created by nerds for nerds. The Google management probably hoped that it would one day take on the likes of Facebook. Being a nerd I don’t mind the complexity and learning curve – in fact, that’s what draws me to it. But by definition, nerds are not great social networkers – e.g. I joined Facebook and Twitter to test their APIs. That definitely reduced the appeal of Wave.

Also, the near zero marketing of Wave is no surprise considering how Google makes money – through advertising. That is why they advertise Google Ads and leave the other pet projects live and die on their own. From the days Larry and Sergey started Google, they had been focusing on technical superiority rather than monetisation. That had worked on the search engine since it was so far ahead than the others and the user experience is extremely simple. The contrary is true for Wave. So to let Wave catch up with its competitors purely based on its own merit without any marketing will require some time – much longer than the 1 year that Google had given it so far. Even the search engine did not catch on on day one – Larry and Sergey spent their doctorate years in uni to develop the product.

Now that the Wave is flushed down the toilet, I wonder what the Google people in Sydney are doing these days.

Virtual Machines

2010-07-25T16:14:00.001+10:00

I recently beefed up my desktop machine at home by upgrading to 4GB RAM, 1TB WD Green Caviar hard disk and nVidia GeForce 7600 series. I also installed Windows 7 Ultimate 64-bit on my AMD Athlon 3500+.

One reason I upgraded my 6-year-old machine is that I want to run virtual machines and try various cloud computing and data grid frameworks.

As soon as I delved into virtual machine technologies, I found that my AMD CPU which has Socket 939, does not support hardware virtualisation. The list of AMD Athlon 64 CPUs show that AMD-V only started from Socket AM2. So that basically rules out the possibility of installing 64-bit guest OS on any of the VM software.

The VM s/w I tried are Oracle/Sun VM VirtualBox 3.2.6, VMWare Player 3.1.0 and VMWare Server 2.0. My understanding is that they are all free.

I first installed Ubuntu 10.04 on VirtualBox. The installation was very smooth. All the devices are installed without problem – sound, network, disk drives, USB devices etc.

Before, VMWare Player was only designed to play a guest OS image. But since version 3, you can also create VM using the player. The experience with VMWare Player is not much different from VirtualBox. The result was equally good. I felt that the VM seems a little faster in VMWare. Also, VMWare Player allows you to connect and disconnect devices on the fly, such as network adapter, printer, sound card, disk drives, USB devices. This is something more flexible than VirtualBox.

In fact, before I installed VMWare Player, I also tried VMWare Server (you cannot install both on the same host machine). The idea of VMWare Server is quite good – it runs as a services (Windows services, or Unix daemon) so that user can connect to it using web browser. All the actions are done on the browser – the VM console is installed as a browser plug-in. However, my user experience was not so good with VMWare Server. I installed the same Ubuntu iso image on it successfully, but it could not connect to internet (although there is connectivity between the host machine and the guest VM), and sound card was not detected. I could not add these h/w from outside the guest either as they were all greyed out on VM’s configurations page on WMWare Server.

Now I am happily running Ubuntu 10.04 on VirtualBox and Linux Mint 9 Isadora – both 32-bit of course .

Eclipse Galileo vs. NetBeans 6.8

2010-06-07T22:33:00.002+10:00

I have read and heard about the praises on both Eclipse and NetBeans on how performant they each are, especially when a new release comes out. Today I had a chance to witness the two with my own eyes and compare them.

I received a hideous wsdl file containing 700+ operations and hundreds of data types. I first tried to generate the java client files in NetBeans using JAXB, but it complained about some duplicated type. So I tried with Eclipse (Galileo). It generated the Java client files using Axis2 without any problems. The files and organised in packages according to the wsdl file and everything was neat and compiled normally.

Didn’t want to give up on NetBeans so I tried to generate the Java client using JAX-RPC style. To do this I had to install the missing plugins, which I did. Then I started to generate the files, NetBeans seemed stuck – the GUI became unresponsive. After a good half an hour or more it came back and reported out of memory error. I realised that it had generated the java files but could not compile due to low memory. So I restarted it and tried to compile again, then it complained about a code being too large – to be fair this is the problem of wsimport which is used by NetBeans to generate the web services client files. So I manually split the big SerializerRegistry method (containing 9000+ lines and probably went over the 64k limit on method size) into two and recompiled. This time it worked. But the whole code editor remains sluggish.

Examining the memory usage of both Eclipse and NetBeans with their respective web services client project open, Eclipse occupies about 255M and NetBeans 520M.

My experience tells me that tooling wise Asix2 is better than JDK’s own web services tools; Eclipse Galileo is more memory efficient than NetBeans 6.8.

Nokia Lost the Plot

2010-04-11T11:19:00.001+10:00

Nokia has abandoned the 4-digit naming convention for its mobile phone models and adopted four series of N, E, C, X instead. The N series is supposed to be the top notch flag-ship products of Nokia with the latest and greatest technologies. The X series are supposed to be the device for social, entertainment and youth. However, the way Nokia allocates the features to their devices is quite messy and inconsistent.

The 5800 XM is supposed to be a great music player. However, it does not have the Home Media application although the machine is equipped with uPnP capability. Also, it allows you to configure SIP settings but there is no built-in SIP client! On the contrary, N95 has both features built-in.

I was ecstatic when I heard about Nokia freeing up its Ovi Map including voice turn by turn navigation since late January 2010. So I quickly went to Nokia’s official maps site and installed them on my N95. In order to install Ovi Map, it had to upgrad my N95’s software version and wiped out all my applications, history and address book during the process! So I had to spend days to re-patch my phone, re-install all the software and games, re-download and reload the map and voice files and restore from a 2-year-old address book backup… But hey the Ovi Map (v3.01 09wk44 b01) got installed and actually worked. But my joy was short lived. My expired trial license for navigation did not get renewed or removed. So when I tried to start navigation, Ovi Map will nag me about purchasing licenses and if I don’t it will quit navigation. So meanwhile I have to feel content with TomTom 6 and its 3-year-old Australian map.

Frustrated, I did some digging and found out that the ‘Free Forever’ maps is not compatible with my N95 – as it is not in the device list. Yet you can still download the software and maps for it – it is just not free! However, Nokia’s marketing machine does not tell you this. No wonder so many N95 users share this frustration.

A glimpse of hope remains for my 5800 XM as it is in the Ovi Map compatible devices list. To install the free version (v3.03 which only works on S60v3 FP2 and S60v5 and it is designed/optimised for touch screen user interface) I need to upgrade the phone’s software to 31.0.008 or greater. I have yet to do so as I want to make sure I back things up before the upgrade…

Using VoIP on Nokia

2010-04-04T11:21:00.001+10:00

For years my family had relied on calling cards for making ‘cheap’ overseas calls. But these days most of the calling card services are notoriously unreliable and those companies downright unscrupulous – they promise you thousands of minutes, but you’d be lucky if it lasts more than an hour. Sometimes the company will even wind up before you can use your newly purchased calling card!

The good news is that there are more nimble telco providers in Australia that provide value for money. The ones that are suitable for my family’s needs are Lebara and PennyTel. Lebara offers cheap rates to overseas calls especially to landlines but with a 25c flagfall charge. PennyTel on the other hand has better rates to overseas overall and does not have any flagfall charges.

I like PennyTel for its competitive rates and its usage of standard SIP technology as its VoIP platform. SIP is available on many Nokia phones. On my N95 it is as simple as following the instructions provided on PennyTel. Then you can call any landland numbers (need use either area code + number or country code + area code + number). Receiving calls to your PennyTel number also works well (calling from a PennyTel ATA device). However, I did find that calling other PennyTel numbers (either landline or account number) using its softphone or SIP clients on my mobile did not work – it either would not connect or gave me announcements…

Using SIP on XM 5800 is another story. Although the phone does have a place for you to configure SIP settings (Menu -> Settings -> Connectivity -> Admin. Settings -> SIP Settings) but there is no built-in SIP client software. Alas, XM 5800 is not in the table. So I installed Fring on my XM 5800 and the SIP configuration is even simpler – only 3 fields to fill.

On PC (Windows 7) I have tried SIP Communicator and X-Lite. SIP Communicator supports IM and VoIP from various social networks while X-Lite is specialised in SIP. Both worked well. But X-Lite seems more polished. I have used eyeBeam – the commercial version of X-Lite for work and quite liked it – it has hundreds of parameters you can tweak provided that you know your SIP and RTP well.

Whatever Happened to Programming?

2010-03-18T12:24:00.001+11:00

Mike Taylor wrote two insightful essays about programming a few days ago. Are we stuck in a tedious loop of gluing APIs together? What happened to the creative aspects of programming?

Whatever happened to programming part 1 talks about how the most enjoyable part of a new project occurs early on, when things are just taking shape and the most creative effort takes place. Whatever happened to programming redux is a response to many comments he received so far, with more insightful commentary.

I don’t think libraries or frameworks are boring to use. Actually I find some of them quite insightful and fun to use. Comparing to the old days (for me that was early 1990’s) when I had to spend time writing our own C libraries to implement data structures or do network communications, now I have heaps more than what I will ever need built-in in JRE or .NET or even ActionScript. The upside of these libraries are quality and productivity. The main complaint I have about frameworks is that it can limit your choice – see my gripe about GAE.

I have said before that the IT industry is modelled after real engineering disciplines like built environment. In built environment, when you want to build a house you will have different people with different expertise in their own area to be involved from concept to design to build and test. When an engineer looks at a design, he/she will choose the materials and the trades people will actually ‘glue’ them together. As a builder, you will not expect to make your own glass, bake your own bricks and tiles, grow your own grass, etc. All are readily available for you to pick and choose and ‘glue’ together.

Sure, if you are stranded on a deserted island you will have to start from raw materials to make tools and build simple structures assuming you know (or can teach yourself) how. But that does not have the quality and productivity that are required in a normal environment. Another big driving factor is that for most of us who write commercial applications (not tool smith) we need to serve a business need. IT does not exist without a business diver. In some cases, I felt bored about the frameworks that have been chosen by the company. That is why I do a lot of playing after work

Not Happy, GAE!

2010-03-08T12:23:00.001+11:00

Cloud Computing has been a latest buzz word recently and seemingly will be for the foreseeable future. I had registered for Google App Engine (GAE) a while ago but did not do anything with it until last week.

I downloaded the Google Plugin for Eclipse 3.5 (Galileo) and installed it without any problem. I then decided to port one of my web applications written in ZK 3.6.2 with Spring Security 3.0.2 in its simplest form.

I started by creating a GAE application in Eclipse using the Google plugin’s wizard. I then copied WebContent directory from my source project to the war directory of the target GAE project. I modified the deployment descriptors according to instructions from ZK small talk with the following additions:

appengine-web.xml – I added /**.xml in the static file exclude path list since in my application I generate .xml files to feed them to the XML/SwfGauge package.
applicationContext-security.xml – the Spring Security deployment descriptor. I had to add a URL pattern for “/” explicitly due to the different way of handling it by Tomcat (the server I used for developing the web app) and Jetty (the server bundled in Google Plugin).

The modified application then ran quite successfully as GAE application in Eclipse. Of course, anything that involves creating user thread were disallowed and any Java packages/classes that were not supported by GAE were also not allowed. I thought I could work around those problems later.

Then I deployed the application to the real GAE. The deployment process was error free. I was overjoyed and clicked on the URL for my application on the appspot.com. A HTTP 500 – Server error was returned without any explanation. I did a little fiddling and my first page was displayed if I remove Spring Security, but then all my included pages (I store all my pages in WEB-INF/pages directory) were not shown… I would need at least a few dozen of trial and error before I can pin-point the problem but my internet connection quota is running out fast! So I will call it quits for now.

While cloud computing gives you great promises it is also a shackle and limits your freedom of choice . Although GAE seems to be compatible with many frameworks, the degree of compatibility varies. For example, in the list ZK framework is COMPATIBLE with GAE, but if you look at all the features and packages that are included in ZK you will quickly realise that it is not true – server push requires separate thread and so does jFreeChart and some other bundled packages; captcha uses java.awt.Font (which is not allowed in GAE)… So you’d be better off sticking with GAE’s recommended technology stack – servlet/JSP/JSF + JPA/JDO + JAXB + XML, etc. This may be OK for new applications but presents a big obstacle for existing large applications which heavily rely on other frameworks.

Happy New Year of the Tiger!

2010-02-11T12:47:00.001+11:00

VPN Client for Windows 7

2010-01-30T10:44:00.001+11:00

I have to use Cisco VPN Client v4.6.* for work. It works fine on Windows XP. Recently, as one of my Windows XP laptops went kaput, I upgraded to Windows 7 and am loving it. However, the Cisco VPN client would not work anymore. I tried to download a newer version (v5.x) from Cisco, but for several months their online registration just would not work (hard to imagine such a giant company couldn’t get such a simple thing working). So I had to hunt for alternatives.

Now I am using ShrewSoft VPN Client. It is extremely simple and fast to install and configure. The whole experience from download, installation, configuration to use took less than 10 minutes. It supports importing of Cisco VPN connection profiles (.pcf files). It has good online documentation, and its free!

Flash vs. JavaFX

2010-01-26T11:53:00.001+11:00

I wrote the same eCards using JavaFX 1.0 in NetBeans and Flash 10 (with ActionScript 3) in CS4. The contents and algorithms of the eCards are the same but the outcome are quite different.

User Experience

The Flash version has 200 snow flakes and the JavaFX one has 30. The result is that Flash eCard is very smooth and the JavaFX one is jerky.
If your browser does not have the required Flash plug-in, you can still view the rest of the page; but JavaFX forces your browser to go to Sun to download the plugin and you cannot see the other contents of the page.
The first time you view the pages, the .swf file and .jar files will be downloaded to your computer respectively. The time taken to download them are not that much different with Flash version slightly shorter. However, if you view them afterwards, the cached Flash eCard will be played instantaneously without having to download again even after you restart the browser or computer. But the JavaFX version will be downloaded again every time.

Designer Experience

The CS4 environment is a lot easier to use for designers. You can do things graphically or programmatically. In NetBeans you have no choice but to write code for everything – even to manage timelines!
CS4 generates deployment code/files for HTML and AIR. It is really easy to deploy .swf files and the process is simple, robust and mature. I can’t say the same thing about Java Web Start, which is nowhere near production quality.
If you are like me, who wants to learn things from free web resources, then JavaFX is easier since there is only one way to do things – by writing code (this could be a bad thing for some). Most of the tutorials and sample codes work. But the different ActionScript releases are not backward-compatible. Many of the code samples were written in AS 1.0 or 2.0 (since they have been out there for many years) – they do not work in 3.0.

Related Post:

Rise of the Droid

2010-01-22T11:25:00.001+11:00

Traffic to this blog had been pretty flat in the last couple of months. It was not surprising considering the holiday season. However, I noticed an unusual surge of traffic in the last three weeks creating new record highs. I was a bit baffled. The only thing mildly interesting (to me) that I wrote after the New Year was about hacking LinkedIn API. I don't think many netizens are that interested in LinkedIn, let alone LinkedIn API.

A little digging in my custom reports in Google Analytics revealed that my No. 1 visited blog was Consumer Web Services from Android that I wrote in 2008 when the Android 0.9 SDK Beta was released. For about 12 months, that post was not among the top 5 list. Yet soon after Google announced its launch of Android phone people are are beefed up about development for Android platform. Judging from the developer community’s enthusiasm, it won’t take long before tons of Android applications to be available, helping Android’s market share. So I expect to see Android to catch up with iPhone in the next 12-24 months. It will be interesting to see how the multi-vendor strategy work out against Apple’s iPhone.

Kudos to Google!

Happy New Year 2010

2010-01-15T21:36:00.009+11:00

This is a Flash port of my eCard first written in JavaFX last year.

The snow fall script is stolen from http://r3dux.org/2010/01/actionscript-3-0-particle-systems-2-snow-effect/

The algorithm of the snow fall between the the JavaFX version and this one are pretty similar. The background photo was taken in Darling Harbour on Christmas eve 2009.

Related Post:

Happy New Year 2009

Hacking LinkedIn API

2010-01-02T15:31:00.009+11:00

Note [2011-03-24]: As promised, I have updated the source code. Now it works! Please see my other post which supercedes this one - Hacking LinkedIn API - Take 2.

Note [2010-03-12]:LinkedIn has changed the login page(s) so that the HTML scraping code below no longer works. However, I do believe that the approach still works. I will publish an update when I have some time after hours. Stay tuned.

I was quite excited to discover that LinkedIn was exposing APIs for 3rd-party applications to tap into its data. I downloaded the LinkedIn-J 0.1 SNAPSHOT-2009-12-30 which is a beta java wrapper of the APIs from Google Code and tried the example posted at LinkedIn Developer Network forum. The sample worked quite well as intended.

However, the paradigm of the oauth procedure which is adopted by LinkedIn APIs is convoluted and the LinkedIn authorisation process assumes your application is a web application. As part of the authorisation process, it forces the user to login directly to LinkedIn web page (shown below) to get a PIN and feed the PIN to the application to continue the process.

Once logged in successfully, a PIN is given in the next HTML page:

This is no good for machine-to-machine type of integration. I just want to retrieve LinkedIn data from the backend without forcing the user to be a LinkedIn member or login twice. So I decided to bypass this extra login.

Looking (view source) at the above login form it is a simple HTML FORM:

...
                          Email Address:                 
                 LinkedIn Password:                 


                 Access Duration:                                  


 or Return to MyApplication

I forgot my password             




                                                  
...

Note that the FORM has the following INPUT elements (including the hidden ones and the submit button):

email – use my login to LinkedIn
password – use my password to LinkedIn
duration – set to 0
access – set to –3
agree – set to true
extra – empty
authorize – set to Grant Access

We can easily populate this form and submit it in Java.

...
DataOutputStream dataOut;
      
        try {
            URL url = new URL(authUrl);
            //URL url = new URL("https://api.linkedin.com/uas/oauth/authorize");
            HttpURLConnection con = (HttpURLConnection)url.openConnection();
            con.setRequestMethod("POST");
            con.setUseCaches(false);
            con.setDoInput(true);
            con.setDoOutput(true);
            con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

            dataOut = new DataOutputStream(con.getOutputStream());

            dataOut.writeBytes("email=myname%40mycompany.com&password=mypassword&duration=0&access=-3&agree=true&extra=&authorize=Grant Access");
            dataOut.flush();
            dataOut.close();

            //SSLException thrown here if server certificate is invalid
            String returnedHtml=convertStreamToString(con.getInputStream());
            System.out.println(returnedHtml);
...

If successful, the returnedHTML will contain the PIN (as shown in screenshot above). So it is simply a matter of scraping that HTML string to get the 5-digit PIN and feed it to the next step of the authorisation process. The following is the modified Beta Java SignPost Sample Code to bypass the LinkedIn login page:

// LinkedIn SignPost Sample Code
// Adapted by Taylor Singletary
// from Twitter SignPost Sample Code ( http://oauth-signpost.googlecode.com/files/OAuthTwitterExample.zip )
// Tested and Functional with attached SignPost JAR
// YMMV

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;

import java.net.HttpURLConnection;
import java.net.URL;
import java.util.logging.Level;

import oauth.signpost.OAuth;
import oauth.signpost.OAuthConsumer;
import oauth.signpost.OAuthProvider;
import oauth.signpost.basic.DefaultOAuthConsumer;
import oauth.signpost.basic.DefaultOAuthProvider;
import oauth.signpost.signature.SignatureMethod;


public class Main {
    static public String getPin(String authUrl) {
        DataOutputStream dataOut;
      
        try {
            URL url = new URL(authUrl);
            //URL url = new URL("https://api.linkedin.com/uas/oauth/authorize");
            HttpURLConnection con = (HttpURLConnection)url.openConnection();
            con.setRequestMethod("POST");
            con.setUseCaches(false);
            con.setDoInput(true);
            con.setDoOutput(true);
            con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

            dataOut = new DataOutputStream(con.getOutputStream());

            dataOut.writeBytes("email=myname%40mycompany.com&password=mypassword&duration=0&access=-3&agree=true&extra=&authorize=Grant Access");
            dataOut.flush();
            dataOut.close();

            //SSLException thrown here if server certificate is invalid
            String returnedHtml=convertStreamToString(con.getInputStream());
            System.out.println(returnedHtml);
            /* extract the pin from the html string. the block looks like this
        <div class="content">
          You have successfully authorized MyApplication
          Please enter the following security code to enable full access

          <p align="center"><b>12345</b></p>
        </div>
             * It turns out that the whole html string only contains one 'p align="center"'
             * also it seems that the pin is always 5-digit long,
             * so we will just crudely detect that string and get the pin out.
             * A proper HTML parser should be used in a real application.
             */
            int i=returnedHtml.indexOf("center\"><b>");
            String pin = returnedHtml.substring(i+11, i+11+5);
            System.out.println("pin="+pin);
            return pin;
        } catch (IOException ex) {
            ex.printStackTrace();
            return null;
        }
    }
    public static void main(String[] args) throws Exception {
        OAuthConsumer consumer = new DefaultOAuthConsumer(
                "YourConsumerKey",
                "YourConsumerSecret",
                SignatureMethod.HMAC_SHA1);

        OAuthProvider provider = new DefaultOAuthProvider(consumer,
                "https://api.linkedin.com/uas/oauth/requestToken",
                "https://api.linkedin.com/uas/oauth/accessToken",
                "https://api.linkedin.com/uas/oauth/authorize");

        System.out.println("Fetching request token from LinkedIn...");

        // we do not support callbacks, thus pass OOB
        String authUrl = provider.retrieveRequestToken(OAuth.OUT_OF_BAND);

        System.out.println("Request token: " + consumer.getToken());
        System.out.println("Token secret: " + consumer.getTokenSecret());
/*
        System.out.println("Now visit:\n" + authUrl
                + "\n... and grant this app authorization");
        System.out.println("Enter the PIN code and hit ENTER when you're done:");

        BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
        String pin = br.readLine();
*/
        System.out.println("Now getting PIN from:\n" + authUrl);
      
        String pin = getPin(authUrl);
        System.out.println("Fetching access token from LinkedIn...");

        provider.retrieveAccessToken(pin);

        System.out.println("Access token: " + consumer.getToken());
        System.out.println("Token secret: " + consumer.getTokenSecret());

        URL url = new URL("http://api.linkedin.com/v1/people/~:(id,first-name,last-name,picture-url,headline)");
        HttpURLConnection request = (HttpURLConnection) url.openConnection();

        consumer.sign(request);

        System.out.println("Sending request to LinkedIn...");
        request.connect();
        String responseBody = convertStreamToString(request.getInputStream());

        System.out.println("Response: " + request.getResponseCode() + " "
                + request.getResponseMessage() + "\n\n" + responseBody);
       
    }

    // Stolen liberally from http://www.kodejava.org/examples/266.html
    public static String convertStreamToString(InputStream is) {
        /*
         * To convert the InputStream to String we use the BufferedReader.readLine()
         * method. We iterate until the BufferedReader return null which means
         * there's no more data to read. Each line will appended to a StringBuilder
         * and returned as String.
         */
        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        StringBuilder sb = new StringBuilder();

        String line = null;
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "\n");
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        return sb.toString();
    }
}

Happy New Year!

2009-12-31T22:47:00.001+11:00

As I look back the year that was 2009 I must say that I can’t complain much. I have achieved not too far from what I had expected. I can summarise my achievements in 2009 as following

Work – I still got my job!
After Work – I learnt and used the ZK framework quite extensively and got conducted in the ZK Hall of Fame and ZK Blogsphere.
Hobby – I have had much improvement in my piano playing compared to last year (when I just started learning). I rewarded myself with a brand new Kawai K3.
Family – I had always travelled extensively in my previous jobs. So 2009 was the year that I spent the most time with my family since I started a family!

Have a happy and prosperous 2010!

Related Posts:

Enterprise Architect

2009-12-23T11:39:00.001+11:00

In any IT forum, the topic of roles and skill sets of architects is a sure-fire flame bait. There are many roles with the title of ‘architect’ in the IT industry and they can be quite different. I summarise the IT architect jobs into the following categories:

glorified software engineers/business analysts – some employers and employment agencies alike beef up the job position to attract more experienced applicants; of course, some employees/applicants do the same to their profile to attract better pay.
domain focused solution designers – these architects/designers focus on specific domains of the IT space. The boundaries may vary depending on how you slice and dice the space: infrastructure, security, integration; service fulfilment, network performance, inventory management, etc.
enterprise wide architect – responsible for IT governance, IT strategy alignment with the business strategy. It overarches the various domains mentioned in point 2 above.

The Enterprise Architect falls into the 3rd category. Not all organisations have or need to have such a dedicated role – a travel agency with 3 staff may well outsource their IT operations and the role of EA to an external party. The role of EA is a product of a mature organisation (or mature org wannabees) that understands the importance of IT to their business. Wikipedia has a pretty detailed description of what EA is. It uses the analogy of city planner (the EA) and domain specific designers and engineers in the building industry. This is a very good analogy and one that I often use. After all, the very term of architect is borrowed from built environment discipline.

It is understandable that many people including IT professionals do not understand the roles of architects, especially EA. The main reason for this lack of understanding is that in many organisations the role of architect is shared with others – e.g. development team lead taking on the responsibility of system architect. When it comes to EA it becomes more illusive because not all companies have one. So it’s no wonder we see in various forums silly questions like should EA be doing coding (btw, my answer to this question is that EA should not do coding as his/her day job; but it certainly helps for EA to learn new technologies to better understand them – so coding afterhours is great for EAs).

The IT industry is very young and immature comparing to other disciplines liking medicine or built environment. As a result of reaching maturity, it is inevitable for the industry to become more and more specialised. Therefore, the roles and jobs become more and more fine-grained, focused and well-defined. No doubt, with IT becoming more mature, the roles of architect especially EA will become better defined and clearer to more people.

My Carbon Footprint

2009-12-15T16:26:00.001+11:00

As Copenhagen becomes the hotspot of global warming spat for the second week, it seems less likely anything substantial can be achieved by the summit. However, one positive effect it has is for sure – it has raised the awareness of the global warming issue across the world.

Just a few months ago there was a blackout in my house due to an unseasonal thunderstorm. I was home alone and suddenly felt so bored – without electricity there was no form of entertainment in the house! No TV, music, internet, game consoles and not even cooking! Whereas such blackouts were quite frequent where I lived during my childhood yet I quite enjoyed it for we got to have candles and play shadow puppets… So I decided to have a good old traditional form of entertainment by buying a piano.

Out of curiosity, I took part in calculating my carbon footprint using an Ecological Footprints calculator adopting Australian model. It turns out that it would take 2.4 planet Earths to support my lifestyle if everyone on Earth lived like me, which is below the Australian average of 3 according to WWF Australia.

However, if I factor in the fact that many of my consumption of resources are actually shared by my family, it only takes less than one Earth, which is ideal.

Looking at the result, about half of the footprint comes from food. I tried to recalculate it with total vegan, the result was actually 8% worse. So my love of meet is actually good for the planet :)

Another thing is that I fly a lot due to work. If I did not fly at all, my carbon footprint would reduce by about 8%. The good thing is that I don’t have to commute to work every day, which offsets my extra carbon emission.

There are many things people can do to improve the situation. I think it all bog down to 3 things: be frugal, share, recycle. These concepts are all familiar to IT professionals like myself because we have to create systems that are preferment and cheap at the same time, although we call them by different names like algorithm efficiency, resource-pooling, package reuse, etc. Sharing (e.g. taking public transport) and recycling have received a lot of public attention. But one thing that is often overlooked is frugality (even in the IT world as hardware becomes cheaper and cheaper). Frugality is a virtue in east Asian traditional cultures. However, due to the western influence and economic boom in the region this virtue is in danger of being lost. When everything is being labelled with and measured in monetary units, it is very easy to miss the actual impact of wastage.

The Wall

2009-11-11T01:14:00.007+11:00

My last visit to the Berlin Wall was in 1998 while sections of the wall was still standing and Checkpoint Charlie was a tourist attraction. Today, as the west celebrates the 20th aniversary of the fall of the Berlin Wall, the Chinese netizens are joking about the Great Firewall of China (or GFW for short). I used to be sceptical about censorship in China - sure they block a few sites, but who cares? The Chinese can still visit majority of the websites, right? My view changed during my visit to China last month and personal experience of the wrath of the GFW.

I first tried to access my blogger site and my picasa photo albums in an internet cafe in Tianjin. It turned out that they were not accessible. In fact, the whole blogger.com was inaccessible! I thought this might be unique to that internet cafe, or Tianjin city. But I was wrong.

After a 25-hour train ride, I arrived in the southern city of GuangZhou. I experienced the same problems! People told me that even for unblocked sites, certain contents (i.e. parts of the web page) of those sites can still be blocked.

Now I understand why my blogger site has so few hits from China. The few red dots shown on the ClustrMaps are probably the result of automated search engines on the outside of the GFW.

Curiously, MSN sites (blog, photos, file sharing, etc.) are not blocked by the GFW. Maybe Microsoft had better political connections than Google. Now I have to create new photo albums on MSN just for the Chinese!

Bulk Photo Scaler

2009-10-24T22:29:00.003+11:00

After my holiday I captured hundreds of photos. I want to upload some of them to my online photo album. Before uploading the photos I need to scale the photos to a smaller size to speed up the file transfer and save my internet bandwidth. A popular bulk image processor - BIMP can be found from Cerebral Synergy.

I have some pretty unique requirements to bulk scale these photos. They have been captured using different cameras - N95, XM5800, Canon, Olympus, Pentax, etc. - each having different image resolutions and therefore sizes. So I cannot just apply a single factor to all the pictures. Also, I have some panorama pictures shot with Panoman which are very wide. So I cannot apply a single size to all pictures - I need to keep the aspect ratio. I decided to write a little bulk image scaler myself.

The basic image resizer is pretty simple using Java:

...
 BufferedImage img=ImageIO.read(new File(fileName));
 if(img==null)
  return;
   
 int w=(int)(img.getWidth()*factor);
 int h=(int)(img.getHeight()*factor);
 String imageType=getFileSuffix(fileName);
 Image scaledImg=img.getScaledInstance(w, h, Image.SCALE_DEFAULT);
 BufferedImage bi=new BufferedImage(w, h, img.getType());
 bi.createGraphics().drawImage(scaledImg, 0, 0,null);
 ImageIO.write(bi, 
  imageType, 
  new File(newFileName));
...

It is not the most efficient way to do this but it is simple and works. To keep aspect ratio is simply a matter of using the new width and the original picture's aspect ratio to calculate the new height:

 double ratio=(double)img.getHeight() / img.getWidth();
 int h=(int) (ratio*w);

The full source code is PhotoScaler.java and ScaleParameters.java. It still needs some refactoring and cleaning up. The usage of the utility:

Usage: java com.laws.photo.PhotoScaler directory_name [-f scale_factor] | [-s w
h] | [-k w]
-f scale by factor
-s scale by size, i.e. width and height
-k keep aspect ratio, use w for width and calculate the height
e.g. java PhotoScaler c:\temp\photos -f 0.5
     java PhotoScaler /tmp/photos -s 800 600
     java PhotoScaler c:\temp\photos -k 800

Confession of a Piano Shopper

2009-09-22T12:41:00.007+10:00

I have been piano shopping in the last few weeks. During this time I have innundated myself with all sorts of information from all sources that I could find - dealers, community forums, pianists and educators. I have been working in the telco industry for many years and some of which were in the sales process. I found striking similarities between my piano shopping experience and the telco sales process (or perhaps sales in any industry). Many of the behaviour and psychology that I have been critical of also manifested in me and I have no intention of correcting them.

Wants vs. Needs

The purpose of buying a piano is two-fold: entertainment for me and education for my 7yo son. I am a self-taught beginner and have no plan of achieving any certification/qualification in music whatsoever. I just enjoy playing. Chances are that I will remain a beginner forever. My son is also a beginner but taught 'properly' in a piano school. He likes playing and needs a proper piano to practise on.

So for beginners like us, a new entry-level(110cm - 118cm) acoustic or a digital piano would be sufficient. However, I don't feel content to have the basic model. I want to have a 'professional' (>120cm) one to start with. Well, my excuse for having a professional model is that there are two players at home and I don't want to upgrade anytime soon. So I might as well have the room for potential growth. But does this justify doubling the price?

Bag a Bargain

So I need to squeeze the vendors to drop the price as much as I can. Normally (in Australia), the RRP of any new piano is inflated by at least 20%. So without any effort, I can get the price down by 20%. Anything below that I will have to work on it. The easiest way is to have 2 vendors out-bid each other.

But what about the value of the piano - the materials, the labor, the quality, the beauty and the enjoyment we can get out of it? Are they worth a few thousand dollors? My opion is a resounding 'yes'. A piece of furniture or even a mobile phone can cost thousands of dollors these days. However, being a buyer, I cannot help exercising my power and try to squeeze that last few dollors, an adjustable stool or an extra tuning from the vendor. Of course, to negotiate effectively, I need to equip myself with knowledge about pianos and the piano market.

Knowledge is Power?

Knowing your products definitly helps in negotiation. I have learnt so much about pianos from all sort of sources in the last few days - information as well as misinformation. I had narrowed down my choices to Yamaha T121 and Kawai K3. A Yamaha dealer told me that the new T121 are fully made in Japan, if anyone tells you it's made in Taiwan don't believe them because the Taiwanese factory had been closed down a few years ago. When I checked online I did find some comments about the T121 being partially made in Taiwan, but all those comments are in or before year 2006. Also, you have to take any online information with a grain of salt (the best online info about piano I found so far is Piano World Forum) When I visited a Kawai dealer, he told me again that the T121 was made in Taiwan and that is why its price is lower. He also sited the The Piano Book (which is an authoritative book on buying pianos) and true enough, the book said so too! When I checked the date of the book, it was last edited in year 2000. So I know in this case, who was giving me the true information.

Conversely, I quite like the Millennium III Action (by using carbon-fibre for certain parts of the action) invented by Kawai, which can increase play speed by up to 16% compared to traditional wood action. But when coming from a Yamaha dealer's mouth, it totally changed taste. He argued that 'the instrument needs to breath just like leather shoes vs. synthetic leather shoes... have you ever seen any musical instrument made from plastic?... carbon-fibre may be good for making spaceships or boats, but that does not mean it's appropriate for piano... if plastic is so good, why doesn't Steinway use it...' Looking at input from both sides it is very clear to me that the MIII does have its advantages and the Yamaha dealer's argument does not hold water. Both Yamaha and Kawai are reputable big piano makers. However, Yamaha is no.1 player in the market - 80% of pianos used in musical institutions in Japan are Yamaha. So naturally, as a no. 2 player, Kawai has to work harder by employing new technologies to achieve better results and lower cost. Having said all that, does the 16% faster response mean anything to me? I would say 80% chance 'no' since I will never reach the skills level required to enjoy the benefit; maybe better odds for my son.

Does all these product knowledge help when choosing a piano? A real player will tell you 'no'. Piano appreciation is very personal. The only ways to pick one are to listen, look and play. All the new technologies, great designs, country of origin and fine-grained spruce from Alaska will not help if you don't like the sound or touch of the overall product.

Apples and Oranges

Having considered the old and new; the Japanese brands, Korean brands and some German brands I have narrowed down to Kawai K3 and Yamaha T121. The short listing process is extremely painful because many of the pianos that I excluded were very good ones but out of my budget. It is also unfair to directly compare most of the models side by side because they were not made to be at the same level. For example, although all models have the same height, the T121, K3 and U1 are very different products targetting different market segments. The specifications for T121 and U1 on Yamaha's web site show almost identical data, yet U1 is almost double the price of T121 and many players swear that it has better sound. It just does not make sense to claim which one is 'better' especially when it's all about personal experience.

Conclusion

With K3 on top of my list and T121 as a backup, I am going to drive a hard bargain after my holiday. Hopefully, I will have a brand new K3 in my living room in a few weeks time.

GPS Navigators on Symbian

2009-08-06T17:26:00.002+10:00

As soon as Nokia XpressMusic 5800 came out in Australia, my wife conveniently lost her HTC touch smartphone. So I bought the XM 5800 for her from Bangkok (and it turned out to be genuine ). Then on the same day she conveniently found her HTC in her friend's car...

So I was charged to install all the must-have software onto the new phone. Since both XM 5800 and my old N95 support GPS, a GPS navigator became the top of the list. I have tried Garmine Mobile XT, Route 66, TomTom and Nokia Map using my N95. I quickly dismissed Garmin since it refused to start the trial period claiming some error requesting the server over the internet (and I am sure my wifi setting was correct and it did get through the internet to attempt to connect to the Garmin server).

I have been using R66 Mobile 7 and then 8 for a while until I successfully installed TomTom v6.02 on my N95. The R66 maps simply don't look good comparing to TT and Nokia Map.

So now I am using TT and Nokia Ovi Map (v3.01) at the same time.

Comparing to Nokia Map, TT is better at navigation. As soon as you type a letter, it intelligently prompts a list of names. When I typed Dolls Point, it automatically points to the beach which is exactly what I wanted. The lane guidance and camera alert work perfectly.

The disadvantage of TT comparing to Nokia Map is in its map browsing capability - perhaps rightly so, considering the fact that TT is a navigator, not a map system.

Nokia Map shines in its usability - you can easily download and configure maps and voice packs in different languages. Its maps are quite pretty and detailed. Here is a screenshot of Nokia Map showing where I am stranded (due to mechanical failure of Royal Brunei Airline flight).

However, the version that I am using does not seem to include camera alert or lane guidance.

When it comes to the XM 5800, there is not much choice because it is running S60 v5 and not equipped with any keyboard. The only choice on XM 5800 for me is Nokia Map.

Using XML/SWF Gauge

2009-07-26T14:47:00.015+10:00

I have been looking for flash widgets to show data in a dashboard. The XML/SWF Gauge has become my best choice so far. It only requires one gauge.swf file which takes in a XML configuration file to instruct it what and how to draw the gauge. I used it to implement a Radar View as part of my dashboard.

I like the simplicity of the XML/SWF Gauge when using it. Yet it generates versatile and visually pleasing results. It also supports dynamic data display and scheduled update/refresh.

Since I used it in a ZK web application, naturally I used ZK to generate the input XML file dynamically. By default XML files are not passed to the ZK Loader to handle. To force it to be passed to ZK Loader I had to add a servlet mapping in web.xml file:


   zkLoader
   *.xml

I can use the same gauge.swf to generate all sorts of views - radar, meter, dial ... all I needed is to generate the appropriate XML file to feed the gauge.swf as shown in the following ZUL file segment.

...


        // ...


 if (view.getName().equals(event.getData())) {
  // recalculate view attributes...
  // refresh:
  gaugeFlash.invalidate();
 }

I do have the following complaints on XML/SWF Gauge:

It does not support concurrent/nested animation.
It does not support event-driven updates (vs. scheduled updates).

If you have come across any good flash gauge packages, please leave a comment.

Of Twitter, Clouds and Google Goo

2009-07-01T23:05:00.003+10:00

I have always thought Twitter is a time waster. However, I have also noticed that there are many companies and individuals using it as a marketing channel to reach the global mass, not to mention the propaganda tool as witnessed in the recent Iranian demonstrations. So instead of subscribing to RSS feeds, following tweets is now the "in" thing to do.

Using a social network for commercial and political gains is not new. We have seen it on Facebook and 2nd Life. However, to use tweets as an input for trading decision sounds dubious. Much of the twitter tweets are just noise and even garbage.

Today, out of curiosity I subscribed to Twitter. Within minutes of me opening my new Twitter account, I already got a follower. To be honest, I was pleasantly surprised and even flattered. Yet when I checked, it turned out to be a prostitute or cyber-pimp pushing some porn site . So to make use of tweets for trading, the system will have to identify which users to follow and filter out the fake, malicious and manipulating users - much like virus scanners rely on their virus database. This is a time-consuming and even labor intensive heuristic process relying on large volume of data. Then it has to filter and analyse the millions of messages per day to extract the useful information.

Let's put aside the ethical issues behind the practice of 'trading on rumors'. To say that a machine can determine market sentiment by reading tweets is at best an overstatement. Even human beings have trouble reading the sentiment in cyberspace, and that is why people have to add all sorts of smileys, emoticons and internet etiquette to assist the reader of the message. Also, the same words can have drastically different intentions and reactions based on different cultural, religional and circumstantial backgrounds. The idea of trawling through the internet to extract marketing information is not new. People have been attempting it on RSS feeds, newsgroups, user forums, etc. for a few years now. However, they are very focused/targetted on certain types of contents and are not of realtime nature -certainly not as ambitious as making trading decisions in real-time.

To apply complex fuzzy logic algorithms on large amount of (current as well as historical statistical) data is very CPU-, memory- and data-intensive. Such jobs are best suited for cloud-computing, which many big players are pushing - Sun, Microsoft, Amazon and Google. A couple of days ago I stumbled upon some cloud-computing PR articles and interviews and found this one - 谷雪梅谈云计算. I realised that the Google chief engineer interviewed in that video was my high school classmate. We affectionately called her 'Goo' back then. I guess now I have to call her the 'Goo of Google'. In that interview, a question was asked about how Google makes money. Well, I believe in 'Knowledge is power' and more so in the information age and that is how Google makes money. Comparing to the new comers (e.g. Bing, WolframAlpha) Google's search algorithm is quite lazy and unsophisticated - it relies on external links or the more you pay the higher the position in the list, yet it has accumulated vast amount of historical data and trained its systems to give better results. So Google has now taken the steps further to sell the infrastructure services and technologies such as GWT, cloud computing, Chrome and Google Wave. It seems Google has better things to do than to recycle the Twitter garbage - for now.

JAXB Custom Data Binding

2009-06-22T13:16:00.009+10:00

In a previous post I experimented consuming WCF web services using various Java WS frameworks and tools. As pointed out by Alex, the one that I missed out was wsimport which is bundled as part of JSE6.

Like many other tools, it supports both Ant task and command line interface (CLI). The CLI for wsimport is quite simple - in my case I generated the source code and client stub library like so:

D:\Program Files\Java\jdk1.6.0_11\bin>wsimport -d /temp/generated -s /temp/gensrc -keep http://localhost/PromoService.svc?wsdl
parsing WSDL...


generating code...

D:\Program Files\Java\jdk1.6.0_11\bin>

All the rest is similar to the results of IntelliJ shown in my previous post. There are two problems with the generated PromoInfo.java which is a data/value object:

string fields are generated as JAXBElement<String>
dateTime fields are generated as XMLGregorianCalendar

I want to use core java data types on the data objects so that they can be easily integrated with other frameworks without having to do conversion. Examining my schema (on http://localhost/PromoService.svc?xsd=xsd2) the PromoInfo complex type is defined as

It is obvious that the xs:string and xs:dateTime were not converted into the desired java types. To solve my problems I specified customised JAXB binding rules in an external file - custombinding.xml like so

The attribute generateElementProperty="false" on line 2 tells wsimport not to generate JAXBElement but to generate native java data types instead.

The javaType element on line 3 defines the binding between "xs:dateTime" and "java.util.Date" because by default xml dateTime binds to javax.xml.datatype.XMLGregorianCalendar as shown here.

Once the binding is defined, rerunning the wsimport tool with the -b switch will produce the desired output:

D:\Program Files\Java\jdk1.6.0_11\bin>wsimport -d /temp/generated -s /temp/gensrc -b /temp/custombinding.xml http://localhost/PromoService.svc?wsdl
parsing WSDL...


generating code...
Note: D:\temp\gensrc\org\w3\_2001\xmlschema\Adapter1.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

This time the generated PromoInfo.java looks much better:

...
public class PromoInfo {

    @XmlElement(name = "PromoDateTime", type = String.class)
    @XmlJavaTypeAdapter(Adapter1 .class)
    @XmlSchemaType(name = "dateTime")
    protected Date promoDateTime;
    @XmlElement(name = "PromoDescription", nillable = true)
    protected String promoDescription;
    @XmlElement(name = "PromoName", nillable = true)
    protected String promoName;
    @XmlElement(name = "PromoVenue", nillable = true)
    protected String promoVenue;
...

A Racist Job Market

2009-06-18T11:33:00.002+10:00

My friend forwarded me an interesting article today - Ethnic names hinder job seeking: report. I have also seen similar reports on Channel 7 two days in a row.

The result of the research shows that job applicants with Anglo-Saxon names received more calls than those with Indigenous, Chinese, Middle Eastern or Italian names. Whether the business owners/employment agencies/HR people are consciously racist or not is debatable. However, it is proven that collectively the job market in Australia is racist.

The rationale behind this phenomenon is complex and multifaceted. Let's assume people are not intentionally or consciously racist for a while. There could be several reasons behind it:

Typecasting - Once someone sees a 'foreign' name, he/she automatically assign a profile to it. Of course, there could be good and bad aspects of this profile depending on the reviewer's personal experiences and belief. The biggest disadvantage of such profiles for foreign names is perhaps the assumption that they do not speak English well. Hence we see many job ads have requirements like "excellent communication skills" to deter non-English speakers from even applying. True, Australia is a country of immigrants and there are many new immigrants who cannot speak English well. However, majority of them are quick and keen learners, especially those who are actively seeking jobs. And let's not forget the vast number of 2nd-, 3rd-... generations of immigrants who identify themselves as true-blue Aussies.

"Not Made Here" syndrome - People are more comfortable with what they are already familiar with. A foreign name automatically triggers off a level of fear and set off the defense mechanism subconsciously (or unconsciously if you prefer Freudism). Many people tend to prefer interacting with people with similar backgrounds and interests forgetting the advantages of diversity, especially in a work environment. There are a large number of small businesses in Australia which have only a handful of employees in the workplace. Hiring people with the same traits/profile that everyone else is comfortable with becomes an important criterion.

Media - The mass media is the most powerful brainwashing machine in the world and often, by criticising their countries of origin the media cast negative lights on the ethnic minorities who live in Australia.

Although we have been told not to judge a book by its cover, people cannot easily shake off the racial prejudice that have been intrinsically wired into our brain and heart as a result of millions of years of evolution. To break down the racial barrier, we have to actively and consciously broaden our horizon; seek more interactions with all walks of life; treat individuals as individuals; do upon others that which you will have them do onto you.

One Year On

2009-06-11T11:40:00.008+10:00

I started this blog one year ago in June 2008. I have been using Google Analytics to track the site. Here are some visitor (mostly from IT and development communities) trends collected in the last year or so.

Browsers used:

OS used:

Browser + OS used:

Java support:

As of June 2009 the top 5 visited posts on this blog site are:

Consuming WCF Web Service Using Java Client, July 2008
Consuming Web Services from Android, August 2008
IE8 + Outlook Web Access = Problems, May 2009
WPF Splash with ProgressBar, July 2008
How To Add Unicode Fonts to N95, September 2008

There are a couple of trends worth noticing:

Firefox has a bigger market share than IE among the technical users and rightly so. It will be interesting to see how Chrome will measure up against the top two. Chrome has certainly great momentum considering it has only been launched 9 months ago and already in 3rd place.
Java support is not as ubiquitous as I first thought. I wouldn't develop my next RIA business application on JavaFX any time soon. So right now I will stick to ZK (and maybe SmartSWT) and wait for HTML5 to take over.
It only took less than a month for the IE8 + Outlook Web Access = Problems post to claim the no. 3 spot. It just shows how bad IE8 is.
The post on number 5 spot has far more comments than the rest. It proves once again that non-technical people are far more social

IE8 + AVG = Problem?

2009-05-22T10:44:00.001+10:00

My IE8 fails to connect to any site from time to time. This happens sporadically and on certain new tabs only. After trawling through the suggestions in this discussion group, I found that the following solutions from fufufufu worked for me.

Work around: if one tab does not connect, try open another one. Some tabs will eventually work.
Solution: disable my AVG Anti-Virus Free (v8.5)'s Link Scanner feature (done in AVG's GUI client)

Interestingly, none of these problems with IE had ever occurred to my trusted FireFox.

IE8 + Outlook Web Access = Problems

2009-05-19T10:02:00.001+10:00

A couple of days ago I upgraded my old IE6 to IE8 because IE6 refused to display one of my ZK pages and spat out an error message box.

As I had to use Outlook Web Access (OWA) daily (which is the only reason why I still have IE on my machine), I ran into trouble straight away after the upgrade:

Any editing screens in OWA - e.g. editing a mail was incredibly slow. This was fixed by clicking the Options view in OWA and scroll down to E-mail Security section and click Re-install button. This installed the latest S/MIME ActiveX.
When downloading Office 2007 file types (docx, xlsx, pptx, etc.), OWA attempts to download them as .zip files. This was fixed by adding my OWA web site URL to the Trusted sites list in IE8 (Internet Options -> Security -> Trusted sites) as suggested here.
Cannot send or reply any more! - After working for a couple of days, OWA played up again: when replying mail I got Error: ‎(0x8000ffff)‎: Catastrophic failure; when sending new mail, I got error messagebox saying access denied... Digging through Google, others have experienced the same problem but no solutions. :( The workaround is to uninstall the OWA S/MIME component (as shown in point 1 above)!

Now I know what makes dogs run around and around chasing their own tails - it's their breakfast!

Server Push + Event in ZK

2009-05-14T22:12:00.007+10:00

I recently implemented a dashboard type of web application in ZK to display a collection of KPI metrics data using various views, for example

Dashboard view - by using some flash dial/meter widgets
Tree view - see screenshot below
Google map view - showing geographical metrics on the map

I want to use ZK's server push feature to dynamically update the views only when the corresponding metric value has been changed and detected from the back end/server side. The examples that are available from ZK small talks invariably pass the widget(s) to be updated to the worker thread and have them updated by the server side. This approach does not quite work in my application for the following reasons:

The various views (see screenshot above) are opened/closed by the user dynamically. Therefore, I don't know which widgets are visible and should be updated;
There are too many widgets to be updated - there could be dozens or hundreds of metrics displayed in the view. So one approach could be to pass the whole view to the server side and have it figure out which widgets to update. I feel that the server side shouldn't be bothered with such a responsibility and the shear number/volume of the widgets involved in the view could render this infeasible.

So I took an alternative approach by combining the server push and event posting that are available in ZK framework.

The main page (main.zul file) contains the code to turn on and off server push as part of its zscript:

void startServerPush() {
 if(!desktop.isServerPushEnabled()){
     desktop.enableServerPush(true);
 }
 MetricsUpdateThread mut = new MetricsUpdateThread(user, desktop);
 mut.start();
}

void endServerPush() {
 if(desktop.isServerPushEnabled()){
     desktop.enableServerPush(false);
 }
}

The MetricsUpdateThread class is shown below:

public class MetricsUpdateThread extends Thread {
 private static final int DELAY_TIME=2000; // 2 second.
 private Desktop _desktop;
 private boolean _ceased;
 
 public MetricsUpdateThread(Desktop desktop) {
  this._desktop=desktop;
 }
 public void updateChangedMetrics() {
  HashSet<Metric> metrics=new HashSet<Metric>();
  // find all changed metrics and put them in the metrics Set
  ...
  if(metrics.size()>0)
    Events.postEvent(new Event("onMetricChanged", null, metrics));
 }
 
 public void run() {
  if (!_desktop.isServerPushEnabled())
   _desktop.enableServerPush(true);
  try {
   while (!_ceased) {
    Executions.activate(_desktop);
    try {
     updateChangedMetrics();
    } finally {
     Executions.deactivate(_desktop);
    }
    Threads.sleep(DELAY_TIME); // Update delay time
   }
  } catch (DesktopUnavailableException due) {
   //System.out.println("Browser exited.");
  } catch (InterruptedException ex) {
   //System.out.println("Server push interrupted.");
  } catch (IllegalStateException ie) {
   //System.out.println("Server push ends.");
  } finally {
   if (_desktop.isServerPushEnabled()) {
    _desktop.enableServerPush(false);
    //System.out.println("Server push disabled.");
   }
  }
 }
}

Notice the Events.postEvent() in the updateChangedMetrics() method, which broadcasts an event if there are any changed metrics. These events are then handled by the corresponding view's zscript. For example, the treeview above has the event handler at its root component like so:

...


 metrics = event.getData();
 for(Metric metric : metrics) {
  tcMovement = win.getFellowIfAny(metric.getId()+"_move");
  if(tcMovement!=null) {
   tcMovement.setImage(Util.constructMovementImage(metric));
    
   tcValue = win.getFellow(metric.getId()+"_value");
   tcValue.setLabel(metric.getValueAsString(metric.getCurrentValue()));
  }
 }

...

...

This approach of combining Server Push and Event broadcasting achieves the effect that I wanted. However, I can't help feeling that it is a bit complicated. So I wonder whether there is a better, simpler or more standard approach to achieve the same user experience in ZK.

Master-Detail View in ZK

2009-04-26T22:43:00.007+10:00

In my latest ZK application, I implemented a typical master-detail view: a split bar in the middle; a tree view on the left and a table/grid on the right. Whenever an item in the treeview is clicked, the grid on the right-hand-side is updated to display the details of the clicked treeview item. The only thing 'special' about this application is that I factored the master (treeview) and details (grid) views as separate files and the main window/page uses <include> to put them together:

<?page title="" contentType="text/html;charset=UTF-8"?>
<zk>
<window border="none" width="100%" height="100%">
 <hbox spacing="0" width="100%" height="100%">
  <include src="analytical.zul"/>
  <splitter collapse="before"/>
  <include src="metricDetails.zul"/>
 </hbox>
</window>
</zk>

The <include> complicates things slightly: the master view which is firing the event has no visibility of who is going to handle the event. Therefore, the event's target field is set to null so that the event is broadcast to all root-level components on the page - including the included details view.

The event sending code is shown below. Note that the treeview of the master view is built using model and renderers and the event sending code is embedded in the renderer.

public class MetricTreeitemRenderer implements TreeitemRenderer {

 @Override
 public void render(Treeitem item, Object data) throws Exception {
  SimpleTreeNode t = (SimpleTreeNode)data;
  Metric metric = (Metric)t.getData();
  ... // construct Treecells
  Treerow tr = null;
  ... // construct the Treerow
  
  final Event e=new Event("onMetricClick", null, metric);
  tr.addEventListener(Events.ON_CLICK, new EventListener(){

   @Override
   public void onEvent(Event arg0) throws Exception {
    Events.postEvent(e);
   }
   
  });
 }
}

The event handling side is part of the included details view file:

<?page title="Metric Details" contentType="text/html;charset=UTF-8"?>
<?init class="org.zkoss.zkplus.databind.AnnotateDataBinderInit" ?>
<zk>
<window id="win" title="Metric Details" border="none" width="100%" height="100%">
<attribute name="onMetricClick">
... handle the event by populating the grid 
</attribute>
<grid id="grid" vflex="true" width="100%" height="100%">

  <columns sizable="true">
   <column label=""/>
   <column label=""/>
  </columns>
  <rows>
...
  </rows>
 </grid>
</window>
</zk>

Note that even though the <window> is inside of the <zk> tag, it is still the root component (since it has no parent component in the .zul file). Also, although the .zul file has been included in the main file, it seems that it still has its own life cycle and its root component is unchanged.

Error: Not enough storage...

2009-04-09T22:34:00.004+10:00

For the last few days I was having problems on my IE6. Every time I use Outlook WebMail to open a modal window - e.g. bringing up the address book, I would be greeted with an error message box saying 'not enough storage is available to complete this operation'. I tolerated the problem for all this time because I could still send mails and attach files, etc. But today, I had to use IE to create a travel request for my trip to Delhi and this problem is stopping me from filling in the request form (and the travel request web application does not work on Firefox). So I was stuck and the clock is ticking for I need to finish the travel request today.

After some digging from Google I found the solution: it was caused by the User Agent string being more than 260 characters long. Huh? I was gobsmacked when I saw this. But it really works: after I deleted all entries in the 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent\Post Platform', (don't forget to close all instances of IE after that) my problem disappeared. Then I did some experiment to figure out what was happening.

First of all, it is well known among web designers/programmers that different browsers render the pages differently although HTML and CSS have been standardised for many years. Hence, the User Agent string is checked by many web applications to find out what browser is being used. In IE, the user agent can be retrieved using a simple javascript function: navigator.userAgent. The way Microsoft IE gets the user agent string is by appending all the values under the above registry entry. You can see this by creating some new values under the above registry entry. You may get a user agent string like: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; New Value #1; lsfjsafj sf;dsfj sakf lksjf salkfjas lkfjlksfj safkjsa lkfsajf ;lksjflksajf lksa;fj salkfjsa kfdsalk). As more values being inserted in the entry, the user agent string will get longer and longer, until it exceeds 260 characters, then it will simply return: Mozilla/4.0 (compatible; MSIE 6.0), obviously some default value without appending any values from the Windows registry.

That would have been fine if there were no other side effects. Unfortunately, maybe some other Microsoft jscript libraries have not catered for this and produce the 'not enough storage...' problem.

In his blog, James Thompson blamed toolbars and spywares for the extra values in the registry entry. In my case, all the values in the registry entry were from Microsoft - it looks like everytime I upgraded .NET, a new value was created:

So if you have been doggedly upgrading .NET all the way from 1.0 to 3.5 like me, then you would be experiencing the same problem as well. Looks like Microsoft does a more thorough job than those spyware vendors.

Fontrouter Open Sourced

2009-02-17T14:07:00.004+11:00

I learned that Fontrouter has been open sourced under Apache License 2.0 two days ago. The download site has been moved to http://code.google.com/p/fontrouter/. This is both a good news and bad news.

The good news is that there may be more people or organisations to work on the project, hopefully, giving it more resources that it deserves.

The bad news is that oasisfeng may not provide as much support to the general public as he used to, which had been excellent in the past. The once very active Fontrouter Forum is now in ruins - real support activities have dwindled and spams are rampant.

I sincerely hope that the Fontrouter project will continue and grow.

Related posts:

How To Add Unicode Fonts to N95

SyntaxHilighter 2

2009-02-12T16:06:00.002+11:00

I have been using SyntaxHighlighter 1.5 on my blog site from day one. Since I don't have my own file server to host the SyntaxHighlighter files, I have been borrowing the URLs from java.dzone .

A few days ago, I noticed that all my code snippets in my blogs were not being processed. I first thought it was a temporary glitch from blogger.com or java.dzone. Today, I realised that SyntaxHighlighter has released v2.0 and obviously java.dzone had upgraded promptly. So I upgraded my blog site to use v2.0 as well by following the official instructions. I also noticed that the author of SyntaxHighlighter is kind enough to host the files from the package which is good for bloggers like me (who do not have their own file servers on the internet).

Facebook API in Erlang

2009-02-11T16:20:00.017+11:00

I have heard of Facebook APIs and Facebook applications so today I decided to check out what they can do.

Based on documentation on Facebook, an Application is something that is hosted on an external server (such as your ISP or company web/application server) and can be invoked/accessed from a Facebook page. This seems very disappointing - I first thought a Facebook Application was something you can create using Facebook widgets and be hosted on Facebook servers. This model is not appealing to me, then again, I am not an advertiser.

Nevertheless, I decided to give Facebook API a try. Looking through the list of supported languages, Java was discontinued by Facebook (what do you expect from a PHP shop?!) So the natural choice is Erlang using Erlang2Facebook client library. Before using the library, I had to prepare my environment:

Install the latest Erlang - OTP R12B (v5.6.5). My old OTP R11B would not work because the library uses some new functions from the standard library (e.g. decode_packet).
Download mochiweb_util.erl and mochijson2.erl as they are used by the erlang_facebook library.
Download the erlang_facebook.erl

The Facebook APIs are RESTful services which support both GET and POST methods. Most API calls require a signature input parameter which is a MD5 hash of all the input parameters concatenated alphabetically. This is explained in Facebook Developers Wiki. Also, many API calls require the uid or session_key as input parameters. It is a bit convoluted to get the session key:

Create your own Application by following the instructions from Facebook so that you will get your own API Key, Application Secret, etc.
To get a session_key value, you have to get auth_token by accessing the URL: http://www.facebook.com/login.php?api_key=1f5f..., which will forward to your application's host's URL with an input parameter for the auth_key. In my case, it forwards to the URL: http://romenlaw.blogspot.com/?auth_token=e1761... So now I have an auth_token.
Once I have the auth_token, I can call facebook.auth.getSession API to get the session_key and uid

In Erlang, this is shown below:

Erlang (BEAM) emulator version 5.6.5 [smp:2] [async-threads:0]

Eshell V5.6.5  (abort with ^G)
1> c("/projects/facebook_erl/erlang_facebook.erl").
{ok,erlang_facebook}
2> 
2> c("/projects/facebook_erl/mochiweb_util.erl").
{ok,mochiweb_util}
3> 
3> c("/projects/facebook_erl/mochijson2.erl").
{ok,mochijson2}
4> 
4> [ApiKey, Secret, AppId]=["1fef...", "99f2...", "41..."].
...
9> erlang_facebook:custom(ApiKey, Secret, "facebook.auth.getSession", [{"auth_token", "c92b9...this is the auth_token copied from step 2 above"}]).
{struct,[{<<"session_key">>,
          <<"2.Ryk_v_nVtG...">>},
         {<<"uid">>,109...},
         {<<"expires">>,123...}]}
...
19> erlang_facebook:custom(ApiKey, Secret, "facebook.users.getLoggedInUser", [{"session_key", SessionKey}]).
109...(same as the uid returned by getSession call above)
28> [Fid]=erlang_facebook:custom(ApiKey, Secret, "facebook.friends.get", [{"uid", "1092201851"}]).
[598...]
31> erlang_facebook:custom(ApiKey, Secret, "facebook.friends.get", [{"uid", "598..."}]).            
{struct,[{<<"error_code">>,10},
         {<<"error_msg">>,
          <<"Application does not have permission for this action">>},
         {<<"request_args">>,
...
36> erlang_facebook:custom(ApiKey, Secret, "facebook.users.getInfo", [{"uids", "109..."},{"fields", "uid, first_name, last_name, name, sex, birthday, affiliations, locale, profile_url, proxied_email"}]).
[{struct,[{<<"affiliations">>,
           [{struct,[{<<"nid">>,67...},
                     {<<"name">>,<<"Australia">>},
                     {<<"type">>,<<"region">>},
                     {<<"status">>,<<>>},
                     {<<"year">>,0}]}]},
          {<<"birthday">>,null},
          {<<"first_name">>,<<"Romen">>},
          {<<"last_name">>,<<"Law">>},
          {<<"name">>,<<"Romen Law">>},
          {<<"sex">>,null},
          {<<"uid">>,109...},
          {<<"locale">>,<<"en_US">>},
          {<<"profile_url">>,
           <<"http://www.facebook.com/people/Romen-Law/109...">>},
          {<<"proxied_email">>,
           <<"apps+419..."...>>}]}]

Happy 牛 Year!

2009-01-23T14:53:00.005+11:00

Happy New Year of the Ox!

Source code available here.

JavaFX MediaPlayer Memory Leak?

2009-01-21T22:00:00.006+11:00

I wrote an e-card using JavaFX 1.0 to celebrate the upcoming Chinese New Year. It's a typical little multimedia applet with some animation, music and sound effect - supposedly perfect for JavaFX. However, I found that the memory usage is steadily climbing even when there is no activity (animation) happening on the canvas. I refactored, double-checked, triple-checked my source code several times to make sure that there was no unnecessary object creation and to reuse objects (by changing the opacity) every time - but to no avail.

Then I did a little experiment and found out that the number-one culprit could be the javafx.scene.media.MediaPlayer. The test program has a MediaPlayer and blank canvas with two buttons - the Start button to play the media/music; and the Stop button to stop the music. The source code for this simple test is shown below.

package testjavafx;

import javafx.animation.Interpolator;
import javafx.animation.KeyFrame;
import javafx.animation.Timeline;
import javafx.ext.swing.SwingButton;
import javafx.scene.media.Media;
import javafx.scene.media.MediaPlayer;
import javafx.scene.media.MediaView;
import javafx.scene.Scene;
import javafx.scene.text.Font;
import javafx.scene.text.Text;
import javafx.stage.Stage;

var m=MediaPlayer {
    autoPlay: false
    repeatCount: MediaPlayer.REPEAT_FOREVER
    media: Media {
        source: "{__DIR__}bubugao.mp3"
    }
};
Stage {
    title: "Application title"
    width: 250
    height: 250
    scene: Scene {
        content: [
            MediaView {
                mediaPlayer: m
            }
            SwingButton {
                translateY:100
                text: "Start"
                action: function() {  
                    m.play();
                }
            },
            SwingButton {
                translateY: 150
                text: "Stop"
                action: function() {  
                    m.stop();
                }
            }
        ]
    }
}

Profiling the application, I found the same memory usage pattern: memory usage climbs steadily with every time the media is played (i.e. pressing the Start button). The heap graphs below are captured from NetBeans 6.5.

The notes on the first graph (on the left) are explained below:

play - the Start button was pressed
GC - garbage collection was forced by pressing the GC icon several times in NetBeans
stop - the Stop button was pressed
GC - garbage collection was forced

The notes on the second graph (on the right) are explained below:

GC - garbage collection was forced by pressing the GC icon several times in NetBeans
play - the Start button was pressed several times quickly
stop - the Stop button was pressed
GC - garbage collection was forced

The MediaPlayer also has very limited support on media formats - it does not support wave files, or MPEG-2.5 sound files... so that I couldn't use most of the sound-effect files available on the internet. So this is Sun's solution to multimedia applications?!

Moon Monsters in JavaFX

2009-01-07T11:13:00.001+11:00

The Moon Monsters demo shows up as the first sample in Microsoft's Silverlight 1.0 Gallery. I thought it'd be great to test-drive JavaFX by porting this demo from Silverlight to JavaFX 1.0.

This seemingly simple demo actually touches on quite a few areas in the core stength of the JavaFX APIs - 2D graphics, data binding and input events handling. While the JavaFX port of the Moon Monsters is a pretty faithful implementation of the original features, it is not 100% complete. The following are not implemented here:

The graphics for paintbrush and keyboard are not included because it is too laborious to copy the coordinates into the JavaFX script.
The HTML Overlay feature is not implemented. I don't know how to do this in JavaFX because unlike Silverlight 1.0, JavaFX is not Javascript based and it is not bound to HTML either. If anyone knows how to do this in JavaFX, please leave a comment.

The source files are available here: alien.zip

Related Posts:

Happy New Year

Problem Mixing SmartSWT and GWT Widgets

2009-01-02T21:40:00.005+11:00

I set out implementing my Address GUI using SmartGWT 1.0b1 together with GWT 1.5.3 for Windows - both are latest releases available for download. The SmartGWT does not have any formal documentation. Information are scattered in Javadoc, showcase and developer/user forums. Although not ideal, the documentation is generally adequate.

I quickly ran into problem with SmartGWT. I wanted to have a menu bar at the top of my application as shown in the following diagram (which is a screenshot of SWT-Ext implementation of the same application).

SmartGWT does have a MenuBar in its API. However, this class does not seem fully implemented - there is no method to add menus (the addMenus() method is missing from the download although it appears in the online javadoc). As a workaround, I decided to use GWT's own MenuBar and MenuItem in combination with SmartGWT's ToolStrip and TabSet widgets as shown below.

I have done the same thing with the SWT-Ext version of the Address GUI (as shown in the first diagram above) using GWT's Menu with SWT-Ext's ToolBar and TabPanel widgets without any problems. However, in SmartGWT, the drop-down menu appears behind the SmartGWT widgets and get obscured by them. So I cannot really use the menu items any more. This result is shown in the above diagram.

It seams that SmartGWT is not so smart after all. [Update 2009-01-03]: Just joking!

Happy New Year

2008-12-31T23:18:00.004+11:00

This e-card was written in JavaFX. The source code can be found here.

If you encounter problems seeing it, I am not surprised.

Related Posts:

Moon Monsters In JavaFX

Merry Christmas!

2008-12-25T14:29:00.004+11:00

2008 has been a disastrous year in many aspects all over the world - from cyclone Nargis to the earth quake in SiChuan China; from Mumbai Massacre to mobocracy in Thailand... and let's not forget the global economic crisis...

Now more to home, following the down-sizing trend a company that I know of has decided to wind down the operation in Sydney just 2 weeks before christmas. As a result, people in the Sydney office have been made redundant. Come to think of it, the holiday season is perfect to fire people. Why? During Christmas, New Year and Australia Day (January 26) people's productivity is not at its peak. Firing them during this period will save money for the company left and right - it is a win-win situation for the company! Being a profit-chasing business entity, it is perfectly logical to 'do the right thing' in the holiday season. After all, humans are nothing but resources to a company - hence the term HR.

In the story, Mr Scrooge came to his senses and became a better human being in the end. But in real life, the reverse is usually true...

Turn Off 3G To Save The Environment!

2008-12-20T20:52:00.006+11:00

My N95 battery could never last more than 16 hours even when sitting idle. I had to recharge it every night. This week when I was in Kathmandu, I was pleasantly surprised: my battery actually lasted over 36 hours during which I snapped more than 70 photos , played 20 minutes of music and had a couple of phone calls (totaling over 15 minutes of talk time maybe). I noticed that there was no 3G network in Nepal (or at least I was not roamed to one). I believe that is why my battery lasted so long. That fact that I live on the edge of the cell probably exacerbated the situation.

So I decided to disable 3G on my phone. Normally, this is a straight forward operation available from the Settings -> Phone menu. However, my phone was from 3 Australia (product code: 0547465) and it was branded - meaning that it was customised for 3 Australia: some 3 applications was installed, some menu items have been disabled... and the menu item to turn off 3G has been disabled. Super keen on saving the environment by extending my battery life I set out to debrand the phone and turn off 3G!

The process of debranding is as following:

Backup the phone using PC Suite
Change the product code by following the instructions. The new product code I used is 0536087. Note that the NSS could not detect any phone on my notebook. In the end I installed NSS on my desktop (also running XP) using root account and that all worked fine. I don't really know why it would not work on my laptop.
Update the firmware using Nokia Software Updater
Restore the phone using PC Suite. The restore is not perfect and I still have to re-arrange/reinstall some applications. And all my messages and contacts are lost!

But it was all worth it. Now under the Tools -> Settings -> Phone -> Network menu I have an extra item: Network mode where I can select GSM only!

On the negative side, there are some major drawbacks upgrading to the 30.x version of the firmware. TRK no longer works and I had to install alternative (but better) solution. Fontrouter does not seem to work any more in the new browser (because it does not have the character encoding that I need) and I got around the problem by installing Opera Mini instead.

The Sorry State of JavaFX 1.0

2008-12-16T12:46:00.014+11:00

The release of JavaFX 1.0 officially marks Sun's defeat in the RIA battle. Why? Because JavaFX 1.0 is inferior to its competitors in all major battlefronts of the RIA war. It looks like a rushed job to make a marketing deadline.

The JavaFX 1.0 APIs

The JavaFX 1.0 is shipped with 2D graphical capabilities and that's pretty much it. Even basic things such as 3D graphics, faster video codecs are labeled with 'coming soon...' on their demo site. Forget about building any business RIAs because there are only a handful of simple Swing widgets at your disposal. On the other hand, people have been using Flash to build business applications for years; Silverlight has got 3D capabilities since v1.0.

Design/Development

The JavaFX plugin for NetBeans is a joke - it's buggy, sluggish and poor in features. One mistyped word or bracket can result in red squiggly lines all over the screen; then it produces unhelpful error hints such as 'sorry I was expecting so and so but I saw such as such...' so that the user has to read through the whole sentence several times and still hard to figure out what the complaint is. What happens to KISS principle?

Sure, Sun had said that they intended the plugin to be used by designers also - hence the 'user friendly' error messages. But let's get serious, how can anyone assume graphical designers will be using computer programming language to define simple things such as Timelines? Sun doesn't have to look far to find a better way - Microsoft has Expression Web to be used by graphical designers, and VS.NET 2008 to be used by programmers when it comes to Silverlight design and development workflow. (The JavaFX plugin for Illustrator is only good for editing shapes).

Deployment

To me there is not much point of deploying RIA as a thick client using Java Web Start. I am only interested in running RIA in a browser. With the new way of deploying applets in Java, JNLP is used, so that the HTML file points to a JNLP file and the JNLP file points to the JAR files and more JNLP files... It's a bit like my multi-hop flight from Sydney to Kathmandu (via Bristane then Bangkok) that took me over 20 hours and 6 of which was waiting at BKK for transit (that that is how I have the time to write this post) all because my company was too cheap to book the Qantas flight.

This is unnecessarily complicated because Sun doesn't want to have JavaFX plugin to the browser. On the other hand, Flash/Silverlight plugins are installed onto the browser so that all you need to specify is the object or embed tag in the HTML file without the hassle of running the myriad javascripts to detect/download the additional framework JAR files every time.

The User Experience

If you are using a browser that does not have the latest Java plugin (that supports JavaFX) installed (such as on my Nokia N95, or using the PC in the Thai Airways lounge at Kathmandu international airport), then you will be greeted with a message box forcing you to redirect to Sun's download website. There is no option to cancel!

Moreover, those javascripts do not report any errors if something goes wrong - it just hangs there with the Java logo spinning around and around forever...

Merry Christmas JavaFX 1.0

2008-12-10T22:40:00.025+11:00

JavaFX 1.0 was finally released last Friday. After over 2 years of brooding, I wonder what Sun has come up with - oh boy what a disappointment the JavaFX IDE (NB plug-in) is! But that is for another post.

The JavaFX 1.0 is no good for developing any RIA business applications for its lack of widgets. Instead it is OK for little toys such as those demos published on javafx.com. Immersed in the holiday season atmosphere, I implemented my Christmas card in JavaFX. A screenshot is shown below.

The Christmas card has the following features:

Transformation - the red and yellow text fly into the screen using a combination of Translation, Rotation and Scale.
Media playback - background music (mp3 file) from Michael Buble - Let it snow.
Animation - snow fall simulation
Drag out - when run as applet in a browser, the card can be dragged out of the browser.

The Project Structure

The project was created using NetBeans 6.5 with JavaFX plug-ins (installed from within NetBeans). I created a JavaFX project called MerryChristmas. The project structure is shown below:

The only files that I added are:

LetItSnow.mp3 - the background music song
tree.png - the background image: photo of Christmas tree shot at Darling Harbour on December 8, 2008.
Main.fx - the main JavaFX file containing the scene and its contents
SnowFall.fx - the CustomNode simulating snow fall and snow flakes (using circle)

The Main Contents

The contents of the main scene are two Texts, one background image, one background music and the snow fall (implemented as CustomNode).

The texts and their animation (actually transformation) are created as shown below. Notice the transforms which use binding data changed by the Timeline definition.

...
var x:Number;
var y: Number;
var scaleX : Number;
var scaleY:Number;
var angle:Number;

Timeline {
    repeatCount: 1
    keyFrames: [
        at (0s) {x=>180; y => 250; scaleX => 0.0; scaleY => 0.0; angle => -180},
        at (10s) {x=> 0; y => 0;
        scaleX => 1.0; scaleY => 1.0;
        angle => 0
        tween Interpolator.EASEOUT},
    ]
}.play();
...
var lighting = Lighting{
    light: DistantLight{
        azimuth: 60,
        elevation: 70
    }
    surfaceScale: 5
};

Stage {
    ...
    scene: Scene {
        content: [
            ...
            Text {
                content: "Merry Christmas & \nHappy New Year!"
                font: Font.font("Arial Bold", FontWeight.BOLD, 40)
                textOrigin: TextOrigin.TOP
                textAlignment: TextAlignment.CENTER
                x: 10
                y: 20
                transforms: bind[
                    Translate{
                        x:x
                        y:y}
                    Scale{
                        x:scaleX
                        y:scaleY}
                    Rotate{
                        angle: -angle
                        pivotX: 180
                        pivotY: 20 }
                ]
                fill: Color.RED
                effect: lighting
            },
            Text {
                content: "From Romen"
                font: Font.font("Times", FontWeight.BOLD, 40)
                textOrigin: TextOrigin.TOP
                x: 90
                y: 470
                transforms: bind[
                    Translate{
                        x:x
                        y:y}
                    Scale{
                        x:scaleX
                        y:scaleY}
                    Rotate{
                        angle: angle
                        pivotX: 180
                        pivotY: 450}
                ]
                fill: Color.YELLOW
                effect: lighting
            },
...

Any suggestions on how to make unicode text work are welcome.

The background image and music are easy enough:

...
            ImageView {
                image: Image {
                    url: "{__DIR__}tree.png"
                }
                x:0,
                y:0
            },
...
            MediaView {
                mediaPlayer: MediaPlayer {
                    autoPlay: true
                    media: Media {
                        source: "{__DIR__}LetItSnow.mp3"
                    }
                }
            },
...

Drag Out

To enable drag out when run as an applet in browser, I added the following extension to the Stage:

    extensions: [
        AppletStageExtension {
            shouldDragStart: function(e): Boolean {
                return e.primaryButtonDown;
            }
            useDefaultClose: false
        }
    ]

Also I had to specify in the Project's properties dialog and check the Draggable checkbox so that it will generate the .html file with the draggable set to true. The generated .html file snippet:

...

...

Snow Fall

All snow related classes are in the SnowFall.fx file. I used white circle as the snow flake:

public class SnowFlake extends Circle {
    init {
        fill = Color.WHITE;
        radius = 3 + Math.random() * 3;
        opacity= Math.random() * 0.5 + 0.5;
    }
}

The SnowFall class takes 3 attributes:

height, width - these specify the size of the canvas so that all the snow flakes can spread out in the canvas making it more realistic
numberOfFlakes - the number of snow flakes to be drawn and animated. The more flakes, the more threads JavaFX will have to create and manage under the hood.

The animation of the snow flakes are a bit tricky. Initially all the snow flakes are generated and randomly placed around the canvas; then, they start to fall. The movement of the snowflakes are as such: every second the snow flake will move left (negative) or right (positive) by a random amount, and move down by a random amount - i.e. the horizontal movement (x) can be either positive or negative, the vertical movement (y) is positive only. This is shown below:

                            KeyFrame {
                                ...
                                values: [
                                    x =>
                                    x + (Math.random() - 0.5)
                                    y =>
                                    y + Math.random() tween Interpolator.LINEAR
                                ]
...

When the snow flake hits either side of the boundary, it sort of bounces back a little; when it moves off the bottom of the canvas, it is wrapped around and re-appears at the top of the canvas and falls down again. This is achieved by using action of Keyframe which are executed at the beginning of each key frame.

...
                               action: function() {
                                    if(flake.centerY+y >height) {
                                        y=0; flake.centerY=0;
                                    }
                                    if(flake.centerX+x>width) {
                                        x=0; flake.centerX=width;
                                    }
                                    if(flake.centerX+x<0) {
                                        x=0; flake.centerX=0;
                                    }
                                }
...

The SnowFall class generates all the snow flakes in a for loop. Each snow flake is accompanied by a Timeline controlling its animation. The number of frames in each Timeline is also randomised (between 60 and 120 seconds per cycle), to avoid the situation where all snow flakes suddenly reset at the same time.

for ( i in [1..numberOfFlakes]) {
...
                    def timer = 60+60*Math.random();
                    Timeline {
                        repeatCount: Timeline.INDEFINITE
                        keyFrames: [ for (j in [0..timer]) {
                            KeyFrame {
...

Want to see the Christmas card in action? Well, you will have to wait till Christmas! [Update NYe2008]Now that it is NYE 2008, click here to run the e-card.

Meanwhile, here are the complete source code.

SnowFall.fx

/*
 * SnowFall.fx
 *
 * Created on 10/12/2008, 09:55:19
 */

package merrychristmas;

import java.lang.Math;
import javafx.animation.Interpolator;
import javafx.animation.Timeline;
import javafx.scene.CustomNode;
import javafx.scene.Group;
import javafx.scene.Node;
import javafx.scene.paint.Color;
import javafx.scene.shape.Circle;
import javafx.animation.KeyFrame;
import java.lang.System;

/**
 * @author ROMENL
 */

public class SnowFall extends CustomNode {
    public var height:Number;
    public var width:Number;
    public var numberOfFlakes:Number;

    override function create():Node {
        return Group {
            content: [
                for ( i in [1..numberOfFlakes]) {
                    var x:Number;
                    var y:Number;
                    x=1;
                    y=1;
                    var flake = SnowFlake{
                        centerX: Math.random() * width
                        centerY: Math.random() * height
                        translateX: bind x;
                        translateY: bind y;
                    }

                    def timer = 60+60*Math.random();
                    Timeline {
                        repeatCount: Timeline.INDEFINITE
                        keyFrames: [ for (j in [0..timer]) {
                            KeyFrame {
                                time: bind Duration.valueOf(j * 1000)
                                values: [
                                    x =>
                                    x + (Math.random() - 0.5)
                                    y =>
                                    y + Math.random() tween Interpolator.LINEAR
                                ]
                                action: function() {
                                    if(flake.centerY+y >height) {
                                        y=0; flake.centerY=0;
                                    }
                                    if(flake.centerX+x>width) {
                                        x=0; flake.centerX=width;
                                    }
                                    if(flake.centerX+x<0) {
                                        x=0; flake.centerX=0;
                                    }
                                }
                            }
                        }
                        ]
                    }.play();
                    flake
                }]
            
        };
    }
}
public class SnowFlake extends Circle {
    init {
        fill = Color.WHITE;
        radius = 3 + Math.random() * 3;
        opacity= Math.random() * 0.5 + 0.5;
    }
}

Main.fx

/*
 * Main.fx
 *
 * Created on 9/12/2008, 15:29:24
 */

package merrychristmas;

import javafx.animation.*;
import javafx.scene.effect.*;
import javafx.scene.effect.light.*;
import javafx.scene.image.*;
import javafx.scene.input.MouseEvent;
import javafx.scene.media.Media;
import javafx.scene.media.MediaPlayer;
import javafx.scene.media.MediaView;
import javafx.scene.paint.Color;
import javafx.scene.Scene;
import javafx.scene.text.*;
import javafx.scene.transform.Rotate;
import javafx.scene.transform.Scale;
import javafx.scene.transform.Translate;
import javafx.stage.AppletStageExtension;
import javafx.stage.Stage;

/**
 * @author ROMENL
 */
var width=390;
var height=540;
var x:Number;
var y: Number;
var scaleX : Number;
var scaleY:Number;
var angle:Number;

Timeline {
    repeatCount: 1//Timeline.INDEFINITE
    keyFrames: [
        at (0s) {x=>180; y => 250; scaleX => 0.0; scaleY => 0.0; angle => -180},
        at (10s) {x=> 0; y => 0;
        scaleX => 1.0; scaleY => 1.0;
        angle => 0
        tween Interpolator.EASEOUT},
    ]
}.play();

var lighting = Lighting{
    light: DistantLight{
        azimuth: 60,
        elevation: 70
    }
    surfaceScale: 5
};

Stage {
    title: "Merry Xmas JavaFX"
    width: width
    height: height
    scene: Scene {
        content: [
            ImageView {
                image: Image {
                    url: "{__DIR__}tree.png"
                }
                x:0,
                y:0
            },
            Text {
                content: "Merry Christmas & \nHappy New Year!"
                font: Font.font("Arial Bold", FontWeight.BOLD, 40)
                textOrigin: TextOrigin.TOP
                textAlignment: TextAlignment.CENTER
                x: 10
                y: 20
                transforms: bind[
                    Translate{
                        x:x
                        y:y}
                    Scale{
                        x:scaleX
                        y:scaleY}
                    Rotate{
                        angle: -angle
                        pivotX: 180
                        pivotY: 20 }
                ]
                fill: Color.RED
                effect: lighting
            },
            Text {
                content: "From Romen"
                font: Font.font("Times", FontWeight.BOLD, 40)
                textOrigin: TextOrigin.TOP
                x: 90
                y: 470
                transforms: bind[
                    Translate{
                        x:x
                        y:y}
                    Scale{
                        x:scaleX
                        y:scaleY}
                    Rotate{
                        angle: angle
                        pivotX: 180
                        pivotY: 450}
                ]
                fill: Color.YELLOW
                effect: lighting
            },
            MediaView {
                mediaPlayer: MediaPlayer {
                    autoPlay: true
                    media: Media {
                        source: "{__DIR__}LetItSnow.mp3"
                    }
                }
            },
            SnowFall {
                height: height
                width: width
                numberOfFlakes: 30
            }
        ]
    }
    extensions: [
        AppletStageExtension {
            shouldDragStart: function(e): Boolean {
                return e.primaryButtonDown;
            }
            useDefaultClose: false
        }
    ]
}

Related Posts:

The Long Tail Theory

2008-12-04T11:04:00.007+11:00

In the Telco business we often talk about the long tail products/services illustrated in the following diagram.

The idea is that for a service provider to stay competitive, it has to deploy and experiment with a large number of products and see how they are accepted by the market. There will be a small number of popular ones and the rest will be less popular as indicated in the long tail. So, over time, the service provider will kill the proven unpopular products and replace them with new/other products. You may wonder, why do they bother with the long tail since they are not that popular anyway? There are two major reasons:

The service providers are too paranoid not to provide those services because they want to stay competitive - anything that their competitors have, they also want it.
Other long tail graphs replace the Popularity label with Revenue. It's not hard to see that the total area under the Long Tail curve (i.e. total revenue) could add up to an handsome amount.

How is this theory relevant to me? Well, being an IT professional, I am also paranoid about my knowledge and skills being stale. So I have to invest my time and effort on updating and expanding my knowledges constantly. With all these new technologies and frameworks popping up all the time, I feel like a headless chook all the time.

The best way to learn a new language/framework/library is to actually use it. 'Hello World' just doesn't cut it. Yet a full blown application such as Web Mail or Pet Shop may be too big or too long for the purpose of learning or proof of concept. That is why I have my own little pet project - Address (among others), which serves as the use cases for my learning and experimentation.

For the products that I like and feel useful in my work, I will keep them in my toolbelt and probably move them up to the head. For the ones that are irrelavent to me, I drop them from the Long Tail picture altogether. There are also products that have died (e.g. Orion Server) or dying and being replaced by others (e.g. GWT-Ext) in the game of evolution. But it does not mean that my time spent on them were wasted. I still learn and accumulate a great deal of experiences so that when I encounter a new product, I can always draw on these experiences and view it in a better light.

GWT vs. ZK - SLOC Count

2008-12-01T17:13:00.006+11:00

I often read about how compact the ZK code was compared to GWT. I couldn't quite believe it. I thought it must have been different design patterns and style of coding used. So I decided to do it myself - I implemented my pet project in both GWT-Ext (GWT 1.4.6 and GWT-Ext 0.9.3 in December 2007) and ZK (ZK 3.5.1 with ZK Studio 0.9.0 in November 2008) and compared the Source Line of Code (SLOC) counts using Code Count tool.

Before I show the SLOC comparison, I need to give a brief introduction of the little demo program that I wrote. The program is a GUI to show various capabilities of the Address package, which has a service tier and DAO implementations behind it. The Address package provides a meta-data driven way of managing address records (see here for the data model). The concerned application here is only the GUI portion. The GUI has the following screens (left=ZK, right=GWT):

CRUD

This screen is used for Create, Retrieve, Update and Delete (CRUD) of the Address records from/to the database.

Find

This screen is used to call the corresponding Address services to find addresses either by keywords or by location.

Compare

This screen is used to call the corresponding Address services to compare two given Address records. Note that the ZK implementation supports drag-and-drop - i.e. I can drag a row from the list of addresses and drop it onto the text box and the text box will be populated with the content of the address (as a string). On the other hand, the GWT implementation does not support DnD because at the time of writing it, GWT-Ext did not support this kind of DnD yet (and I am not about to rewrite it using new version of GWT-Ext now considering Sanjiv has left the project... but that's another story).

Preferences

This screen is used to manage user preferences of the GUI application. Cookies are used as the storage.

In my SLOC count, I divided the source code into the following groups:

Main View - this includes the main screens: CRUD, Find and Compare tab panels.
Preference View - the user preference window
Model/Controller - the code behind the screen components, including models, event handlers, interaction with the service tier, helper/utilities
Splasher - this is the Splasher screen displayed at the beginning of the application. Only ZK version has implemented this (using index.zul and automatically forwarding to the main view after initialising the Address service); other versions do not have this screen.

Note that in ZK, all the views are a result of drag-and-drop from the components palette with modifications by hand.

The physical and logical SLOC counts are shown below.

The results speak for themselves.

UPDATE [2008-12-08]:

The GWT implementation source code can be found here: Address_gwt.zip
The ZK implementation source code can be found here: Address_zk.rar

GWT/GWT-Ext screenshots have also been added after the ZK ones (i.e. GWT screenshots are on the right-hand side).

ZK - First Impression

2008-11-27T15:10:00.002+11:00

I first heard of ZK from several developer forums, where one of the authors of ZK posted articles comparing ZK with GWT trying to prove that ZK was superior. Ever since then, I have been wanting to learn/use ZK for several months now.

I was pleasantly surprised by the maturity and high quality of ZK from the first moment I started using it - I didn't expect it to have a drag-and-drop graphical editor - something that not many free web frameworks have out of the box. I believe it is a real contender of GWT.

The praises of ZK can be found easily and I will not repeat them here. Instead, I will outline the reservations that I have about ZK hoping to get advices from the community.

ZK Studio Installation

When I followed download instructions on ZK site to install the ZK Studio directly from Eclipse, it turned out to be an older version of the plugin. It took me a while to find out why my ZK Studio did not have the ZUL Graphical Editor mentioned in the documentation. I had to download the plugin (v0.9.x) and install it onto Eclipse manually.

ZUL Graphical Editor

The ZUL Graphical Editor is not really an editor at the moment. It only serves as a previewer. I believe ZK team is actively working on improving the editor right now.

Debugging

I am not aware of any special debugger for ZK. If something goes wrong, I have to inspect the web server's log file to get the stack trace - cannot jump to the line of source code automatically. Moreover, if the error occurs at the ZUL script, the trace shows BeanShell exceptions, which can be harder to locate the real position of the error in the ZUL file.

Centralised ZScript File

I understand that zscript can be isolated in its own file and be included into the .zul file, so that the script is not littered in the presentation code. However, I find that to completely separate the script and the ZUL code is not practical. For example, to specify the a renderer for a component, I need to instantiate the renderer and assign it to a variable, then put the variable to the components renderer attribute. For example,

<zscript>
import com.laws.address.zk.*;

renderer=new AddressRenderer();
</zscript>
   <listbox id="crudAddressList" mold="paging" pageSize="10" 
   itemRenderer="${renderer}">
    <listhead sizable="true">
     <listheader label="ID" width="30px"/>
     <listheader forEach="${afts}" >
      <attribute name="label">${each.name}</attribute>
     </listheader>
    </listhead>
   </listbox>

However, the bit of script that instantiates and assigns the new renderer cannot be separated into another file (unless a function is created to do it, which looks odd and impractical).

Event Handling

The event handling in ZK is strange - the sender has to specify the target for the event. The whole idea of events is so that the sender (or publisher) is not aware of who is going to consume the event, or how many consumers, and what the consumer will do with it. But in ZK, the target of the event is specified up front by the sender - i.e. the sender knows who should receive the event. I shall elaborate this point in another post in more detail.

To be fair, ZK does allow the target of the event to be null when posting an event. However, the event is only accessible from top-level components in this scenario.

Server Round Trips for Static Contents

Unlike GWT, even static contents in ZK will involve server decision (a bit like Echo 2). For example, upon clicking a button or menu, my application pops up an 'About' message box displaying static contents. The processing of the button click and the display of the message box involves the server and incurs a round-trip to the server. This certainly has performance penalty and puts presentation logic and load onto the server (which, IMO should be placed in the client tier - i.e. the browser). I know, there has been debate about whether the GWT way is better. I personally prefers the GWT way - especially considering it is broadband era now.

Overall, my experience with ZK has been quite positive. I think it is among my top 3 web RIA frameworks.

N95 Must Haves: Panoman

2008-11-22T14:55:00.002+11:00

I travel quite a bit due to the nature of my job. I used to bring my camera along on every trip. But due to my forgetful nature, and the bulkiness of the normal cameras, I stopped using them on my trips. Instead, I began to use my mobile phone camera, especially after I got my N95.

N95 must have one of the best phone cameras around. I feel that it is even better than many of the cheap digital cameras. It has a decent Carl Zeiss lens, 5M pixels resolution and intelligent image processing software built-in.

When traveling, many of my photos are scenery. Panorama pictures are the best to capture the landscape and the atmosphere of the new places that I visit. So far the best panorama software I found is Panoman. It is extremely easy to use: simple press start and move the handset horizontally, then press stop when you are done. The rest is handled by the Panoman software. Most importantly, the result is superb.

What Do VB & Applet Have In Common?

2008-11-07T12:33:00.003+11:00

No, I am not talking about Victoria Bitter, which I quite like. I am talking about Visual Basic and Java Applets. They both have a bad reputation due to historical reasons, which is hampering their adoption.

In the heyday of IT, VB was probably the most widely used programming language at least in the Windows world. IT professionals and IT professional wannabes all flocked to write software to cash in on the golden era of IT. I still remember one of the managers at Australian Taxation Office (ATO) initiated an IT project and wrote the software in VB/Visual Foxpro by himself together with a public servant who had "13 years of programming experience" but also had trouble finding keys on the keyboard.

Today, VB has become VB.NET and the developer community is more mature. Still, there has been much discussion about why people perceive VB.NET to be inferior to C#. Many people choose C# over VB or even move from VB to C# simply because it will make them look better or making them feel better about themselves - real programmers use 'C-like' languages.

Unlike VB, the Java Applet never reached popularity since the beginning because those were the dial-up days. I can see three major hurdles for Java Applet in those days: slow internet links meaning downloading the JAR files was a big problem; 'complexity' in installing the JRE plug-in; awkward look-and-feel and lack of good frameworks.

Today, things have drastically improved: pervasive broadband access, improved browser plug-in capabilities, 80% new PCs are pre-installed with JRE, improved JRE (6 update 10), the imminent release of JavaFX 1.0 and more other applet-based frameworks. Still people doubt whether these frameworks can compete with Silverlight and Flex. I don't see the word 'applet' get mentioned much by Sun or JavaFX's promoters. The FUD is just too deeply entrenched in people's minds.

At the end of the day, great technology does not thrive on its own merit. The human factor has the final say. It's a bit like Wall Street.

A Musical Note

2008-11-03T18:02:00.010+11:00

I spent last month at home on holiday (is that an oxymoron?) feeling lethargic as people do on holidays. As a result I did not touch my computer much and only spent a few hours learning Erlang.

However, I did get to learn music and ended up spending 1-2 hours per day in front of the electronic keyboard that I bought for my kids . I have to admit, I am not good at music at all - I did use to excel in singing in primary school; but since my voice broke I lost all enthusiasm on music - my parents sending me to a violin class where I was the only boy did not help either. Until last month, as I fiddled around with the piano/keyboard, I revived my thirst for knowledge on music and piano-playing. It turned out that piano-playing is really easy to learn (at least to start) if you are a fast touch-typist. So I relearned the basic music theory 101 by following the wonderful book Total Piano. As an IT professional geek, I couldn't help finding many striking similarities between the musical and computing worlds.

First of all there are the patterns in music. As a matter of fact, all musical pieces are based on establishing some patterns and then repeating them - there are quite a few symbols in the staff to represent repetitions (i.e. the GOTO statement). It seems like composers are lazier than system designers Also, there are chords and each chord has many variations - broken, harmonic, minor, etc. Again, it's all about repetition and reusing.

Then there is the arrangements. The same music can be (re)arranged to adapt to different environments/scenarios - e.g. to suit different musical instruments (e.g. piano solo, string quartet), or to suit different players (e.g. beginner vs. pro) etc. Thanks to simple arrangements of many great works included in the book, I was able to play some very interesting pieces and keep getting fully engaged in the learning process. By the way, how many AJAX'ian web frameworks have you come across recently?

Well-known musicians also remake other people's songs to boost their own career. Yesterday, I read an interview with Seal and his upcoming new album of remakes of classics that 'appealed' to him. In it, Seal revealed that quite a few young people in their 20s had not heard the original version of the songs. So Seal's version was the first time that they heard them and Seal kind of re-introduced the great classics back to the young generations. I hope the youngsters don't mistake Madona's American Pie with the original.

Every now and then I can hear curses coming from across the office when my colleagues cannot find some features in their MS Office 2007. I have mixed feelings about the ribbon interface of Office 2007 - I like the look and feel but hate the fact that so many features have been rearranged so that everytime I want to use them it's Easter all over again. I am sure if someone has never used the older versions, he/she will like the new ribbon interface.

Perhaps learning from the closed-door design experiences, Microsoft is openning up their design process for some of their major products - e.g. Enginnering Windows 7. On the other hand, music creation is usually an extremely personal experience, unless you are charged to write some propaganda piece or the national anthem (I will vote for Waltzing Matilda anytime for Aussie national anthem)!

Meanwhile, I'd better get back to practise the beautiful piano solo arrangement of Beijing 2008 Olympic theme song - You and Me.

Erlang Mnesia - Take 2

2008-10-21T17:57:00.006+11:00

After my first attempt of using Erlang Mnesia, I decided to apply what I have learned so far to refactor and improve my Erlang Mnesia implementation of the Address database and Data Access Layer (DAL).

Redesigning The Database

This time I de-normalised part of the database design -

to duplicate the hierarchy_order field from address_field_type to address_field table
removed the record id fields from the 3 entities:
- address_field_type is a reference data table with small amount of data. I use the locale_country + hierarchy_order as a combined key implicitly
- address and address_field both have the location_code field which is unique for each record, so I used location_code as their primary key

The address.hrl file:

-record(address_field_type, {
 locale_country,
 name,
 default_value,
 hierarchy_order,
 display_order,
 validation_rules,
 suffix} ).
 
-record(address_field, {
 location_code,
 value,
 locale_country,  %% denormailsed from address_field_type
 hierarchy_order  %% denormalised from address_field_type
 }).
 
-record(address, {
 location_code,
 status
 }).
 
-record(address_addressField, {
 address_locationCode,
 address_field_locationCode
 }).

Creating the database:

create_db() ->
 mnesia:create_table(address_field_type,
  [
   {type, bag}, %% use bag because no single-filed primary key now
   {ram_copies,[nonode@nohost]},
   {index, [hierarchy_order]},
   {attributes, record_info(fields, address_field_type)}
  ]),
 mnesia:create_table(address_field,
  [
   {ram_copies,[nonode@nohost]},
   {attributes, record_info(fields, address_field)}
  ]),
 mnesia:create_table(address,
  [
   {ram_copies,[nonode@nohost]},
   {attributes, record_info(fields, address)}
  ]),
 mnesia:create_table(address_addressField,
  [
   {type, bag},
   {ram_copies,[nonode@nohost]},
   {index, [address_field_locationCode]},
   {attributes, record_info(fields, address_addressField)}
  ]).

The above changes do reduce the functionality of the Address application very slightly but it's a worthy trade-off as it simplifies the design - now I don't need the oid table/record any more along with all the functions and code revolving the oids.

Also, by de-normialising, i.e. duplicating the hierarchy_order field into address_field table, the sorting of the address_field list is made easy and efficient (without having to look up the database in every iteration). This is demonstrated in the code below.

Simplified Sorting

The sorting of address_field list has reduced from 3 functions previously to the use of lists:keysort/2 due to the duplication of the hierarchy_order field from address_field_type into address_field in the data/entity model.

sort_afs(Afs) when is_list(Afs) ->
 %% 5th field of the address_field tuple is hierarchy_order
 lists:keysort(5, Afs).

Simplified Data Insertion

By skipping any record id handling, the code for insertion of records is considerably simplified. This is most evident from the insert_address/2 function (compared to the previous version):

insert_aft(Aft) when is_record(Aft, address_field_type) ->
 Fun = fun() ->
  mnesia:write(Aft)
 end,
 mnesia:transaction(Fun),
 Aft.
 
insert_address(A, Afs) when is_record(A, address) and is_list(Afs) ->
 {NewA, NewAfs} = address:generate_location_code(A, Afs),
 Fun = fun() ->  
  % create the new address record
  mnesia:write(NewA),
  
  % now insert/update into address_field 
  % and insert into address_addressField table
  lists:foreach( fun(Af) ->
    mnesia:write(Af),
    A_Af=#address_addressField{ 
     address_locationCode=NewA#address.location_code,
     address_field_locationCode=Af#address_field.location_code
    },
    mnesia:write(A_Af)
   end,
   NewAfs),
  {NewA, NewAfs}
 end,
 mnesia:transaction(Fun).

Simplified Data Queries

By using dirty_read and dirty_index_read functions unnecessary transactions are avoided.

find_afts(LocaleCountry) when is_list(LocaleCountry) ->
 mnesia:dirty_read({address_field_type, LocaleCountry}).

find_address(Id) when is_integer(Id) ->
 [A] = mnesia:dirty_read({address, Id}),
 A.
 
find_address_field(Id) when is_integer(Id) ->
 [Af] = mnesia:dirty_read({address_field, Id}),
 Af.
 
find_address_field_codes(A_code) ->
 % the read will return a list of tuples, we want the 3rd field (the AF_locationCode) 
 % of each tuple and put them in a list.
 [ element(3, A) || A <- mnesia:dirty_read({address_addressField, A_code}),
  true ].
 
find_address_codes(Af_code) ->
 F = fun() ->
  mnesia:select(address_addressField, [{
   #address_addressField{ address_locationCode='$1',
    address_field_locationCode=Af_code, 
    _='_' },
    [], ['$1'] }] 
  )
 end,
 {atomic, Results}=mnesia:transaction(F),
 Results.
 
find_address_codes2(Af_code) ->
 F = fun() ->
  Q = qlc:q([Aaf#address_addressField.address_locationCode
   || Aaf <- mnesia:table(address_addressField),
   Aaf#address_addressField.address_field_locationCode==Af_code]),
  qlc:e(Q)
 end,
 {atomic, Results}=mnesia:transaction(F),
 Results.
 
find_address_codes3(Af_code) ->
 Aafs = mnesia:dirty_index_read(address_addressField, Af_code, 
  #address_addressField.address_field_locationCode),
 % the second element of the tuple is the address_locationCode
 [ element(2, Aaf) || Aaf <- Aafs, true ].

In the above code, find_address_codes/1, find_address_codes2/1, find_address_codes3/1 do the same thing but are implemented using different approaches. The dirty read one is the simplest.

Conclusion

Just by thinking in Erlang, the resulting database design has been simplified - less table and fields; the source lines of code (SLOC) has been reduced by 30%; most importantly, the programming logic is now simpler and easier to read/maintain.

However, this does not mean that Erlang and Mnesia are the best way to implement this kind of database applications (where normalisation is important for the business logic).

Erlang Mnesia

2008-10-18T22:35:00.008+11:00

Carrying out my Erlang learning plan, I embarked on the porting exercise of my Address DAO package into Erlang Mnesia. In fact, it is more than porting of the data access functions. It is also a porting of the Address database itself from RDBMS into Mnesia.

Creating the Mnesia Schema

The Entity Relationship Diagram (ERD) of the database schema is shown below.

To create the tables in Mnesia, I defined the following records, each corresponding to a table.

-record(address_field_type, {id,
 name,
 locale_country,
 default_value,
 hierarchy_order,
 display_order,
 validation_rules,
 suffix} ).
 
-record(address_field, {id,
 value,
 address_field_type_id,
 location_code}).
 
-record(address, {id,
 status,
 location_code}).
 
-record(address_addressField, {
 address_id,
 address_field_id
 }).
-record(oids, {
 name,
 id
 }).

Note that there is an extra record oids. This record/table is required because Mnesia does not support auto-incrementing fields usually used as primary key generation. Therefore this extra table is used to store the current key value - similar to Oracle's sequence.

Creating the tables are shown below. The oids table is also populated with seed values.

create_db() ->
 mnesia:create_table(address_field_type,
  [
   {ram_copies,[nonode@nohost]},
   {attributes, record_info(fields, address_field_type)}
  ]),
 mnesia:create_table(address_field,
  [
   {ram_copies,[nonode@nohost]},
   {index, [location_code]},
   {attributes, record_info(fields, address_field)}
  ]),
 mnesia:create_table(address,
  [
   {ram_copies,[nonode@nohost]},
   {index, [location_code]},
   {attributes, record_info(fields, address)}
  ]),
 mnesia:create_table(address_addressField,
  [
   {type, bag},
   {ram_copies,[nonode@nohost]},
   {index, [address_field_id]},
   {attributes, record_info(fields, address_addressField)}
  ]),
 mnesia:create_table(oids,
  [
   {attributes, record_info(fields, oids)}
  ]),
 Fun = fun() ->
  mnesia:write(#oids{ name=address, id=0 }),
  mnesia:write(#oids{ name=address_field_type, id=0 }),
  mnesia:write(#oids{ name=address_field, id=0})
 end,
 mnesia:transaction(Fun).

Simple CRUD

The address_field_type is a table for storing reference data, which does not rely on other tables. So the C and R of the CRUD is very simple. From address_db.erl file:

%% returns {atomic, Oid} or {aborted, Reason}
generate_oid(TableName) when is_atom(TableName) ->
 Fun = fun() ->
  [Oid] = mnesia:read(oids, TableName, write),
  %% because Erlang only supports single assignment
  %% I have to create new variables every time the value changes.
  NewId=Oid#oids.id+1,
  New = Oid#oids{ id=NewId },
  mnesia:write(New),
  NewId
 end,
 mnesia:transaction(Fun).
  
insert_aft(Aft) when is_record(Aft, address_field_type) ->
 Fun = fun() ->
  {atomic, Id}=generate_oid(address_field_type),
  New = Aft#address_field_type{ id=Id },
  mnesia:write(New),
  New
 end,
 mnesia:transaction(Fun).

find_aft(Id) when is_integer(Id) ->
 {atomic, [Aft]} = mnesia:transaction(fun()-> mnesia:read({address_field_type, Id}) end),
 Aft.

There are a few things worth noting:

Although the mnesia:read and mnesia:select functions can be read-only, they still need to be wrapped in a function and called by mnesia:transaction/1. This does not make sense to me.
The mnesia:transaction(Fun) returns {aborted, Reason} or {atomic, Result} where Result is whatever the return result of the Fun is. Therefore, the insert_aft/1 above returns New - the new address_field_type record with the id field populated.
Erlang is single assignment language - i.e. a variable can only be assigned value once. Therefore, if I want to assign the Aft#address_field_type.id, I have to create a whole new address_field_type record as in the insert_aft/1 above. I have to say, I am not a fan of this single assignment - it defeats the half of the purpose of having variables.

A Slightly More Complicated CRUD

The address entity relies on address_field, which in turn depends on address_field_type. Also, if address records share the same address_field values, then the address_field record should be shared and not be duplicated in the database. Therefore, the management of the address records are more complicated than those of address_field_type.

The creation of address records is shown below.

insert_address(A, Afs) when is_record(A, address) and is_list(Afs) ->
 {A2, Afs2} = address:generate_location_code(A, Afs),
 Fun = fun() ->
  % populate the address_field records' id field
  NewAfs=lists:foldl( fun(Af, NewAfs) ->
    % check if the same Af is already in database
    Result = mnesia:select(address_field, [{
     #address_field{ id='$1',
      location_code=Af#address_field.location_code, 
      _='_' },
     [], ['$1'] }] 
    ),
    if length(Result)==0 -> % Af not in DB, so insert it
     {atomic, Id}=generate_oid(address_field);
    true -> % Af already exists in database
     Id=lists:nth(1, Result)
    end,
    NewAf = Af#address_field{ id=Id },
    lists:append(NewAfs, [NewAf])
   end,
   [],
   Afs2),
  
  % create the new address record
  {atomic, AddressId} = generate_oid(address),
  NewA = A2#address{ id = AddressId },
  mnesia:write(NewA),
  
  % now insert/update into address_field 
  % and insert into address_addressField table
  lists:foreach( fun(Af) ->
    mnesia:write(Af),
    A_Af=#address_addressField{ 
     address_id=AddressId,
     address_field_id=Af#address_field.id
    },
    mnesia:write(A_Af)
   end,
   NewAfs),
  {NewA, NewAfs}
 end,
 mnesia:transaction(Fun).

From the address.erl file:

%% applying quick sort to AddressField list.
sort_afs([]) -> [];
sort_afs([Pivot|Tail]) when is_record(Pivot, address_field) ->
 sort_afs([Af || Af <- Tail, 
  compare_aft(address_db:find_aft(Af#address_field.address_field_type_id),
   address_db:find_aft(Pivot#address_field.address_field_type_id)) < 0])
  ++ [Pivot] ++
  sort_afs([Af || Af <- Tail, 
  compare_aft(address_db:find_aft(Af#address_field.address_field_type_id),
   address_db:find_aft(Pivot#address_field.address_field_type_id)) >= 0]).

compare_aft(Aft1, Aft2) when 
 is_record(Aft1, address_field_type) and is_record(Aft2, address_field_type) ->
 Aft1#address_field_type.hierarchy_order - Aft2#address_field_type.hierarchy_order.

generate_location_code(A, AddressFields) when is_record(A, address) 
 and is_list(AddressFields) and (length(AddressFields)>0) ->
 Afs=generate_location_code(AddressFields),
 FirstAf=lists:nth(1, Afs),
 { 
  A#address{ location_code=FirstAf#address_field.location_code },
  Afs
 }.

generate_location_code(Afs) ->
 ReverseSortedAfs=lists:reverse(sort_afs(Afs)),
 generate_location_code(ReverseSortedAfs, [], []).
 
generate_location_code([Head|Tail], String, NewAfs) when is_record(Head, address_field) ->
 StringValue = Head#address_field.value ++ String,
 Code=erlang:phash2(StringValue),
 generate_location_code(Tail, 
  StringValue,
  [ Head#address_field{ location_code=Code } ] ++ NewAfs);  
generate_location_code([], _String, NewAfs) ->
 NewAfs.

The following functions are queries from the address_db.erl file:

find_address(Id) when is_integer(Id) ->
 {atomic, [A]} = mnesia:transaction(fun()-> mnesia:read({address, Id}) end),
 A.
 
find_address_field(Id) when is_integer(Id) ->
 {atomic, [Af]} = mnesia:transaction(fun()-> mnesia:read({address_field, Id}) end),
 Af.
 
find_address_field_ids(A_id) ->
 mnesia:transaction(fun()-> mnesia:read({address_addressField, A_id}) end).
 
find_address_ids(Af_id) ->
 F = fun() ->
  mnesia:select(address_addressField, [{
   #address_addressField{ address_id='$1',
    address_field_id=Af_id, 
    _='_' },
    [], ['$1'] }] 
  )
 end,
 mnesia:transaction(F).
 
find_address_ids2(Af_id) ->
 F = fun() ->
  Q = qlc:q([Aaf#address_addressField.address_id 
   || Aaf <- mnesia:table(address_addressField),
   Aaf#address_addressField.address_field_id==Af_id]),
  qlc:e(Q)
 end,
 mnesia:transaction(F).

Things that I want to highlight from the above are:

Unlike Java or C#, Erlang does not have a sortable list out-of-the-box. Instead, I had to write functions that sort my list of address_field records - these are: sort_afs, compare_aft
When searching using the primary key field, mnesia:read is used; when searching using other fields, either use mnesia:select or QLC. In the above code, the functions find_address_ids/1 and find_address_ids2/1 do the same thing. However, the result (a list) given by find_address_ids2/1 are reversed.
If a record with the same primary key already exists in the table, the mnesia:write either creates (if table type if set) or updates (if table type is bag) the record in the table.
Again, due to the single assignment rule, the lists:append(List, AnotherList) does not change the List; instead, the new list is the return value of lists:append/2. This is just counter-intuitive.

Related Posts:

Erlang Mnesia - Take 2

Erlang ODBC

2008-10-13T21:29:00.009+11:00

Continuing my Erlang journey, I decided to try with Erlang ODBC by porting one of my toy applications. Perhaps not a wise choice since working with database (especially relational database) is not a strength of Erlang, at least for now.

After almost 10 years and being spoiled by myriad of Object-Relational Mapping (ORM) frameworks, using ODBC again feels like a giant leap backward - 3 generations back to be exact (ADO.Net, ADO, ODBC). So the overall development experience is not that great. This is not to say that Erlang is not a great language, but Erlang as a platform is quite narrow in scope (this makes sense considering Erlang's strong telco heritage).

So here is my porting exercise. Note that the code below is my feeble attempt at using/learning Erlang by reading the man-pages alone, so if you have suggestions to improve it, please drop a comment. Also, it is not robust since it does not handle error conditions.

Background

The application (or component) that I am working on is a Data Access Objects (DAO) package which I originally developed in Java using various ORM frameworks and then C# using various 3rd-party ORM and ADO.NET frameworks. The data model is quite simple, consisting 3 entities: AddressFieldType (AFT for short), AddressField (AF for short) and Address. The relationships among them are as following:

AFT - AF has a one-to-many relationship
Address - AF has a many-to-many relationship

The partial entity relationship diagram (ERD) can be found in my previous blog.

Here, I attempt to implement some of the interfaces from the DAO package using Erlang ODBC.

The Design

Since Erlang is not an object oriented language, I cannot call my module DAO, so I call it DAL (Data Access Layer) instead borrowing from Microsoft terminology. I have two simple modules:

address_dal: implementation of some of the interfaces retrieving records from the Address database (using MySQL 5.x with ODBC driver 5.1.5 for win32 downloaded from MySQL).
orm: ORM here means ODBC-Record Mapping. There are two types of functions in this module: those that provide ODBC template functions; and those that convert ODBC returned data into Erlang Records defined in address.hrl file.

Records

I want to work with Erlang records rather than tuples returned by ODBC calls. So I created these records to represent the domain model. From the address.hrl file:

-record(address_field_type, {id,
 name,
 locale_country,
 default_value,
 hierarchy_order,
 display_order,
 validation_rules,
 suffix} ).
 
-record(address_field, {id,
 value,
 address_field_type_id,
 location_code}).
 
-record(address, {id,
 status,
 location_code,
 address_fields = [#address_field{}]}).

Notice the address record has a list of address_field records, representing half of the many-to-many relationship.

ODBC Template

I need to surround my SQL queries with database connection and disconnection so that these boiler plate codes do not get scattered everywhere in my business logic. Also, if the SQL queries results in error, Erlang breaks the ODBC connection (I am not sure if this is Erlang's fault or ODBC's). So I have to reconnect before my next SQL query anyway.

So in my orm.erl I have these ODBC template/boiler-plate functions:

-define(CONNECTION_STRING, "DSN=Address-MySQL;UID=root").

connect() ->
 case proplists:is_defined(odbc,application:which_applications()) of
  false ->
   application:start(odbc); % pre R12 way of starting odbc;
  _Else ->
   false % do nothing
 end,
 odbc:connect(?CONNECTION_STRING, [
  {auto_commit, off}, 
  {scrollable_cursor, off} 
 ]).
 
sql_query(Query) ->
 {ok, Ref}=connect(),
 ResultTuple = odbc:sql_query(Ref, Query),
 odbc:disconnect(Ref),
 ResultTuple.
 
param_query(Query, Params) ->
 {ok, Ref}=connect(),
 ResultTuple = odbc:param_query(Ref, Query, Params),
 odbc:disconnect(Ref),
 ResultTuple.

Note that Erlang ODBC is based on ODBC v3 which supports connection pooling which has been turned on. So surrounding the SQL queries with connection and disconnection should not incur performance penalty (although I have not explicitly tested it).

One thing I hate about Erlang's odbc:connect() is that it does not give any comprehensive error reasons if the connection fails - it always says "No SQL-driver information available. Connection to database failed." Again, I am not sure if this is Erlang's or ODBC's fault.

Converting into Records

The ODBC query functions return the results as tuples or list of tuples. I want to convert them into recdords so that I can access the fields of the records more easily - e.g. #address.id.

I tried two approaches implementing the record constructor functions. The first one is more portable since it does not rely on the field position but uses column name as a clue to map to the record's fields. This is shown below from the orm.erl file mapping the AddressFieldType query results to their corresponding address_field_type record type:

construct_aft_records(ColNames, Rows) when erlang:is_list(ColNames) and (length(ColNames)>0)
 and erlang:is_list(Rows) ->
 if (length(Rows)>0) ->
  lists:foldl(fun(R, AftRecords) ->
    lists:append(AftRecords, [construct_aft_record(ColNames, R)])
   end,
   [], % initial value of AftRecords list
   Rows
  );
 true->
  []
 end.

   
construct_aft_record(ColNames, Row) when erlang:is_list(ColNames) and (length(ColNames)>0)
 and erlang:is_tuple(Row) and (size(Row) == length(ColNames)) ->
 Map=lists:foldl(fun(N, Map) ->
  lists:append(Map, 
   [{string:to_upper(lists:nth(N, ColNames)), 
   element(N,Row)}]
  )
  end,
  [], % initial value of Map is []
  lists:seq(1, length(ColNames))
 ),
 #address_field_type{
  id=extract_map_value(Map, "ADDRESSFIELDTYPEID"),
  name=extract_map_value(Map, "NAME"),
  locale_country=extract_map_value(Map, "LOCALECOUNTRY"),
  default_value=extract_map_value(Map, "DEFAULTVALUE"),
  hierarchy_order=extract_map_value(Map, "HIERARCHYORDER"),
  display_order=extract_map_value(Map, "DISPLAYORDER"),
  validation_rules=extract_map_value(Map, "VALIDATIONRULES"),
  suffix=extract_map_value(Map, "SUFFIX")
 }.

 
extract_map_value(Map, Key) ->
  element(2, element(2,lists:keysearch(Key, 1, Map))).

The second approach is lazier as it relies on the field position as specified in the SELECT SQL statement. This is shown below from the orm.erl file.

find_addresses(Query, Params) ->
 {ok, Ref}=connect(),
 {selected, _, Rows} = odbc:param_query(Ref, Query, Params),
 % now for each address Row fetch its addressField records
 AddressRecords = lists:foldl(fun(A, Records) ->
   AddressId=element(1,A),
   AddressStatus=element(2,A),
   AddressLocationCode=element(3,A),
   {selected, _, AfRows}=odbc:param_query(Ref,
    "SELECT af.addressFieldId, af.value, af.addressFieldTypeId,"
    "       af.locationCode, aft.hierarchyOrder"
    " FROM Address.addressField as af, Address.AddressFieldType as aft,"
         "      Address.address_addressField as aaf"
    " WHERE aaf.addressId=?"
      "   AND af.addressFieldId=aaf.addressFieldId"
      "   AND aft.addressFieldTypeId=af.addressFieldTypeId"
            " ORDER BY aft.hierarchyOrder",
            [{sql_integer, [AddressId]}]
           ),
           AfRecords = lists:foldl(fun(Af, AddressFieldRecords) ->
             AfRecord=#address_field{
              id=element(1, Af),
      value=element(2, Af),
      address_field_type_id=element(3, Af),
      location_code=element(4, Af)
             },
             lists:append(AddressFieldRecords, [AfRecord])
            end,
            [],
            AfRows
           ),
           AddressRecord= #address{
            id=AddressId,
    status=AddressStatus,
    location_code=AddressLocationCode,
    address_fields=AfRecords
   },
   lists:append(Records, [AddressRecord])
          end,
  [],
  Rows
 ),
 odbc:disconnect(Ref),
 AddressRecords.

Implementing the DAL Interfaces

Now that the boiler-plate code is done, I can implement the DAL interface methods, oops! I mean functions. Here is the address_dal.erl file.

%% Implementation of the Address Data Access Layer (DAL) interfaces.
-module(address_dal).
-export([find_aft/0, find_aft/1]).
-export([find_addresses_in_af/1, find_addresses_in_location/1]).
-include("address.hrl").
 
find_aft() ->
 {selected, ColNames, Rows} = orm:sql_query("SELECT * from Address.AddressFieldType"),
 orm:construct_aft_records(ColNames, Rows).
 
find_aft(LocaleCountry) ->
 {selected, ColNames, Rows} = orm:param_query("SELECT * from Address.AddressFieldType "
  "WHERE localeCountry=?", 
  [{{sql_varchar, 64}, [LocaleCountry]} ]
 ),
 orm:construct_aft_records(ColNames, Rows).


find_addresses_in_af(AddressFieldId) ->
 orm:find_addresses( 
  "SELECT a.addressID, a.status, a.locationCode"
  " FROM Address.Address as a, Address.Address_AddressField as aaf"
  " WHERE aaf.ADDRESSFIELDID = ?"
  " AND a.ADDRESSID=aaf.ADDRESSID",
  [{sql_integer, [AddressFieldId]}] 
 ).
 
% the input parameter LocationCode needs to be a string due to the ODBC 
% to Erlang datatype mapping - hence the test for is_list().
find_addresses_in_location(LocationCode) when is_list(LocationCode) ->
 orm:find_addresses("SELECT a.addressID, a.status, a.locationCode"
  " FROM Address.Address as a, Address.AddressField as af, Address.Address_AddressField as aaf"
  " WHERE af.LOCATIONCODE = ?"
  "   AND a.ADDRESSID=aaf.ADDRESSID"
         "   AND af.ADDRESSFIELDID=aaf.ADDRESSFIELDID",
  [{{sql_numeric, 32, 0}, [LocationCode]}] 
 ).

Notice that in the find_aft functions I can use 'SELECT *...' because the orm:construct_aft_records/2 function uses the column names to map to the address_field_type record; on the other hand, the find_addresses... functions have to use 'SELECT a.addressID, a.status, a.locationCode...' and in that strict order because the orm:find_addresses/2 function expects the columns to be in those positions.

Dealing with Databases in Erlang

2008-10-09T16:01:00.004+11:00

Correction: Thanks to Brandon's comment below, there is also Erlang ODBC which comes with the OTP distribution. This is the 'official' gateway to the RDBMS world. I had a go at it in this post. There is also CouchDB from Apache, which is still in incubation.

Currently there are basically two ways to use database in Erlang - Mnesia and ErlyDB. There is also the prospect of edbc. But nothing is available yet from the project.

Mnesia is the native DBMS in Erlang. It is bundled with Erlang/OTP distribution and is written in Erlang as well. It was originally developed for telco switching applications by Ericsson. Therefore, it boasts a lot of non-functional features - in-memory database, distributed database, high performance (for read), making changes without downtime, etc. However, all these benefits are at the cost of sacrificing many of the basic functionality of a traditional RDBMS - such as referential integrity check, data type validation/check, normalisation support, triggers, stored procedures, etc.

ErlyDB is essentially Yariv's one-man effort. It's a code generator to generate the data access layer code in Erlang to work with RDBMS (and perhaps Mnesia). Currently supported RDBMS are MySQL and Postgress. It's primary use case was ErlyWeb from the same author. It looked promising. However, the whole ErlyWeb and ErlyDB/ErlySQL seem to have lost its momentum: The last ErlyWeb release was over a year ago; ErlyDB is no longer a project on its own right, therefore, no information on its releases; promised features from Yariv's blog have not eventuate after 1 to 2 years.

So given the database landscape in Erlang, the most viable approach is to use Mnesia if you don't mind the lack of database features and tools. I will give it a try by porting my prototype Address database into Mnesia.

Consuming RESTful Service with Erlang

2008-10-05T17:04:00.019+11:00

Having been working in the telco industry for over 10 years, I can't help feeling a bit ashamed for not having learned Erlang. Erlang was developed by Ericsson - the leading Swedish-based network equipment provider, and has been used in many of the network switching equipment produced by Ericsson. Erlang has gained more traction recently, especially in the last year or two.

Here, I write an Erlang program to consume a demo RESTful service that I developed a couple of months ago. The Erlang code is based on example by Dave Thomas - the author of Programming Ruby. The detail of the RESTful service is available in my previous post.

There are two services that I want to consume.

The first one accepts a social web site URL and scrapes the page for the person's interests in movies and music to return a list of comma-delimited keywords. For example,

http://localhost:8080/SvdemoRestful/resources/webPageKeywords?url=http://localhost/someWebPage.html

if any keywords are found, then they are returned in the body of the HTTP response as a string; otherwise, the HTTP body is empty. For example,

folk songs, pop music, chinese music ,battle, action, comedy

Notice the spaces in the above string. The second service I want to consume accepts the above string to search for any matching promotion in a mock up XML database. If any is found then the XML string will be returned in the HTTP body; otherwise, the string <Empty/> is returned in the HTTP body. For example, the following URL will return the promotion information in XML below:

http://localhost:8080/SvdemoRestful/resources/promo/jazz

the returned XML:

<Promotion>
  <Code>802</Code> 
  <Name>Jazz Night</Name> 
  <Description>Jazz lovers' do not miss this once in a lifetime opportunity.</Description> 
  <Venue>The Jazz Club</Venue> 
  <DateTime>2008-10-30 21:00:00</DateTime> 
  <Tags>Jazz</Tags> 
</Promotion>

Now, let's do this in Erlang.

Create a file called svdemoClient.erl. The following code will consume the first RESTful service:

-module(svdemoClient).
-export([get_keywords/1]).

-define(BASE_URL, "http://localhost:8080/SvdemoRestful/resources").
-define(PROMO_URL, ?BASE_URL ++ "/promo/").
-define(KEYWORDS_URL, ?BASE_URL "/webPageKeywords"). % also works without ++

keywords_url_for(Url) -> ?KEYWORDS_URL ++ "?url=" ++ Url.
get_keywords(Url) ->
 URL = keywords_url_for(Url),
 { ok, {_Status, _Header, Body} } = http:request(URL),
 Body.

In Erlang, function names and Atoms must start with lower-case letters; variable names must start with upper-case letters or underscore (meaning the variable is not used/read).

The -define() macro in Erlang is similar to #define in C/C++. In the above example, after defining BASE_URL, any occurance of ?BASE_URL will be replaced with the string "http://localhost:8080/SvdemoRestful/resources".

The get_keywords() function returns the body of the HTTP response from requesting the given Url. The Body is either a comma-delimited string, or an empty collection. Executing the above code in Erlang:

127> c("d:/projects/svdemoErl/svdemoClient.erl").                            
{ok,svdemoClient}
128> svdemoClient:get_keywords("http://localhost/myWebPage.htm").     
"folk songs, pop music, chinese music ,battle, action, comedy "
129> svdemoClient:get_keywords("http://localhost/someOtherPage.htm").
[]
130>

To consume the second RESTful service, the search_promo() function is added.

promo_url_for(Keywords) -> ?PROMO_URL ++ utils:url_encode(Keywords).
search_promo(Keywords) ->
 URL = promo_url_for(Keywords), 
 { ok, {_Status, _Header, Body} } = http:request(URL),
 
 %%% Now that the XML is in the Body variable, let's parse it.
 if
  Body == "<Empty/>" ->
   not_found;
  true ->
   {ParseResult, _Misc} = xmerl_scan:string(Body),
   [ #xmlText{value=Code} ] = xmerl_xpath:string("//Code/text()", ParseResult),
   [ #xmlText{value=Name} ] = xmerl_xpath:string("//Name/text()", ParseResult),
   [ #xmlText{value=Description} ] = xmerl_xpath:string("//Description/text()", ParseResult),
   [ #xmlText{value=Venue} ] = xmerl_xpath:string("//Venue/text()", ParseResult),
   [ #xmlText{value=DateTime} ] = xmerl_xpath:string("//DateTime/text()", ParseResult),
   { Code, Name, Description, Venue, DateTime }
 end.

Erlang/OTP download comes with XML parser and XPath support in the xmerl application, which is not part of the Erlang standard library (stdlib). To use the XML functions, the header file must be included:

-include_lib("xmerl/include/xmerl.hrl").

Note that the keywords contain spaces, which must be URL-encoded before passing to Erlang's http:request() function. I stole the url_encode() function from YAWS and put it in utils.erl file.

To string the two service consumptions together:

search_promo_from_url(Url) ->
 Keywords=get_keywords(Url),
 if
  Keywords == [] ->
   not_found;
  true ->
   search_promo(Keywords)
 end.

Calling the function in Erlang shell:

126> svdemoClient:search_promo_from_url("http://localhost/MyWebPage.htm")
. 
{"801",
 "Batman The Dark Knight",
 "\n\t\t\tMeet stars in Batman in person - Chritian Bale, Michael Caine.\n\t\t",

 "Star City",
 "2008-7-30 10:00:00"}

The final svdemoClient.erl file:

-module(svdemoClient).
-export([get_keywords/1, search_promo/1, search_promo_from_url/1]).
-include_lib("xmerl/include/xmerl.hrl").

-define(BASE_URL, "http://localhost:8080/SvdemoRestful/resources").
-define(PROMO_URL, ?BASE_URL ++ "/promo/").
-define(KEYWORDS_URL, ?BASE_URL "/webPageKeywords"). % also works without ++

keywords_url_for(Url) -> ?KEYWORDS_URL ++ "?url=" ++ Url.
get_keywords(Url) ->
 URL = keywords_url_for(Url),
 { ok, {_Status, _Header, Body} } = http:request(URL),
 Body.

promo_url_for(Keywords) -> ?PROMO_URL ++ utils:url_encode(Keywords).
search_promo(Keywords) ->
 URL = promo_url_for(Keywords), 
 { ok, {_Status, _Header, Body} } = http:request(URL),
 
 %%% Now that the XML is in the Body variable, let's parse it.
 if
  Body == "<Empty/>" ->
   not_found;
  true ->
   {ParseResult, _Misc} = xmerl_scan:string(Body),
   [ #xmlText{value=Code} ] = xmerl_xpath:string("//Code/text()", ParseResult),
   [ #xmlText{value=Name} ] = xmerl_xpath:string("//Name/text()", ParseResult),
   [ #xmlText{value=Description} ] = xmerl_xpath:string("//Description/text()", ParseResult),
   [ #xmlText{value=Venue} ] = xmerl_xpath:string("//Venue/text()", ParseResult),
   [ #xmlText{value=DateTime} ] = xmerl_xpath:string("//DateTime/text()", ParseResult),
   { Code, Name, Description, Venue, DateTime }
 end.
 
search_promo_from_url(Url) ->
 Keywords=get_keywords(Url),
 if
  Keywords == [] ->
   not_found;
  true ->
   search_promo(Keywords)
 end.

The utils.erl file (copied from YAWS):

-module(utils).
-export([integer_to_hex/1, url_encode/1]).

integer_to_hex(I) ->
     case catch erlang:integer_to_list(I, 16) of
         {'EXIT', _} ->
             old_integer_to_hex(I);
         Int ->
             Int
     end.
 
 
old_integer_to_hex(I) when I<10 ->
     integer_to_list(I);
old_integer_to_hex(I) when I<16 ->
     [I-10+$A];
old_integer_to_hex(I) when I>=16 ->
     N = trunc(I/16),
     old_integer_to_hex(N) ++ old_integer_to_hex(I rem 16).
 

url_encode([H|T]) ->
     if
         H >= $a, $z >= H ->
             [H|url_encode(T)];
         H >= $A, $Z >= H ->
             [H|url_encode(T)];
         H >= $0, $9 >= H ->
             [H|url_encode(T)];
         H == $_; H == $.; H == $-; H == $/; H == $: -> % FIXME: more..
             [H|url_encode(T)];
         true ->
             case integer_to_hex(H) of
                 [X, Y] ->
                     [$%, X, Y | url_encode(T)];
                 [X] ->
                     [$%, $0, X | url_encode(T)]
             end
     end;
 
url_encode([]) ->
     [].

Hancock: Good job!

2008-09-28T11:19:00.005+10:00

Last week's experience remindes me of scenes from the movie Hancock.

The marketing guy Ray tought Hancock how to behave in front of the crowd - here are the dialogues from the scene.

Ray Embrey: So you've used the door, the building's still intact, people are happy you've arrived, they feel safe now, there's an officer there and he's done a good job, so you might want to tell him he's done a good job.
Hancock: What the hell did I have to come for Ray if he's done a good job?

So later when Hancock when to the crime scene after the police failed to contain the situation, he repeatedly said 'Good job!' to everyone, including the policewoman who was injured and pinned down by enemy fire.

Hancock: [to pinned-down cop] Good job! Do I have permission to touch your body?
Female Cop: Yes!
Hancock: It's not sexual. Not that you're not an attractive woman. You're actually a very attractive woman and...
Female Cop: [screaming] Get me the hell out of here!

Those were the my favorite scenes from the movie. Do the above scenes look familiar in your workplace?

Why Not Chrome?

2008-09-26T11:01:00.003+10:00

I installed and tried Google Chrome recently and my first impression of Chrome as a browser is quite average. As I said before, there is nothing revolutionary about Chrome at the surface. It's really the backend functionality that Google is betting on. For now, my biggest complaint about Chrome is that it installs in user's home directory rather than a shared directory so that all users can use (maybe there is or will be an option to install it in shared mode and I just don't know it).

At the moment, each browser has its niche market: IE is pre-installed on all Windows based machines including Windows Mobile; Firefox has a great plug-in framework so that its functionality and usability can be greatly extended beyond just being a browser; Opera has over 100 million installations on mobile phones; Safari is the browser of choice on Mac...

So far, my favourite browser is Firefox v3 mostly because of the plug-ins. As a blogger, the ScribeFire blogger plug-in for Firefox is a must-have. I use it to do my blogging rather than using the Blogger's own web pages (thanks to Blogger's APIs). Another plug-in I use is the Adblock Plus where you can selectively block advertisement contents by clicking on the 'block' tab that Adblock Plus inserted over them.

IE still remains the widely used browser and I still use it. I only use it because certain plug-ins (ActiveX) are only available on IE. The most popular plug-in is probably the Microsoft Outlook for IE since most companies these days use Outlook as their email solution.

Google is also taking this approach by integrating Chrome with other Google services so that Chrome will be the best browser when it comes to Google (and its partner) provided services. Therefore, I don't see any single browser replacing all others. It will be more of a case of using certain browser for certain use cases.

Turn N95 Into A Ping-Pong Bat

2008-09-22T12:28:00.003+10:00

As I was searching for some games for my N95, I stumbled upon the Accelerometer support for N95. If you have seen the iPhone ads on TV, you may have been impressed by the feature of the applications changing their display orientation after the iPhone has been rotated. I must say I was very impressed after playing with my friend's iPhone. Little did I know that my N95 also has the capability to do this. It's just that Nokia did not put any application on it to utilise the accelerometer.

There are a couple of applications developed by Nokia Research center which take advantage of the accelorometer - Moving Ball and Activity Monitor, both available from the Nokia Research Center web site.

I have downloaded and tried the Moving Ball demo. It was great! As I move/rotate my handset in any direction, the ball moves or bounces following my motion. If I hold the phone like a ping-pong ball bat and swing it, the ball displayed on screen actually bounce up and down against the 'bat'.

According to these videos, the French developer Samir has made some cool applications using this technology - RotateMe and Nokmote. I particularly like Nokemote where it allows the user to control the application by moving the handset without having to press any buttons/keys - e.g. in the music player, tilt left to rewind, tilt right to fast-forward, bounce it down and up to select song, etc. Unfortunately, the author's web site is down.

RotateMe on N95 Uploaded by soueldi

Nokmote Caricato da soueldi

There are others of course: There is a dedicated forum on N95Users suggesting all sorts of cool uses with videos to show; NiiMe is too shaky to be practical; FlipSilent and ShakeSMS are pretty good, but I am not sure if they drain the battery; Landscape Pro, but it's not free; Alas, all I can do with a great piece of technology is to use it as a ping-pong ball bat!

Related Posts: How To Add Unicode Fonts to N95

The Shortest Self-Printing Code

2008-09-19T21:23:00.001+10:00

#!/bin/ksh
cat `whence $0`

Could this be the shortest self-printing code?

Of Tatami, jMaki and Others...

2008-09-13T17:53:00.009+10:00

Tatami 1.2 Beta was released a few days ago.

Both GWT and Dojo are Javascript frameworks for creating Rich Internet Applications (RIA). The GWT team focuses on the foundation of the framework and does not bother with making state-of-the-art cool widgets. While Dojo has a mature and good-looking widgets library. Wouldn't it be good if the two can be seamlessly integrated at the API and runtime levels so that one can leverage on the advantages of both frameworks? That's exactly what Tatami offers.

Tatami is a widget/component library for GWT. What makes it interesting is that it is a GWT wrapper of Dojo components. With Tatami, the Dojo widgets become GWT widgets; the Dojo utilities become GWT helper classes. If you don't have the stomach for editing CSS+XHTML+Javascript directly, but still want to use the Dojo components, then Tatami is good news. The development experience is back to Java and that's it.

Another web framework which provides Dojo wrapper is the jMaki framework. While jMaki has a nifty NetBeans plug-in, the development experience is improved somewhat. However, as a developer, you'd still have to deal with a mixture of JSP, Javascript and Java: the page templates are written in JSP, the glue code is written in Javascript for event handlers (using a pub-sub mechanism), the server side code is written as Java servlets - maybe this hodge-podge is really what they mean by mash-up.

Another wrapper is the GWT-EXT, which is a GWT wrapper for EXT/JS. I have blogged about the EXT/JS, EXT-GWT and GWT-EXT and the licensing issues revolving EXT/JS previously. Well, the licensing of Dojo is very liberal, without the GPL issues introduced by EXT/JS. Thankfully, Tatami is LGPL.

How To Add Unicode Fonts to Symbian

2008-09-09T15:53:00.014+10:00

[Update 2009-02-17]: Fontrouter project has been open sourced. Some of the download URLs in this post may have been changed. See here for more details.

If you are like me, who got a Nokia N95 with only English but wants to view contents in other languages - e.g. Chinese, Japanese, Korean (CJK), Russian, Hindi, etc. then you will be in a bit of strife.

According to Nokia forum, there is no way that you can install an extra language package on Nokia. You'd have to go to Nokia Care to have them install it for you, for a fee! Well, from my previous encounters with Nokia Care (here in Sydney), they are bunch of slow, bureaucratic, idiotic waste of space. So that option is a no-no for me.

Fortunately, the Symbian guru oasisfeng has created a wonderful software called FontRouter. The latest incarnation of the software is the FontRouter LT. Because the software is in testing stage, there is no full documentation on how to install and use it. There are pieces of information everywhere in the forum and mostly in Chinese. It took me a day to figure out the end-to-end process of getting it to work on my N95. So I feel that I should record it down and share it. I believe it should work for all S60 v3, S80 and S90 Nokia phones.

Step 1 - Downloading the Software

There are many versions on the oasisfeng web site, the version that I used was the FontRouter LT for Symbian 9 Beta (Build 20071109) Open Signed Online测试版.

Also download the TrueType font from the same web site: 【TrueType】方正隶变 GBK字体. This contains all the CJK characters(should contain other european characters too). Note that I tried using other Chinese TrueType fonts and my handset failed to start. So experiment with other .ttf files only after you have successfully installed and tested with this one. I am currently using the 方正准圆(FZY3JW.TTF) which is a Chinese true type font and its English characters are very close to the original font on my Nokia N95.

Notice that all Nokia phones belong to a certain profile (to make it easy for software developers to select and test the target platform/handsets). The N95 is an S60 3rd edition phone and runs on Symbian version 9. You can find this out from the phone: Tools -> Utilities -> About. Also notice that the version of FontRouter that I used was "Open Signed Online". What does this mean? Read on...

Step 2 - Signing the Software

Symbian is pretty rigorous when it comes to security. Software needs to be digitally signed before installing onto the phone. You can try installing the .sis file just as it is and your phone will reject it and complain that the digital certificate/signature is invalid or something of that nature. To sign the file you just downloaded, you will have to go to www.symbiansigned.com. On its front page, there is a URL for Open Signed Online (Beta). Click this URL and fill in the form on the following page. Note:

The IMEI number uniquely identifies your handset (e.g. if your phone is stolen, you can report it to the carrier and they can bar the IMEI from the network so that the thief cannot use it). You can find it out by 'dialing' *#06#
Email - make sure you use a valid email address because you will need to retrieve email from Symbian to carry out the next steps.
The Application is the .sis file you just downloaded from FontRuter.

Upon successful submission of the form, Symbian will send you an email with a URL to confirm the submission. Click it.

Then Symbian will send you another email with a URL to download the signed version of the .sis file. Just click the URL and save the file.

Step 3 - Installation

Now you can install the signed version of the software that you just downloaded from Symbian using PC Suite. Make sure you install it on the Memory Card, NOT on the Phone Memory (otherwise, if it hangs your handset, you will not be able to bring your phone back to life). During installation, it will warn you that your phone is incompatible with the software, just ignore it and continue.

Then copy the fzlb_gbk.ttf file downloaded in Step 1 into the Memory Card's \data\Fonts directory. You will notice that the file FontRouter.ini has also been created in this directory by the software installation process.

Now it is time to modify the FontRouter.ini file. You can use the Windows Nodepad to edit a copy of the file and then copy it onto the handset using PC Suite. Modify the two lines of the .ini file as following:

FixFontMetrics=0
FixCharMetrics=0

changed the values from 0 to 1 like this:

FixFontMetrics=1
FixCharMetrics=1

and leave everything else the same. If you don't make the changes, the fonts will be curtailed either from the top or bottom.

That's it. Phew! Now it's time to test it - bounce the phone (i.e. switch it off then on).

Euphoria!

There are a few must-read guides that I highly recommend:

Related posts: Turn N95 Into Ping-Pong Bat

Why Google Chrome?

2008-09-05T23:22:00.010+10:00

The release of Google Chrome beta is indead exciting news. Chrome is going to be bigger than iPhone! What makes it exciting is not the GUI - in fact, it looks and behaves pretty much just like any other browser. It is what's under the hood that make people (OK, developers) pumped about the new browser.

Google made it very clear on why they built a browser: "What we really needed was not just a browser, but also a modern platform for web pages and applications, and that's what we set out to build." So it is clear that Google intends to make Chrome to host and run next generation rich internet applications (RIA). Google's RIA platform is the Google Web Toolkit (GWT), which just had version 1.5 released a few days ago (Developer's Guide here). JavaFX folks are also excited about Chrome as shown in this article.

Google is a big proponent of Javascript. GWT 'compiles' Java code into Javascript and uses Javascript in runtime. A sophisticated GWT application can result in a large Javascript file, which poses a performance problem for the initial download of the application (a similar problem faced by Java Applets, which Java 6 Update 10 solves by allowing downloading relevant JARs only, without having to download the whole JRE libraries). Google Gears is another component that Google pushes to increase performance of Javascript applications. Yet, users have to download and install it separately if they want to take advantage of it. Therefore, it is natural for Google to come up with a browser as a unified platform for great support of Javascript. From Chrome web site, Google stated: "We also built V8, a more powerful JavaScript engine, to power the next generation of web applications that aren't even possible in today's browsers." Here we can see a glimpse of what to expect from Google on the Chrome, Gears and GWT fronts. GWT applications will run faster on Chrome than any other browser, thanks to Javascript optimisation on Chrome.

No wonder people are speculating that Microsoft is contemplating on suing Google for antitrust!

The Sorry State of ADO.NET Entity Framework 1.0

2008-09-03T23:54:00.006+10:00

A few months ago I test-drove the ADO.NET Entity Framework 1.0 Beta 3. While I enjoyed certain aspects of EF, such as LINQ and graphical modelling, the overall experience of using EF hasn't been that great. In fact, my very first blog on this site was dedicated to EF bashing. My major complaint about EF was its intrusive approach to ORM - instead of using PONO (like in Hibernate), it generates bloated relational database-centric data entity models. This is against the domain driven design method. Another problem I encountered was its shoddy implementation of lazy loading (or lack of it, as a matter of fact).

I then realised that I was not alone in feeling disenchanted by EF. In the same month of my blog, several hundred people have signed up on the ADO.NET Entity Framework Vote of No-Confidence open letter, many of whom are from Microsoft's Most Valued Partners (MVP). In last month's issue of Visual Studio Magazine, the article A Vote for Transparency gave a blow-by-blow account of events on the controversy surrounding EF, including the aformentioned open letter. So Microsoft decided to make the EF version 2 design exercise a more transparent process. This no doubt is a step forward. However, it is too little too late for EF v1. Looks like EF v1 will not receive any significant improvements or address any of the communities concerns in the open letter. In fact, I doubt that any EF design improvements have been made into .NET Framework 3.5 SP1 (released in August 2008) since EF 1.0 Beta 3 (released in December 2007).

On a side note: although some people whine about JCP program being slow (e.g. the Java closure debate) and sometimes committee-driven, comparing to the traditional closed-door design pattern like EF v1, JCP is a great leap forward and the result speaks for itself. The JCP specifications are more readily accepted and adopted by the community than a proprietary one like EF.

The Sorry State of JavaFX

2008-08-25T09:52:00.014+10:00

[Update: since JavaFX 1.0 has come out, this article is out-dated. See The Sorry State of JavaFX 1.0 instead.]

It's been over a year since F3 has been rebadged JavaFX in May 2007. I had enjoyed programming in JavaFX script and it really is easy to learn and enables rapid application development.

JavaFX is fine for its original goal of delivering a 'media' stack for the Java platform. However, as JavaFX is more and more being pitched to compete with Silverlight and Flash to be Sun's answer to Rich Internet Application (RIA) platform, it is falling short.

A major requirement of a RIA platform is to be able to run the application seamlessly in a web browser - i.e. using the browser as a canvas, not just using the browser to download the fat client. JavaFX does not do that. Both Silverlight and Flash's home pages promote their own technologies by using them. But what does javafx.com use? Javascript! So far almost none of the JavaFX demos show an application running in browser. For the few that do show, such as the Applet Example, when I tried it using javafx-sdk1.0pre1, all I got was a blank applet canvas. (I did manage to run a JavaFX application as an Applet from HTML via trial and error though).

Also, no support for being inlineable with HTML will decrease its flexibility as a RIA. Tey Chui gave a comprehensive argument on this issue.

Sun has been promising a browser javafx plugin for a while now. I do hope they can deliver it together with the proper 1.0 release. Talking about the releases, I was really surprised by the drastic changes made in the current 1.0pre1 release. It has pervasively changed the JavaFX script syntax so that your previous JavaFX code in the last 2 years simply will not compile anymore. Furthermore, the bundled JARs have also been totally changed - the javafx.ui.* simply disappeared! This suggests a deeper underlying architectural change.

Well, making big changes may not be a bad thing, especially considering JavaFX' beta status. However, the documentation available in supporting the developer community to cope with the change is totally inadequate. Sun does have JavaFX Reference documents. However, the current state of these documents are nothing near to be complete or even adequate. For example, the Migration Guide does not tell you what had happed to 'trigger on new' or nullable datatypes (e.g. int?) or recommendations for replacing javafx.ui.* with the scene graph libraries...

For now, I will put my JavaFX exercises on halt and wait for the final v1.0 release.

Consuming Web Services from Android

2008-08-22T13:07:00.021+10:00

Earlier this week, Google released Android 0.9 SDK Beta. As usual, I couldn't wait to try it out.

Unlike JavaME (especially MIDP 1.0), Android is targetting mobile devices with more grunt - it's bundled with many open source libraries, such as Apache Harmony (open source Java SE), Unicode library, JUnit, Apache Commons, ASN.1 libarary and more from Bouncy Castle, kXML, etc. The android.jar file is 11MB!

It is time to write a web service consumer on Android, to consume my WCF web service as well as RESTful service that I developed for a demo.

The Android does not contain any tools to help building SOAP based web service clients. Google is a proponent of REST services. It is no surprise that the SDK is not bundled with any SOAP-related tools. An alternative is to add kSOAP 2 to my test project. But I quickly dismissed the idea as my web service built in WCF is not JSR-172 compliant.

Like my exercise in JavaME, I decided to call the RESTful version of the same service built in NetBeans 6.1 instead.

I created a new Android project - SvdemoAndroid, using Eclipse 3.4 with the Android plugin installed. Because my Android Activity will be accessing the internet for the HTTP connection (although it's hosted on my machine), I had to add the line <uses-permission android:name="android.permission.INTERNET" /> to my AndroidManifest.xml file. Now the file looks something like:



    <uses-permission android:name="android.permission.INTERNET" /> 
    
        
            
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />

Android is bundled with Apache HttpClient 4. So invoking a REST service is simply a matter of openning up a HTTP link using the HttpClient and process the response. The following two methods show the consumption of two different RESTful services. The first method getKeywords() calls a service returning a plain text on HTTP.

        static String SERVER_HOST="192.168.2.100";
 static int SERVER_PORT = 8080;
 static String URL1 = "/SvdemoRestful/resources/mySpace?url=http://wpickup02/MySpace_Com_John.htm";
 static String URL2 = "/SvdemoRestful/resources/promo";
 /**
     * Call the RESTful service to get a list of keywords from the web page.
     * @param target - the target HTTP host for the RESTful service.
     * @return - comma delimited keywords. May contain spaces. If no keywords found, return null.
     */
    private String getKeywords(HttpHost target) {
        String keywords=null;
     HttpEntity entity = null;
     HttpClient client = new DefaultHttpClient();
     HttpGet get = new HttpGet(URL1);
     try {
   HttpResponse response=client.execute(target, get);
   entity = response.getEntity();
   keywords = EntityUtils.toString(entity);
  } catch (Exception e) {
   e.printStackTrace();
  } finally {
   if (entity!=null)
    try {
     entity.consumeContent();
    } catch (IOException e) {}
  }
  return keywords;
    }

The second method calls another service (on URL2). This service returns a XML string containing the details of a Promotion. Unlike JavaME, Android SDK is bundled with DOM parser. Since my XML string is pretty short I decided to use DOM instead of kXML's pull parser. Note that PromoInfo is just a data object holding all the promotion information in its fields.

/**
     * Call the REST service to retrieve the first matching promotion based
     * on the give keywords. If none found, return null.
     * @param target - the target HTTP host for the REST service.
     * @param keywords - comma delimited keywords. May contain spaces.
     * @return - PromoInfo that matches the keywords. If error or no match, return null.
     */
    private PromoInfo searchPromo(HttpHost target, String keywords) {
     if(keywords==null)
      return null;
     
     PromoInfo promo=null;
     Document doc = null;
     HttpClient client = new DefaultHttpClient();
     HttpGet get = new HttpGet(URL2+"/"+keywords.replaceAll(" ", "%20"));
     try {
      DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
   DocumentBuilder builder = factory.newDocumentBuilder();
   HttpEntity entity = client.execute(target, get).getEntity();
   
   doc = builder.parse(entity.getContent());
   NodeList promotions = doc.getElementsByTagName("Promotion");
   
   if(promotions!=null) {
    Node promoNode = promotions.item(0);
    
    if (promoNode!=null) {
     promo=new PromoInfo();
     NodeList nodeList = promoNode.getChildNodes();
     int len = nodeList.getLength();
     for(int i=0; i<len; i++) {
      Node node = nodeList.item(i);
      String value = this.getNodeValue(node);
      if("Name".equals(node.getNodeName())) {
       promo.setName(value);
      } else if("Description".equals(node.getNodeName())) {
       promo.setDescription(value);
      } else if("Venue".equals(node.getNodeName())) {
       promo.setVenue(value);
      } else if("Name".equals(node.getNodeName())) {
       promo.setName(value);
      } else if("DateTime".equals(node.getNodeName())) {
       promo.setDateTime(value);
      }
     }
    }
   }
  } catch (Exception e) {
   e.printStackTrace();
   promo=null;
  } finally {
   
  }
  return promo;
    }
    private String getNodeValue(Node node) {
     NodeList children = node.getChildNodes();
     if(children.getLength()>0) {
      return children.item(0).getNodeValue();
     } else
      return null;
    }

Now that I have invoked the web services and got the data back to my Android phone, it is time to display them on screen. Android adopts a similar approach to Microsoft WPF, where you can define the view layout and styles in a XML file, so that the Java code is freed up from screen rendering code and can focus more on the business logic.

I created a style.xml file under the {project}/res/values directory to define the styles for my screen widgets.



    <style name="LabelText">
        18sp
        #fff
        fill_parent 
        wrap_content 
        5px
    </style>
    <style name="ContentText">
        14sp
        #000
        #e81
        true
        
        fill_parent 
        wrap_content 
    </style>

Then my view layout XML can use the styles. Here is the {project}/res/layout/main.xml file:



    <TextView style="@style/LabelText" android:text="Name:"/>
    <TextView style="@style/ContentText" android:id="@+id/tvName" />
    <TextView style="@style/LabelText" android:text="Description:"/>
    <TextView style="@style/ContentText" android:id="@+id/tvDescription" />
    <TextView style="@style/LabelText" android:text="Venue:"/>
    <TextView style="@style/ContentText" android:id="@+id/tvVenue" />
    <TextView style="@style/LabelText" android:text="Date:"/>
    <TextView style="@style/ContentText" android:id="@+id/tvDate" />

Note that by adding the android:text="@+id/tvName", the Android Eclipse Plugin will automatically regenerate the R.java file to add fields into the id class, so that you can reference the widget from Java code using findViewById(R.id.tvName). Here is the code for the onCreate() method:

/** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        HttpHost target = new HttpHost(SERVER_HOST, SERVER_PORT, "http");
        String keywords = getKeywords(target);
        
        setContentView(R.layout.main);
        if(keywords!=null) {
         PromoInfo promo = searchPromo(target, keywords);
         if(promo!=null) {
          ((TextView)this.findViewById(R.id.tvName)).setText(promo.getName());
          ((TextView)this.findViewById(R.id.tvDescription)).setText(promo.getDescription());
          ((TextView)this.findViewById(R.id.tvDate)).setText(promo.getDateTime());
          ((TextView)this.findViewById(R.id.tvVenue)).setText(promo.getVenue());
         }
        }
    }

Running the application:

Consuming Web Services By JavaME

2008-08-19T22:17:00.024+10:00

The J2ME was designed for very resource-constrained hardware. As the devices get more and more powerful (N95, iPhone, HTC Android phones), the JavaME's Mobile Information Device Profiles (MIDP) desperately need an upgrade. Before JSR-271 (MIDP 3.0) is finalised Sun has recognised the poor UI capability of JavaME and released the Light Weight UI Toolkit (LWUIT) to stay competitive. Another good 3rd-party framework is J2ME Polish.

Here I attempt to call the same web services that I developed as part of a demo using JavaME and then display the results using LWUIT.

I first tried to consume the web services developed using WCF. The Sun's Wireless Toolkit (WTK2.5.2) comes with a utility to generate SOAP client proxy from WSDL. (An alternative is to use KSOAP2). However, JavaME only supports a subset of JAXP and JAX-RPC as defined in JSR-172. My web service data contract contains a DateTime field, which violates JSR-172. Also it is mapped to java.util.Calendar, which is not supported by JavaME. So I could not call the SOAP web service as it is from JavaME. I guess I will have to wait for MIDP 3.0 or even later. In any case, full SOAP web services are too much of a heavy weight for mobile devices. It is better to use RESTful services.

Fortunately, I have also developed a RESTful version of the services using NetBeans 6.1 under the project SvdemoRestful. So I created a MIDP 2.0 project named SvdemoWUIT in NetBeans and included the LWUIT JAR in the project.

There are two quick and easy ways to consume RESTful services:

The first way is to use one of NetBeans MIDP wizards to create "Mobile Client to Web Application". This wizard will take a Web Application (or a Web App project in NB) to discover the RESTful services and generates a pair of servlet (in the web app project) and mobile client (in the MIDP project) code to wrap the service. The screenshot shows the generated files highlighted in red. Using the generated mobile client in the MIDlet:

// create an instance of the mobile client proxy
WebToMobileClient client=new WebToMobileClient();

// consume the web service via the client
String keywords = client.getText("http://localhost/MySpace_Com_John.htm");
System.out.println("keywords:"+keywords);
String promoXml = client.getXml(keywords);
System.out.println("promo:"+promoXml);

The corresponding stdout shows:

keywords:folk songs, pop music, chinese music ,battle, action, comedy
promo:801Batman The Dark Knight
                        Meet stars in Batman in person - Chritian Bale, Michael Caine.
                Star City2008-7-30 10:00:00Batman, action

There seemed to be a bug in the wizard: it only generated methods from one web service resource of the SvdemoRestful project although I specified both resources. Therefore, I had to modify the Utility.java and WebToMobileClient.java by hand to add the missing methods.

The second way is to use the javax.microedition.io.HttpConnection and deal with the low level stuff myself. I needed to parse the Promotion XML and to populate the PromoInfo object. JavaME only offers SAX for XML parsing, so I had to get the InputStream from the HttpConnection and pass it to the SAXParser. SAX uses callback mechanism. Therefore, a XmlHandler class had to be defined.

The MIDlet code snippet:

public class SvdemoWUITMidlet extends MIDlet implements ActionListener {
    static String promoUrl="http://localhost:8080/SvdemoRestful/resources/promo";
    // data object to hold the XML contents in its corresponding fields
    PromoInfo promo=null;
    public PromoInfo getPromo() {
        return this.promo;
    }
    public void setPromo(PromoInfo promo) {
        this.promo=promo;
    }
    public void startApp() {
        Display.init(this);
        HttpConnection hc=null;

        WebToMobileClient client=new WebToMobileClient();
        try {
            String keywords = client.getText("http://localhost/MySpace_Com_John.htm");
            System.out.println("keywords:"+keywords);
            String promoXml = client.getXml(keywords);
            System.out.println("promo:"+promoXml);
           
            hc=(HttpConnection)Connector.open(promoUrl + StringUtil.replace(keywords," ", "%20"));
            parse(hc.openInputStream());

            initialiseForm();
        } catch (IOException ex) {
            ex.printStackTrace();
        } finally {
            try { if (hc!=null) hc.close(); }
            catch(IOException ignored) {}
        }
    }

    private void parse(InputStream is) {
        SAXParserFactory factory = SAXParserFactory.newInstance();
        try {
            SAXParser saxParser = factory.newSAXParser();
            InputSource inputSource = new InputSource(is);
            saxParser.parse(is,new XmlHandler(this));

        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
...

The XmlHandler.java file:

/*
 * To change this template, choose Tools  Templates
 * and open the template in the editor.
 */

package svdemo;

import java.util.Stack;
import org.xml.sax.Attributes;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;

/**
 *
 * @author ROMENL
 */
public class XmlHandler extends DefaultHandler {
    static String TAG_PROMOTION="Promotion";
    static String TAG_NAME="Name";
    static String TAG_DESCRIPTION="Description";
    static String TAG_VENUE="Venue";
    static String TAG_DATETIME="DateTime";
   
    private SvdemoWUITMidlet midlet;
    private PromoInfo promo=null;
    private Stack tagStack=new Stack();
    public XmlHandler(SvdemoWUITMidlet midlet) {
        this.midlet=midlet;
    }

    public void startDocument() throws SAXException {}
    public void startElement(String uri, String localName, String qName,
    Attributes attributes) throws SAXException {
        if(TAG_PROMOTION.equals(qName)) {
            promo=new PromoInfo();
        }
        tagStack.push(qName);
    }
    public void characters(char[] ch, int start, int length)
            throws SAXException  {
        String chars = new String(ch, start, length).trim();

        if(chars.length() > 0) {
            String qName = (String)tagStack.peek();
     
            if (TAG_NAME.equals(qName)) {
                promo.setName(chars);
            } else if(TAG_DESCRIPTION.equals(qName)){
                promo.setDescription(chars);
            } else if(TAG_VENUE.equals(qName)) {
                promo.setVenue(chars);
            } else if(TAG_DATETIME.equals(qName)) {
                promo.setDateTime(chars);
            }
        }
    }

    public void endElement(String uri, String localName, String qName,
    Attributes attributes) throws SAXException  {
        tagStack.pop();
    }

    public void endDocument() throws SAXException {
        midlet.setPromo(promo);
    }
}

Finally, presenting the retrieved data on the mobile device - the initialiseForm() method in the MIDlet:

    private void initialiseForm() {
        Form f = new Form("JME LWUIT WS Demo!");
        
        f.setTransitionInAnimator(Transition3D.createCube(2000, false));
        f.setTransitionOutAnimator(Transition3D.createCube(2000, true));
        f.getStyle().setBgColor(0x0000ff);
        
        if(promo!=null) {
            f.addComponent(createLabel("Name:"));
            f.addComponent(createTextArea(1, promo.getName()));

            f.addComponent(createLabel("Description:"));
            f.addComponent(createTextArea(2, promo.getDescription()));

            f.addComponent(createLabel("Venue:"));
            f.addComponent(createTextArea(1, promo.getVenue()));

            f.addComponent(createLabel("Date:"));
            f.addComponent(createTextArea(1, promo.getDateTime()));
        }
        f.show();
      
        Command exitCommand = new Command("Exit");
        f.addCommand(exitCommand);
        f.setCommandListener(this);
    }
    
    private Label createLabel(String labelText) {
        Label label=new Label(labelText);
        Font font=Font.createSystemFont(Font.FACE_SYSTEM, 
                Font.STYLE_BOLD, Font.SIZE_MEDIUM);
        label.getStyle().setFont(font);
        return label;
    }
    
    private TextArea createTextArea(int rows, String text) {
        TextArea ta=new TextArea(rows ,20, TextArea.ANY);
        ta.setEditable(false);
        ta.setText(text);
        return ta;
    }

The following screenshots show the LWUIT 3D cube transition effect and the MIDlet output form respectively.

Consuming WCF Web Service Using Groovy Client

2008-08-18T12:05:00.016+10:00

Last month, I developed some simple web services using WCF and consumed it using various web service frameworks in Java. Now I have implemented the web service client using GroovyWS, which is an add-on module to Groovy. It uses CXF under the hood.

To add the GroovyWS library, I simply copied its JAR file groovy-all-1.5.6.jar into the Groovy's lib directory (it can either go in {groovy.home}/lib or {user.home}/.groovy/lib according to the groovy-starter.conf file). Then run the following script from Groovy Console:

import groovyx.net.ws.WSClient
def proxy = new WSClient("http://localhost/PromoService.svc?wsdl", 
                this.class.classLoader)
        
def keywords = proxy.GetKeywordsFromMySpace("http://someurl/somepage.htm")
if(keywords!=null) {
    if (proxy.HasPromo(keywords)) {
        def promo = proxy.GetFirstPromo(keywords)
        println "Promo: ${promo?.promoName?.getValue()}, ${promo?.promoDateTime}"
    }
}

gives the result:

BindingInfo = [BindingInfo http://schemas.xmlsoap.org/wsdl/soap/]
o = SOAPBinding ({http://schemas.xmlsoap.org/wsdl/soap/}binding):
required=null
transportURI=http://schemas.xmlsoap.org/soap/http
Promo: Batman The Dark Knight, 2008-07-30T10:00:00

That's all!

I then re-did it using Eclipse 3.4 with Groovy Plug-in to create the Groovy project by following the instructions from the same web site. Then I added the JAR files into the dependency:

ant.jar from Apache Ant 1.7.0 - this is needed by GroovyWS but not included in its distribution
groovy-all-1.5.6.jar

The Groovy code for the test client is very simple:

package svdemo

import groovyx.net.ws.WSClient

/**
 * @author ROMENL
 *
 */
public class Program{

 /**
  * @param args
  */
 public static void main(def args){
  def proxy = new WSClient("http://localhost/PromoService.svc?wsdl", 
    Program.class.classLoader)
  
  def keywords = proxy.GetKeywordsFromMySpace("http://someurl/somepage.htm")
  if(keywords!=null) {
   if (proxy.HasPromo(keywords)) {
    def promo = proxy.GetFirstPromo(keywords)
    println "Promo: ${promo?.promoName?.getValue()}, ${promo?.promoDateTime}"
   }
  }
 }
}

Compiling then running the program yielded:

18/08/2008 12:19:33 PM org.apache.cxf.endpoint.dynamic.DynamicClientFactory outputDebug
INFO: Created classes: org.tempuri.GetFirstPromo, org.tempuri.GetFirstPromoResponse, org.tempuri.GetKeywordsFromMySpace, org.tempuri.GetKeywordsFromMySpaceResponse, org.tempuri.HasPromo, org.tempuri.HasPromoResponse, org.tempuri.ObjectFactory, com.microsoft.schemas._2003._10.serialization.ObjectFactory, org.datacontract.schemas._2004._07.svdemo.ObjectFactory, org.datacontract.schemas._2004._07.svdemo.PromoInfo
BindingInfo = [BindingInfo http://schemas.xmlsoap.org/wsdl/soap/]
o = SOAPBinding ({http://schemas.xmlsoap.org/wsdl/soap/}binding):
required=null
transportURI=http://schemas.xmlsoap.org/soap/http
Promo: Batman The Dark Knight, 2008-07-30T10:00:00

Much of the output was produced by CXF. Only the last line was produced by the test program. As you can see, the major difference between the Groovy version and the Java version using CXF is that in Java, you will use the wsdl2java tool to generate the Java client proxy and the data contract classes up front; but in Groovy, all that is automatic when you run the script. This makes the Groovy code very simple and small. However, the downside of this is that the IDE (Eclipse) cannot know the methods and members of the class up front. Therefore, there is no code-completion (intellisense) support for the business methods. Since I heavily rely on code-completion of the IDEs, this is a big disadvantage about dynamic languages for me (I do not memorise all the class members and methods names, and I do not read all the API docs either. I usually try to figure out the methods I need by looking at their names).

There are a couple of things I had to do for my code to run without errors:

I had to upgrade my JRE used by Eclipse to a 1.6.0_u10rc. Eclipse automatically addes a bunch of JARs from the JRE, including the resources.jar, which includes the JAXB classes. In an earlier version of my JRE (1.6.0_u2) the JAXB was version 2.0, but the CXF bundled in GroovyWS 0.3.1 requires v2.1. However, if you are not using Eclipse (e.g. using the Groovy Console directly), this step is not required
The CXF bundled with GroovyWS 0.3.1 had problem generating the code from WSDL - it complained about a namespace "doesnt contain ObjectFactory.class or jaxb.index". It turned out to be a CXF bug and had been fixed in CXF 2.1.1, thanks to a post by Bernie. After I replaced the org/apache/cxf branch of the groovyws-all-0.3.1.jar using the contents in cxf-2.1.1.jar, it was smooth sailing.

ORM Frameworks and DBA Friendliness

2008-08-16T21:18:00.005+10:00

Database programming used to directly rely on tools provided by the database vendors, such as ESQL-C from Informix, Pro*C from Oracle and Sybase. Using these tools makes the business logic (C/C++ code) mingled with and dependent on the database design and logic (SQL code).

The emergence of Object-Relational Mapping (ORM) frameworks solved this problem by decoupling the code and the database assets. Nowadays, when designing software, it's not a matter of using ORM or not for the designer, but rather which ORM framework to use.

Most ORM or entity frameworks (Hibernate, JPA, Entity Framework) out there generate SQL statements automatically and hide the internals from the developers and rightly so. However, this creates another problem: to optimise the SQL statements, it is usually a joint effort between the developers and DBAs (or data architects) - the DBA should have the final say in how to shape the SQL queries and to design/modify the database schema to suit the usage pattern of the application(s). Now that the SQL statements are auto-generated by the framework and the DBA cannot design or examine the SQL code up front, the control of the DBA is diminished.

This loss of control may not be a problem for custom-built/one-time applications, or applications with a simply data model. However, if you are dealing with a large/complex product that has to be implemented in different environments, then it is crucial to have full control over the database deisgn and deployment and how it is to be used.

To have the visibility to the SQL statements up front, there seems to be two options:

resort back to JDBC/ADO.NET... to mix the SQL statements with the business logic code - not advisable
use a framework that exposes the SQL statements - iBATIS SQLMap

Strictly speaking, iBATIS is not an ORM framework - it maps the objects with SQL statement query input/output. There are two distinct advantages of using SQL Map:

The database-specific code (SQL statements, connection strings, etc.) can be centralised and totally separated from the core (Java/C#...) code. This makes the code decoupled from the database and more portable across different databases;
All database CRUD are specified in native SQL which are DBA friendly, making it easy for the DBA to review and participate in the design of these statements.

Because iBATIS does not map objects with tables, it makes it ideal for applicaitons that have to use legacy databases which did not adhere to object oriented design or have totally different structure from the object model.

Therefore, if you are involved in software product development (as opposed to custom-built one-time projects), DBA friendliness should be a factor to be considered when deciding an ORM framework.

The Life and Death of OC4J

2008-08-12T11:29:00.010+10:00

Once upon a time, two smart Swedes wrote a J2EE server called Orion and formed a company Ironflare to sell it. Orion server was totally different from all the other J2EE containers out there: it's light weight, blazing fast and dirt cheap (only a couple of grand USD for a commercial license, which was unheard of in the heyday of IT and internet)! It soon gained traction and almost gained a cult status in the J2EE world.

Then came the 21st century. In 2001, Oracle licensed Orion and created OC4J. The two smart Swedes from Ironflare moved with it and worked on OC4J for Oracle. Therefore, the Orion server and the website was virtually dead. However, Orion's heart still beats inside OC4J's body - you could even see the words 'orion' inside some of OC4J's configuration files. OC4J then became the core of Oracle Application Server for over half a decade, during which time the Oracle AS had grown to a humongous beast.

In January 2008, Oracle announced the merger with BEA, which created great FUD among OC4J and Weblogic communities - they were speculating whether Oracle will kill off Weblogic or OC4J. Yesterday, Oracle finally sounded the death knell for OC4J by announcing the release of Oracle Weblogic Server 10g R3 - it threw its weight behind WLS and is abandoning OC4J. This will no doubt create panick among OC4J user communities. I worked for a company whose core technology was Oracle Forms (yes, you heard that right!) and they used to be relatively calm about it because Oracle offered a migration path from Oracle Forms to its JSF framework, and for the meantime you can still run the Oracle Forms on Oracle AS 9i as Java applets (no longer supported in 10g though). Now that OC4J will disappear, that will certainly motivate them to migrate off the Oracle Forms technology sooner.

I don't believe the two smart Swedes will go back to resurrect Orion server because there are several free JavaEE servers nowadays, which take away the incentive for people to use Orion. Besides, Orion has been disconnected with the Java community for so long, it's fan base has all but evaporated.

Orion & OC4J

Beijing 2008

2008-08-11T14:23:00.008+10:00

The Beijing 2008 Olympic opened with the most spectacular opening ceremony ever. It was China's coming-out party and China spared no effort and expense to impress the world - and indeed it did.

To me, the weeekend was pretty much an Olympic weekend - watching the events on TV and internet, reading the latest news and results from the Beijing2008.com website.

The website is managed by China's largest information portal - sohu.com. Sohu deployed 600 quad-core Intel Xeon powered servers to host the website, plus 300 more as backup servers. The massive infrastructure supports millions of concurrent users and billions of hits a day. On the software side, they used Apache 2.2.3 running on Red Hat Linux and the web technology is the good old shtml! This is an interesting software stack and all open source and free. It's no secret that the Chinese government favours Linux - who doesn't?! The combination of shtml over Apache is quite appropriate for websites like this: contents are manually input and updated; the need for a database is very low.

shtml is good for using with template pages and then dynamically include the corresponding contents from other files. There is really no need for a full-blown database server for an Olympic website - the Game results are not particularly large or complicated, they are not even uniformly structured (each event has its own rules and ways of expressing the results). All you need is a good content management system (and indeed Sohu used one, which likely uses a database server itself) due to the frequent and current updates to the contents and their distribution to the large number of servers.

I still remember the Sydney 2000 Games where IBM was contracted to provide the IT infrastructure. Unsurprisingly, IBM used Websphere and Java technology to implement the official website. With such resource-hungry beasts, I'd image they needed at least double the amount of hardware to support the same level of service (even if they were as powerful as the quad-core Xeon servers).

Killer Frameworks vs. Killer Apps

2008-08-08T14:18:00.007+10:00

I came across this post on Java Lobby the other day, which put a grin on my face. Basically the author of the post was questioning how come there are so many programming libraries and frameworks out there and people talk about them so much, but no one seem to talk about business applications with the same level of enthusiasm.

This phenomenon is something that I observe everyday yet never realised it until reading the post. Now that I come to think about it, I can find several reasons for this.

First of all, the web sites such as Java Lobby cater for the software developer communities. Software developers tend to value technical knowledge more than business (domain) knowledge. After all, domain knowledge experts are usually titled business analysts, subject matter experts, and the like. The role of developers is mainly solving problems with technical tools. If the same business problem can be solved by non-technical means, e.g. lobbying, re-organising, developers are kept out of the exercise. So it's no wonder developers talk about technical tools all the time.

Of course, there are developers who are also interested in business knowledge and excel in it. However, the very nature of domain knowledge is, well... domain-specific. This means that what you know and what you blog about may not be understood or appreciated by the rest of the developer community. Therefore, this kind of discussions are rare. (When I tell people that I work in the Telco industry in a birthday party, they automatically assume that I know how to install telephone lines into their houses).

Also, the IT industry is very dynamic and people move around fairly frequently. So after such moves, the domain knowledge may also change and new killer app ideas will germinate and the old killer app ideas wither. Without longevity, it is pretty hard to grow an idea into a full-blown product.

Another reason for the phenomenon is that discussion about business products are not as generic as talking about technology such as Java which actually covers many topics. By comparison, the area covered by a business product is very small. I am sure you can find blogs about great gaming ideas on gaming sites. But I doubt that the same gaming site also talk about billing applications. Therefore, their audience is also quite limited.

Why Not Five-teen?

2008-08-05T22:38:00.006+10:00

My four-year-old son has just learnt how to count. As a newbie, he has not memorised all the numbers. So sometimes, he improvises and derives the number words that he needed. For example, when he sees '15', he would read it as 'five-teen'. He is OK with the numbers that are 'regular', such as 14, 16, 17... But he could not readily read the irregular ones all the time, such as 11 and 12.

At this young age, he has not been 'tainted' by all the variations and irregularities of the English spelling and pronunciation rules. It is natural that he follows the 'logical' way to come up what he thinks and seemingly to be correct.

The English language is full of exceptions and variations in its rules of spelling, pronunciation and grammar. These inconsistencies make English not an easylanguage to learn comparing to many others.

When it comes to computing languages and APIs, consistency also determines whether the language or API is easy to learn and use (although now people are talking about more advanced topics, such as fluency). Let's take GUI frameworks as an example.

When building graphical user interfaces, we need to put some widgets/components onto the canvas or another widget. So there is a parent-child relationship: the parent is the container, the child is the object being put inside the container. There are two ways of saying this relationship:

parent.addChild(child)
child.setParent(parent) or specify the parent in the child's constructor

Most GUI frameworks use the first syntax - such as Swing, Windows Forms, WPF, ASP.NET, GWT, Echo2. For example, in Swing:

JButton button=new JButton("Clear");
button.setActionCommand(command);
button.setToolTipText(tooltip);
button.addActionListener(this);
...
JPanel buttonPanel=new JPanel(new FlowLayout());
buttonPanel.add(button);

In these GUI frameworks, the same widget cannot be added to more than one container. If you did, then it either complains at runtime or gives undesired odd results. But the API allows you to make these mistakes and your code compiles without error. So this is an inconsistency in those framework APIs.

To fix this problem, SWT/JFace takes the second approach by using the constructor:

Composite buttons=new Composite(this, SWT.NONE);
...
Button clearButton=new Button(buttons, SWT.PUSH);

This way, you cannot mistakenly set more than one parent for the widget.

There are other languages and frameworks that take a third approach by creating the child widget inside the code block of the parent. For example, in Wicket:

public abstract class BaseAddressForm extends Form {
...
  void initForm() {
  ...
    add(new Button("clear", new Model("Clear")) {
      @Override
      public void onSubmit() {
        log.info(getModelObject());
        resetModel();
      }
    });
  }
...
}

Similarly, in JavaFX script:

...
bottom: FlowPanel {
  content: [
    Button {
      text: "Clear"
      action: operation() {
        model.selectedAddressIndex=-1;
      }
    },
...

The above are exmaples within a single framework. When we have to deal with multiple frameworks the problem becomes more obvious. Normally, we don't expect different frameworks/languages to follow the same rules. However, when one framework is an add-on/extension to another, our expectation changes. For example, GWT-Ext is an add-on to GWT. However, its API naming conventions are not consistent with that of GWT: in GWT the method name getText() is used to get the text label of a widget (e.g. a label, button, etc.); but in GWT-Ext the method is called getValue(). Also, in GWT, the Command is widely used (in a similar way to the Action class in JFace). However, in GWT-Ext event listeners have to be explicitly created...

Don't get me wrong, I love using GWT and GWT-Ext. But these little inconsistencies hinder productivity when using this mix.

RESTful Web Service

2008-08-02T14:11:00.019+10:00

I built a SOAP web service using WCF as a part of a demo. However, as it turned out, the client software I was given can only handle HTTP as transport (the WCF service I did accepts requests directly from TCP payload). So I had to convert it into a RESTful web services. I have basicly a couple of options to implement the RESTful web services:

Modify the existing WCF project to support REST style according to MSDN
Create a ASP.NET project and reuse the code that I created in the WCF project.
Create a Java web project and rewrite the code.

The 1st option seems the best choice on the surface. But there are still too much of hand-rolling of boiler-plate type of code to be done in option 1. You have to process the HTTP request method using if/switch ... yuck.

The 2nd option will save me some time because I can reuse the LINQ to XML code. But the code is very short, so the saving is not big. Again, the disadvantage of this option is that I have to create all the virtual path mapping (extending the VirtualPathProvider class) myself. The time to spend on building the ASP.NET web application from scratch will outweigh the savings on reusing the bit of XML query code (although there is a pretty good sample by Doug Seven from where I can steal some code.)

NetBeans (v6.1) has a RESTful web services (JSR-311) plugin using Jersey libraries. The wizard can create all the boilerplate codes and configurations required to get a RESTful service up and running by a few simple steps. The downside of this approach is that I have to rewrite the web service implementation methods - XML query, HTML scraping... However, the code for those is quite simple and should be easy to rewrite.

So, ASP.NET is lagging behind in the area of RESTful web services. I guess to get the similar features offered by JSR-311 and NetBeans, one has to wait for the upcoming MVC Framework for ASP.NET. An excellent tutorial and example project using the MVC Framework can be found here.

Since time is of essense in this exercise, I chose option 3.

I started by creating a Java Web Application (called SvdemoRestful) in NetBeans. Then adding a RESTful web service is simply right-clicking the project and choose New -> RESTful Web Services from Patterns... NetBeans also offers a wizard to create RESTful Web Services from Entity Class. I did not need to use Entity Class because I am going to use XPath to query a simple XML file (as my database) and return the same XML snippet to the service consumer (i.e. I don't even need to convert the XML into a Java bean);and I don't want to use the ORM tools and create Persistent Objects. Then it's just a matter of filling in the self-explanatory fields in the wizard.

Once this is done, the skeleton code is generated along with all the necessary web configuration settings (e.g. url mapping). I want to have my RESTful service to have URI like below so that the GET method will return the first promotion that matches the keywords (actually, it is better to use query parameters instead of path parameters for keyword searching services):

http://hostname/SvdemoRestful/resources/promo/jazz, action

To achieve this, the @Path annotation is defined as

@Path("promo/{keywords}")

I used XPath to retrieve the Promotion record from the data.xml file. I need to make the keyword matching case in-sensitive. In XPath 2.0 there is an upper-case() function, but it does not seem to be supported by the XPath libraries. So I have to use the following XPath statement:

//Promotion[contains(translate(Tags,'abcdefghijklmnopqrstuvwxyz','ABCDEFGHIJKLMNOPQRSTUVWXYZ'),'keywordsToBeMatched')]

as opposed to

//Promotion[contains(upper-case(Tags),'keywordsToBeMatched')]

So the resulting code for the promo RESTful web service:

/*
 *  PromoResource
 *
 */

package svdemo;

import java.util.logging.Level;
import java.util.logging.Logger;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.UriInfo;
import javax.ws.rs.ProduceMime;
import javax.ws.rs.ConsumeMime;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.GET;
import javax.ws.rs.PathParam;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Node;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathFactory;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;

/**
 * REST Web Service
 *
 * @author ROMENL
 */

@Path("promo")
public class PromoResource {
    static String EMPTY="";
    @Context
    private UriInfo context;

    /** Creates a new instance of PromoResource */
    public PromoResource() {
    }

    /**
     * Retrieves representation of an instance of svdemo.PromoResource
     * The URL parameters are:
     * keywords - comma delimited words to be matched
     * @return an xml string representing the PromoInfo
     */
    @GET
    @Path("{keywords}")
    @ProduceMime("text/xml")
    public String getXml(@PathParam("keywords") String keywords ) {
        if (keywords == null || keywords.length()==0) {
            return EMPTY;
        }
        String[] words = keywords.split(",");
        try {
            DocumentBuilderFactory domFactory = DocumentBuilderFactory.newInstance();
            domFactory.setNamespaceAware(true);
            DocumentBuilder builder = domFactory.newDocumentBuilder();
            Document doc =  builder.parse("data.xml");
            
            XPathFactory factory = XPathFactory.newInstance();
            XPath xpath = factory.newXPath();
            for(String keyword : words) {
                XPathExpression expr = xpath.compile(
                        "//Promotion[contains(translate(Tags,'abcdefghijklmnopqrstuvwxyz','ABCDEFGHIJKLMNOPQRSTUVWXYZ'),'"
                        +keyword.trim().toUpperCase()+"')]");
                Node result = (Node)expr.evaluate(doc, XPathConstants.NODE);
                if(result!=null) {
                    StringBuffer sb=new StringBuffer("");
                    NodeList nodes = result.getChildNodes();
                    for(int i=0; i<nodes.getLength(); i++) {
                        Node node = nodes.item(i);
                        if(!"#text".equals(node.getNodeName())){
                            sb.append("<"+node.getNodeName()+">");
                            sb.append(node.getTextContent());
                            sb.append("");
                        }
                    }
                    sb.append("");
                    return sb.toString();
                }
            }
            return EMPTY;
        } catch (Exception ex) {
            Logger.getLogger(PromoResource.class.getName()).log(Level.SEVERE, null, ex);
            return EMPTY;
        }
    }

    /**
     * PUT method for updating or creating an instance of PromoResource
     * @param content representation for the resource
     * @return an HTTP response with content of the updated or created resource.
     */
    @PUT
    @ConsumeMime("application/xml")
    public void putXml(String content) {
    }
}

In another service I needed to pass in a URL so that the service will get the web page of the URL and scrape the HTML code (yes, scraping HTML is a bad practice, but I don't have any alternatives and it's OK for demo purposes). The HTML scraping is easily handled by using the javax.swing.text.html.HTMLEditorKit. The problem is that the input parameter to this RESTful service contains forward-slashes ('/') and it will confuse the web server if I give the URLs like so:

http://hostname/SvdemoRestful/resources/webPageKeywords/http://localhost/someWebPage.html
http://hostname/SvdemoRestful/resources/webPageKeywords/'http://localhost/someWebPage.html'

According to JSR-311, the above urls should be specified as @Path("webPageKeywords/{url:.+}"), but it did not work. I tried different values of the limited and encode attributes of the @Path(), but to no avail. So I had to resort to passing the parameters as the URL's query parameter:

http://hostname/SvdemoRestful/resources/webPageKeywords?url=http://localhost/someWebPage.html

The corresponding @Path annotation is just @Path("webPageKeywords"). The corresponding service implementation code snippet:

/**
 * REST Web Service
 *
 * @author ROMENL
 */

@Path("webPageKeywords")
public class WebPageKeywordsResource {
    @Context
    private UriInfo context;

    /** Creates a new instance of WebPageKeywordsResource */
    public WebPageKeywordsResource () {
    }

    /**
     * Retrieves representation of an instance of svdemo.WebPageKeywordsResource 
     * @return empty string if no keywords found; otherwise, comma delimited keywords
     */
    @GET
    @ProduceMime("text/plain")
    public String getText(@QueryParam("url") String urlString) {
        if(urlString==null || urlString.length()==0)
            return "";
        try {
            URL url = new URL(urlString);
            HTMLEditorKit kit = new HTMLEditorKit();
            HTMLDocument doc = (HTMLDocument) kit.createDefaultDocument();
            doc.putProperty("IgnoreCharsetDirective", Boolean.TRUE);
            Reader HTMLReader = new InputStreamReader(url.openConnection().getInputStream());     
            kit.read(HTMLReader, doc, 0);
            String keywords="";
            Element e=doc.getElement("ProfileMusic");
            if(e!=null) {
                keywords=doc.getText(e.getStartOffset(), e.getEndOffset()-e.getStartOffset());
            }
            e=doc.getElement("ProfileFilms");
            if(e!=null) {
                keywords+=","+doc.getText(e.getStartOffset(), e.getEndOffset()-e.getStartOffset());
            }
            return keywords;
        } catch (Exception ex) {
            Logger.getLogger(WebPageKeywordsResource .class.getName()).log(Level.SEVERE, null, ex);
            return "";
        }
    }

    /**
     * PUT method for updating or creating an instance of WebPageKeywordsResource 
     * @param content representation for the resource
     * @return an HTTP response with content of the updated or created resource.
     */
    @PUT
    @ConsumeMime("text/plain")
    public void putText(String content) {
    }
}

I am looking forward to the new ASP.NET MVC Framework and will definitely rewrite these services using it.

Telco APIs

2008-07-30T20:16:00.009+10:00

The communications industry (telecommunications and internet) must have the largest number of standards specifications - from the layering of the network model, to individual protocols inside each layer. Even the same standard can be defined by more than one standard bodies - ITU-T, ESTI, ANSI, etc. The standards are very necessary in the communications industry. Afterall, communications are all about being able to interwork/interact with each other. So a common language is needed.

Traditionally, the standards stayed at the protocol level - i.e. to ensure that different systems/subsystems can communicate with each other. Typical exmaples are SS7 (and SS7 based protocols), VoIP protocols (SIP, H.323). More recently, as the industry matures, standard programming APIs have become prevalent, e.g. Parlay/X, OSS/J, JAIN APIs. These APIs enables inter-operability among different systems and system vendors. It also cuts down the effort of individual software vendors because much of the specification and documentation have already been done by the community/expert groups. Leveraging these standard APIs increases the software vendors' productivity and ensures smoother integration with other systems.

I have been involved in service development using legacy Service Creation Environment (SCE), which provides proprietory drag-and-drop type of 'programming' environment (think Visio). The legacy SCE boasts that no programming experience is needed and only drawing of the flow diagram is required. On the surface, this seems an attractive paradigm (apart from lack of documentation - zero documentation really), however, once you need to go beyond the surface, it's nothing but pure frustration. When you are stuck, you have to ask the vendor to provide another service building block (SBB) in order to put it onto the flow diagram. On the contrary, in a JAIN SLEE environment, a developer can write new SBBs using Java following the JAIN APIs and architecture. This gives the developer power and flexibility over the legacy approach.