This e-card was written in JavaFX. The source code can be found here.
If you encounter problems seeing it, I am not surprised.Related Posts:
Wednesday, 31 December 2008
Thursday, 25 December 2008
Merry Christmas!
2008 has been a disastrous year in many aspects all over the world - from cyclone Nargis to the earth quake in SiChuan China; from Mumbai Massacre to mobocracy in Thailand... and let's not forget the global economic crisis...
Now more to home, following the down-sizing trend a company that I know of has decided to wind down the operation in Sydney just 2 weeks before christmas. As a result, people in the Sydney office have been made redundant. Come to think of it, the holiday season is perfect to fire people. Why? During Christmas, New Year and Australia Day (January 26) people's productivity is not at its peak. Firing them during this period will save money for the company left and right - it is a win-win situation for the company! Being a profit-chasing business entity, it is perfectly logical to 'do the right thing' in the holiday season. After all, humans are nothing but resources to a company - hence the term HR.
In the story, Mr Scrooge came to his senses and became a better human being in the end. But in real life, the reverse is usually true...
Saturday, 20 December 2008
Turn Off 3G To Save The Environment!
My N95 battery could never last more than 16 hours even when sitting idle. I had to recharge it every night. This week when I was in Kathmandu, I was pleasantly surprised: my battery actually lasted over 36 hours during which I snapped more than 70 photos , played 20 minutes of music and had a couple of phone calls (totaling over 15 minutes of talk time maybe). I noticed that there was no 3G network in Nepal (or at least I was not roamed to one). I believe that is why my battery lasted so long. That fact that I live on the edge of the cell probably exacerbated the situation.
So I decided to disable 3G on my phone. Normally, this is a straight forward operation available from the Settings -> Phone menu. However, my phone was from 3 Australia (product code: 0547465) and it was branded - meaning that it was customised for 3 Australia: some 3 applications was installed, some menu items have been disabled... and the menu item to turn off 3G has been disabled. Super keen on saving the environment by extending my battery life 
I set out to debrand the phone and turn off 3G!
The process of debranding is as following:
- Backup the phone using PC Suite
- Change the product code by following the instructions. The new product code I used is 0536087. Note that the NSS could not detect any phone on my notebook. In the end I installed NSS on my desktop (also running XP) using root account and that all worked fine. I don't really know why it would not work on my laptop.
- Update the firmware using Nokia Software Updater
- Restore the phone using PC Suite. The restore is not perfect and I still have to re-arrange/reinstall some applications. And all my messages and contacts are lost!
On the negative side, there are some major drawbacks upgrading to the 30.x version of the firmware. TRK no longer works and I had to install alternative (but better) solution. Fontrouter does not seem to work any more in the new browser (because it does not have the character encoding that I need) and I got around the problem by installing Opera Mini instead.
Tuesday, 16 December 2008
The Sorry State of JavaFX 1.0
The release of JavaFX 1.0 officially marks Sun's defeat in the RIA battle. Why? Because JavaFX 1.0 is inferior to its competitors in all major battlefronts of the RIA war. It looks like a rushed job to make a marketing deadline.
The JavaFX 1.0 APIs
The JavaFX 1.0 is shipped with 2D graphical capabilities and that's pretty much it. Even basic things such as 3D graphics, faster video codecs are labeled with 'coming soon...' on their demo site. Forget about building any business RIAs because there are only a handful of simple Swing widgets at your disposal. On the other hand, people have been using Flash to build business applications for years; Silverlight has got 3D capabilities since v1.0.
Design/Development
The JavaFX plugin for NetBeans is a joke - it's buggy, sluggish and poor in features. One mistyped word or bracket can result in red squiggly lines all over the screen; then it produces unhelpful error hints such as 'sorry I was expecting so and so but I saw such as such...' so that the user has to read through the whole sentence several times and still hard to figure out what the complaint is. What happens to KISS principle?
Sure, Sun had said that they intended the plugin to be used by designers also - hence the 'user friendly' error messages. But let's get serious, how can anyone assume graphical designers will be using computer programming language to define simple things such as Timelines? Sun doesn't have to look far to find a better way - Microsoft has Expression Web to be used by graphical designers, and VS.NET 2008 to be used by programmers when it comes to Silverlight design and development workflow. (The JavaFX plugin for Illustrator is only good for editing shapes).
Deployment
To me there is not much point of deploying RIA as a thick client using Java Web Start. I am only interested in running RIA in a browser. With the new way of deploying applets in Java, JNLP is used, so that the HTML file points to a JNLP file and the JNLP file points to the JAR files and more JNLP files... It's a bit like my multi-hop flight from Sydney to Kathmandu (via Bristane then Bangkok) that took me over 20 hours and 6 of which was waiting at BKK for transit (that that is how I have the time to write this post) all because my company was too cheap to book the Qantas flight.
This is unnecessarily complicated because Sun doesn't want to have JavaFX plugin to the browser. On the other hand, Flash/Silverlight plugins are installed onto the browser so that all you need to specify is the object or embed tag in the HTML file without the hassle of running the myriad javascripts to detect/download the additional framework JAR files every time.
The User Experience
If you are using a browser that does not have the latest Java plugin (that supports JavaFX) installed (such as on my Nokia N95, or using the PC in the Thai Airways lounge at Kathmandu international airport), then you will be greeted with a message box forcing you to redirect to Sun's download website. There is no option to cancel!
Moreover, those javascripts do not report any errors if something goes wrong - it just hangs there with the Java logo spinning around and around forever...
Wednesday, 10 December 2008
Merry Christmas JavaFX 1.0
JavaFX 1.0 was finally released last Friday. After over 2 years of brooding, I wonder what Sun has come up with - oh boy what a disappointment the JavaFX IDE (NB plug-in) is! But that is for another post.
The JavaFX 1.0 is no good for developing any RIA business applications for its lack of widgets. Instead it is OK for little toys such as those demos published on javafx.com. Immersed in the holiday season atmosphere, I implemented my Christmas card in JavaFX. A screenshot is shown below.
The Christmas card has the following features:
- Transformation - the red and yellow text fly into the screen using a combination of Translation, Rotation and Scale.
- Media playback - background music (mp3 file) from Michael Buble - Let it snow.
- Animation - snow fall simulation
- Drag out - when run as applet in a browser, the card can be dragged out of the browser.
The Project Structure
The project was created using NetBeans 6.5 with JavaFX plug-ins (installed from within NetBeans). I created a JavaFX project called MerryChristmas. The project structure is shown below:
The only files that I added are:
LetItSnow.mp3- the background music songtree.png- the background image: photo of Christmas tree shot at Darling Harbour on December 8, 2008.Main.fx- the main JavaFX file containing the scene and its contentsSnowFall.fx- theCustomNodesimulating snow fall and snow flakes (using circle)
The Main Contents
The contents of the main scene are two Texts, one background image, one background music and the snow fall (implemented as CustomNode).
The texts and their animation (actually transformation) are created as shown below. Notice the transforms which use binding data changed by the Timeline definition.
...
var x:Number;
var y: Number;
var scaleX : Number;
var scaleY:Number;
var angle:Number;
Timeline {
repeatCount: 1
keyFrames: [
at (0s) {x=>180; y => 250; scaleX => 0.0; scaleY => 0.0; angle => -180},
at (10s) {x=> 0; y => 0;
scaleX => 1.0; scaleY => 1.0;
angle => 0
tween Interpolator.EASEOUT},
]
}.play();
...
var lighting = Lighting{
light: DistantLight{
azimuth: 60,
elevation: 70
}
surfaceScale: 5
};
Stage {
...
scene: Scene {
content: [
...
Text {
content: "Merry Christmas & \nHappy New Year!"
font: Font.font("Arial Bold", FontWeight.BOLD, 40)
textOrigin: TextOrigin.TOP
textAlignment: TextAlignment.CENTER
x: 10
y: 20
transforms: bind[
Translate{
x:x
y:y}
Scale{
x:scaleX
y:scaleY}
Rotate{
angle: -angle
pivotX: 180
pivotY: 20 }
]
fill: Color.RED
effect: lighting
},
Text {
content: "From Romen"
font: Font.font("Times", FontWeight.BOLD, 40)
textOrigin: TextOrigin.TOP
x: 90
y: 470
transforms: bind[
Translate{
x:x
y:y}
Scale{
x:scaleX
y:scaleY}
Rotate{
angle: angle
pivotX: 180
pivotY: 450}
]
fill: Color.YELLOW
effect: lighting
},
...
Any suggestions on how to make unicode text work are welcome.
The background image and music are easy enough:
...
ImageView {
image: Image {
url: "{__DIR__}tree.png"
}
x:0,
y:0
},
...
MediaView {
mediaPlayer: MediaPlayer {
autoPlay: true
media: Media {
source: "{__DIR__}LetItSnow.mp3"
}
}
},
...
Drag Out
To enable drag out when run as an applet in browser, I added the following extension to theStage:
extensions: [
AppletStageExtension {
shouldDragStart: function(e): Boolean {
return e.primaryButtonDown;
}
useDefaultClose: false
}
]
Also I had to specify in the Project's properties dialog and check the Draggable checkbox so that it will generate the .html file with the draggable set to true. The generated .html file snippet:
... ...
Snow Fall
All snow related classes are in theSnowFall.fx file. I used white circle as the snow flake:
public class SnowFlake extends Circle {
init {
fill = Color.WHITE;
radius = 3 + Math.random() * 3;
opacity= Math.random() * 0.5 + 0.5;
}
}
The SnowFall class takes 3 attributes:
height, width- these specify the size of the canvas so that all the snow flakes can spread out in the canvas making it more realisticnumberOfFlakes- the number of snow flakes to be drawn and animated. The more flakes, the more threads JavaFX will have to create and manage under the hood.
KeyFrame {
...
values: [
x =>
x + (Math.random() - 0.5)
y =>
y + Math.random() tween Interpolator.LINEAR
]
...
When the snow flake hits either side of the boundary, it sort of bounces back a little; when it moves off the bottom of the canvas, it is wrapped around and re-appears at the top of the canvas and falls down again. This is achieved by using action of Keyframe which are executed at the beginning of each key frame.
...
action: function() {
if(flake.centerY+y >height) {
y=0; flake.centerY=0;
}
if(flake.centerX+x>width) {
x=0; flake.centerX=width;
}
if(flake.centerX+x<0) {
x=0; flake.centerX=0;
}
}
...
The SnowFall class generates all the snow flakes in a for loop. Each snow flake is accompanied by a Timeline controlling its animation. The number of frames in each Timeline is also randomised (between 60 and 120 seconds per cycle), to avoid the situation where all snow flakes suddenly reset at the same time.
for ( i in [1..numberOfFlakes]) {
...
def timer = 60+60*Math.random();
Timeline {
repeatCount: Timeline.INDEFINITE
keyFrames: [ for (j in [0..timer]) {
KeyFrame {
...
Want to see the Christmas card in action? Well, you will have to wait till Christmas! [Update NYe2008]Now that it is NYE 2008, click here to run the e-card.
Meanwhile, here are the complete source code.
SnowFall.fx
/*
* SnowFall.fx
*
* Created on 10/12/2008, 09:55:19
*/
package merrychristmas;
import java.lang.Math;
import javafx.animation.Interpolator;
import javafx.animation.Timeline;
import javafx.scene.CustomNode;
import javafx.scene.Group;
import javafx.scene.Node;
import javafx.scene.paint.Color;
import javafx.scene.shape.Circle;
import javafx.animation.KeyFrame;
import java.lang.System;
/**
* @author ROMENL
*/
public class SnowFall extends CustomNode {
public var height:Number;
public var width:Number;
public var numberOfFlakes:Number;
override function create():Node {
return Group {
content: [
for ( i in [1..numberOfFlakes]) {
var x:Number;
var y:Number;
x=1;
y=1;
var flake = SnowFlake{
centerX: Math.random() * width
centerY: Math.random() * height
translateX: bind x;
translateY: bind y;
}
def timer = 60+60*Math.random();
Timeline {
repeatCount: Timeline.INDEFINITE
keyFrames: [ for (j in [0..timer]) {
KeyFrame {
time: bind Duration.valueOf(j * 1000)
values: [
x =>
x + (Math.random() - 0.5)
y =>
y + Math.random() tween Interpolator.LINEAR
]
action: function() {
if(flake.centerY+y >height) {
y=0; flake.centerY=0;
}
if(flake.centerX+x>width) {
x=0; flake.centerX=width;
}
if(flake.centerX+x<0) {
x=0; flake.centerX=0;
}
}
}
}
]
}.play();
flake
}]
};
}
}
public class SnowFlake extends Circle {
init {
fill = Color.WHITE;
radius = 3 + Math.random() * 3;
opacity= Math.random() * 0.5 + 0.5;
}
}
Main.fx
/*
* Main.fx
*
* Created on 9/12/2008, 15:29:24
*/
package merrychristmas;
import javafx.animation.*;
import javafx.scene.effect.*;
import javafx.scene.effect.light.*;
import javafx.scene.image.*;
import javafx.scene.input.MouseEvent;
import javafx.scene.media.Media;
import javafx.scene.media.MediaPlayer;
import javafx.scene.media.MediaView;
import javafx.scene.paint.Color;
import javafx.scene.Scene;
import javafx.scene.text.*;
import javafx.scene.transform.Rotate;
import javafx.scene.transform.Scale;
import javafx.scene.transform.Translate;
import javafx.stage.AppletStageExtension;
import javafx.stage.Stage;
/**
* @author ROMENL
*/
var width=390;
var height=540;
var x:Number;
var y: Number;
var scaleX : Number;
var scaleY:Number;
var angle:Number;
Timeline {
repeatCount: 1//Timeline.INDEFINITE
keyFrames: [
at (0s) {x=>180; y => 250; scaleX => 0.0; scaleY => 0.0; angle => -180},
at (10s) {x=> 0; y => 0;
scaleX => 1.0; scaleY => 1.0;
angle => 0
tween Interpolator.EASEOUT},
]
}.play();
var lighting = Lighting{
light: DistantLight{
azimuth: 60,
elevation: 70
}
surfaceScale: 5
};
Stage {
title: "Merry Xmas JavaFX"
width: width
height: height
scene: Scene {
content: [
ImageView {
image: Image {
url: "{__DIR__}tree.png"
}
x:0,
y:0
},
Text {
content: "Merry Christmas & \nHappy New Year!"
font: Font.font("Arial Bold", FontWeight.BOLD, 40)
textOrigin: TextOrigin.TOP
textAlignment: TextAlignment.CENTER
x: 10
y: 20
transforms: bind[
Translate{
x:x
y:y}
Scale{
x:scaleX
y:scaleY}
Rotate{
angle: -angle
pivotX: 180
pivotY: 20 }
]
fill: Color.RED
effect: lighting
},
Text {
content: "From Romen"
font: Font.font("Times", FontWeight.BOLD, 40)
textOrigin: TextOrigin.TOP
x: 90
y: 470
transforms: bind[
Translate{
x:x
y:y}
Scale{
x:scaleX
y:scaleY}
Rotate{
angle: angle
pivotX: 180
pivotY: 450}
]
fill: Color.YELLOW
effect: lighting
},
MediaView {
mediaPlayer: MediaPlayer {
autoPlay: true
media: Media {
source: "{__DIR__}LetItSnow.mp3"
}
}
},
SnowFall {
height: height
width: width
numberOfFlakes: 30
}
]
}
extensions: [
AppletStageExtension {
shouldDragStart: function(e): Boolean {
return e.primaryButtonDown;
}
useDefaultClose: false
}
]
}
Related Posts:
Thursday, 4 December 2008
The Long Tail Theory
In the Telco business we often talk about the long tail products/services illustrated in the following diagram.
The idea is that for a service provider to stay competitive, it has to deploy and experiment with a large number of products and see how they are accepted by the market. There will be a small number of popular ones and the rest will be less popular as indicated in the long tail. So, over time, the service provider will kill the proven unpopular products and replace them with new/other products. You may wonder, why do they bother with the long tail since they are not that popular anyway? There are two major reasons:
- The service providers are too paranoid not to provide those services because they want to stay competitive - anything that their competitors have, they also want it.
- Other long tail graphs replace the Popularity label with Revenue. It's not hard to see that the total area under the Long Tail curve (i.e. total revenue) could add up to an handsome amount.
How is this theory relevant to me? Well, being an IT professional, I am also paranoid about my knowledge and skills being stale. So I have to invest my time and effort on updating and expanding my knowledges constantly. With all these new technologies and frameworks popping up all the time, I feel like a headless chook all the time.
The best way to learn a new language/framework/library is to actually use it. 'Hello World' just doesn't cut it. Yet a full blown application such as Web Mail or Pet Shop may be too big or too long for the purpose of learning or proof of concept. That is why I have my own little pet project - Address (among others), which serves as the use cases for my learning and experimentation.
For the products that I like and feel useful in my work, I will keep them in my toolbelt and probably move them up to the head. For the ones that are irrelavent to me, I drop them from the Long Tail picture altogether. There are also products that have died (e.g. Orion Server) or dying and being replaced by others (e.g. GWT-Ext) in the game of evolution. But it does not mean that my time spent on them were wasted. I still learn and accumulate a great deal of experiences so that when I encounter a new product, I can always draw on these experiences and view it in a better light.
Monday, 1 December 2008
GWT vs. ZK - SLOC Count
I often read about how compact the ZK code was compared to GWT. I couldn't quite believe it. I thought it must have been different design patterns and style of coding used. So I decided to do it myself - I implemented my pet project in both GWT-Ext (GWT 1.4.6 and GWT-Ext 0.9.3 in December 2007) and ZK (ZK 3.5.1 with ZK Studio 0.9.0 in November 2008) and compared the Source Line of Code (SLOC) counts using Code Count tool.
Before I show the SLOC comparison, I need to give a brief introduction of the little demo program that I wrote. The program is a GUI to show various capabilities of the Address package, which has a service tier and DAO implementations behind it. The Address package provides a meta-data driven way of managing address records (see here for the data model). The concerned application here is only the GUI portion. The GUI has the following screens (left=ZK, right=GWT):
CRUD
This screen is used for Create, Retrieve, Update and Delete (CRUD) of the Address records from/to the database.
Find
This screen is used to call the corresponding Address services to find addresses either by keywords or by location.
Compare
This screen is used to call the corresponding Address services to compare two given Address records. Note that the ZK implementation supports drag-and-drop - i.e. I can drag a row from the list of addresses and drop it onto the text box and the text box will be populated with the content of the address (as a string). On the other hand, the GWT implementation does not support DnD because at the time of writing it, GWT-Ext did not support this kind of DnD yet (and I am not about to rewrite it using new version of GWT-Ext now considering Sanjiv has left the project... but that's another story).
Preferences
This screen is used to manage user preferences of the GUI application. Cookies are used as the storage.
In my SLOC count, I divided the source code into the following groups:
- Main View - this includes the main screens: CRUD, Find and Compare tab panels.
- Preference View - the user preference window
- Model/Controller - the code behind the screen components, including models, event handlers, interaction with the service tier, helper/utilities
- Splasher - this is the Splasher screen displayed at the beginning of the application. Only ZK version has implemented this (using
index.zuland automatically forwarding to the main view after initialising the Address service); other versions do not have this screen.
Note that in ZK, all the views are a result of drag-and-drop from the components palette with modifications by hand.
The physical and logical SLOC counts are shown below.
The results speak for themselves.
UPDATE [2008-12-08]:
- The GWT implementation source code can be found here: Address_gwt.zip
- The ZK implementation source code can be found here: Address_zk.rar
Thursday, 27 November 2008
ZK - First Impression
I first heard of ZK from several developer forums, where one of the authors of ZK posted articles comparing ZK with GWT trying to prove that ZK was superior. Ever since then, I have been wanting to learn/use ZK for several months now.
I was pleasantly surprised by the maturity and high quality of ZK from the first moment I started using it - I didn't expect it to have a drag-and-drop graphical editor - something that not many free web frameworks have out of the box. I believe it is a real contender of GWT.
The praises of ZK can be found easily and I will not repeat them here. Instead, I will outline the reservations that I have about ZK hoping to get advices from the community.
ZK Studio Installation
When I followed download instructions on ZK site to install the ZK Studio directly from Eclipse, it turned out to be an older version of the plugin. It took me a while to find out why my ZK Studio did not have the ZUL Graphical Editor mentioned in the documentation. I had to download the plugin (v0.9.x) and install it onto Eclipse manually.
ZUL Graphical Editor
The ZUL Graphical Editor is not really an editor at the moment. It only serves as a previewer. I believe ZK team is actively working on improving the editor right now.
Debugging
I am not aware of any special debugger for ZK. If something goes wrong, I have to inspect the web server's log file to get the stack trace - cannot jump to the line of source code automatically. Moreover, if the error occurs at the ZUL script, the trace shows BeanShell exceptions, which can be harder to locate the real position of the error in the ZUL file.Centralised ZScript File
I understand that zscript can be isolated in its own file and be included into the .zul file, so that the script is not littered in the presentation code. However, I find that to completely separate the script and the ZUL code is not practical. For example, to specify the a renderer for a component, I need to instantiate the renderer and assign it to a variable, then put the variable to the components renderer attribute. For example,
<zscript>
import com.laws.address.zk.*;
renderer=new AddressRenderer();
</zscript>
<listbox id="crudAddressList" mold="paging" pageSize="10"
itemRenderer="${renderer}">
<listhead sizable="true">
<listheader label="ID" width="30px"/>
<listheader forEach="${afts}" >
<attribute name="label">${each.name}</attribute>
</listheader>
</listhead>
</listbox>
However, the bit of script that instantiates and assigns the new renderer cannot be separated into another file (unless a function is created to do it, which looks odd and impractical).
Event Handling
The event handling in ZK is strange - the sender has to specify the target for the event. The whole idea of events is so that the sender (or publisher) is not aware of who is going to consume the event, or how many consumers, and what the consumer will do with it. But in ZK, the target of the event is specified up front by the sender - i.e. the sender knows who should receive the event. I shall elaborate this point in another post in more detail.
To be fair, ZK does allow the target of the event to be null when posting an event. However, the event is only accessible from top-level components in this scenario.
Server Round Trips for Static Contents
Unlike GWT, even static contents in ZK will involve server decision (a bit like Echo 2). For example, upon clicking a button or menu, my application pops up an 'About' message box displaying static contents. The processing of the button click and the display of the message box involves the server and incurs a round-trip to the server. This certainly has performance penalty and puts presentation logic and load onto the server (which, IMO should be placed in the client tier - i.e. the browser). I know, there has been debate about whether the GWT way is better. I personally prefers the GWT way - especially considering it is broadband era now.
Overall, my experience with ZK has been quite positive. I think it is among my top 3 web RIA frameworks.
Saturday, 22 November 2008
N95 Must Haves: Panoman
I travel quite a bit due to the nature of my job. I used to bring my camera along on every trip. But due to my forgetful nature, and the bulkiness of the normal cameras, I stopped using them on my trips. Instead, I began to use my mobile phone camera, especially after I got my N95.
N95 must have one of the best phone cameras around. I feel that it is even better than many of the cheap digital cameras. It has a decent Carl Zeiss lens, 5M pixels resolution and intelligent image processing software built-in.
When traveling, many of my photos are scenery. Panorama pictures are the best to capture the landscape and the atmosphere of the new places that I visit. So far the best panorama software I found is Panoman. It is extremely easy to use: simple press start and move the handset horizontally, then press stop when you are done. The rest is handled by the Panoman software. Most importantly, the result is superb.
Friday, 7 November 2008
What Do VB & Applet Have In Common?
No, I am not talking about Victoria Bitter, which I quite like. I am talking about Visual Basic and Java Applets. They both have a bad reputation due to historical reasons, which is hampering their adoption.
In the heyday of IT, VB was probably the most widely used programming language at least in the Windows world. IT professionals and IT professional wannabes all flocked to write software to cash in on the golden era of IT. I still remember one of the managers at Australian Taxation Office (ATO) initiated an IT project and wrote the software in VB/Visual Foxpro by himself together with a public servant who had "13 years of programming experience" but also had trouble finding keys on the keyboard.
Today, VB has become VB.NET and the developer community is more mature. Still, there has been much discussion about why people perceive VB.NET to be inferior to C#. Many people choose C# over VB or even move from VB to C# simply because it will make them look better or making them feel better about themselves - real programmers use 'C-like' languages.
Unlike VB, the Java Applet never reached popularity since the beginning because those were the dial-up days. I can see three major hurdles for Java Applet in those days: slow internet links meaning downloading the JAR files was a big problem; 'complexity' in installing the JRE plug-in; awkward look-and-feel and lack of good frameworks.
Today, things have drastically improved: pervasive broadband access, improved browser plug-in capabilities, 80% new PCs are pre-installed with JRE, improved JRE (6 update 10), the imminent release of JavaFX 1.0 and more other applet-based frameworks. Still people doubt whether these frameworks can compete with Silverlight and Flex. I don't see the word 'applet' get mentioned much by Sun or JavaFX's promoters. The FUD is just too deeply entrenched in people's minds.
At the end of the day, great technology does not thrive on its own merit. The human factor has the final say. It's a bit like Wall Street.
Monday, 3 November 2008
A Musical Note
I spent last month at home on holiday (is that an oxymoron?) feeling lethargic as people do on holidays. As a result I did not touch my computer much and only spent a few hours learning Erlang.
However, I did get to learn music and ended up spending 1-2 hours per day in front of the electronic keyboard that I bought for my kids . I have to admit, I am not good at music at all - I did use to excel in singing in primary school; but since my voice broke I lost all enthusiasm on music - my parents sending me to a violin class where I was the only boy did not help either. Until last month, as I fiddled around with the piano/keyboard, I revived my thirst for knowledge on music and piano-playing. It turned out that piano-playing is really easy to learn (at least to start) if you are a fast touch-typist. So I relearned the basic music theory 101 by following the wonderful book Total Piano. As an IT professional geek, I couldn't help finding many striking similarities between the musical and computing worlds.
First of all there are the patterns in music. As a matter of fact, all musical pieces are based on establishing some patterns and then repeating them - there are quite a few symbols in the staff to represent repetitions (i.e. the GOTO statement). It seems like composers are lazier than system designers Also, there are chords and each chord has many variations - broken, harmonic, minor, etc. Again, it's all about repetition and reusing.
Then there is the arrangements. The same music can be (re)arranged to adapt to different environments/scenarios - e.g. to suit different musical instruments (e.g. piano solo, string quartet), or to suit different players (e.g. beginner vs. pro) etc. Thanks to simple arrangements of many great works included in the book, I was able to play some very interesting pieces and keep getting fully engaged in the learning process. By the way, how many AJAX'ian web frameworks have you come across recently?
Well-known musicians also remake other people's songs to boost their own career. Yesterday, I read an interview with Seal and his upcoming new album of remakes of classics that 'appealed' to him. In it, Seal revealed that quite a few young people in their 20s had not heard the original version of the songs. So Seal's version was the first time that they heard them and Seal kind of re-introduced the great classics back to the young generations. I hope the youngsters don't mistake Madona's American Pie with the original.
Every now and then I can hear curses coming from across the office when my colleagues cannot find some features in their MS Office 2007. I have mixed feelings about the ribbon interface of Office 2007 - I like the look and feel but hate the fact that so many features have been rearranged so that everytime I want to use them it's Easter all over again. I am sure if someone has never used the older versions, he/she will like the new ribbon interface.
Perhaps learning from the closed-door design experiences, Microsoft is openning up their design process for some of their major products - e.g. Enginnering Windows 7. On the other hand, music creation is usually an extremely personal experience, unless you are charged to write some propaganda piece or the national anthem (I will vote for Waltzing Matilda anytime for Aussie national anthem)!
Meanwhile, I'd better get back to practise the beautiful piano solo arrangement of Beijing 2008 Olympic theme song - You and Me.
Tuesday, 21 October 2008
Erlang Mnesia - Take 2
After my first attempt of using Erlang Mnesia, I decided to apply what I have learned so far to refactor and improve my Erlang Mnesia implementation of the Address database and Data Access Layer (DAL).
Redesigning The Database
This time I de-normalised part of the database design -- to duplicate the
hierarchy_orderfield fromaddress_field_typetoaddress_fieldtable - removed the record id fields from the 3 entities:
address_field_typeis a reference data table with small amount of data. I use thelocale_country+hierarchy_orderas a combined key implicitlyaddressandaddress_fieldboth have thelocation_codefield which is unique for each record, so I usedlocation_codeas their primary key
address.hrl file:
-record(address_field_type, {
locale_country,
name,
default_value,
hierarchy_order,
display_order,
validation_rules,
suffix} ).
-record(address_field, {
location_code,
value,
locale_country, %% denormailsed from address_field_type
hierarchy_order %% denormalised from address_field_type
}).
-record(address, {
location_code,
status
}).
-record(address_addressField, {
address_locationCode,
address_field_locationCode
}).
Creating the database:
create_db() ->
mnesia:create_table(address_field_type,
[
{type, bag}, %% use bag because no single-filed primary key now
{ram_copies,[nonode@nohost]},
{index, [hierarchy_order]},
{attributes, record_info(fields, address_field_type)}
]),
mnesia:create_table(address_field,
[
{ram_copies,[nonode@nohost]},
{attributes, record_info(fields, address_field)}
]),
mnesia:create_table(address,
[
{ram_copies,[nonode@nohost]},
{attributes, record_info(fields, address)}
]),
mnesia:create_table(address_addressField,
[
{type, bag},
{ram_copies,[nonode@nohost]},
{index, [address_field_locationCode]},
{attributes, record_info(fields, address_addressField)}
]).
The above changes do reduce the functionality of the Address application very slightly but it's a worthy trade-off as it simplifies the design - now I don't need the oid table/record any more along with all the functions and code revolving the oids.
Also, by de-normialising, i.e. duplicating the hierarchy_order field into address_field table, the sorting of the address_field list is made easy and efficient (without having to look up the database in every iteration). This is demonstrated in the code below.
Simplified Sorting
The sorting ofaddress_field list has reduced from 3 functions previously to the use of lists:keysort/2 due to the duplication of the hierarchy_order field from address_field_type into address_field in the data/entity model.
sort_afs(Afs) when is_list(Afs) -> %% 5th field of the address_field tuple is hierarchy_order lists:keysort(5, Afs).
Simplified Data Insertion
By skipping any record id handling, the code for insertion of records is considerably simplified. This is most evident from theinsert_address/2 function (compared to the previous version):
insert_aft(Aft) when is_record(Aft, address_field_type) ->
Fun = fun() ->
mnesia:write(Aft)
end,
mnesia:transaction(Fun),
Aft.
insert_address(A, Afs) when is_record(A, address) and is_list(Afs) ->
{NewA, NewAfs} = address:generate_location_code(A, Afs),
Fun = fun() ->
% create the new address record
mnesia:write(NewA),
% now insert/update into address_field
% and insert into address_addressField table
lists:foreach( fun(Af) ->
mnesia:write(Af),
A_Af=#address_addressField{
address_locationCode=NewA#address.location_code,
address_field_locationCode=Af#address_field.location_code
},
mnesia:write(A_Af)
end,
NewAfs),
{NewA, NewAfs}
end,
mnesia:transaction(Fun).
Simplified Data Queries
By usingdirty_read and dirty_index_read functions unnecessary transactions are avoided.
find_afts(LocaleCountry) when is_list(LocaleCountry) ->
mnesia:dirty_read({address_field_type, LocaleCountry}).
find_address(Id) when is_integer(Id) ->
[A] = mnesia:dirty_read({address, Id}),
A.
find_address_field(Id) when is_integer(Id) ->
[Af] = mnesia:dirty_read({address_field, Id}),
Af.
find_address_field_codes(A_code) ->
% the read will return a list of tuples, we want the 3rd field (the AF_locationCode)
% of each tuple and put them in a list.
[ element(3, A) || A <- mnesia:dirty_read({address_addressField, A_code}),
true ].
find_address_codes(Af_code) ->
F = fun() ->
mnesia:select(address_addressField, [{
#address_addressField{ address_locationCode='$1',
address_field_locationCode=Af_code,
_='_' },
[], ['$1'] }]
)
end,
{atomic, Results}=mnesia:transaction(F),
Results.
find_address_codes2(Af_code) ->
F = fun() ->
Q = qlc:q([Aaf#address_addressField.address_locationCode
|| Aaf <- mnesia:table(address_addressField),
Aaf#address_addressField.address_field_locationCode==Af_code]),
qlc:e(Q)
end,
{atomic, Results}=mnesia:transaction(F),
Results.
find_address_codes3(Af_code) ->
Aafs = mnesia:dirty_index_read(address_addressField, Af_code,
#address_addressField.address_field_locationCode),
% the second element of the tuple is the address_locationCode
[ element(2, Aaf) || Aaf <- Aafs, true ].
In the above code, find_address_codes/1, find_address_codes2/1, find_address_codes3/1 do the same thing but are implemented using different approaches. The dirty read one is the simplest.
Conclusion
Just by thinking in Erlang, the resulting database design has been simplified - less table and fields; the source lines of code (SLOC) has been reduced by 30%; most importantly, the programming logic is now simpler and easier to read/maintain.
However, this does not mean that Erlang and Mnesia are the best way to implement this kind of database applications (where normalisation is important for the business logic).
Saturday, 18 October 2008
Erlang Mnesia
Carrying out my Erlang learning plan, I embarked on the porting exercise of my Address DAO package into Erlang Mnesia. In fact, it is more than porting of the data access functions. It is also a porting of the Address database itself from RDBMS into Mnesia.
Creating the Mnesia Schema
The Entity Relationship Diagram (ERD) of the database schema is shown below.
To create the tables in Mnesia, I defined the following records, each corresponding to a table.
-record(address_field_type, {id,
name,
locale_country,
default_value,
hierarchy_order,
display_order,
validation_rules,
suffix} ).
-record(address_field, {id,
value,
address_field_type_id,
location_code}).
-record(address, {id,
status,
location_code}).
-record(address_addressField, {
address_id,
address_field_id
}).
-record(oids, {
name,
id
}).
Note that there is an extra record oids. This record/table is required because Mnesia does not support auto-incrementing fields usually used as primary key generation. Therefore this extra table is used to store the current key value - similar to Oracle's sequence.
Creating the tables are shown below. The oids table is also populated with seed values.
create_db() ->
mnesia:create_table(address_field_type,
[
{ram_copies,[nonode@nohost]},
{attributes, record_info(fields, address_field_type)}
]),
mnesia:create_table(address_field,
[
{ram_copies,[nonode@nohost]},
{index, [location_code]},
{attributes, record_info(fields, address_field)}
]),
mnesia:create_table(address,
[
{ram_copies,[nonode@nohost]},
{index, [location_code]},
{attributes, record_info(fields, address)}
]),
mnesia:create_table(address_addressField,
[
{type, bag},
{ram_copies,[nonode@nohost]},
{index, [address_field_id]},
{attributes, record_info(fields, address_addressField)}
]),
mnesia:create_table(oids,
[
{attributes, record_info(fields, oids)}
]),
Fun = fun() ->
mnesia:write(#oids{ name=address, id=0 }),
mnesia:write(#oids{ name=address_field_type, id=0 }),
mnesia:write(#oids{ name=address_field, id=0})
end,
mnesia:transaction(Fun).
Simple CRUD
Theaddress_field_type is a table for storing reference data, which does not rely on other tables. So the C and R of the CRUD is very simple. From address_db.erl file:
%% returns {atomic, Oid} or {aborted, Reason}
generate_oid(TableName) when is_atom(TableName) ->
Fun = fun() ->
[Oid] = mnesia:read(oids, TableName, write),
%% because Erlang only supports single assignment
%% I have to create new variables every time the value changes.
NewId=Oid#oids.id+1,
New = Oid#oids{ id=NewId },
mnesia:write(New),
NewId
end,
mnesia:transaction(Fun).
insert_aft(Aft) when is_record(Aft, address_field_type) ->
Fun = fun() ->
{atomic, Id}=generate_oid(address_field_type),
New = Aft#address_field_type{ id=Id },
mnesia:write(New),
New
end,
mnesia:transaction(Fun).
find_aft(Id) when is_integer(Id) ->
{atomic, [Aft]} = mnesia:transaction(fun()-> mnesia:read({address_field_type, Id}) end),
Aft.
There are a few things worth noting:
- Although the
mnesia:readandmnesia:selectfunctions can be read-only, they still need to be wrapped in a function and called bymnesia:transaction/1. This does not make sense to me. - The
mnesia:transaction(Fun)returns{aborted, Reason}or{atomic, Result}whereResultis whatever the return result of theFunis. Therefore, theinsert_aft/1above returnsNew- the newaddress_field_typerecord with theidfield populated. - Erlang is single assignment language - i.e. a variable can only be assigned value once. Therefore, if I want to assign the
Aft#address_field_type.id, I have to create a whole newaddress_field_typerecord as in theinsert_aft/1above. I have to say, I am not a fan of this single assignment - it defeats the half of the purpose of having variables.
A Slightly More Complicated CRUD
Theaddress entity relies on address_field, which in turn depends on address_field_type. Also, if address records share the same address_field values, then the address_field record should be shared and not be duplicated in the database. Therefore, the management of the address records are more complicated than those of address_field_type.
The creation of address records is shown below.
insert_address(A, Afs) when is_record(A, address) and is_list(Afs) ->
{A2, Afs2} = address:generate_location_code(A, Afs),
Fun = fun() ->
% populate the address_field records' id field
NewAfs=lists:foldl( fun(Af, NewAfs) ->
% check if the same Af is already in database
Result = mnesia:select(address_field, [{
#address_field{ id='$1',
location_code=Af#address_field.location_code,
_='_' },
[], ['$1'] }]
),
if length(Result)==0 -> % Af not in DB, so insert it
{atomic, Id}=generate_oid(address_field);
true -> % Af already exists in database
Id=lists:nth(1, Result)
end,
NewAf = Af#address_field{ id=Id },
lists:append(NewAfs, [NewAf])
end,
[],
Afs2),
% create the new address record
{atomic, AddressId} = generate_oid(address),
NewA = A2#address{ id = AddressId },
mnesia:write(NewA),
% now insert/update into address_field
% and insert into address_addressField table
lists:foreach( fun(Af) ->
mnesia:write(Af),
A_Af=#address_addressField{
address_id=AddressId,
address_field_id=Af#address_field.id
},
mnesia:write(A_Af)
end,
NewAfs),
{NewA, NewAfs}
end,
mnesia:transaction(Fun).
From the address.erl file:
%% applying quick sort to AddressField list.
sort_afs([]) -> [];
sort_afs([Pivot|Tail]) when is_record(Pivot, address_field) ->
sort_afs([Af || Af <- Tail,
compare_aft(address_db:find_aft(Af#address_field.address_field_type_id),
address_db:find_aft(Pivot#address_field.address_field_type_id)) < 0])
++ [Pivot] ++
sort_afs([Af || Af <- Tail,
compare_aft(address_db:find_aft(Af#address_field.address_field_type_id),
address_db:find_aft(Pivot#address_field.address_field_type_id)) >= 0]).
compare_aft(Aft1, Aft2) when
is_record(Aft1, address_field_type) and is_record(Aft2, address_field_type) ->
Aft1#address_field_type.hierarchy_order - Aft2#address_field_type.hierarchy_order.
generate_location_code(A, AddressFields) when is_record(A, address)
and is_list(AddressFields) and (length(AddressFields)>0) ->
Afs=generate_location_code(AddressFields),
FirstAf=lists:nth(1, Afs),
{
A#address{ location_code=FirstAf#address_field.location_code },
Afs
}.
generate_location_code(Afs) ->
ReverseSortedAfs=lists:reverse(sort_afs(Afs)),
generate_location_code(ReverseSortedAfs, [], []).
generate_location_code([Head|Tail], String, NewAfs) when is_record(Head, address_field) ->
StringValue = Head#address_field.value ++ String,
Code=erlang:phash2(StringValue),
generate_location_code(Tail,
StringValue,
[ Head#address_field{ location_code=Code } ] ++ NewAfs);
generate_location_code([], _String, NewAfs) ->
NewAfs.
The following functions are queries from the address_db.erl file:
find_address(Id) when is_integer(Id) ->
{atomic, [A]} = mnesia:transaction(fun()-> mnesia:read({address, Id}) end),
A.
find_address_field(Id) when is_integer(Id) ->
{atomic, [Af]} = mnesia:transaction(fun()-> mnesia:read({address_field, Id}) end),
Af.
find_address_field_ids(A_id) ->
mnesia:transaction(fun()-> mnesia:read({address_addressField, A_id}) end).
find_address_ids(Af_id) ->
F = fun() ->
mnesia:select(address_addressField, [{
#address_addressField{ address_id='$1',
address_field_id=Af_id,
_='_' },
[], ['$1'] }]
)
end,
mnesia:transaction(F).
find_address_ids2(Af_id) ->
F = fun() ->
Q = qlc:q([Aaf#address_addressField.address_id
|| Aaf <- mnesia:table(address_addressField),
Aaf#address_addressField.address_field_id==Af_id]),
qlc:e(Q)
end,
mnesia:transaction(F).
Things that I want to highlight from the above are:
- Unlike Java or C#, Erlang does not have a sortable list out-of-the-box. Instead, I had to write functions that sort my list of
address_fieldrecords - these are:sort_afs, compare_aft - When searching using the primary key field,
mnesia:readis used; when searching using other fields, either usemnesia:selectorQLC. In the above code, the functionsfind_address_ids/1andfind_address_ids2/1do the same thing. However, the result (a list) given byfind_address_ids2/1are reversed. - If a record with the same primary key already exists in the table, the
mnesia:writeeither creates (if table type ifset) or updates (if table type isbag) the record in the table. - Again, due to the single assignment rule, the
lists:append(List, AnotherList)does not change the List; instead, the new list is the return value oflists:append/2. This is just counter-intuitive.
Related Posts:
Monday, 13 October 2008
Erlang ODBC
Continuing my Erlang journey, I decided to try with Erlang ODBC by porting one of my toy applications. Perhaps not a wise choice since working with database (especially relational database) is not a strength of Erlang, at least for now.
After almost 10 years and being spoiled by myriad of Object-Relational Mapping (ORM) frameworks, using ODBC again feels like a giant leap backward - 3 generations back to be exact (ADO.Net, ADO, ODBC). So the overall development experience is not that great. This is not to say that Erlang is not a great language, but Erlang as a platform is quite narrow in scope (this makes sense considering Erlang's strong telco heritage).
So here is my porting exercise. Note that the code below is my feeble attempt at using/learning Erlang by reading the man-pages alone, so if you have suggestions to improve it, please drop a comment. Also, it is not robust since it does not handle error conditions.
Background
The application (or component) that I am working on is a Data Access Objects (DAO) package which I originally developed in Java using various ORM frameworks and then C# using various 3rd-party ORM and ADO.NET frameworks. The data model is quite simple, consisting 3 entities: AddressFieldType (AFT for short), AddressField (AF for short) and Address. The relationships among them are as following:
- AFT - AF has a one-to-many relationship
- Address - AF has a many-to-many relationship
The partial entity relationship diagram (ERD) can be found in my previous blog.
Here, I attempt to implement some of the interfaces from the DAO package using Erlang ODBC.
The Design
Since Erlang is not an object oriented language, I cannot call my module DAO, so I call it DAL (Data Access Layer) instead borrowing from Microsoft terminology. I have two simple modules:
address_dal: implementation of some of the interfaces retrieving records from the Address database (using MySQL 5.x with ODBC driver 5.1.5 for win32 downloaded from MySQL).orm: ORM here means ODBC-Record Mapping. There are two types of functions in this module: those that provide ODBC template functions; and those that convert ODBC returned data into Erlang Records defined inaddress.hrlfile.
Records
I want to work with Erlang records rather than tuples returned by ODBC calls. So I created these records to represent the domain model. From the address.hrl file:
-record(address_field_type, {id,
name,
locale_country,
default_value,
hierarchy_order,
display_order,
validation_rules,
suffix} ).
-record(address_field, {id,
value,
address_field_type_id,
location_code}).
-record(address, {id,
status,
location_code,
address_fields = [#address_field{}]}).
Notice the address record has a list of address_field records, representing half of the many-to-many relationship.
ODBC Template
I need to surround my SQL queries with database connection and disconnection so that these boiler plate codes do not get scattered everywhere in my business logic. Also, if the SQL queries results in error, Erlang breaks the ODBC connection (I am not sure if this is Erlang's fault or ODBC's). So I have to reconnect before my next SQL query anyway.
So in my orm.erl I have these ODBC template/boiler-plate functions:
-define(CONNECTION_STRING, "DSN=Address-MySQL;UID=root").
connect() ->
case proplists:is_defined(odbc,application:which_applications()) of
false ->
application:start(odbc); % pre R12 way of starting odbc;
_Else ->
false % do nothing
end,
odbc:connect(?CONNECTION_STRING, [
{auto_commit, off},
{scrollable_cursor, off}
]).
sql_query(Query) ->
{ok, Ref}=connect(),
ResultTuple = odbc:sql_query(Ref, Query),
odbc:disconnect(Ref),
ResultTuple.
param_query(Query, Params) ->
{ok, Ref}=connect(),
ResultTuple = odbc:param_query(Ref, Query, Params),
odbc:disconnect(Ref),
ResultTuple.
Note that Erlang ODBC is based on ODBC v3 which supports connection pooling which has been turned on. So surrounding the SQL queries with connection and disconnection should not incur performance penalty (although I have not explicitly tested it).
One thing I hate about Erlang's odbc:connect() is that it does not give any comprehensive error reasons if the connection fails - it always says "No SQL-driver information available. Connection to database failed." Again, I am not sure if this is Erlang's or ODBC's fault.
Converting into Records
The ODBC query functions return the results as tuples or list of tuples. I want to convert them into recdords so that I can access the fields of the records more easily - e.g. #address.id.
I tried two approaches implementing the record constructor functions. The first one is more portable since it does not rely on the field position but uses column name as a clue to map to the record's fields. This is shown below from the orm.erl file mapping the AddressFieldType query results to their corresponding address_field_type record type:
construct_aft_records(ColNames, Rows) when erlang:is_list(ColNames) and (length(ColNames)>0)
and erlang:is_list(Rows) ->
if (length(Rows)>0) ->
lists:foldl(fun(R, AftRecords) ->
lists:append(AftRecords, [construct_aft_record(ColNames, R)])
end,
[], % initial value of AftRecords list
Rows
);
true->
[]
end.
construct_aft_record(ColNames, Row) when erlang:is_list(ColNames) and (length(ColNames)>0)
and erlang:is_tuple(Row) and (size(Row) == length(ColNames)) ->
Map=lists:foldl(fun(N, Map) ->
lists:append(Map,
[{string:to_upper(lists:nth(N, ColNames)),
element(N,Row)}]
)
end,
[], % initial value of Map is []
lists:seq(1, length(ColNames))
),
#address_field_type{
id=extract_map_value(Map, "ADDRESSFIELDTYPEID"),
name=extract_map_value(Map, "NAME"),
locale_country=extract_map_value(Map, "LOCALECOUNTRY"),
default_value=extract_map_value(Map, "DEFAULTVALUE"),
hierarchy_order=extract_map_value(Map, "HIERARCHYORDER"),
display_order=extract_map_value(Map, "DISPLAYORDER"),
validation_rules=extract_map_value(Map, "VALIDATIONRULES"),
suffix=extract_map_value(Map, "SUFFIX")
}.
extract_map_value(Map, Key) ->
element(2, element(2,lists:keysearch(Key, 1, Map))).
The second approach is lazier as it relies on the field position as specified in the SELECT SQL statement. This is shown below from the orm.erl file.
find_addresses(Query, Params) ->
{ok, Ref}=connect(),
{selected, _, Rows} = odbc:param_query(Ref, Query, Params),
% now for each address Row fetch its addressField records
AddressRecords = lists:foldl(fun(A, Records) ->
AddressId=element(1,A),
AddressStatus=element(2,A),
AddressLocationCode=element(3,A),
{selected, _, AfRows}=odbc:param_query(Ref,
"SELECT af.addressFieldId, af.value, af.addressFieldTypeId,"
" af.locationCode, aft.hierarchyOrder"
" FROM Address.addressField as af, Address.AddressFieldType as aft,"
" Address.address_addressField as aaf"
" WHERE aaf.addressId=?"
" AND af.addressFieldId=aaf.addressFieldId"
" AND aft.addressFieldTypeId=af.addressFieldTypeId"
" ORDER BY aft.hierarchyOrder",
[{sql_integer, [AddressId]}]
),
AfRecords = lists:foldl(fun(Af, AddressFieldRecords) ->
AfRecord=#address_field{
id=element(1, Af),
value=element(2, Af),
address_field_type_id=element(3, Af),
location_code=element(4, Af)
},
lists:append(AddressFieldRecords, [AfRecord])
end,
[],
AfRows
),
AddressRecord= #address{
id=AddressId,
status=AddressStatus,
location_code=AddressLocationCode,
address_fields=AfRecords
},
lists:append(Records, [AddressRecord])
end,
[],
Rows
),
odbc:disconnect(Ref),
AddressRecords.
Implementing the DAL Interfaces
Now that the boiler-plate code is done, I can implement the DAL interface methods, oops! I mean functions. Here is the address_dal.erl file.
%% Implementation of the Address Data Access Layer (DAL) interfaces.
-module(address_dal).
-export([find_aft/0, find_aft/1]).
-export([find_addresses_in_af/1, find_addresses_in_location/1]).
-include("address.hrl").
find_aft() ->
{selected, ColNames, Rows} = orm:sql_query("SELECT * from Address.AddressFieldType"),
orm:construct_aft_records(ColNames, Rows).
find_aft(LocaleCountry) ->
{selected, ColNames, Rows} = orm:param_query("SELECT * from Address.AddressFieldType "
"WHERE localeCountry=?",
[{{sql_varchar, 64}, [LocaleCountry]} ]
),
orm:construct_aft_records(ColNames, Rows).
find_addresses_in_af(AddressFieldId) ->
orm:find_addresses(
"SELECT a.addressID, a.status, a.locationCode"
" FROM Address.Address as a, Address.Address_AddressField as aaf"
" WHERE aaf.ADDRESSFIELDID = ?"
" AND a.ADDRESSID=aaf.ADDRESSID",
[{sql_integer, [AddressFieldId]}]
).
% the input parameter LocationCode needs to be a string due to the ODBC
% to Erlang datatype mapping - hence the test for is_list().
find_addresses_in_location(LocationCode) when is_list(LocationCode) ->
orm:find_addresses("SELECT a.addressID, a.status, a.locationCode"
" FROM Address.Address as a, Address.AddressField as af, Address.Address_AddressField as aaf"
" WHERE af.LOCATIONCODE = ?"
" AND a.ADDRESSID=aaf.ADDRESSID"
" AND af.ADDRESSFIELDID=aaf.ADDRESSFIELDID",
[{{sql_numeric, 32, 0}, [LocationCode]}]
).
Notice that in the find_aft functions I can use 'SELECT *...' because the orm:construct_aft_records/2 function uses the column names to map to the address_field_type record; on the other hand, the find_addresses... functions have to use 'SELECT a.addressID, a.status, a.locationCode...' and in that strict order because the orm:find_addresses/2 function expects the columns to be in those positions.
Thursday, 9 October 2008
Dealing with Databases in Erlang
Correction: Thanks to Brandon's comment below, there is also Erlang ODBC which comes with the OTP distribution. This is the 'official' gateway to the RDBMS world. I had a go at it in this post. There is also CouchDB from Apache, which is still in incubation.
Currently there are basically two ways to use database in Erlang - Mnesia and ErlyDB. There is also the prospect of edbc. But nothing is available yet from the project.
Mnesia is the native DBMS in Erlang. It is bundled with Erlang/OTP distribution and is written in Erlang as well. It was originally developed for telco switching applications by Ericsson. Therefore, it boasts a lot of non-functional features - in-memory database, distributed database, high performance (for read), making changes without downtime, etc. However, all these benefits are at the cost of sacrificing many of the basic functionality of a traditional RDBMS - such as referential integrity check, data type validation/check, normalisation support, triggers, stored procedures, etc.
ErlyDB is essentially Yariv's one-man effort. It's a code generator to generate the data access layer code in Erlang to work with RDBMS (and perhaps Mnesia). Currently supported RDBMS are MySQL and Postgress. It's primary use case was ErlyWeb from the same author. It looked promising. However, the whole ErlyWeb and ErlyDB/ErlySQL seem to have lost its momentum: The last ErlyWeb release was over a year ago; ErlyDB is no longer a project on its own right, therefore, no information on its releases; promised features from Yariv's blog have not eventuate after 1 to 2 years.
So given the database landscape in Erlang, the most viable approach is to use Mnesia if you don't mind the lack of database features and tools. I will give it a try by porting my prototype Address database into Mnesia.
Sunday, 5 October 2008
Consuming RESTful Service with Erlang
Having been working in the telco industry for over 10 years, I can't help feeling a bit ashamed for not having learned Erlang. Erlang was developed by Ericsson - the leading Swedish-based network equipment provider, and has been used in many of the network switching equipment produced by Ericsson. Erlang has gained more traction recently, especially in the last year or two.
Here, I write an Erlang program to consume a demo RESTful service that I developed a couple of months ago. The Erlang code is based on example by Dave Thomas - the author of Programming Ruby. The detail of the RESTful service is available in my previous post.
There are two services that I want to consume.
The first one accepts a social web site URL and scrapes the page for the person's interests in movies and music to return a list of comma-delimited keywords. For example,
http://localhost:8080/SvdemoRestful/resources/webPageKeywords?url=http://localhost/someWebPage.html
if any keywords are found, then they are returned in the body of the HTTP response as a string; otherwise, the HTTP body is empty. For example,
folk songs, pop music, chinese music ,battle, action, comedy
Notice the spaces in the above string. The second service I want to consume accepts the above string to search for any matching promotion in a mock up XML database. If any is found then the XML string will be returned in the HTTP body; otherwise, the string <Empty/> is returned in the HTTP body. For example, the following URL will return the promotion information in XML below:
http://localhost:8080/SvdemoRestful/resources/promo/jazz
the returned XML:
<Promotion> <Code>802</Code> <Name>Jazz Night</Name> <Description>Jazz lovers' do not miss this once in a lifetime opportunity.</Description> <Venue>The Jazz Club</Venue> <DateTime>2008-10-30 21:00:00</DateTime> <Tags>Jazz</Tags> </Promotion>
Now, let's do this in Erlang.
Create a file called svdemoClient.erl. The following code will consume the first RESTful service:
-module(svdemoClient).
-export([get_keywords/1]).
-define(BASE_URL, "http://localhost:8080/SvdemoRestful/resources").
-define(PROMO_URL, ?BASE_URL ++ "/promo/").
-define(KEYWORDS_URL, ?BASE_URL "/webPageKeywords"). % also works without ++
keywords_url_for(Url) -> ?KEYWORDS_URL ++ "?url=" ++ Url.
get_keywords(Url) ->
URL = keywords_url_for(Url),
{ ok, {_Status, _Header, Body} } = http:request(URL),
Body.
In Erlang, function names and Atoms must start with lower-case letters; variable names must start with upper-case letters or underscore (meaning the variable is not used/read).
The -define() macro in Erlang is similar to #define in C/C++. In the above example, after defining BASE_URL, any occurance of ?BASE_URL will be replaced with the string "http://localhost:8080/SvdemoRestful/resources".
The get_keywords() function returns the body of the HTTP response from requesting the given Url. The Body is either a comma-delimited string, or an empty collection. Executing the above code in Erlang:
127> c("d:/projects/svdemoErl/svdemoClient.erl").
{ok,svdemoClient}
128> svdemoClient:get_keywords("http://localhost/myWebPage.htm").
"folk songs, pop music, chinese music ,battle, action, comedy "
129> svdemoClient:get_keywords("http://localhost/someOtherPage.htm").
[]
130>
To consume the second RESTful service, the search_promo() function is added.
promo_url_for(Keywords) -> ?PROMO_URL ++ utils:url_encode(Keywords).
search_promo(Keywords) ->
URL = promo_url_for(Keywords),
{ ok, {_Status, _Header, Body} } = http:request(URL),
%%% Now that the XML is in the Body variable, let's parse it.
if
Body == "<Empty/>" ->
not_found;
true ->
{ParseResult, _Misc} = xmerl_scan:string(Body),
[ #xmlText{value=Code} ] = xmerl_xpath:string("//Code/text()", ParseResult),
[ #xmlText{value=Name} ] = xmerl_xpath:string("//Name/text()", ParseResult),
[ #xmlText{value=Description} ] = xmerl_xpath:string("//Description/text()", ParseResult),
[ #xmlText{value=Venue} ] = xmerl_xpath:string("//Venue/text()", ParseResult),
[ #xmlText{value=DateTime} ] = xmerl_xpath:string("//DateTime/text()", ParseResult),
{ Code, Name, Description, Venue, DateTime }
end.
Erlang/OTP download comes with XML parser and XPath support in the xmerl application, which is not part of the Erlang standard library (stdlib). To use the XML functions, the header file must be included:
-include_lib("xmerl/include/xmerl.hrl").
Note that the keywords contain spaces, which must be URL-encoded before passing to Erlang's http:request() function. I stole the url_encode() function from YAWS and put it in utils.erl file.
To string the two service consumptions together:
search_promo_from_url(Url) -> Keywords=get_keywords(Url), if Keywords == [] -> not_found; true -> search_promo(Keywords) end.
Calling the function in Erlang shell:
126> svdemoClient:search_promo_from_url("http://localhost/MyWebPage.htm")
.
{"801",
"Batman The Dark Knight",
"\n\t\t\tMeet stars in Batman in person - Chritian Bale, Michael Caine.\n\t\t",
"Star City",
"2008-7-30 10:00:00"}
The final svdemoClient.erl file:
-module(svdemoClient).
-export([get_keywords/1, search_promo/1, search_promo_from_url/1]).
-include_lib("xmerl/include/xmerl.hrl").
-define(BASE_URL, "http://localhost:8080/SvdemoRestful/resources").
-define(PROMO_URL, ?BASE_URL ++ "/promo/").
-define(KEYWORDS_URL, ?BASE_URL "/webPageKeywords"). % also works without ++
keywords_url_for(Url) -> ?KEYWORDS_URL ++ "?url=" ++ Url.
get_keywords(Url) ->
URL = keywords_url_for(Url),
{ ok, {_Status, _Header, Body} } = http:request(URL),
Body.
promo_url_for(Keywords) -> ?PROMO_URL ++ utils:url_encode(Keywords).
search_promo(Keywords) ->
URL = promo_url_for(Keywords),
{ ok, {_Status, _Header, Body} } = http:request(URL),
%%% Now that the XML is in the Body variable, let's parse it.
if
Body == "<Empty/>" ->
not_found;
true ->
{ParseResult, _Misc} = xmerl_scan:string(Body),
[ #xmlText{value=Code} ] = xmerl_xpath:string("//Code/text()", ParseResult),
[ #xmlText{value=Name} ] = xmerl_xpath:string("//Name/text()", ParseResult),
[ #xmlText{value=Description} ] = xmerl_xpath:string("//Description/text()", ParseResult),
[ #xmlText{value=Venue} ] = xmerl_xpath:string("//Venue/text()", ParseResult),
[ #xmlText{value=DateTime} ] = xmerl_xpath:string("//DateTime/text()", ParseResult),
{ Code, Name, Description, Venue, DateTime }
end.
search_promo_from_url(Url) ->
Keywords=get_keywords(Url),
if
Keywords == [] ->
not_found;
true ->
search_promo(Keywords)
end.
The utils.erl file (copied from YAWS):
-module(utils).
-export([integer_to_hex/1, url_encode/1]).
integer_to_hex(I) ->
case catch erlang:integer_to_list(I, 16) of
{'EXIT', _} ->
old_integer_to_hex(I);
Int ->
Int
end.
old_integer_to_hex(I) when I<10 ->
integer_to_list(I);
old_integer_to_hex(I) when I<16 ->
[I-10+$A];
old_integer_to_hex(I) when I>=16 ->
N = trunc(I/16),
old_integer_to_hex(N) ++ old_integer_to_hex(I rem 16).
url_encode([H|T]) ->
if
H >= $a, $z >= H ->
[H|url_encode(T)];
H >= $A, $Z >= H ->
[H|url_encode(T)];
H >= $0, $9 >= H ->
[H|url_encode(T)];
H == $_; H == $.; H == $-; H == $/; H == $: -> % FIXME: more..
[H|url_encode(T)];
true ->
case integer_to_hex(H) of
[X, Y] ->
[$%, X, Y | url_encode(T)];
[X] ->
[$%, $0, X | url_encode(T)]
end
end;
url_encode([]) ->
[].
Related Posts:
Sunday, 28 September 2008
Hancock: Good job!
Last week's experience remindes me of scenes from the movie Hancock.
The marketing guy Ray tought Hancock how to behave in front of the crowd - here are the dialogues from the scene.
Ray Embrey: So you've used the door, the building's still intact, people are happy you've arrived, they feel safe now, there's an officer there and he's done a good job, so you might want to tell him he's done a good job.
Hancock: What the hell did I have to come for Ray if he's done a good job?
So later when Hancock when to the crime scene after the police failed to contain the situation, he repeatedly said 'Good job!' to everyone, including the policewoman who was injured and pinned down by enemy fire.
Hancock: [to pinned-down cop] Good job! Do I have permission to touch your body?
Female Cop: Yes!
Hancock: It's not sexual. Not that you're not an attractive woman. You're actually a very attractive woman and...
Female Cop: [screaming] Get me the hell out of here!
Those were the my favorite scenes from the movie. Do the above scenes look familiar in your workplace?
Friday, 26 September 2008
Why Not Chrome?
I installed and tried Google Chrome recently and my first impression of Chrome as a browser is quite average. As I said before, there is nothing revolutionary about Chrome at the surface. It's really the backend functionality that Google is betting on. For now, my biggest complaint about Chrome is that it installs in user's home directory rather than a shared directory so that all users can use (maybe there is or will be an option to install it in shared mode and I just don't know it).
At the moment, each browser has its niche market: IE is pre-installed on all Windows based machines including Windows Mobile; Firefox has a great plug-in framework so that its functionality and usability can be greatly extended beyond just being a browser; Opera has over 100 million installations on mobile phones; Safari is the browser of choice on Mac...
So far, my favourite browser is Firefox v3 mostly because of the plug-ins. As a blogger, the ScribeFire blogger plug-in for Firefox is a must-have. I use it to do my blogging rather than using the Blogger's own web pages (thanks to Blogger's APIs). Another plug-in I use is the Adblock Plus where you can selectively block advertisement contents by clicking on the 'block' tab that Adblock Plus inserted over them.
IE still remains the widely used browser and I still use it. I only use it because certain plug-ins (ActiveX) are only available on IE. The most popular plug-in is probably the Microsoft Outlook for IE since most companies these days use Outlook as their email solution.
Google is also taking this approach by integrating Chrome with other Google services so that Chrome will be the best browser when it comes to Google (and its partner) provided services. Therefore, I don't see any single browser replacing all others. It will be more of a case of using certain browser for certain use cases.
Related Posts: Why Google Chrome?Monday, 22 September 2008
Turn N95 Into A Ping-Pong Bat
As I was searching for some games for my N95, I stumbled upon the Accelerometer support for N95. If you have seen the iPhone ads on TV, you may have been impressed by the feature of the applications changing their display orientation after the iPhone has been rotated. I must say I was very impressed after playing with my friend's iPhone. Little did I know that my N95 also has the capability to do this. It's just that Nokia did not put any application on it to utilise the accelerometer.
There are a couple of applications developed by Nokia Research center which take advantage of the accelorometer - Moving Ball and Activity Monitor, both available from the Nokia Research Center web site.
I have downloaded and tried the Moving Ball demo. It was great! As I move/rotate my handset in any direction, the ball moves or bounces following my motion. If I hold the phone like a ping-pong ball bat and swing it, the ball displayed on screen actually bounce up and down against the 'bat'.
According to these videos, the French developer Samir has made some cool applications using this technology - RotateMe and Nokmote. I particularly like Nokemote where it allows the user to control the application by moving the handset without having to press any buttons/keys - e.g. in the music player, tilt left to rewind, tilt right to fast-forward, bounce it down and up to select song, etc. Unfortunately, the author's web site is down.
RotateMe on N95
Uploaded by soueldi
There are others of course: There is a dedicated forum on N95Users suggesting all sorts of cool uses with videos to show; NiiMe is too shaky to be practical; FlipSilent and ShakeSMS are pretty good, but I am not sure if they drain the battery; Landscape Pro, but it's not free; Alas, all I can do with a great piece of technology is to use it as a ping-pong ball bat!
Friday, 19 September 2008
The Shortest Self-Printing Code
#!/bin/ksh cat `whence $0`
Could this be the shortest self-printing code?
Saturday, 13 September 2008
Of Tatami, jMaki and Others...
Tatami 1.2 Beta was released a few days ago.
Both GWT and Dojo are Javascript frameworks for creating Rich Internet Applications (RIA). The GWT team focuses on the foundation of the framework and does not bother with making state-of-the-art cool widgets. While Dojo has a mature and good-looking widgets library. Wouldn't it be good if the two can be seamlessly integrated at the API and runtime levels so that one can leverage on the advantages of both frameworks? That's exactly what Tatami offers.
Tatami is a widget/component library for GWT. What makes it interesting is that it is a GWT wrapper of Dojo components. With Tatami, the Dojo widgets become GWT widgets; the Dojo utilities become GWT helper classes. If you don't have the stomach for editing CSS+XHTML+Javascript directly, but still want to use the Dojo components, then Tatami is good news. The development experience is back to Java and that's it.
Another web framework which provides Dojo wrapper is the jMaki framework. While jMaki has a nifty NetBeans plug-in, the development experience is improved somewhat. However, as a developer, you'd still have to deal with a mixture of JSP, Javascript and Java: the page templates are written in JSP, the glue code is written in Javascript for event handlers (using a pub-sub mechanism), the server side code is written as Java servlets - maybe this hodge-podge is really what they mean by mash-up.
Another wrapper is the GWT-EXT, which is a GWT wrapper for EXT/JS. I have blogged about the EXT/JS, EXT-GWT and GWT-EXT and the licensing issues revolving EXT/JS previously. Well, the licensing of Dojo is very liberal, without the GPL issues introduced by EXT/JS. Thankfully, Tatami is LGPL.
Tuesday, 9 September 2008
How To Add Unicode Fonts to Symbian
[Update 2009-02-17]: Fontrouter project has been open sourced. Some of the download URLs in this post may have been changed. See here for more details.
If you are like me, who got a Nokia N95 with only English but wants to view contents in other languages - e.g. Chinese, Japanese, Korean (CJK), Russian, Hindi, etc. then you will be in a bit of strife.
According to Nokia forum, there is no way that you can install an extra language package on Nokia. You'd have to go to Nokia Care to have them install it for you, for a fee! Well, from my previous encounters with Nokia Care (here in Sydney), they are bunch of slow, bureaucratic, idiotic waste of space. So that option is a no-no for me.
Fortunately, the Symbian guru oasisfeng has created a wonderful software called FontRouter. The latest incarnation of the software is the FontRouter LT. Because the software is in testing stage, there is no full documentation on how to install and use it. There are pieces of information everywhere in the forum and mostly in Chinese. It took me a day to figure out the end-to-end process of getting it to work on my N95. So I feel that I should record it down and share it. I believe it should work for all S60 v3, S80 and S90 Nokia phones.
Step 1 - Downloading the Software
There are many versions on the oasisfeng web site, the version that I used was the FontRouter LT for Symbian 9 Beta (Build 20071109) Open Signed Online测试版.Also download the TrueType font from the same web site: 【TrueType】方正隶变 GBK字体. This contains all the CJK characters(should contain other european characters too). Note that I tried using other Chinese TrueType fonts and my handset failed to start. So experiment with other .ttf files only after you have successfully installed and tested with this one. I am currently using the 方正准圆(FZY3JW.TTF) which is a Chinese true type font and its English characters are very close to the original font on my Nokia N95.
Notice that all Nokia phones belong to a certain profile (to make it easy for software developers to select and test the target platform/handsets). The N95 is an S60 3rd edition phone and runs on Symbian version 9. You can find this out from the phone: Tools -> Utilities -> About. Also notice that the version of FontRouter that I used was "Open Signed Online". What does this mean? Read on...
Step 2 - Signing the Software
Symbian is pretty rigorous when it comes to security. Software needs to be digitally signed before installing onto the phone. You can try installing the .sis file just as it is and your phone will reject it and complain that the digital certificate/signature is invalid or something of that nature. To sign the file you just downloaded, you will have to go to www.symbiansigned.com. On its front page, there is a URL for Open Signed Online (Beta). Click this URL and fill in the form on the following page. Note:
- The IMEI number uniquely identifies your handset (e.g. if your phone is stolen, you can report it to the carrier and they can bar the IMEI from the network so that the thief cannot use it). You can find it out by 'dialing' *#06#
- Email - make sure you use a valid email address because you will need to retrieve email from Symbian to carry out the next steps.
- The Application is the
.sisfile you just downloaded from FontRuter.
Upon successful submission of the form, Symbian will send you an email with a URL to confirm the submission. Click it.
Then Symbian will send you another email with a URL to download the signed version of the .sis file. Just click the URL and save the file.
Step 3 - Installation
Now you can install the signed version of the software that you just downloaded from Symbian using PC Suite. Make sure you install it on the Memory Card, NOT on the Phone Memory (otherwise, if it hangs your handset, you will not be able to bring your phone back to life). During installation, it will warn you that your phone is incompatible with the software, just ignore it and continue.
Then copy the fzlb_gbk.ttf file downloaded in Step 1 into the Memory Card's \data\Fonts directory. You will notice that the file FontRouter.ini has also been created in this directory by the software installation process.
Now it is time to modify the FontRouter.ini file. You can use the Windows Nodepad to edit a copy of the file and then copy it onto the handset using PC Suite. Modify the two lines of the .ini file as following:
FixFontMetrics=0 FixCharMetrics=0changed the values from 0 to 1 like this:
FixFontMetrics=1 FixCharMetrics=1and leave everything else the same. If you don't make the changes, the fonts will be curtailed either from the top or bottom.
That's it. Phew! Now it's time to test it - bounce the phone (i.e. switch it off then on).
Euphoria!
There are a few must-read guides that I highly recommend:
Subscribe to:
Posts (Atom)
