How to Call AWS API from Snowflake

My Snowflake free account expires tomorrow. So I thought I should get the external integration happening today.

Following the length instructions from Snowflake turns out not to be enough. I got stuck at the input and output parameter formats between SF and the Lambda function. I finally got it working and here they are:

The Lambda function in Python:

import json

print('Loading function')

dict={2000:"Sydney", 2218:"Allawah", 3000 : "Melbourne"}

def lambda_handler(event, context):

    print(f"event={event}")

    # normal channel the event looks like: {'body': '{ "data": [ [ 0, 43, "page" ], [ 1, 42, "life, the universe, and everything" ] ] }'}

    # from SF, event looks like: {'data': [[0, 2000]]}

    if "body" in event:

        postcode=event["body"]["data"][0][1]

    else:

        postcode=event["data"][0][1]

    city="unknown"

    print(f"postcode={postcode}")

    if postcode in dict:

        city=dict[postcode]

    output=[0, city]

   

    array_of_rows_to_return = [ ]

    array_of_rows_to_return.append(output)

    json_compatible_string_to_return = json.dumps({'data' : array_of_rows_to_return})

   

    return {'data': array_of_rows_to_return}

The Snowflake side:

create or replace external function postcode_to_city(postcode int)

    returns array

    api_integration = demonstration_external_api_integration_01

    as 'https://....execute-api.ap-southeast-2.amazonaws.com/dev';

select postcode_to_city(2000) as city;

The result:

Like It or Not - Azure Function

 👍Here is what I like about Azure Functions:

1. Resource Group - the concept of resource group is a good one. It allows me to see all the related resources in the same screen. In the Numbers Game example, it lists the related App Service Plan, Function App and Application Insights on the same screen (on Portal and VSCode). Also, if I want to delete the whole thing, I can just delete the resource group. That's just a few of the goodies that RG offers.
2. Function App - this is another layer of the hierarchy that can group all the related functions under the same app. Settings like Keys, Logging, CORS can be specified at this level and apply to all the functions of the app. However, this is a double-edged sword (see below).
3. Portal GUI is very intuitive
4. Console and the other Development Tools - I can go into the console to see what's happening behind the scene. (OK, for serverless you shouldn't have to care about the server or what's behind the scene. But I am curious).
5. Code is quite performant - speed of execution is pretty impressive - around 500ms - 800ms to solve the game. 
6. Although not required for my function, I like the fact that Azure allows you to scale up by changing the App Service Plan. It would have been even nicer to be able to combine AWS' concurrency control with the scaling options here. 

👎What I don't quite like about Azure Functions:

1. Portal GUI (and VSCode) loading of the screens are quite slow - loading of code takes a couple of seconds (with only 350 - 400 lines of code). However, the App Service Editor development tool from the portal is much quicker. (in fact, the Azure CLI/Powershell feel very slow in general).
2. The language/Stack setting is at Function App level. Therefore, I am stuck with the same language for all functions underneath. In my example, the main code of solve game was developed a few years ago in Javascript, so it's natural to choose Node.js. However, the GetNumbersGameHtml function would be much easier to write in Python taking advantage of its """ multi-line string syntax. 

My overall impression about Azure Function is very positive albeit the tooling may still need some polishing and some missing features like data mapping and validation declaratively outside of the code.

Numbers Game on Azure Functions

Porting the Numbers Game onto Azure Function is so easy!

Just like in the AWS Lambda approach, I created two Azure Functions in the same Function App:

• GetNumbersGameHtml - triggered by HTTP GET
• SolveNumbersGame - triggered by HTTP POST

Both functions are configured with the same Function Key value.

Code for GetNumbersGameHtml:

module.exports = async function (context, req) {
    const key= req.headers["x-functions-key"] || req.query.code;

    context.res = {
        // status: 200, /* Defaults to 200 */
        headers: {
            "Content-Type": "text/html"
        },
        body: myHtml+key+myHtml2
    };
}

const myHtml = '<!DOCTYPE html>\n'+
'<html>\n'+...

Code for SolveNumbersGame:

module.exports = async function (context, req) {
    const e=req.body;
    var numbers=[e.n1,e.n2,e.n3,e.n4,e.n5,e.n6], target=e.target;
    context.log("solving: "+JSON.stringify(e));
	var answer=solveAllGames(numbers, target);
    
    context.res = {
        // status: 200, /* Defaults to 200 */
        body: JSON.stringify(answer)
    };
}
...

Although the speed of Azure's CLI/Powershell/Portal is very slow comparing to AWS, the Function execution is blazingly fast - between 500ms-800ms - this is on the lowest App Service Plan (F1). The AWS Lambda function can be 10x as slow, usually between 3-8s - I had to increase the timeout setting of the function. In fact, to cater for the long wait time on the web GUI, I downloaded and customised a nifty spinner like below. I kind of enjoy seeing it going 4-5 cycles before it disappears when the solution is displayed; but with Azure, I can hardly see it complete one cycle! (a bit disappointed 😋).

Please wait while getting the solutions...

100

25

5

4

7

9

AWS REST APIs offer fine-grained control such as Model, Mapping Template, which are missing in Azure Function. On the other hand, the way Azure uses Keys for the Function App and Function is very simple to use and effective.

Two Lambdas Are Better Than One

 To refactor the single Lambda function and API previously, I created two Lambda functions:

1. getNumbersGameHtml - to return the HTML string, which is the GUI page of the Numbers Game single page web application.
2. solveNumbersGame - to return the array of solutions (strings of math expressions)

The Python code for getNumbersGameHtml function:

def lambda_handler(event, context):
    return {
        'body': myHtml
    }
    
myHtml = ('<!DOCTYPE html>\n'
'<html>\n'
'<head profile=\"http://www.w3.org/2005/10/profile\"><link rel=... />...\n'
'<body>\n'
'...'
'	<input type="submit" value="Solve Game" onClick="if (validate_inputs()) {solve_game();} else alert(\'Please ensure all numbers and targets are filled in and with valid values.\\nHover mouse over input boxes for rules details.\');" />&nbsp;&nbsp;&nbsp;&nbsp;    \n'
'	<input type="button" value="Reset Game" onClick="reset_game()"/> \n'
'...')

# alternatively
myHtml="""<!DOCTYPE html>
<html>
<head profile=\"http://www.w3.org/2005/10/profile\"><link rel=... />...
<body>
...
	<input type="submit" value="Solve Game" onClick="if (validate_inputs()) {solve_game();} else alert(\'Please ensure all numbers and targets are filled in and with valid values.\\nHover mouse over input boxes for rules details.\');" />&nbsp;&nbsp;&nbsp;&nbsp;
	<input type="button" value="Reset Game" onClick="reset_game()"/>
...
"""

The web page looks like this:

Manually input numbers Auto with 4 large numbers Auto with 3 large numbers Auto with 2 large numbers Auto with 1 large numbers Auto with all small numbers

    

---

Copyright (c) Romen Law. All rights reserved.

The Javascript code for solveNumbersGame function:

exports.handler = async (event) => {
	var e=event;
	var numbers=[e.n1,e.n2,e.n3,e.n4,e.n5,e.n6], target=e.target;
	var answer=solveAllGames(numbers, target);
	
    const response = {
        statusCode: 200,
        body: JSON.stringify(answer)
    };
    
    return response;
};

An API is created at the API Gateway with two methods implemented:

• GET method invokes the getNumbersGameHtml function
• at the GET - Integration Response, for response 200, 
• add a Header Mappings entry with Response header of content-type and the Mapping value of 'text/html' (notice the single quote - it represents literal string in the mapping language).
• at Mapping Templates, add an entry for Content-Type of text/html with template code of $input.path('body') (don't forget to press the grey Save button at the bottom before the blue Save button at the top).
• at the GET - Method Response, for HTTP status 200 
• add header of content-type
• add Response model of Empty for content-type text/html
• POST method invokes the solveNumbersGame function
• at the POST - Method Request, Request Body: 
• set the Request Validator field to Validate body and click the tick.
• add NumbersInput model for application/json. This will enforce the input data format and some validation rules
• at the POST - Integration Response, Mapping Templates: add template code $input.path('body') for content type application/json. This will extract the 'body' field from the response of the Lambda function and return it as the API's response.

The NumbersInput model:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "n1": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n2": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n3": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n4": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n5": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n6": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "target": {
      "type": "integer", "minimum": 100, "maximum": 999
    }
  },
  "required": [
    "n1",
    "n2",
    "n3",
    "n4",
    "n5",
    "n6",
    "target"
  ]
}

See also:

• Numbers Game - Part 1 - The Journey Begins
• Numbers Game - Part 2 - A Dog's Breakfast
• Numbers Game - Part 3 - Putting Things Right
• Numbers Game - Part 4 - De-duplicate Mathematically Equivalent Expressions
• Numbers Game - Part 5 - Opps
• Numbers Game - Part 6 - AWS Lambda
• Numbers Game - Part 7 - AWS APIs

One Lambda to Serve Them All

 The really cheap way to host a dynamic web site on AWS is by using the combination of S3 and Lambda:

• S3 to store the static contents such as HTML files, graphics, CSS, javascript files
• Lambda and API Gateway to serve the dynamic contents by providing APIs to the javascripts of the web site.

I went one step further with my Numbers Game single page web application by serving everything from a Lambda function without having to use S3. This is because

1. I want the web site to stay totally free. Lambda is perpetually free tier and I don't get charged as long as my volumes stay under the threshold. However, S3 is not free after the first 12 months.
2. The static content for my web page is relatively simple - one HTML file actually. My own Javascripts are embedded in the page; 3rd-party contents (CSS, javascripts, icons) are pointed to their corresponding URLs so the browser will simply gets them from the internet.
3. I don't want to enable CORS of my AWS API, so my web page (and its javascript) and the APIs must be served by the same domain. 

This is very easy to do with AWS Lambda: my Lambda function handles two types of requests - GET and POST (of course, you can refactor these to let the API layer do some mapping and use two Lambda functions... ). 

• for GET, the Lambda function returns the HTML string. This string was prepared by feeding my static HTML file to some online converter to get the big string with all the double quotes (along with some other characters) escaped - the online tool's result was not perfect, I had to hand fix something that it missed. This HTML page basically allows the user to enter the input numbers and performs some validation according to the game's rules. Once submitted, it will invoke the API using POST
• for POST, the Lambda function invokes the javascript functions to calculate the results and return them as a JSON string. The HTML simply displays the results on the same page.

The Lambda function:

exports.handler = async (event) => {
	console.log("event:"+JSON.stringify(event));
	var answer="no or unsupported event";
	var response;
	if(event.httpMethod == "GET") {
		response = {
	        statusCode: 200,
	        headers: {
	        	"content-type": "text/html"
	        },
	        body: myHtml,
	    };
	} else if (event.httpMethod == "POST" || event.routeKey.startsWith("POST")) { // supporting REST APIs and HTTP APIs
		if(event.body != null) {
			var e=JSON.parse(event.body);
			var numbers=[e.n1,e.n2,e.n3,e.n4,e.n5,e.n6], target=e.target;
			answer=solveAllGames(numbers, target);
			//answer=solveGame(numbers, target);
		}
	    response = {
	        statusCode: 200,
	        body: JSON.stringify(answer),
			headers: {
	        	"Access-Control-Allow-Origin": "*"
	        }
	    };
	}
    console.log(response);
    return response;
};
//... more code here ... where myHtml stores the big string converted from my static HTML file.

This approach works quite well. However, it is an abomination and wrong in so many places:

• it's mixing network layer logic with business logic - in the code it's handling http methods which should be the responsibility of the API Gateway!
• At the API, when only the ANY is mapped, the event passed into the Lambda is at a higher level - the Lambda function has to traverse into the 'body' to get the input parameters. 
• The JSON.parse(event.body) is very cumbersome. I should be able to just get the input numbers from the event itself as a JSON object, not as a string. 
• the lambda function cannot be used as-is for other triggering types (because it explicitly handles HTTP methods). e.g. cannot test in AWS console.

It's time to refactor... 💨

See also:

• Numbers Game - Part 1 - The Journey Begins
• Numbers Game - Part 2 - A Dog's Breakfast
• Numbers Game - Part 3 - Putting Things Right
• Numbers Game - Part 4 - De-duplicate Mathematically Equivalent Expressions
• Numbers Game - Part 5 - Opps
• Numbers Game - Part 6 - AWS Lambda
• Numbers Game - Part 7 - AWS APIs

Loving Powershell

I have always loved Unix shell scripting and Unix shells in general. I especially like the shortcut keys in shell environment and you can choose different modes - it's emacs mode for me anytime. They are such productivity boost - it makes me cringe to see someone painstakingly pressing the back-arrow all the way to the front of the line just to fix a typo (and very often even in training videos). 😖

When it comes to Powershell, I am a newbie. I only started using it quite recently. However, I was pleasantly surprised on how powerful its construct is and how easy it is to manipulate objects in Powershell.

In Unix, the output of the commands are strings. So when you pipe them to the next command you are doing string manipulation - hence, sed, awk, cut... become very useful. However, in Powershell, they are objects - you can directly manipulate the objects themselves without having to rely on helper commands to locate the right fields all the time. 

Here are some examples to demonstrate it. In AWS, it is often required to assign security policies to users or roles, etc. However, those policy names are long and not easy to remember (for me anyway). So I end up querying those policy names and ARNs quite all the time. For example: search for S3 related security policies - 

1. using AWS-Tools for Powershell it is easy to filter based on the fields/attributes/properties and choose which one to display by directly manipulating the object's attributes:

PS C:\WINDOWS\system32> Get-IAMPolicies | Where-Object {$_.PolicyName -like "*S3*"} | Select-Object -Property PolicyName, arn

PolicyName                                              Arn                                                                    
----------                                              ---                                                                    
AmazonDMSRedshiftS3Role                                 arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role           
AmazonS3FullAccess                                      arn:aws:iam::aws:policy/AmazonS3FullAccess                             
QuickSightAccessForS3StorageManagementAnalyticsReadOnly arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageMan...
AmazonS3ReadOnlyAccess                                  arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess                         
AmazonS3OutpostsFullAccess                              arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess                     
S3StorageLensServiceRolePolicy                          arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy
IVSRecordToS3                                           arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3                 
AmazonS3OutpostsReadOnlyAccess                          arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess                 



PS C:\WINDOWS\system32> 

See how neat the results are laid out! What I also like about PS is that its case insensitive - even using -like "*s3*" will return the same results. 👍

The huge complaint about AWS Tools for Powershell is its installation process - it took over 1 hour in one of my laptops over proxy, and about 5 minutes on another.

2. using AWS CLI native query capability

PS C:\WINDOWS\system32> aws iam list-policies --query 'Policies[?contains(PolicyName,`S3`)].[PolicyName, Arn]' --output text
AmazonDMSRedshiftS3Role	arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role
AmazonS3FullAccess	arn:aws:iam::aws:policy/AmazonS3FullAccess
QuickSightAccessForS3StorageManagementAnalyticsReadOnly	arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly
entAnalyticsReadOnly
AmazonS3ReadOnlyAccess	arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
AmazonS3OutpostsFullAccess	arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess
S3StorageLensServiceRolePolicy	arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy
IVSRecordToS3	arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3
AmazonS3OutpostsReadOnlyAccess	arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess

PS C:\WINDOWS\system32>

If it weren't for the syntax highlighter of this web page, I'd feel dizzy looking for the item that I want...😵  The AWS CLI is strictly case-sensitive and very unforgiving! 😠

3. using combination of AWS CLI query and awk

$ aws iam list-policies --query 'Policies[?contains(PolicyName, `S3`)]' --output text | awk '{print $9, $1}'
AmazonDMSRedshiftS3Role arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role
AmazonS3FullAccess arn:aws:iam::aws:policy/AmazonS3FullAccess
QuickSightAccessForS3StorageManagementAnalyticsReadOnly arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly
AmazonS3ReadOnlyAccess arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
AmazonS3OutpostsFullAccess arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess
S3StorageLensServiceRolePolicy arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy
IVSRecordToS3 arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3
AmazonS3OutpostsReadOnlyAccess arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess

Identical result as the previous one. As you can see, the awk command assumes the desired fields are the 9th and 1st fields. This makes the code fragile. 😱

I used WSL for the above command. If using Git Bash for Windows, then put the following before running any aws CLI commands:

$ export HOME=$USERPROFILE

This is needed for aws to locate the configuration and credential files.

4. using combination of AWS CLI and jq

$ aws iam list-policies --query 'Policies[?contains(PolicyName, `S3`)]' | jq '.[] | {PolicyName, Arn}'
{
  "PolicyName": "AmazonDMSRedshiftS3Role",
  "Arn": "arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
}
{
  "PolicyName": "AmazonS3FullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
{
  "PolicyName": "QuickSightAccessForS3StorageManagementAnalyticsReadOnly",
  "Arn": "arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
}
{
  "PolicyName": "AmazonS3ReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
{
  "PolicyName": "AmazonS3OutpostsFullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess"
}
{
  "PolicyName": "S3StorageLensServiceRolePolicy",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy"
}
{
  "PolicyName": "IVSRecordToS3",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3"
}
{
  "PolicyName": "AmazonS3OutpostsReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess"
}

or...

$ aws iam list-policies | jq '.Policies[] | select(.PolicyName | contains("S3")) | {PolicyName, Arn}'
{
  "PolicyName": "AmazonDMSRedshiftS3Role",
  "Arn": "arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
}
{
  "PolicyName": "AmazonS3FullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
{
  "PolicyName": "QuickSightAccessForS3StorageManagementAnalyticsReadOnly",
  "Arn": "arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
}
{
  "PolicyName": "AmazonS3ReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
{
  "PolicyName": "AmazonS3OutpostsFullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess"
}
{
  "PolicyName": "S3StorageLensServiceRolePolicy",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy"
}
{
  "PolicyName": "IVSRecordToS3",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3"
}
{
  "PolicyName": "AmazonS3OutpostsReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess"
}
$ aws iam list-policies | jq '.Policies[] | select(.PolicyName | contains("S3")) | .PolicyName, .Arn'
"AmazonDMSRedshiftS3Role"
"arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
"AmazonS3FullAccess"
"arn:aws:iam::aws:policy/AmazonS3FullAccess"
"QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
"arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
"AmazonS3ReadOnlyAccess"
"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
"AmazonS3OutpostsFullAccess"
"arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess"
"S3StorageLensServiceRolePolicy"
"arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy"
"IVSRecordToS3"
"arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3"
"AmazonS3OutpostsReadOnlyAccess"
"arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess"

2c Worth of Notes

The following nuggets of knowledge costed me US2c (AUD3c) by trial and error playing with EC2 outside of the 12-month free tier.

• after modifying (creating new versions of) launch templates make sure to make it default, or be specific of the version number when launching a new EC2 instance. Otherwise, the changes will not take effect.
• It's better to copy and paste when creating User Data - I made the simple mistake of typing 'ec2_user' instead of 'ec2-user'. Also consult https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/ to change the frequency of the execution of the user data.
• the user data is run as root.

On another note, while I was playing with the EC2 launch template, my WSL played up - the DNS setting stopped working on the Linux VMs, and WSL kept overwriting the /etc/resolv.conf file even with /etc/wsl.conf created with 

[network]
generateResolvConf = false

It turns out that WSL has to be terminated to make the wls.conf take effect; alternatively, prevent the /etc/resolv.conf from being writable by

chattr +i /etc/resolve.conf

Numbers Game - Part 6 - AWS Lambda

 Porting the numbers game to AWS Lambda is really simple. I created the Lambda function using AWS Console and added an API as trigger.

The AWS Lambda function for my numbers game is called getExprs. By adding a few lines of code in the generated index.js, and append my Javascript functions in the same file, it was working in no time. The index.js file:

exports.handler = async (event) => {
    console.log("event:"+JSON.stringify(event));
    var answer="no event";
    if(event.body != null) {
	var e=JSON.parse(event.body);
	var numbers=[e.n1,e.n2,e.n3,e.n4,e.n5,e.n6], target=e.target;
	answer=solveAllGames(numbers, target);
	//answer=solveGame(numbers, target);
    }
    const response = {
        statusCode: 200,
        body: JSON.stringify(answer),
        //body: answer,
    };
    console.log(answer);
    return response;
};
// my functions appended here...

Testing it with Postman:

 

Testing it in Python:

import requests

body={
  "n1": 75,
  "n2": 100,
  "n3": 2,
  "n4": 5,
  "n5": 9,
  "n6": 10,
  "target": 110
}
try:
    response = requests.post('https://xxxxxxxxxx.execute-api.ap-southeast-2.amazonaws.com/default/getExprs', json=body)
    response.raise_for_status()
    print(response.body.json());
except requests.exceptions.HTTPError as errh:
    print(errh)
except requests.exceptions.ConnectionError as errc:
    print(errc)
except requests.exceptions.Timeout as errt:
    print(errt)
except requests.exceptions.RequestException as err:
    print(err)

Results are:

======================= RESTART: C:\pcd_tmpl32\call.py =======================

['100+10', '2*5+100', '(2+9)*10', '(9-2)*5+75', '(75-10)+5*9', '5*9-(10-75)', '(100+5)+10/2', '(10-5)*2+100', '100*2-9*10', '100/10*(2+9)', '(100/5-9)*10', '100/5+9*10', '(5*9+10)*2', '((75+100)+5*9)/2', '(75-5)/(9-2)+100', '(75+9)/2*5-100', '((100-75)*9-5)/2', '75*2/(5+10)+100', '(75*2/10+100)-5', '75*2/10-(5-100)', '(75-5)+(100/2-10)', '(75/5+100)-10/2', '((75+5)/10+100)+2', '75/5-(10/2-100)', '(75+5)/(10-2)+100', '(75*10-100*2)/5', '((100*2-75)-5)-10', '100*2-((75+5)+10)', '(100*2-75)-(5+10)', '(100/2-(5-75))-10', '(75-100)*(5-9)+10', '(75-100)+(5+10)*9', '((75+5)+10)/9+100', '(((100-75)-5)-9)*10', '(100*9/75+10)*5', '(5*9+75)-100/10', '5*9-(100/10-75)', '75*2+(5-9)*10', '75*2-(9-5)*10', '(2-9)*(5-10)+75', '(100*2*5-10)/9', '100/5-10)*(2+9', '(100+9)+2*5/10', '(100+9)+10/2/5', '100+10)*(2*5-9', '(100/10+5*9)*2', '2*5-(9-10)*100', '(75+100)-((2+9)*5+10)', '((75+100)-(2+9)*5)-10', '(75+100)-(2*10+5*9)', '(75-100)+(2*10-5)*9', '((75+100)-(5-10)*9)/2', '(75+100)-((9-2)*10-5)', '((75+100)+(10-5)*9)/2', '(75*2+100*9)/10+5', '75*2+100/10*(5-9)', '75*2-100/10*(9-5)', '(75-5)-(100/2-9*10)', '75/5+(9*10+100)/2', '(75/5-9)*10+100/2', '((75+5)-10)/(9-2)+100', '(75*9-100)/5-10/2', '(((75-9)*2-100)-10)*5', '(((75+10)/5+100)+2)-9', '(75+10)/5-((9-100)-2)', '(100-75)-((2-9)-10)*5', '100*2*5*9/75-10', '100*2*9/75/5-10', '(100/2*9/75+5)*10', '(100*9/75*2+5)*10', '(5*9+75)/(2+10)+100']

>>> 

One gripe I have about Lambda functions is that they are supposed to support multiple triggers - even the AWS Console shows so. However, the event that are passed into Lambda can have different structures depending on the channel/triggering type. For example, when integrated with a POST method of a REST API, the payload is the event - i.e. you can get the payload like n1=event.n1; however, when integrated with a POST method of a HTTP API, the payload is in the body - i.e. you can get the payload like n1=event.body.n1. This makes the Lambda function pretty much channel/trigger specific - you have to write another version of the Lambda function if you want to expose it to another trigger/channel.

See also:

• Numbers Game - Part 1 - The Journey Begins
• Numbers Game - Part 2 - A Dog's Breakfast
• Numbers Game - Part 3 - Putting Things Right
• Numbers Game - Part 4 - De-duplicate Mathematically Equivalent Expressions
• Numbers Game - Part 5 - Opps
• Numbers Game - Part 7 - One Lambda to Serve Them All

AWS Certification

 I was over the moon when I passed the AWS Practitioner's Certification exam last Saturday and couldn't believe how easy it was, well if you have read the materials. I finished the 90-minute test in half an hour.

The reason for my excitement of passing the exam was not because the certification itself but the journey to get there. I first started the first version of the online course in June/July this year. It was a 6-7 hour long multi-part video + presentation course. Being the first time, I really sat through the whole thing, didn't want to miss anything. So it took me two days to go through the materials. 

Then I was told that there was a 2nd version. So I re-did the course. This time it was more of a revision, and I got through it in about one day. Thinking I was ready, I booked for an exam. This was a remotely monitored exam in one of the test centers, where there were a couple of web cams mounted on the computer so that the person sitting in some unknown location can watch you remotely. I had to scan my ID on the scanner, fill in my details on screen, agree on the T&C's... several times only to be told that there was some technical issues and the remote monitor could not see any video streaming from the test center. By that time, I had already lost 30 minutes, so the test was canceled and rescheduled. So I wasted about half a day on that aborted test including traveling time.

Then a couple of weeks ago I was told that there was a 3rd version of the course and promised to be heaps better than the previous ones. So I re-did the online course, but this time running the video at x2 speed and finished it in a couple of hours.

After that, I felt I was ready and scheduled my exam... the rest is history.

The exam mostly makes sure that you know which AWS service is available and for what purpose. So reading through the overview pages of each AWS product is really helpful. In my exam the following AWS services are mentioned Glue, Athena, Cost Optimizer, Cost Explorer, Concierge Support Team, LightSail, Redshift, Storage Gateway, Connect... apart from the mundane ones that you always hear about - EC2, S3, RDS, DynamoDB, caching services, VPC, IGW/TGW, Route 53, CloudeWatch, CloudeFormation, Cloude Trail, Lambda, CloudFront, etc.

The certification badge is good for 3 years, after which I would have to do another exam ?! 

Getting Cloudy

Last week I attended the Amazon Web Services APAC Tour in Sydney. It was quite helpful to see the AWS in action and their customer testimonials in real life.

There is no doubt that the cloud environment can help many IT departments and internet-oriented businesses. The main benefits are cost reduction and improved scalability and reliability. Unlike many IT fads in previous years, there is real momentum and benefits of adopting the cloud approach for many businesses and government organisations alike.

When it comes to telco BSS and OSS applications, the use case is quite different from many other industries. At the bottom level, many BSS and OSS applications need to be connected to the network in real-time. This can be real-time charging, service control for BSS and real-time network performance management for OSS. These real-time interfaces are usually dedicated links using telco standards (e.g. SS7 based), which conventional cloud providers do not support. So you will have no choice but to host these applications locally.

For the non-real-time (i.e. offline) telco applications, many of them need to access the vast repository of usage records, transaction records and network events. A telco may choose to host only the front-office web applications in the cloud and have them retrieve the necessary data stored locally over the internet; or take advantage of the cloud storage and ship all the transaction data to the cloud over the internet in a batch mode frequently. Either way, the requirement on the WAN link between the telco and the cloud data centers is very high. Let’s assume 1 million subscribers with 0.5BHCA and the CDR size at 750 bytes (note that CDRs from the 3GPP Bi interface may be much larger). This produces 375MB of CDR data a day. When it comes to OSS network management, the alarm repository can easily go into gigabytes and terabytes.

There may be a compelling business case for small telcos who do not have large amount of data and a slow growth model. Still, reliable WAN link with high bandwidth is required. These links may not be cheap and are charged on an ongoing basis – unlike the local computer servers which are one-off expenditure.

On the contrary, large telcos do not have problem with network links because in many cases, they own the network. Yet their data volume can also be orders of magnitude higher than the small ones. If you are not physically located in the same country or continent with their cloud provider’s data centers, then having a reliable WAN link becomes even more expensive if it’s possible at all – because the submarine cables are owned by consortiums, not any single telcos.

Also, beware of the pricing model of the cloud providers. They charge for each virtual machine instance, each I/O, storage, etc. The other day, I played with AWS EC2 by creating a supposedly free Linux based micro-instance, and snooping around by changing directories and doing  ‘ls’, etc. before terminating the instance. To my surprise, there were total of 13,473 I/Os, which I could not account for. I got charged for a total of 5 cents for less than an hour of activities (most idle any way). So, take a close look at the pricing model of the cloud provider, it may turn out to be much cheaper to buy your own Windows servers to host your Outlook than doing it in the cloud.

So a middle ground solution is for a telco to host their own cloud environment and use this for their own BSS/OSS systems as well as providing cloud services to their customers. Quite a few telcos have already taken this route. Of course, hosting a cloud is expensive in terms of CAPEX and OPEX. There must be a good business case to do so.

Related Articles:

• Here are a few reasons why *not* to use the cloud
• Down Goes The Internet… Again. Amazon EC2 Outage Takes Down Foursquare, Instagram, Quora, Reddit, Etc
• Outage hits Microsoft CRM Online, Office 365 customers

Google WAVES Good-Bye

I was shocked when I saw the above heading on one of the news items yesterday. I remember I saw an online video of a demo of Wave mid last year. I was very curious and anxious to try it. However, I could not find out how to join! It was apparently by invitation only. Then I forgot about it – until yesterday. So that online video turned out to be my first and last encounter with the great Wave.

Someone sited top 11 reasons why Wave failed. I believe the top reason for me is the missing marketing of the product. Every now and then I would receive promotional emails on my Gmail account about Google Ads. But there has been nothing about Wave. I have resisted the temptation to put it on my blog thus far – mainly because I am too lazy and the traffic to my blog is very low (maybe 60 per day and half of them are from crawlers).

If you take a closer look to how Google works internally, it does not seem so surprising. It is part of Google’s corporate policy to have their employees spend up to 20% of their time to work on any thing that they think interesting. If the management thinks the project has potential, the company will resource and fund it. No doubt Wave was born out of such conditions. So it’s created by nerds for nerds. The Google management probably hoped that it would one day take on the likes of Facebook. Being a nerd I don’t mind the complexity and learning curve – in fact, that’s what draws me to it. But by definition, nerds are not great social networkers – e.g. I joined Facebook and Twitter to test their APIs. That definitely reduced the appeal of Wave.

Also, the near zero marketing of Wave is no surprise considering how Google makes money – through advertising. That is why they advertise Google Ads and leave the other pet projects live and die on their own. From the days Larry and Sergey started Google, they had been focusing on technical superiority rather than monetisation. That had worked on the search engine since it was so far ahead than the others and the user experience is extremely simple. The contrary is true for Wave. So to let Wave catch up with its competitors purely based on its own merit without any marketing will require some time – much longer than the 1 year that Google had given it so far. Even the search engine did not catch on on day one – Larry and Sergey spent their doctorate years in uni to develop the product.

Now that the Wave is flushed down the toilet, I wonder what the Google people in Sydney are doing these days.

Virtual Machines

I recently beefed up my desktop machine at home by upgrading to 4GB RAM, 1TB WD Green Caviar hard disk and nVidia GeForce 7600 series. I also installed Windows 7 Ultimate 64-bit on my AMD Athlon 3500+.

One reason I upgraded my 6-year-old machine is that I want to run virtual machines and try various cloud computing and data grid frameworks.

As soon as I delved into virtual machine technologies, I found that my AMD CPU which has Socket 939, does not support hardware virtualisation. The list of AMD Athlon 64 CPUs show that AMD-V only started from Socket AM2. So that basically rules out the possibility of installing 64-bit guest OS on any of the VM software.

The VM s/w I tried are Oracle/Sun VM VirtualBox 3.2.6, VMWare Player 3.1.0 and VMWare Server 2.0. My understanding is that they are all free.

I first installed Ubuntu 10.04 on VirtualBox. The installation was very smooth. All the devices are installed without problem – sound, network, disk drives, USB devices etc.

Before, VMWare Player was only designed to play a guest OS image. But since version 3, you can also create VM using the player. The experience with VMWare Player is not much different from VirtualBox. The result was equally good. I felt that the VM seems a little faster in VMWare. Also, VMWare Player allows you to connect and disconnect devices on the fly, such as network adapter, printer, sound card, disk drives, USB devices. This is something more flexible than VirtualBox.

In fact, before I installed VMWare Player, I also tried VMWare Server (you cannot install both on the same host machine). The idea of VMWare Server is quite good – it runs as a services (Windows services, or Unix daemon) so that user can connect to it using web browser. All the actions are done on the browser – the VM console is installed as a browser plug-in. However, my user experience was not so good with VMWare Server. I installed the same Ubuntu iso image on it successfully, but it could not connect to internet (although there is connectivity between the host machine and the guest VM), and sound card was not detected. I could not add these h/w from outside the guest either as they were all greyed out on VM’s configurations page on WMWare Server.

Now I am happily running Ubuntu 10.04 on VirtualBox and Linux Mint 9 Isadora – both 32-bit of course .

Not Happy, GAE!

Cloud Computing has been a latest buzz word recently and seemingly will be for the foreseeable future. I had registered for Google App Engine (GAE) a while ago but did not do anything with it until last week.

I downloaded the Google Plugin for Eclipse 3.5 (Galileo) and installed it without any problem. I then decided to port one of my web applications written in ZK 3.6.2 with Spring Security 3.0.2 in its simplest form.

I started by creating a GAE application in Eclipse using the Google plugin’s wizard. I then copied WebContent directory from my source project to the war directory of the target GAE project. I modified the deployment descriptors according to instructions from ZK small talk with the following additions:

1. appengine-web.xml – I added /**.xml in the static file exclude path list since in my application I generate .xml files to feed them to the XML/SwfGauge package.
2. applicationContext-security.xml – the Spring Security deployment descriptor. I had to add a URL pattern for “/” explicitly due to the different way of handling it by Tomcat (the server I used for developing the web app) and Jetty (the server bundled in Google Plugin).

The modified application then ran quite successfully as GAE application in Eclipse. Of course, anything that involves creating user thread were disallowed and any Java packages/classes that were not supported by GAE were also not allowed. I thought I could work around those problems later.

Then I deployed the application to the real GAE. The deployment process was error free. I was overjoyed and clicked on the URL for my application on the appspot.com. A HTTP 500 – Server error was returned without any explanation.  I did a little fiddling and my first page was displayed if I remove Spring Security, but then all my included pages (I store all my pages in WEB-INF/pages directory) were not shown… I would need at least a few dozen of trial and error before I can pin-point the problem but my internet connection quota is running out fast! So I will call it quits for now.

While cloud computing gives you great promises it is also a shackle and limits your freedom of choice . Although GAE seems to be compatible with many frameworks, the degree of compatibility varies. For example, in the list ZK framework is COMPATIBLE with GAE, but if you look at all the features and packages that are included in ZK you will quickly realise that it is not true – server push requires separate thread and so does jFreeChart and some other bundled packages; captcha uses java.awt.Font (which is not allowed in GAE)… So you’d be better off sticking with GAE’s recommended technology stack – servlet/JSP/JSF + JPA/JDO + JAXB + XML, etc.  This may be OK for new applications but presents a big obstacle for existing large applications which heavily rely on other frameworks.

Of Twitter, Clouds and Google Goo

I have always thought Twitter is a time waster. However, I have also noticed that there are many companies and individuals using it as a marketing channel to reach the global mass, not to mention the propaganda tool as witnessed in the recent Iranian demonstrations. So instead of subscribing to RSS feeds, following tweets is now the "in" thing to do.

Using a social network for commercial and political gains is not new. We have seen it on Facebook and 2nd Life. However, to use tweets as an input for trading decision sounds dubious. Much of the twitter tweets are just noise and even garbage.

Today, out of curiosity I subscribed to Twitter. Within minutes of me opening my new Twitter account, I already got a follower. To be honest, I was pleasantly surprised and even flattered. Yet when I checked, it turned out to be a prostitute or cyber-pimp pushing some porn site . So to make use of tweets for trading, the system will have to identify which users to follow and filter out the fake, malicious and manipulating users - much like virus scanners rely on their virus database. This is a time-consuming and even labor intensive heuristic process relying on large volume of data. Then it has to filter and analyse the millions of messages per day to extract the useful information.

Let's put aside the ethical issues behind the practice of 'trading on rumors'. To say that a machine can determine market sentiment by reading tweets is at best an overstatement. Even human beings have trouble reading the sentiment in cyberspace, and that is why people have to add all sorts of smileys, emoticons and internet etiquette to assist the reader of the message. Also, the same words can have drastically different intentions and reactions based on different cultural, religional and circumstantial backgrounds. The idea of trawling through the internet to extract marketing information is not new. People have been attempting it on RSS feeds, newsgroups, user forums, etc. for a few years now. However, they are very focused/targetted on certain types of contents and are not of realtime nature -certainly not as ambitious as making trading decisions in real-time.

To apply complex fuzzy logic algorithms on large amount of (current as well as historical statistical) data is very CPU-, memory- and data-intensive. Such jobs are best suited for cloud-computing, which many big players are pushing - Sun, Microsoft, Amazon and Google. A couple of days ago I stumbled upon some cloud-computing PR articles and interviews and found this one - 谷雪梅谈云计算. I realised that the Google chief engineer interviewed in that video was my high school classmate. We affectionately called her 'Goo' back then. I guess now I have to call her the 'Goo of Google'. In that interview, a question was asked about how Google makes money. Well, I believe in 'Knowledge is power' and more so in the information age and that is how Google makes money. Comparing to the new comers (e.g. Bing, WolframAlpha) Google's search algorithm is quite lazy and unsophisticated - it relies on external links or the more you pay the higher the position in the list, yet it has accumulated vast amount of historical data and trained its systems to give better results. So Google has now taken the steps further to sell the infrastructure services and technologies such as GWT, cloud computing, Chrome and Google Wave. It seems Google has better things to do than to recycle the Twitter garbage - for now.