Saturday 27 March 2021

Two Lambdas Are Better Than One

 To refactor the single Lambda function and API previously, I created two Lambda functions:

  1. getNumbersGameHtml - to return the HTML string, which is the GUI page of the Numbers Game single page web application.
  2. solveNumbersGame - to return the array of solutions (strings of math expressions)
The Python code for getNumbersGameHtml function:


def lambda_handler(event, context):
    return {
        'body': myHtml
    }
    
myHtml = ('<!DOCTYPE html>\n'
'<html>\n'
'<head profile=\"http://www.w3.org/2005/10/profile\"><link rel=... />...\n'
'<body>\n'
'...'
'	<input type="submit" value="Solve Game" onClick="if (validate_inputs()) {solve_game();} else alert(\'Please ensure all numbers and targets are filled in and with valid values.\\nHover mouse over input boxes for rules details.\');" />&nbsp;&nbsp;&nbsp;&nbsp;    \n'
'	<input type="button" value="Reset Game" onClick="reset_game()"/> \n'
'...')

# alternatively
myHtml="""<!DOCTYPE html>
<html>
<head profile=\"http://www.w3.org/2005/10/profile\"><link rel=... />...
<body>
...
	<input type="submit" value="Solve Game" onClick="if (validate_inputs()) {solve_game();} else alert(\'Please ensure all numbers and targets are filled in and with valid values.\\nHover mouse over input boxes for rules details.\');" />&nbsp;&nbsp;&nbsp;&nbsp;
	<input type="button" value="Reset Game" onClick="reset_game()"/>
...
"""
The web page looks like this:

    


Copyright (c) Romen Law. All rights reserved.

The Javascript code for solveNumbersGame function:

exports.handler = async (event) => {
	var e=event;
	var numbers=[e.n1,e.n2,e.n3,e.n4,e.n5,e.n6], target=e.target;
	var answer=solveAllGames(numbers, target);
	
    const response = {
        statusCode: 200,
        body: JSON.stringify(answer)
    };
    
    return response;
};
An API is created at the API Gateway with two methods implemented:
  • GET method invokes the getNumbersGameHtml function
    • at the GET - Integration Response, for response 200, 
      • add a Header Mappings entry with Response header of content-type and the Mapping value of 'text/html' (notice the single quote - it represents literal string in the mapping language).
      • at Mapping Templates, add an entry for Content-Type of text/html with template code of $input.path('body') (don't forget to press the grey Save button at the bottom before the blue Save button at the top).
    • at the GET - Method Response, for HTTP status 200 
      • add header of content-type
      • add Response model of Empty for content-type text/html
  • POST method invokes the solveNumbersGame function
    • at the POST - Method Request, Request Body
      • set the Request Validator field to Validate body and click the tick.
      • add NumbersInput model for application/json. This will enforce the input data format and some validation rules
    • at the POST - Integration Response, Mapping Templates: add template code $input.path('body') for content type application/json. This will extract the 'body' field from the response of the Lambda function and return it as the API's response.
The NumbersInput model:


{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "n1": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n2": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n3": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n4": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n5": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "n6": {
      "type": "integer", "minimum": 1, "maximum": 100
    },
    "target": {
      "type": "integer", "minimum": 100, "maximum": 999
    }
  },
  "required": [
    "n1",
    "n2",
    "n3",
    "n4",
    "n5",
    "n6",
    "target"
  ]
}
See also:

Sunday 21 March 2021

One Lambda to Serve Them All

 The really cheap way to host a dynamic web site on AWS is by using the combination of S3 and Lambda:

  • S3 to store the static contents such as HTML files, graphics, CSS, javascript files
  • Lambda and API Gateway to serve the dynamic contents by providing APIs to the javascripts of the web site.
I went one step further with my Numbers Game single page web application by serving everything from a Lambda function without having to use S3. This is because
  1. I want the web site to stay totally free. Lambda is perpetually free tier and I don't get charged as long as my volumes stay under the threshold. However, S3 is not free after the first 12 months.
  2. The static content for my web page is relatively simple - one HTML file actually. My own Javascripts are embedded in the page; 3rd-party contents (CSS, javascripts, icons) are pointed to their corresponding URLs so the browser will simply gets them from the internet.
  3. I don't want to enable CORS of my AWS API, so my web page (and its javascript) and the APIs must be served by the same domain. 
This is very easy to do with AWS Lambda: my Lambda function handles two types of requests - GET and POST (of course, you can refactor these to let the API layer do some mapping and use two Lambda functions... ). 
  • for GET, the Lambda function returns the HTML string. This string was prepared by feeding my static HTML file to some online converter to get the big string with all the double quotes (along with some other characters) escaped - the online tool's result was not perfect, I had to hand fix something that it missed. This HTML page basically allows the user to enter the input numbers and performs some validation according to the game's rules. Once submitted, it will invoke the API using POST
  • for POST, the Lambda function invokes the javascript functions to calculate the results and return them as a JSON string. The HTML simply displays the results on the same page.
The Lambda function:

exports.handler = async (event) => {
	console.log("event:"+JSON.stringify(event));
	var answer="no or unsupported event";
	var response;
	if(event.httpMethod == "GET") {
		response = {
	        statusCode: 200,
	        headers: {
	        	"content-type": "text/html"
	        },
	        body: myHtml,
	    };
	} else if (event.httpMethod == "POST" || event.routeKey.startsWith("POST")) { // supporting REST APIs and HTTP APIs
		if(event.body != null) {
			var e=JSON.parse(event.body);
			var numbers=[e.n1,e.n2,e.n3,e.n4,e.n5,e.n6], target=e.target;
			answer=solveAllGames(numbers, target);
			//answer=solveGame(numbers, target);
		}
	    response = {
	        statusCode: 200,
	        body: JSON.stringify(answer),
			headers: {
	        	"Access-Control-Allow-Origin": "*"
	        }
	    };
	}
    console.log(response);
    return response;
};
//... more code here ... where myHtml stores the big string converted from my static HTML file.

This approach works quite well. However, it is an abomination and wrong in so many places:
  • it's mixing network layer logic with business logic - in the code it's handling http methods which should be the responsibility of the API Gateway!
  • At the API, when only the ANY is mapped, the event passed into the Lambda is at a higher level - the Lambda function has to traverse into the 'body' to get the input parameters. 
  • The JSON.parse(event.body) is very cumbersome. I should be able to just get the input numbers from the event itself as a JSON object, not as a string. 
  • the lambda function cannot be used as-is for other triggering types (because it explicitly handles HTTP methods). e.g. cannot test in AWS console.
It's time to refactor... 💨

See also:

Monday 8 March 2021

Loving Powershell

I have always loved Unix shell scripting and Unix shells in general. I especially like the shortcut keys in shell environment and you can choose different modes - it's emacs mode for me anytime. They are such productivity boost - it makes me cringe to see someone painstakingly pressing the back-arrow all the way to the front of the line just to fix a typo (and very often even in training videos). 😖

When it comes to Powershell, I am a newbie. I only started using it quite recently. However, I was pleasantly surprised on how powerful its construct is and how easy it is to manipulate objects in Powershell.

In Unix, the output of the commands are strings. So when you pipe them to the next command you are doing string manipulation - hence, sed, awk, cut... become very useful. However, in Powershell, they are objects - you can directly manipulate the objects themselves without having to rely on helper commands to locate the right fields all the time. 

Here are some examples to demonstrate it. In AWS, it is often required to assign security policies to users or roles, etc. However, those policy names are long and not easy to remember (for me anyway). So I end up querying those policy names and ARNs quite all the time. For example: search for S3 related security policies - 

1. using AWS-Tools for Powershell it is easy to filter based on the fields/attributes/properties and choose which one to display by directly manipulating the object's attributes:
PS C:\WINDOWS\system32> Get-IAMPolicies | Where-Object {$_.PolicyName -like "*S3*"} | Select-Object -Property PolicyName, arn

PolicyName                                              Arn                                                                    
----------                                              ---                                                                    
AmazonDMSRedshiftS3Role                                 arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role           
AmazonS3FullAccess                                      arn:aws:iam::aws:policy/AmazonS3FullAccess                             
QuickSightAccessForS3StorageManagementAnalyticsReadOnly arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageMan...
AmazonS3ReadOnlyAccess                                  arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess                         
AmazonS3OutpostsFullAccess                              arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess                     
S3StorageLensServiceRolePolicy                          arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy
IVSRecordToS3                                           arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3                 
AmazonS3OutpostsReadOnlyAccess                          arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess                 



PS C:\WINDOWS\system32> 

See how neat the results are laid out! What I also like about PS is that its case insensitive - even using -like "*s3*" will return the same results. 👍

The huge complaint about AWS Tools for Powershell is its installation process - it took over 1 hour in one of my laptops over proxy, and about 5 minutes on another.

2. using AWS CLI native query capability
PS C:\WINDOWS\system32> aws iam list-policies --query 'Policies[?contains(PolicyName,`S3`)].[PolicyName, Arn]' --output text
AmazonDMSRedshiftS3Role	arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role
AmazonS3FullAccess	arn:aws:iam::aws:policy/AmazonS3FullAccess
QuickSightAccessForS3StorageManagementAnalyticsReadOnly	arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly
entAnalyticsReadOnly
AmazonS3ReadOnlyAccess	arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
AmazonS3OutpostsFullAccess	arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess
S3StorageLensServiceRolePolicy	arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy
IVSRecordToS3	arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3
AmazonS3OutpostsReadOnlyAccess	arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess

PS C:\WINDOWS\system32>
If it weren't for the syntax highlighter of this web page, I'd feel dizzy looking for the item that I want...😵  The AWS CLI is strictly case-sensitive and very unforgiving! 😠

3. using combination of AWS CLI query and awk
$ aws iam list-policies --query 'Policies[?contains(PolicyName, `S3`)]' --output text | awk '{print $9, $1}'
AmazonDMSRedshiftS3Role arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role
AmazonS3FullAccess arn:aws:iam::aws:policy/AmazonS3FullAccess
QuickSightAccessForS3StorageManagementAnalyticsReadOnly arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly
AmazonS3ReadOnlyAccess arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
AmazonS3OutpostsFullAccess arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess
S3StorageLensServiceRolePolicy arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy
IVSRecordToS3 arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3
AmazonS3OutpostsReadOnlyAccess arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess
Identical result as the previous one. As you can see, the awk command assumes the desired fields are the 9th and 1st fields. This makes the code fragile. 😱

I used WSL for the above command. If using Git Bash for Windows, then put the following before running any aws CLI commands:
$ export HOME=$USERPROFILE
This is needed for aws to locate the configuration and credential files.

4. using combination of AWS CLI and jq

$ aws iam list-policies --query 'Policies[?contains(PolicyName, `S3`)]' | jq '.[] | {PolicyName, Arn}'
{
  "PolicyName": "AmazonDMSRedshiftS3Role",
  "Arn": "arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
}
{
  "PolicyName": "AmazonS3FullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
{
  "PolicyName": "QuickSightAccessForS3StorageManagementAnalyticsReadOnly",
  "Arn": "arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
}
{
  "PolicyName": "AmazonS3ReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
{
  "PolicyName": "AmazonS3OutpostsFullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess"
}
{
  "PolicyName": "S3StorageLensServiceRolePolicy",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy"
}
{
  "PolicyName": "IVSRecordToS3",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3"
}
{
  "PolicyName": "AmazonS3OutpostsReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess"
}
or...

$ aws iam list-policies | jq '.Policies[] | select(.PolicyName | contains("S3")) | {PolicyName, Arn}'
{
  "PolicyName": "AmazonDMSRedshiftS3Role",
  "Arn": "arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
}
{
  "PolicyName": "AmazonS3FullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
{
  "PolicyName": "QuickSightAccessForS3StorageManagementAnalyticsReadOnly",
  "Arn": "arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
}
{
  "PolicyName": "AmazonS3ReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}
{
  "PolicyName": "AmazonS3OutpostsFullAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess"
}
{
  "PolicyName": "S3StorageLensServiceRolePolicy",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy"
}
{
  "PolicyName": "IVSRecordToS3",
  "Arn": "arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3"
}
{
  "PolicyName": "AmazonS3OutpostsReadOnlyAccess",
  "Arn": "arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess"
}
$ aws iam list-policies | jq '.Policies[] | select(.PolicyName | contains("S3")) | .PolicyName, .Arn'
"AmazonDMSRedshiftS3Role"
"arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
"AmazonS3FullAccess"
"arn:aws:iam::aws:policy/AmazonS3FullAccess"
"QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
"arn:aws:iam::aws:policy/service-role/QuickSightAccessForS3StorageManagementAnalyticsReadOnly"
"AmazonS3ReadOnlyAccess"
"arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
"AmazonS3OutpostsFullAccess"
"arn:aws:iam::aws:policy/AmazonS3OutpostsFullAccess"
"S3StorageLensServiceRolePolicy"
"arn:aws:iam::aws:policy/aws-service-role/S3StorageLensServiceRolePolicy"
"IVSRecordToS3"
"arn:aws:iam::aws:policy/aws-service-role/IVSRecordToS3"
"AmazonS3OutpostsReadOnlyAccess"
"arn:aws:iam::aws:policy/AmazonS3OutpostsReadOnlyAccess"

Thursday 4 March 2021

Android Not-So-Smart TV

 On my TCL Android TV, the USB files are displayed on screen in reverse chronological order based on the file creation date, by default. To order by file name, I have to explicitly go to the menu and select it every time. This is really inconvenient. What's worse - the TV can only sort the file names as a string without considering numeric parts of the string - e.g. file 2.* will appear before 11.*. 

So for me to watch TV series or training videos, I 'd have to manually look for the next file to watch. This is not smart at all.

To get around the problem I recursively updated all my files' creation date to reversely align with their file names (with consideration of the numerical part of the file name).



$filepath="E:\1_TV\Show\[romen.com] - Block & Chain - A Complete Introduction"
ls -literalPath $filepath -Recurse | Sort-Object -descending { [regex]::Replace($_.Name, '\d+', { $args[0].Value.PadLeft(20) }) } | forEach-Object { 
    Write-Host "processing file" $_
    $_.CreationTime = Get-Date
    $_.LastWriteTime = Get-Date
    sleep 1
    }

Tuesday 2 March 2021

2c Worth of Notes

The following nuggets of knowledge costed me US2c (AUD3c) by trial and error playing with EC2 outside of the 12-month free tier.
  • after modifying (creating new versions of) launch templates make sure to make it default, or be specific of the version number when launching a new EC2 instance. Otherwise, the changes will not take effect.
  • It's better to copy and paste when creating User Data - I made the simple mistake of typing 'ec2_user' instead of 'ec2-user'. Also consult https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/ to change the frequency of the execution of the user data.
  • the user data is run as root.
On another note, while I was playing with the EC2 launch template, my WSL played up - the DNS setting stopped working on the Linux VMs, and WSL kept overwriting the /etc/resolv.conf file even with /etc/wsl.conf created with 

[network]
generateResolvConf = false
It turns out that WSL has to be terminated to make the wls.conf take effect; alternatively, prevent the /etc/resolv.conf from being writable by
chattr +i /etc/resolve.conf