Calling A Python Lambda Function From Pelican
Posted on Sun 09 August 2020 in Tech • 6 min read
The Problem - Collecting User Emails From A Static Site
As I write more blog posts on this site, I wanted to make a way for readers to sign up for updates and notifications about posts. There's tools that let you create a subscribe link on your site, like ConvertKit or MailChimp. I chose to make my own, for two main reasons:
- I control and keep the data
- It seemed like a fun problem to solve
Without getting into a whole debate about buy-vs-build, I chose to build in spite of it likely taking longer and being worse. I considered this more of a learning opportunity that I could tinker with, and not shipping user emails off to another company (no matter how much I trust them) seems like a good thing. (Keen observers will note that I'm technically shipping the data off to AWS, but I consider a hosted database different from a mailing list service.)
The Solution - AWS Lambda and AWS DynamoDB
I'm partial to using AWS technologies for personal tasks, since it helps me learn about them in order to use them for my day job. The site is already hosted on AWS in an S3 bucket using Cloudfront as the CDN, so it's not a big stretch to use other AWS tools as well. I mention this because it's not necessarily the right choice for anyone else. Using Netlify or Github Pages to host your Pelican site might affect the choices for the rest of your tech stack.
The general flow I want is a separate subscription page with a form for the user to fill out. The submitted form goes to a Python Lambda function. The Lambda function adds the data to a DynamoDB table, then sends the user's browser to a "thank you" page. I also want to be notified in my personal slack channel when someone signs up. Let's get started!
Setting Up The Lambda Function
From the AWS Lambda console, I created a new function. I chose Python 3.8 as my runtime, as that was the latest Python available at the time. I selected the option for the execution role to "Create a new role with basic Lambda permissions", as I'd modify those permissions later to allow the API Gateway to access the function and for the function to access the DynamoDB table.
Here's the basics of the function that takes data that was POSTed to the function, makes sure that the 'email' field existed (as that will be our database primary key), and enters it as a row in a DynamoDB table:
import json
import boto3
import urllib.parse
def lambda_handler(event, _):
postdata = urllib.parse.parse_qs(event['body'])
try:
if 'email' in postdata:
dynamo_dict = {}
for key in postdata:
if type(postdata[key][0]) is str:
dynamo_dict[key] = {'S': postdata[key][0]}
elif type(postdata[key]) is int:
dynamo_dict[key] = {'N': postdata[key][0]}
else:
# Not a string or int, ignoring for now
print("I don't know what to do with " + key + " and " + postdata[key][0])
dynamo_response = client.put_item(TableName='myDynamoDBTable', Item=dynamo_dict)
else:
no_primary_key_error = "Not formed properly, try again! You currently need email"
print(no_primary_key_error)
return no_primary_key_error
except Exception as e:
print (e)
return {
"statusCode": 200,
"headers": {
"Content-Type": "application/json",
},
"body": dynamo_response,
}
Setting Up the API Gateway
In order to actually call this function, I set up an API Gateway to trigger the Lambda. Near the top of the Lambda page, there's a "designer" section, with the lambda function in the middle. On the right side, you can add a trigger. I chose "API Gateway", then chose to "Create an API". I tried doing "HTTP API" but couldn't get it to work, so I ended up with "REST API" with the Security choice as "Open". Everything else can be left as the defaults, and this will update the IAM permissions automatically. The trigger on the lambda has a Details arrow that can be expanded, showing the "API endpoint" URL. I need this for the form.
Creating the Pelican Form
I've got a customized Pelican template that I'm using, so I created a new "subscribe.html" template. In the template, I put an HTML form that will submit to the URL I got from the API Gateway above. Here's the details of the template:
{% extends "base.html" %}
{% block meta %}
{{ super() }}
{% endblock %}
{% block title %} – {{ page.title }}{% endblock %}
{% block content %}
<div id="subscribe">
<form method="POST" action="API GATEWAY URL HERE">
<input name="redirect" type="hidden" value="{{ SITEURL }}/{{ page.slug }}">
<input name="url" type="hidden" value="{{ page.slug }}">
<div><label>First Name: <input name="first_name" type="text"></label></div>
<div><label>Last Name: <input name="last_name" type="text"></label></div>
<div><label>E-mail: <input name="email" type="email"></label></div>
<button type="submit">Submit</button>
</form>
</div>
{% endblock %}
I then created an essentially blank page that used this template, and put it in my content/pages directory:
Title: Subscribe
Date: 2020-08-04 00:00
Modified: 2020-08-04 00:00
Template: subscribe
Creating the DynamoDB Table
To store user emails and names, I chose to use DynamoDB. I debated storing this in a flat file in S3 or using a relational database, but I was curious about DynamoDB and it seemed like a good use case for this. I created the new table with a Primary Key of 'email'. In the settings, I changed the 'Read/write capacity mode' to 'on-demand', as I don't know how many people will signing up, so I just wanted it to write when necessary. I made sure the name of the table was the same as I had defined in the Lambda function.
I then had to edit the IAM permissions to allow the Lambda function to access the DynamoDB table. I used the IAM builder to create a new policy that just allowed read and write access to DynamoDB, with the 'resource' field set to the ARN string of my DynamoDB table. This policy ended up looking like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"dynamodb:BatchGetItem",
"dynamodb:BatchWriteItem",
"dynamodb:ConditionCheckItem",
"dynamodb:PutItem",
"dynamodb:DeleteItem",
"dynamodb:GetItem",
"dynamodb:Scan",
"dynamodb:Query",
"dynamodb:UpdateItem"
],
"Resource": "arn:aws:dynamodb:us-east-1:123456789:table/mydynamotable"
}
]
}
I likely could be more limiting on my list of permissions, but I wanted to keep options open for the Lambda to do some other potential tasks, while still being limited to just that table. I then attached that policy to the IAM Role that the Lambda was using, so it had 2 separate IAM policies attached.
Bonus - Thank You Page
At this point the subscription link worked when I deployed my Pelican site, but it would just end at a page that had the DynamoDB response pasted into the browser window. To fix this, I changed the return of my lambda function to this:
returnsite='''<meta http-equiv="Refresh" content="0; url='https://mysite/thanks'" />'''
return {
"statusCode": 200,
"headers": {
"Content-Type": "text/html",
},
"body": returnsite,
}
This creates a new HTML site with a single meta tag that just refreshes the browser after a 0 second delay to a "thanks" page at my site. The "thanks" page I created was just a basic Pelican page that is hidden so it doesn't show on the menu:
Title: Thanks
Date: 2020-08-04 00:00
Modified: 2020-08-04 00:00
Status: hidden
##Thanks for subscribing!
Bonus 2 - Signup Notifications In Slack
I have my own Slack organization that I use for various notifications about Google Drive, Github, Twitter, ParkMyCloud, and more. I wanted a notification to be sent when the Lambda function was triggered, so I created a Webhook connection in my Slack channel using this setup and saved the Webhook URL. I then added this to my Lambda function:
from urllib.request import Request, urlopen
slack_url = 'my_webhook_url'
slack_text = "New subscriber!"
slack_text += "\nEmail = "+postdata['email'][0]
slack_text += "\nFirst name = "+postdata['first_name'][0]
slack_text += "\nLast name = "+postdata['last_name'][0]
slack_message = {
'text': slack_text,
'username': "AWS Lambda",
'icon_emoji': ':tophat:'
}
data = json.dumps(slack_message).encode('ascii')
req = Request(slack_url, data)
response = urlopen(req)
The Resulting Lambda Function
Here's what my final Lambda function ended up being:
import json
import boto3
import urllib.parse
from urllib.request import Request, urlopen
def lambda_handler(event, _):
postdata = urllib.parse.parse_qs(event['body'])
print(postdata)
client = boto3.client('dynamodb')
try:
if 'email' in postdata:
print("email in postdata")
dynamo_dict = {}
for key in postdata:
if type(postdata[key][0]) is str:
dynamo_dict[key] = {'S': postdata[key][0]}
elif type(postdata[key]) is int:
dynamo_dict[key] = {'N': postdata[key][0]}
else:
# Not a string or int, ignoring for now
print("I don't know what to do with " + key + " and " + postdata[key][0])
print(dynamo_dict)
dynamo_response = client.put_item(TableName='mytable', Item=dynamo_dict)
print(dynamo_response)
#return dynamo_response
else:
no_primary_key_error = "Not formed properly, try again! You currently need email"
print(no_primary_key_error)
return no_primary_key_error
except Exception as e:
print (e)
# Post to my slack channel on slack
slack_url = 'https://hooks.slack.com/services/blahblah'
slack_text = "New subscriber!"
slack_text += "\nEmail = "+postdata['email'][0]
slack_text += "\nFirst name = "+postdata['first_name'][0]
slack_text += "\nLast name = "+postdata['last_name'][0]
slack_message = {
'text': slack_text,
'username': "AWS Lambda",
'icon_emoji': ':tophat:'
}
data = json.dumps(slack_message).encode('ascii')
req = Request(slack_url, data)
response = urlopen(req)
print(response)
# make a blank webpage that redirects to the thanks page after submitting
returnsite='''<meta http-equiv="Refresh" content="0; url='https://mysite/thanks'" />'''
return {
"statusCode": 200,
"headers": {
"Content-Type": "text/html",
},
"body": returnsite,
}