May, 2020 | Andrew Smith

Processing IoT Data using Amazon Kinesis

In this post I’m using Kinesis Data Streams, Kinesis Data Firehose and Kinesis Data Analytics to process data received from an IoT sensor.

The inbound IoT data stream is written to Redshift, has rolling averages calculated over it which are again written to Redshift, and is filtered to discover data that’s […] “Processing IoT Data using Amazon Kinesis”

Running ETL Jobs using Amazon Athena

Amazon Athena supports the CREATE TABLE AS and INSERT INTO statements, which can be used to transform data between a source and a destination.

In this post I’m using them to optimise the storage of data that is received into S3 as files of JSON objects. Specifically, I’m using CREATE TABLE AS (CTAS) to […] “Running ETL Jobs using Amazon Athena”

Integrating AWS Glue Jobs with S3 Notifications and EventBridge

In my last post I created a standalone AWS Glue job for loading and transforming data from S3 into Redshift.

Here, I’m expanding that scenario by integrating the job with S3 notifications, Lambda functions, EventBridge rules and SNS messaging. Specifically this will […] “Integrating AWS Glue Jobs with S3 Notifications and EventBridge”