Processing IoT Data using Amazon KinesisIn this post I’m using Kinesis Data Streams, Kinesis Data Firehose and Kinesis Data Analytics to process data received from an IoT sensor. The inbound IoT data stream is written to Redshift, has rolling averages calculated over it which are again written to Redshift, and is filtered to discover data that’s […] “Processing IoT Data using Amazon Kinesis” |
Month: May 2020
Running ETL Jobs using Amazon AthenaAmazon Athena supports the CREATE TABLE AS and INSERT INTO statements, which can be used to transform data between a source and a destination. In this post I’m using them to optimise the storage of data that is received into S3 as files of JSON objects. Specifically, I’m using CREATE TABLE AS (CTAS) to […] “Running ETL Jobs using Amazon Athena” |
Integrating AWS Glue Jobs with S3 Notifications and EventBridgeIn my last post I created a standalone AWS Glue job for loading and transforming data from S3 into Redshift. Here, I’m expanding that scenario by integrating the job with S3 notifications, Lambda functions, EventBridge rules and SNS messaging. Specifically this will […] “Integrating AWS Glue Jobs with S3 Notifications and EventBridge” |