Skip to content

Andrew Smith

Software Engineer / Architect

  • Home
  • Bio/Contact

Month: September 2016

The Work Pile Pattern

The Work Pile Pattern

The Work Pile Pattern is a design approach that can be used to implement elastic horizontal scalability for SSIS processing.

In contrast to parallelism that can be hard-coded into the SSIS control flow or data flow (e.g. using the Balanced Data Distributor), a Work Pile implementation allows […] “The Work Pile Pattern”

Author Andrew SmithPosted on Posted on 23rd September 201627th May 2020Categories UncategorisedTags ETL, SQL Server, SSIS
Staging the ETL

Staging the ETL

Whilst a key strength of SSIS is the in-memory pipeline used to implement its data flow, the staging of data to disk remains an important strategy, and can be essential for meeting some requirements.

In this post I look at alternative reasons for staging data within SSIS-based ETL processes, together with an […] “Staging the ETL”

Author Andrew SmithPosted on Posted on 14th September 201626th May 2020Categories UncategorisedTags ETL, SQL Server, SSIS
Auditing ETL Processes

Auditing ETL Processes

In the context of ETL, support for auditing can be driven by a variety of requirements.

In some cases a complete audit trail of all changes made to a data item may be required, such that the path from source to destination can be interrogated in full. In others, a simple check that the number of rows […] “Auditing ETL Processes”

Author Andrew SmithPosted on Posted on 4th September 201626th May 2020Categories UncategorisedTags ETL, SQL Server, SSIS

Posts navigation

About

I’m Andrew Smith and I’m an experienced software professional with an employment history spanning a variety of industries and including blue-chip organisations such as Ford Motor Company, PricewaterhouseCoopers and IQVIA.

I’m currently architecting and implementing an AWS-based IoT system, including network and security design, RDS database implementation and the provision of high availability and disaster recovery capabilities.

Whilst this website was initially written to document research I was doing throughout the Microsoft SQL Server product suite, it’s now oriented around how best to use Amazon Web Services (AWS) to deliver solutions to problems in enterprise computing.

Contact

Email:
andrew@andrewleesmith.co.uk
Linked In:

Popular Tags

  • AWS (8)
  • ETL (20)
  • Python (6)
  • SQL Server (38)
  • SSIS (17)

Recent Posts

  • Optimising Query Execution for Amazon Redshift Spectrum
  • Processing IoT Data using Amazon Kinesis
  • Running ETL Jobs using Amazon Athena
  • Integrating AWS Glue Jobs with S3 Notifications and EventBridge
  • Loading data into Amazon Redshift using AWS Glue

Archives

  • June 2020
  • May 2020
  • April 2020
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • Home
  • Bio/Contact
Andrew Smith Proudly powered by WordPress
Close Bitnami banner
Bitnami