September, 2016 | Andrew Smith

The Work Pile Pattern

The Work Pile Pattern is a design approach that can be used to implement elastic horizontal scalability for SSIS processing.

In contrast to parallelism that can be hard-coded into the SSIS control flow or data flow (e.g. using the Balanced Data Distributor), a Work Pile implementation allows […] “The Work Pile Pattern”

Staging the ETL

Whilst a key strength of SSIS is the in-memory pipeline used to implement its data flow, the staging of data to disk remains an important strategy, and can be essential for meeting some requirements.

In this post I look at alternative reasons for staging data within SSIS-based ETL processes, together with an […] “Staging the ETL”

Auditing ETL Processes

In the context of ETL, support for auditing can be driven by a variety of requirements.

In some cases a complete audit trail of all changes made to a data item may be required, such that the path from source to destination can be interrogated in full. In others, a simple check that the number of rows […] “Auditing ETL Processes”