The Work Pile PatternThe Work Pile Pattern is a design approach that can be used to implement elastic horizontal scalability for SSIS processing. In contrast to parallelism that can be hard-coded into the SSIS control flow or data flow (e.g. using the Balanced Data Distributor), a Work Pile implementation allows […] “The Work Pile Pattern” |
Month: September 2016
Staging the ETLWhilst a key strength of SSIS is the in-memory pipeline used to implement its data flow, the staging of data to disk remains an important strategy, and can be essential for meeting some requirements. In this post I look at alternative reasons for staging data within SSIS-based ETL processes, together with an […] “Staging the ETL” |
Auditing ETL ProcessesIn the context of ETL, support for auditing can be driven by a variety of requirements. In some cases a complete audit trail of all changes made to a data item may be required, such that the path from source to destination can be interrogated in full. In others, a simple check that the number of rows […] “Auditing ETL Processes” |