DataStage development for the Centralized Data Hub project. The CDH architecture uses DataStage based on sequential files, and a number of jobs were developed, to offer patterns to other developers to use in rapid development projects (via Containers). Verification of primary keys (from a sequential file) was performed via a Sort Stage, adding a key change parameter into the flow, allowing the Copy Stage to split a key failure output as well as a source data output.
DataStage development for the Financial Data Mart, Star Schema project. Multiple fact tables populated through a number of DataStage jobs (based on multiple time dimensions). Numerous data sets and UDB tables joined within the DataStage Join stage, including a number of Build Op stages to perform transformations. Resulting fact job populates a 300 column fact table supporting 3.5 million rows. Dimension tables are built hanging off the fact table and refreshed on a daily basis, again through the use of DataStage jobs.
DataStage development for the You Owe Me (YOM) data warehouse project. Reception of 61 full refresh and 2 incremental daily feeds, flow into 63 DataStage jobs which perform business defined transformations through multiple Build Ops stages.
DataStage development for the Timer Database data mart. This data mart supports a single fact table with 12 dimension tables. The fact table is built via the flow of data from a Sequential File stage, outer joining to multiple UDB tables landing into a daily refreshed fact table. The 12 dimension tables are refreshed daily as well, via 12 DataStage jobs.
DataStage development of 285 daily data feed Customer Survey project. Use of the Funnel stage to union the common data to 45 unique formats. Then developed Build Ops stages for each of the 45 to implement business rules. The 45 tables again are then consolidated into 4 tables, extracted from the enterprise data warehouse and delivered to the business data mart.
DataStage development for multiple projects performing a Multiload Stage to refresh data in the Teradata Active Data Warehouse. ETL transformation is performed via the Build Op stage, and the Teradata native tables are refreshed on a daily basis.
Multiple maintenance projects performed against numerous DataStage production jobs. Some efforts include the splitting of the Sequential File stage output to perform round-robin output, writing to each partition, modification of Build Ops stages to strip hexadecimal characters from source data, and more.