Our client had acquired various companies which helped them where they are today, a pioneer in commercial insurance underwriting but that means they had also acquired a distributed system of data management platforms, so they needed a centralized data store which can eventually feed their reporting and financial systems for better revenue reporting and decision making.
It took 4.5 years to build such a system, and my job as a senior data engineer was to develop new data pipelines, help other data engineers, also data architects in designing the data warehouse.
Stats
- 6 Heterogeneous data sources
- 1.2 Million CRUD operations every day
- 110 Dimensions & 35 Facts
- 250 batch jobs
- 24 hours of processing time
Technologies/Skills used:
- SQL Server Integration Services(SSIS)
- MS SQL Server
- Python
- SQL
- Data Warehouse Techniques
- Data Modelling
- Data Mining
- CI/CD using TFS(now Azure DevOps Services)