Project Darwin
Darwin is an ongoing Enterprise project, with new and ongoing requirements. Some of the requirements completed so far include getting data from various data sources where data sources are Marketing Engines e.g. Facebook Ads, LinkedIn Ads, Google Analytics, Google Search Ads 360, Pinterest, Salesforce Marketing Cloud, etc. , merge it and process data according to client needs. The system has multiple web connectors which ingest data to the system and process it as and when the data comes and store it in databases which is used for creating dashboards and other high level analysis tools. Dotsquares has implemented various Azure services including Azure DevOps as per client’s needs to Mange, develop, test, monitor and deploy
Introduction
Requirements
- The first requirement for this project was to improve the data processing architecture. The previous architecture had some loops holes and it required immense production support due to lot of data load failures. These loops holes included lack of planning of data processing, lack of triggers and a planned Architecture. Dotsquares team helped by understanding the requirements by doing brain storming sessions with key stakeholders, further working on architecture to improve the data processing framework, finally changed the framework for data processing initially for 1 data source.
- Dotsquares team helped in development of new connectors (Data source) like LinkedIn, LinkedIn Ads, YouTube, YouTube Ads, Email, and Salesforce while some of them are in pipeline like Pinterest. Team started with requirement gathering followed by data model design to map out any & all required metrics and calculations behind metrics. Further development of required connector as per the discussed metrics and model.
- Revamp all other connectors into new processing framework.
- This requirement focused in giving public access to master database to Darwin clients. Further with access to master data in secured way, customers are able to store brand wise data & see detailed reports on Power BI embedded.
- Improved framework
- Less data processing time
- No Data failures
- Important alerts and reports
- Time zone-based triggers
- New Synchronous and Asynchronous APIs
- Managed Incremental load & Data Pipelines
- Improved Data model
- Process improvement
- Error Logging
- Azure SQL Database
- Tackled Data Concurrency
- AAS refresh Activity
- Log Analytics
- New connectors in new data processing framework
- Desired metrics, reports and analytics
- Integration with Power BI
- Public DB access in secured way
- Ad hoc Power BI reports
- Advanced Role management & RBAC
- Import Dataset in Power BI workspace
- IP whitelisting
Technology stack
- ETL- Databricks
- Azure Data Factory
- Sync Functions
- Service Bus
- ADLS Gen2
- Azure Blob Storage
- Azure DevOps
- Azure APIs
- Power BI & Power BI Embedded
- Service Principle
- Azure SQL Data warehouse
- Key Vault
How does Dotsquares help with Azure?
From building, deploying to managing the applications with Azure Services, we at Dotsquares have done it all. With a team of efficient professionals who’ve got the knowledge of integrating the Microsoft Azure services in the client’s project, we provide end-to-end Azure services.
Production Support
From building, deploying to managing the applications with Azure Services, we at Dotsquares have done it all. With a team of efficient professionals who’ve got the knowledge of integrating the Microsoft Azure services in the client’s project, we provide end-to-end Azure services.
Technical Architecture
