Steps for phase 1
- Finding an initial experimental data model
- Make an MVP work with the new backend with a sample dataset
- Run some experiments
- Scaling/performance improvements
- Change the data model and indexes accordingly
- Finding the hosting strategy (single server vs multiple instance)
- Populate with prod data and testing
- Deployment
Phase 2:
- Support to work with incremental data stream