Steps for phase 1
- Finding an initial experimental data model
- Make an MVP (with the new backend and a sample dataset)
- Run some experiments
- Scaling/performance improvements
- Change the data model and indexes accordingly
- Finding the hosting strategy (single server vs multiple instance)
- Populate with prod data and testing
- Deployment
Phase 2:
- Support to work with incremental data stream