Imperial College London Accelerates Dementia Research with Databricks Platform
Imperial College London modernized its dementia research platform by integrating IoT, clinical, and research data using Databricks. This new architecture separates workloads, enhances data access via Unity Catalog, and empowers non-technical users to explore patient insights. The platform significantly reduced data integration timelines from six months to one month, accelerating model development and improving dementia care.
- →New Data Architecture for Scalability and Modularity
- →Enhanced Data Governance and Access with Unity Catalog
- →Accelerated Research-to-Production Workflow
- →Modernizing Dementia Research Data Platform
- →Improved Data Accessibility for Non-Technical Users
Features (3) ›
- New Data Architecture for Scalability and Modularity
The team re-architected the platform to separate operational and analytics workloads, ingesting IoT data into Delta Lake on Azure Data Lake Storage and refining it through bronze, silver, and gold layers. This modular approach ensures scalability without impacting operational systems, while EHR systems remain optimized for clinical interoperability using FHIR standards.
- Enhanced Data Governance and Access with Unity Catalog
Centralized governance was implemented using Unity Catalog, providing fine-grained access control for research teams, studies, and collaborators. Databricks now serves as the dedicated analytics layer, offering a unified environment for data exploration, model building, and collaboration, independent of production workflows.
- Accelerated Research-to-Production Workflow
The platform now facilitates a research-to-production workflow, with Unity Catalog tracking dataset usage and identifying valuable assets. Analytical pipelines are code-hardened and made reusable, reducing duplicated effort and providing researchers with standardized templates for working with complex datasets.
Enhancements (1) ›
- Improved Data Accessibility for Non-Technical Users
Databricks dashboards now present IoT device health, behavioral trends, and cohort-level insights more intuitively. Embedded dashboard integrations are being tested within monitoring systems to provide clinicians with direct access to insights within their existing tools.
Notes (2) ›
- Modernizing Dementia Research Data Platform
Imperial College London's dementia research team faced challenges with an aging data platform struggling to scale with growing IoT, clinical, and research data volumes. Issues included competing workloads, tightly coupled storage and compute, and limited data access for researchers and non-technical stakeholders, delaying translation of research into clinical practice.
- Significant Improvements in Integration and Development Timelines
The modernized platform has demonstrated substantial gains, including integrating new IoT data sources in as little as one month (down from six months) and reducing model development time to approximately one month. The platform maintained 100% uptime during migration and is experiencing rapid data growth and increasing adoption among non-technical users.
https://www.databricks.com/blog/how-imperial-college-london-accelerating-dementia-research-modern-data-platform