Cloud Platform Scalability & Optimisation
An intelligence-driven data science and behavior change company was experiencing scalability and performance bottlenecks in its existing data architecture. They needed to scale and optimize their platform to collect and analyze data from mobile devices.
The company had built an AWS-based cloud platform to process data from their mobile application SDK by collecting smartphone sensor data. This platform was designed according to a Lambda architecture with a speed layer for real-time processing and a batch layer for the daily processing of sensor data. While the platform leveraged cloud computing and modern technologies, there were still many performance and scalability bottlenecks, and this way, only a limited number of users were supported.
Contact usLook behind the scenes
Solution
In only three months, Klarrio
Results
200
From 4K to 200K persistent connections
50
more concurrent connections
0
difference in cloud infrastructure costs
Behind the scenes
Our customer built an AWS-based cloud platform to process data from their mobile application SDK by collecting smartphone sensor data.
It was designed according to a lambda architecture with both a speed layer for real-time processing and a batch layer for daily processing of sensor data.
After a cloud architecture analysis and load tests, we improved the messaging layer from 4k to 200k and supported persistent connections using Scala, Apache-Spark, Kafka and elasticsearch.
The Technology
- Scala
- Apache Spark
- Kafka
- Elasticsearch
The Expertise
- Technical optimizations
- Data science code adjustments
- Messaging layer improvement
The Expert
“At Klarrio we learned the hard way that the real scalability challenge goes beyond optimizing the CPU and memory of the current processing jobs. We found that the real issues lied in the resources that cannot be easily scaled up, such as network, single point of failures, recovering from failed components, read/write postgresql limitations, …
We really had to think outside the box here. Besides the technical optimizations, we also had to adjust the data science code written in Python, and solve queuing problems in the APIs between micro services as well as in Kafka caused by bursty traffic patterns.”
”Solving scalability issues without additional compute resources and costs is a tough, but fun challenge for any data engineer.
Bruno De BusCTO
Join us!
Want to work on similar projects?
Introverts and extroverts, geeks, nerds, and digital poets... Klarrio is the perfect place to learn and teach, experiment and brainstorm, exercise your brain, and feed your passion. Surrounded by people with amazing, world-changing talents.
We're hiring
Contact us!
We're your one-stop cloud-native partner
Big Data engineering. Data science. Data Analytics. Site Reliability Engineering. Consulting. And customized Open Source projects for companies of all sizes. What can Klarrio do for you today?
Contact us
Other Projects
Just a few projects examples.
