How we helped Argus create a data processing solution that effectively handles 100,000 files daily
Argus Data Insights wanted us to work with them as their software development partner on yet another amazing project – an extension of the data lake.
125 years of experience. Catering to clients from all across the world, including Germany, Switzerland, France, and Spain. Providing custom smart media insights. Argus Data Insights represents a group of companies that provides solutions for active media work, media monitoring, and media analysis.
At Vega IT, we pride ourselves in being a part of their success as their technical partner.
First, we worked with the client's team on implementing natural language processing models to create an inventive engine for detailed reports and rich insights. Delighted with our collaboration and innovative way of thinking, Argus wanted us to join them on another amazing project – developing the extension of their data lake.
We used NLP to help Argus develop an intuitive engine for detailed reports and rich insights.
By using natural language processing models, we drammatically improved the quality of the media insights Argus could offer its clients. We focused on the four core areas - sentiment analysis, named entity recognition, target dominance, and language detection.
Read moreThe challenge: Reusing the existing system for new data processing
When they started working with us as their tech partner, Argus Data Insights already had a legacy system that processed data. Their primary goal was the transfer of this data flow from the legacy system to a newer system. By leveraging their existing infrastructure, their goal was to optimize resources, minimize expenses, and boost efficiency.
This involved integrating the new data source into the current system and ensuring it could manage the data without the issues faced by the legacy system. To do that, they first needed to eliminate their legacy relational database, which often struggled under the load of constant queries. Their goal was to switch to a new system, which logs data in a different database that is more search- and analytics-friendly.
They were looking for experienced engineers who would perform detailed analyses with them and help them find a way to process new print data, leveraging the existing infrastructure.
The team: A five-person team to drive the client’s success
The client was looking for a team of expert engineers who had thorough experience in handling big data for companies similar to theirs. That is why they chose us as their long-term technical partner.
To meet their requirements, we created a team of two software engineers, a DevOps engineer, a delivery manager, and a quality assurance engineer.
Over time, we built strong relationships and effective collaboration with the client. They valued our proactive, efficient approach, which reflected in a range of new ideas and suggestions.
The solution: Getting the most out of their existing system
Before starting the development part, we first provided a discovery process. Our primary goal was to explore the system and identify the best implementation opportunities.
By leveraging the existing infrastructure, we created an extension without needing to change the entire structure. We implemented new components that were critical to us, including EventGrid, EventHub, ServiceBus, new enrichment services inside Kubernetes, and CI/CD pipelines.
Additionally, we helped the client meet the data retention period compliance, which was one of their top priorities. We helped them define and implement a 30-day data retention period.
The results: Streamlined data processing
Proactivity. Determination. Passion. We contributed to the client’s success on multiple levels.
Unlocking fast data processing
We optimized certain parts of the system to process data faster – it is now capable of handling around 100,000 files daily without a hitch. Files are processed in milliseconds. According to our tests, the system can now handle more than 10K files efficiently at the same time and without any performance problems.
Spotless documentation that leads to project success
We helped the client translate the original documentation into English and write new documentation. The goal? Making it easy for everyone working on the project to get on board faster.
The full tech stack
- C#
- .NET
- Azure
- Elasticsearch
- Kibana
- Python
- Kubernetes