INNOVATION & BIG DATA ANALYTICS EMPOWERING A PREDICTIVE WATER SUPPLY MANAGEMENT SYSTEM WITH CLOUDERA AND HADOOP
Although we cannot stop the sudden and irresistible actions of natural forces —so called “acts of god”— we can proactively get ahead of them to prevent the subsequent disasters that can totally disrupt our lives. We call this moving from a reactive model to a predictive model i.e. solving problems “before the fact” instead of “after the fact”. Using Cloudera Distribution Hadoop (CDH) and Big Data Analytics we have clearly made this game changing transition. Leveraging tools for simple ingestion of unstructured/semi-structured datasets from any storage types, easy implementation of customized data crawlers and the extensive support for machine learning we created a low cost solution that enables us to accurately predict the future so we can better prepare for it. In the past we could only react to a flooding emergency due to overloaded and burst pipes, today we can completely prevent it!
Publiacqua supplies and distributes high quality water to 1.3 million Florentines everyday. This is our duty and we are responsible and accountable for the highest standards of service. We all benefit from the life giving powers of water but we are equally aware of its destructive force when not fully controlled and managed. We learned from the past that our services are exposed to a domino like effect of disruption when adverse weather conditions overload the capacity of our water distribution systems. Such occurrences render our services almost helpless and can have devastating knock-on effects on all supply chains and services throughout the region, with very long and costly recovery periods.
Learning to predict
To fortify our service levels against such exposure we decided to innovate beyond the existing monitoring systems that are only capable of alerting limited problems as they unfold. We needed a monitoring, alerting & reporting model that would warn us well in advance of potential weather driven disasters, material flaws and structural degradation problems, giving us enough time to take preventative action. We needed a predictive IT model for our water management system and a trusted guide with proven experience in designing big data analytical solutions to lead the way.
Innovation, expertize and proven experience
We turned to TAI SOLUTIONS, not only for their experience and expertize but for their patience, understanding and commitment to building solutions that scale to stand the test of time. They successfully designed and completed a proof of concept based on big data and machine learning technology platforms.
TAI SOLUTIONS preferred Cloudera’s CDH platform, including Apache Hadoop because of how it meets the demands of large-scale deployments like ours. The simplicity of CDH was also a key part of their design as it delivers so much straight out of the box. And, because Cloudera has integrated Hadoop with many other critical open source projects it helped us create an advanced system to perform end-to-end Big Data workflows.
A combination of Apache Flume for data collection and streaming into Hadoop, and Apache Sqoop for integrating with relational databases gave us the all the heterogeneous connectivity we needed. Apache Spark covered our advanced analytics needs with its in-memory batch and real-time processing. Apache Oozie was added for workflow scheduling, Apache Hive for the batch processing of our Hadoop data and Apache Parquet for our database management formatting.
By comparing and analysing multiple real-time data streams from weather stations, water distribution monitoring sensors and historical data archives we can —now— instantly predict short & long-term water flow changes and respond accordingly, with pinpoint accuracy. Additionally, any non-weather-related changes in water flow quantity and quality can be tracked, immediately triggering appropriate preventive maintenance at a fraction of the previous cost.
“Big Data Analytics is only as good as the quality of your data sources and IoT style monitoring provides us with quality data.
It lets us know exactly how things are being used, how they are performing and what the right action is that we need to take when necessary.
TAI SOLUTIONS helped us put a monitoring system in place that guarantees we are looking at quality data as a foundation for our analytics”
IT Department Publiacqua
Looking to the future with confidence
We cannot imagine our lives without a reliable high quality water supply, and that very notion serves as the foundation of our promise to the citizens and businesses of Florence. Our predictive water supply management system will protect our promise from any harmful torrential rain, deluges and storms in the future. We are confident in building and safeguarding the path forward, holding our duty and our customers interests sacred.
“We put tremendous value on the power of technology and innovation, always raising the bar in search of a new gold standard. We believe that good solutions should be measured by their ability to make breakthroughs, doing what once seemed impossible. Our new predictive water supply management system is a perfect example of this”
Technology & Innovation Publiacqua
“The people at TAI SOLUTIONS helped us build a solution that is not just transformational to our business but more importantly helps protect our customers in times of adverse weather emergencies”
IT Operations Publiacqua
“Open source solutions bring unprecedented levels of flexibility and speed to the table that you don’t get with a propriety style approach to problem solving. Our move to a predictive model to keep our water supply services running in times of crisis can only stay up-to-date in a rapidly changing world by investing in an open source approach”