Data Analytics and Data Lakes, Swamps and Yards

Interesting presentations and discussions at the #Databanking and #CDO event in NY the last two days . There was a variety of opinions around the practical implementations of managing big data infrastructures to support operational business needs as well as data exploration and mining. 

The common themes here were really about having clear business cases and well-defined requirements for the (big) data and associated infrastructure needed by important business functions and user personas, while still providing the appropriate data quality and access controls. 

Any of these initiatives and transformation programs present significant challenges as the banks need to balance i) regulations on data quality, collection, retention, usage, privacy and testing, ii) being innovative in a dynamic and competitive market place , iii) managing the limited capital, time and resources available. Robust change management, risk management and resource optimization are all needed to manage these large scale programs effectively. This is a clear validation for me of what we are doing at IDE-International around our PRIMED change and program management platform. 

There were some specific points that resonated with me I thought I would share:

Derek Strauss,  TD Ameritrade' CDO,  discussed how from his perspective data warehouses and data quality improvement are still needed, but the concept of “Data Lake” seems to be an unnecessary dumping ground for data (and company’s cash).  Instead he introduced the idea of a “Data Marshalling Yard”: an upstream hybrid environment that supports traditional data warehousing and data quality controls together  with a noSQL-based marts for staging, enterprise archiving, application access and exploratory analytics support.

This was interesting as when we look at the work we have done at the Smarter Cities Open Data initiative in Glasgow, providing and accessing a vast variety of disparate data to do exploratory data analytics did not require building a data lake. That allowed us to be agile, quick to develop business opportunities and be very efficient with limited resources. More information here: http://www.ide-international.com/customers/future-city-glasgow/

Credit Suisse have built their own “Semantically derived data lake” (compute / analytics environment and lake), but are very focused on ensuring that all requirements are bound to the actual needs of the business in terms of Market, Customer and Operational Intelligence and Risk+Reputation management -- thus avoiding a “Data Swamp” (!). Very sensible. They also provide context via semantic search and have built an interesting heterogeneous environment of industry technologies to provide a Rich Data Fabric (define, store and query data store that supports temporal views), Risk Calculation fabric (resolve and compute), Risk Data Algebra (distribute, aggregate, slice and dice data), and a Risk Data Portal (visualize and share the data). Finally, they have also built a Domain Specific Language  (DSL) to support transformation and mapping between data sources.

Over at Citibank,  Savy Sriram, Global Head, Retail Risk Data Strategy, discussed a pro-active, design-based approach to strategic end-to-end data planning that engages with stakeholders on a partner basis recognizing that data is a powerful and important asset across the firm. She noted that you must be able to decode meta data and data lineage and making that available to users is imperative to giving them context. Intelligence needs to be brought to data through machine learning, statistical methods and analytics to confirm data quality and the observations (output) from systems. Then, focus SMEs (people) efforts on final review and data attestation (an important topic for CFO, CIO, CEO governance!). So, take intelligence TO the data and institutionalize intelligence FROM the data.

Thanks to The Data Lab for letting me join them at the event, and for introducing me to some great people and promising Scottish companies. Shout out also to Scottish Development International for their ongoing support. Sláinte!!