Agile in Analytics – A Perspective!

I had promised in my earlier blog to have this topic covered separately. Over the last 12-13 years or so I have been coming across this in most of our implementations. Obviously, with the modern-day techniques of delivery & wide-spread adoption of Agile methodologies, the expectations of the business community have also evolved & Analytics initiatives have been no different. What is your perspective? There is no clear right or wrong answer when it comes to applying Agile methodology for Analytics, there could be perspectives, here is mine!

To a layman & to put this in simple way, Agile methodology is all about running in sprints than marathons, having a viable product delivered to the business every 4-5 weeks. Let’s not discuss the advantages etc., we all are aware of those. The daily scrums, stand-up meets are part & parcel of being in an Agile project execution & we have seen how exciting it goes as a team. Did you try applying this to Analytics? We did & there are some observations & obviously some learnings.

Over the years delivering Analytics we have realized that Analytics executions are more like a conveyor belt running thru a workshop, there are certain operations with a hard dependency on prior ones (sometimes a tangled way with the next one as well) & can be completed only in a serial manner. From a design point of view, it is always a top-down approach we follow & a bottom-up for execution. The kind of KPIs asked by business users, give input to the metadata & data marts design, which in turn drives the data engineering design. The design of these components & data engineering becomes a foundation layer of further Analytics stages, hence considered as very critical to the success of the whole initiative. As much as 65-70% of total efforts go in building this foundation, the right way! The dashboards or reports or ML use cases are the toppings over this foundation, which one can roll out as needed based on business priority.

Based on the use case in scope of analytics, the timeline of this foundation layer varies. Finalizing KPIs, personified metrics & the whole foundation layer depends on whom are we serving & what functions are getting served. For e.g., for a CFO’s desk which would have KPIs from the P&L elements, Balance Sheet Ratios & Cash Positions, mostly be dependent on Financials from ERP. The process of agreement over KPIs & metrics with the CFO, design of data marts & data engineering aspect to transform the data from ERP Financials into data marts takes about 4-6 weeks at a minimum.

Now, if you closely look at the use case above, nothing gets delivered to the business for 6+ weeks. Can we call this an Agile implementation? We can’t.

There are some tweaks needed to the original Agile methodology. As a scrum-master or a leader of the process, the key is to get those tweaks communicated to all stakeholders & have everyone on the same level of understanding. 

The 1^st & major tweak is having a longer Sprint 0 – The Foundation Phase, as we talked about. The duration of Sprint 0 would change based on the use case getting delivered & on the industry we are serving to. But our experience says, however complex use case you may have; Sprint 0 need not exceed 8 weeks. But acknowledging the importance of Foundation phase & calling it as Sprint 0 is the key in adopting Agile while delivering Analytics. All standard thumb rules of execution would then be applied to make Sprint 0 as efficient as possible, we need not discuss those here, that’s not the purpose of this blog.

The subsequent sprints would be roll out of reports & dashboards or ML use cases. Here as well, a few tweaks are needed. Analytics is more like a research work, evolves as we see it more or as we experience it more. We have observed the users getting creative once they start looking at initial outcomes. A typical ‘Show & Tell’ stage within Agile terms or a Conference Room Pilot (CRP) stage with users, as we may call it more often is another key aspect. This needs to be a part of Sprint 1 – One needs to reserve 1-2 weeks just to get ‘Show & Tell’ done & conclude with one more round after accommodating feedback. So, your Sprint 1 also needs to be a bit longer than remaining sprints. A lot of aspects come into picture like branding colors, layouts, type of charts & personification while concluding Sprint 1. They provide a skeleton to follow for remaining sprints.

The lieu of our customized approach with Sprint 0 & 1, the Sprint planning phase & associating User stories appropriately gets one going well with adopting Agile in Analytics. Getting a user buy in during the sprint planning phase with the customized nature of Sprint 0 & 1 helps with execution phases & sprints to follow. 

Other processes like Daily stand-up meetings, burndown charts, backlog management etc. would continue as-is regardless of Analytics or otherwise project.

With these tweaks to Agile methodology, the masters don’t consider it as an Agile anymore. For Analytics people like us, we call this as an iterative way if it is not approved by the Agile masters, the point is getting Analytics implemented the right way & we have been doing that regardless of Agile or iterative!

I am waiting to hear your perspective on this.