Agile BI and Analytics

I've worked in several organizations that had a reporting team in the IT organization.  Typically report developers work with the DBA team to respond to requests from business units for custom report development and enhancements to existing reports.  They have some SQL skills, skills using the reporting tools, and some business knowledge.  Like so most others in the IT org, they have too much to do and not much process or structure around how requests to them are prioritized, how their work is delivered, and success criteria.

Frustration on the part of business users often leads to everyone having his or her own private data store in the form of Excel spreadsheets or Access databases.

In one company, the COO had 4 years of month-end data in one spreadsheet file on her local hard drive.  In another large, multi-billion dollar company the CFO presented monthly financial reports to his leadership team consisting of bar and line graphs on powerpoint slides that he manipulated to represent data to fit his narrative of results

Adding to the pressure on reporting organizations lately is the plethora of SaaS data visualization and reporting/analytics tools available to end users.  Now, not only do people have their own local data stores, they are able to present information with more sophisticated visualizations that infer validity and analytical rigor.

The answer for IT organizations trying to establish data governance and security while meeting the BI and analytics needs of the organization is a more Agile/Lean approach.

Traditional reporting process has all the attributes of a failed waterfall software development process: up-front design; heavy reliance on technical documentation; and separation of the end user from the solution provider.  Outcomes typically reflect that process: missed expectation; disappointed and frustrated end users and solution providers; rework; and long lead times.

An Agile/Lean approach to BI applies principles and practices that have proven effective in software development, including:

• frequent face-to-face interaction between end users and solution developers
• regular delivery of finished work in short cycles
• prioritizaiton of work according to business value
• within and between delivery cycles, review of the work backlog to re-assess business value and prioritize requests
• visibility to all stakeholders of the work backlog, work in progress, and latest completed work

BI project success in my experience begins with some known legacy report, for example, a month-end financial statement or a weekly operations status report.  The business value of the report is established because it is already integrated in the weekly or monthly business cycle.  A project to migrate the report to a new platform, out of a legacy Excel-based system for example, provides business value in terms of:

1. Reduced time to deliver.  In one example I saw an accounting team reduce the number of hours required to deliver their month-end account statement from 40 to 4, a 90% reduction in time to complete.
2. Improved accuracy.  Any report that relies on manual data manipulation (copy, paste, formulas embedded in cells, etc.) is bound to introduce inadvertent errors.  In the month-end statement of accounts example, errors had been introduced to cells in the spreadsheet and had gone undetected for over a year.  The resulting erroneous results had been perpetuated that entire time.  Correcting that data and putting security and controls around it radically improved data accuracy and ensured accuracy from that point forward.

I've seen a BI organization use swimlanes for each part of the organization, allocate a portion of resources to each lane, and allow those BUs to prioritize their own backlog of requests.  Again, visibility, limited WIP per lane, and small enough sized work items to allow frequent delivery lead to more satisfied BU customers.