Agile BI : A Scrum Adaptation (Part 1 & 2)

Agile BI : A Scrum Adaptation (Part 1 & 2)

Agile BI : a SCRUM adaptation (part 1)

At the end of the year, it is always nice to look back at how we progressed and evolved within the projects of the last year. This year personally, I learned a lot concerning project and team management, which I’m happy to share with you. In the following article, I will focus on 2 different aspects and how we handled them at my current client:

• How we converted business demands in workable tasks
• How we plan and execute BI developments in an agile way

Within the BI team, we have opted for a SCRUM-based approach. We are not adhering to the strict rules of SCRUM, but have taken the aspect we could implement within our current team and organizational structure. The organizational structure itself is not very agile, so we needed to improvise on certain aspects and compromise also a bit on the strict adherence to the methodology.

CONVERT BUSINESS DEMAND IN WORKABLE TASKS

At the client, business demands reached our team in a lot of different ways: a quick discussion in the hallway, a mail sent to one of the team members, a formal request from management, … The most challenging issue with these multiple channels, is that you want to avoid not processing the demands or forget to communicate the progress or next steps to your client.

In order to keep track of these requirements, we created a general mailbox for our BI team and put into place a “BI Demands” Trello board where everybody can register demands and provide updates on it. Trello was our choice of ticketing system, as it is very lightweight and easily adapted to our still evolving process.

When a demand is received by any channel, a new ticket is created in Trello in an “Inbox” list to make sure that all demands are registered. At the creation, we add a number of attributes on which we can filter:

• The domain from which the demand originated (FI, MM, PM, …)
• From which domain the information is requested (for example, a request from FI can contain data from HR)
• Which BI domain takes the lead
• Who was the original requestor
• …

So no actual work is performed yet, we only communicate that the demand has been registered and that we will take it up when possible. This acknowledges the business that they have been heard and that we will plan the next steps when the team has time to take it up.

Depending on the impact and the projected work, it is decided which flow should be followed. If it is a small bug to resolve, the bug can be planned directly in the current or next sprint. If it will take up more time, the project passes through a predefined flow. The flow, as defined below, is not yet fully implemented, but this is where I envision we are working towards.

REQUIREMENTS MEETING

During an initial requirements meeting, we discuss:

• What is the goal of the report or data mart
• Who is the owner of the data, both for the original data and the resulting data mart
• The connections to the source data (we can start from exports, but need original data to start a project)
• What is the requested deadline for the business
• What are the acceptance criteria for the developments. This is an important part, as we need to know when the business will consider the development as finished.

ANALYSIS

After the requirements meeting, an analysis is performed to determine:

• The stories in which the demand can be split, with acceptance criteria per story
• Per story, what will be the deliverable to the business
• Each story is split into a number of steps, which can be translated into tickets for the sprint

After analysis, it is clear whether a project can be taken up and planned or not.

PRIORITY MEETING

As a BI team, we do not unilaterally decide whether a demand is given priority or not. This decision is taken with all the stakeholders (business, sponsor and team). After the priority meeting, it is clear for the team which projects can be included in the next sprint. This priority meeting is planned once every month, about 3 to 5 days before the start of the next sprint to allow time to properly plan.

The result of this process is that the team can perform the sprint planning without involvement of the business and that the work can be balanced within the team.

CONCLUSION (PART 1)

Even though it seems a rigid process to follow, it has provided a certain flexibility and peace within our BI team in the last few months.

At the end of the analysis phase, it is clear for the business which project can be taken up and clear for the development team what the next steps are for that project. The business is also made responsible to be clear in the decision which projects receive priority and have the power to influence the decision in a funded and open way.

In the next part, we will talk about how we structured our team and how we planned the tasks within the sprint and did the follow-up. The workflow presented in this first part is not something that was put into place from the start, but it also grew together with the way we handle the sprints, which you will read about in the next article. More to come soon!

Agile BI: a SCRUM adaptation (Part 2)

ANALYSIS

In the first part of this article, we briefly touched on the subject of Analysis of the demands which were registered. After the initial requirements meeting, we have a good overview of what is requested by the customer and what is can be included in the development for that project. Now it is up to 1 or more members of the BI team to analyse the work before creating the tickets for the development.

During the analysis phase, we make sure we have the connections we need to the data and analyse the sources which need to be used in the development. This means:

• Technical connections are set up
• Data / Risk owner of the source data is aware of the demand and has agreed to provide access. This is formalized in a risk/data owner document and also provided to the authorization team for audit purposes.

Based on the discussions during the requirements meeting, a number of stories can be created, at the end of which a deliverable is provided to the customer. These stories are then divided into a number of tickets, following a workflow we established during the development of these processes:

• Ingestion:
  • Setting up connection to source (creation of technical users, setting up authorizations, …)
  • Setting up SAP BODS jobs or SDA connection to load the data if required
  • If necessary, set up Python scripts for non-DB sources
• Modelization:
  • Creating the HANA models, based on a layer system which was developed in the past (as described in a previous blog https://cubis.be/sap-hana-views-organised-in-layers/?utm_source=social&utm_medium=linkedin&utm_campaign=tim )
  • Models are created based on the dimensional model methodology described in data warehouse techniques from Kimball
  • Tickets can be specified per dimension
  • Different data sources are combined within the model
• Analysis (Data mart)
  • When requested, we create the dimensions and facts in a package which is made available to the customer for self-service
  • And if customer needs a prefabricated view with the star join, this is also created. This results in a data mart view for the users to be consumed.
• Reporting
  • If the customer requires a report, we also add tickets for the creation of a new Power BI report.
  • Tickets are added to request a new Power BI workspace, to which the report can be published
• Authorization
  • We create tickets to make sure we request authorization roles for the client to access the data
  • Also, a ticket is foreseen to make sure the risk owner document (for the resulting data views) is filled in and sent to the authorization team
• Transport
  • Tickets are created to keep track of all the transports which need to be done to production in order to make the new development available

All these tickets are created in the “Product Backlog” Trello board and are linked to the correct story and project. Also, all tickets are estimated in number of mandays. This results in a clearly defined workload and structure, which can be planned in the next sprint.

SPRINTS

At the customer, sprints were organized on a 2-weekly basis. When starting of the sprint, as a result of the priority meeting, there is a clear view of what projects need to be taken up first and following the analysis phase, the work is clearly defined.

SPRINT PLANNING

During the sprint planning, the availability of each member is defined for the next 2 weeks. We do not plan all the days of the sprint however. There are 10 workdays in a sprint, but we limit the available time to 9 days, as there needs to be time for sprint planning, sprint review, meetings and administration. Of those 9 days, we only plan about 80% of the time of each developer, as we need a buffer for calamities or other changes in priority.

When planning the work, the tickets are moved from the “Product Backlog” board to the “Sprint backlog” board, this until all the developers are planned for 80% of the 9 days in the next sprint.

In principle, you can only have 1 ticket in “Ongoing” at a time, but a label has been added to indicate if a developer is “Waiting for feedback” or if the ticket is “On Hold”. This allows the developer to not be blocked during the sprint.

There is also the possibility to add tickets directly in the “To Do” or “Ongoing” lists during the sprint, if there is for example a bug discovered which needs immediate response. These tickets are labelled with “Not foreseen in sprint” in order to keep track and analyse the time spent on them during the sprint review.

The sprint planning meeting usually takes up about 1 hour of time, if all tickets in the product backlog have been well defined. As an additional way to communicate the sprint planning to the business and management, we also create a small mind map to show for each team member what the goals are during the next sprint.

 

 

DELIVERING DURING SPRINT

When we deliver a story during the sprint, we organize a demo with all parties involved in the development. This is only done when we have the following documentation available on our solution site:

• A detailed ERD diagram of all views made available to the customer (even if they don’t have direct access, we explain what was created)
• A data dictionary (at that time still in Excel format) which is used as input in metadata views in HANA. This is a listing per attribute and measure in the available views, showing
  • Corresponding data in source
  • Resulting name/type/… in the view
  • Any comments we have

The solution site is a Sharepoint site of the team, where all projects have their proper Document Set. A second requirement is that all the developments must be present in production and the correct authorizations have to be set up before organizing the handoff meeting.

During the handoff meeting of the story, we include:

• The requestor of the demand
• The data/risk owner of the source data
• The data/risk owner of the resulting work, which is not necessarily the same as the requestor
• The BI team members responsible for the development
• The authorization team

The goal at the end of this meeting is that all developments for that story are completed and everybody is aligned on the result.

SPRINT REVIEW AND RETROSPECTIVE

At the end of each sprint, a review meeting is held to analyse how the sprint went in terms of completion of tickets. All tickets which are not yet finished and are in status “Ongoing” are moved back to the “To Do” list, to be picked up in the next sprint. This means also that the estimated mandays on these tickets can be altered to reflect the correct amount of work still needed.

The tickets in the “Done” list are archived, but as Trello keeps track of all these tickets the information is never lost. We can always check back on the feedback and comments in those tickets at a later time.

We also take some time to do a small retrospective in order to analyse what were the pitfalls / issues during the last sprint and how we can improve the process even further. This is usually a very open and casual discussion.

The sprint review and retrospective normally take up no more than 1 hour of time.

CONCLUSION (PART 2)

Thanks to the work already performed during the analysis phase, it is clear for everybody what needs to be done to deliver a story and how much work can be taken up in the next sprint. Also, because we have estimations of availability and work on the tickets, the planning is fairly straight forward in terms of workload.

We had until now not yet taken up the time to analyse the velocity of the sprints, as the estimations on the tickets were done in mandays. It could however be interesting to add Story Points to the tickets in the backlog, in order to do a retrospective analysis of the velocity during the sprint.

In general, the way we planned the print provided us with a clear definition of what was needed in the next few weeks and the whole team felt comfortable with the amount of work which fell on their shoulders. It even gave us some time to do some non-project related work to further optimize how we handle the overall teamwork and deliverables like the data dictionary, ERD and governance Power BI reports.

We genuinely believe that this methodology gave our team a new motivation and resulted in a major improvement in the quality of developments and deliverables. The work was more structured, even if it was an adaptation of an agile methodology!

Blog By Dries Vlaeminck