To be able to draw the right analysis from data to make strategic, tactical and operational business decisions, a business must have a result-based framework. Data analysis can be done using data from OLAP reports or scorecards.

Operational insights can be obtained using BI/Big Data analytics/predictive analysis/mining models. These insights are critical for business decision-making and may have far-reaching implications on the business’s outcomes.

In a typical software industry, BI/Big Data is often used in conjunction with a waterfall or an iteration model. These models are structured in a hierarchy that starts with analysis and moves on to design, development, and finally deployment. The problem is that most businesses do not consider the risks, costs, and time involved until the end. Businesses must wait until the end of projects to make any necessary corrections. To achieve a higher success rate in BI projects, Agile methodology is necessary.

Related post – 10 Agile certifications to take your career to the next level

BIG DATA’S BIG PROBLEM and THE AGILE SOLUTION

Big data analytics can detect patterns that are difficult to find using traditional analytics tools. Big data analytics is a widely used tool that has led to breakthroughs in many areas, including medical diagnostics, people management and how organizations respond to consumers’ behavior.

Data scientists begin big data projects by developing a theory about a business problem. For example, they might predict the demand for a vehicle model based on past sales and new features. Or how to determine the number of employees that an organization should hire in order to staff a new venture based on existing staffing levels and personnel from similar projects. The algorithms are then tested using a variety of techniques, such as machine learning, optimization or traditional statistics. They may keep refining the theory until they arrive at a valid conclusion if it is proven false. They may decide to abandon the theory and focus on a more urgent business problem.

Big data analytics can produce amazing results. However, we have seen many failures in big data analytics rollouts, especially when they are attempted on a large scale. This is what we learned from observing companies. Sometimes, when problems occur, it is possible for deliverables like any anticipated insights or process improvements to not materialize. Most often, the problem is not with the data, but with the processes used to verify, process and act upon it. (See ” How To Avoid the Big Bad Data Trap“, BCG article June 2015).

The problem lies in the way big data analytics are built. Many are constructed sequentially using the waterfall method of project management, which is commonly used in software development. Data scientists use the waterfall method to acquire, verify and integrate data, develop a model or algorithm for testing it, then act on the results or refine the model. The task you are working on is completed before the next task. This is inefficient. Many times, people spend more time in meetings and managing handoffs that they spend on data-related activities. Sometimes the work is misdirected or wasteful. It is not easy for lay executives to comprehend the final product and it is often delayed.

BIG DATA PROJECTS – USING AGILE

Agile is proving to be a great tool in many other areas than software. These include marketing and consumer goods. (See BCG article ” Agile to Save Retail” October 2018 and ” Taking Agile Transforms Beyond the Tipping point August 2018). We have seen that agile has helped clients to empower their teams and ensure they are in line with the organization’s strategic goals. We believe agile could have many benefits for big data projects, based on these successes.

Rapid Experimentation. In the past, testing happens near the end big data projects. This means that business executives may not see results until then. A team developing a predictive analytics model that will help salespeople convert leads may wait until the end of testing before showing executives the results. But, if results are not presented in a timely manner, it could result in unclear expectations, methods used, and possible outcomes. Agile allows projects to be broken down into smaller chunks that can easily be built and tested quickly. Continuously, teams develop and test MVPs. Business executives can immediately find out if the data analytics are not producing the expected results and make the necessary adjustments to correct the situation. Business executives can ask their team to analyze other data or make modifications. In some cases, they may even abandon the project. These are all steps that will save money and time.

This idea of rapid experimentation was taken to heart by a specialty retailer when it formed an agile team consisting of people from data engineering, marketing, and data science. The team was tasked with creating innovative big data sales and marketing solutions. They used agile sprints to produce incrementally new omnichannel programs every seven days. These new programs result in immediate revenue increases within a short time.

Rapid experimentation is useful for both big data analytics algorithms and the data on which a project is built. It also helps to ensure that the organization understands and can act on the results. A big data project MVP also includes the output and algorithm, but also the business and behavioral changes required to achieve real results. A new process could be developed to notify customers about upcoming appointments. For example, an algorithm that improves dispatch and scheduling for service technicians might also be developed.

Early Customer Feedback. Big data projects are not designed to create mathematical models, but to solve business problems or uncover insights that can lead to benefits for customers. Customers should be included in the project. A representative of the client might be included in the team if the project is for an outside client. A member of the department that is involved in a big-data project for an internal client might also be part of a team. A European oil refinery developed a big-data application that allowed its engineers to optimize their maintenance cycle. For instance, the team that created the app was assigned process and maintenance engineers.

Prioritizing value. Tasks that add value and don’t increase cost are more important than completing them in a set order. The product owner, who is the person responsible for representing customers, can take into consideration the fact that a feature may take longer than expected and not provide a lot more value. The product owner can decide to drop the feature and move on to more valuable or lower-cost items in the project backlog.

Cross-Functionality. Big data projects that are not based on data analysis often fail. Our experience with clients shows that 70% of cross-functional teams’ efforts go beyond data analysis and into business processes, operational behavior, and other types of decision-making. Big data project teams often include people from a wide range of backgrounds to accommodate this scope. See Exhibit 2. (See Exhibit 2.) The team can take decisions without the approval of their bosses. Teams work together in real-time to resolve conflicts, tradeoffs, and compromises. This is why it is important for them all to be in the same place, preferably in the exact same room. While they work on algorithms and data, the teams might also be working to modify operating models or business processes.

This is how the European oil refinery described above used agile methods of working when it integrated them into three large data projects that were part of a larger digital transformation effort. Each project had its own scrum team, with a scrum master and product owner. Other key functions included operations, technology development and asset maintenance. The teams also included data scientists from an external advanced-analytics consultancy firm. With the agile approach and teams, the refinery was able to produce multiple MVPs in a short time span. It also produced a final product much faster than what it could have achieved with its normal product development cycle. Multidisciplinary scrum teams were also a key contributor to a more collaborative corporate culture that greatly increased employee engagement.

People empowerment. A project manager in a traditional big-data project decides what priorities are most important and how these priorities will be met, even though they may not fully understand the development process. People become more involved in their work and more invested in the final outcome when that power is delegated. Agile team members, unlike data scientists who may work on multiple engagements simultaneously, are not split up. Instead, they give all of their time to the team and are more invested in the work. This single focus builds accountability.

It is no surprise that agile companies are more successful in attracting younger workers and digital talent than other companies. These two groups of people prioritize work that gives meaning to their lives.

The SECRETS TO USING AGILE IN BIG DOUBLE DATA

Although agile and big data sound like the perfect combination, it isn’t as simple as it seems. Keep these key factors in mind if you want to be successful.

An algorithm is not a finished product. If it fails to deliver results-oriented output in an understandable way for a business department, or other organization, it can be either beautiful or wasteful. The results of agile teams’ findings should be easily communicated through infographics and dashboards. They should also include in their work the operational and behavioral changes required to achieve real results.

MVPs differ from prototypes. They are usually built using historical data to confirm that an algorithm is capable of doing what it’s supposed to. A prototype is used to create a larger MVP that can be distributed to users if it works. An MVP includes current data, a user-friendly interface and core features. It also contains operating instructions. It must work within a business context and may include changes to operating models or processes.

All stakeholders must be open to the possibility of mistakes. Agile development is iterative. It’s possible for works in progress not to look as good or perform as well. They may be indicative of progress towards a satisfactory solution, despite their imperfections. Stakeholders, who in the past saw only partial versions, may have to accept this imperfection. This could require them to change how they view big data analytics projects. Stakeholders can increase the chances that a project will go from MVP to full scale production by encouraging trial and error.

Not just data scientists. Teams should be made up of a variety of talents, not based on past experience or a standard structure. It is a good idea to have a team that includes data engineers who can prepare data, data scientists who conduct analysis, designers who can present data, and business personnel who are familiar both with the business objectives and the implications of existing processes. It is important to combine people’s talents in a way that makes the whole more than the sum of its parts.

Agile for BigData Projects

The general consensus in the software industry is that BI/Big Data works well with either a waterfall or an iteration model. The industry’s first choice for BI is the non-agile model, due to its success rate (which is lower than the failure ratio). The components needed to complete BI projects are most often already in place.

* ETL packages/mappings

* Data model

* Data Quality

* Reports/Dashboards

The project starts with an analysis of all components, ETL strategy, design and testing of the Data model, and then the development of reports. The process of analyzing, testing and deploying these components can take many months before the business can experience the results and benefits. The following will be the result:

* Once the design has been tested, it is difficult to change it.

* Users cannot see any product that is in use until the end of the day

* Some risks are unknown beyond the last day

* Further delay could impact downstream applications

How can we avoid these problems and make sure that the BI project goes smoothly? Agile is the answer. You may be wondering how agile can you deliver a BI project? This model is more risky than the Iterative or waterfall models. How can one make sure that the BI project is a success? Without a plan, Agile can be even more dangerous than other models. These paragraphs will outline the key to success when using agile for BI projects. This will lead to a wonderful success story.

We will use the insurance domain and its related modules as an example. Each project starts with specific business requirements and related topics to address the problem/pain area.

An insurance policy is a contract between an insurer and policyholder that covers liability for various types of risks. It protects property and assets against damage or loss in the event that they are damaged by fire, theft, and other natural disasters. Many business processes define the business’s nature. Policies, renewal, underwriting, claims and claims are just a few of the important processes. This is the most important question any insurance company should ask:

* How can one analyze the data?

* How much granularity is allowed for the analysis?

* Does one have any analytics to assist in setting premium pricing?

* How can one distinguish between risky and non-risky properties?

Analytics is used to answer some of these questions. Data is spread across many systems in different structured and unstructured formats. Insurance companies face a challenge in transforming these data sets into meaningful insights. These tools and technologies will help firms find the right insights and make key decisions. These data sets can be continuously accessed through Big Data, which allows for the resolution of key data issues, modeling and analysis, as well as continuous data updates.

Modularity in the Agile Concept

Any BI/Big Data program that can be implemented Agile can be adapted to modularity and functionality. Businesses can then reap the benefits of agile. The basic principle of modularity is the foundation of Agile. The backlogs/requirements are collected and collated in a highly structured way.

A backlog, which is where all user stories are stored, is an important part of Agile. These user stories (or backlog) are linked to various business processes. It takes a lot of time to interview business users. This includes brainstorming sessions, surveys, and one-on-one group discussions. Then verification of understanding is done. Once these business processes are identified and confirmed, they will be divided into modules/sub-modules according to their functionality. This is known as mapping. It helps to link modules and business processes. This is an essential component of agile projects. The success of any project will depend on how modular and functional the structure is. This will allow the team to manage any changes at any stage of the project and to efficiently manage the end result.

An agile team must identify module dependencies. It is important to determine the inter-module dependencies based on functionality. This will allow for a key mapping to help understand the relationship between different modules. This helps the agile team in many different ways.

* Knowledge of the impacts on different modules and the changes they bring

* Divide and Rule – Parallel development activity can take place where there are not dependencies between modules. This results in significant reductions in time.

* Seamless integration

* Easy adaptation when requirements change at any stage in the project

Once the modularity, constraints/or dependencies have been identified and baselined it will be possible to prioritize these modules so that both the agile team and the business can decide the order in which they should be developed and deployed.

Conclusion

Companies might be tempted just to jump in, as there are so many benefits to incorporating agile into big-data projects. However, to get the best results, it takes careful planning. This includes a thorough analysis of the requirements and how they would impact existing processes and personnel. A pilot is a great way to get started. A pilot is a good way to start.