Common Barriers to Implementing ML and AI in Large Corporations

The thirst for rapid implementation of ML and AI has spawned the relatively new career of the data scientist. More recently, those with more specialized skills are classified as data engineers or machine learning engineers. The hiring of any of these data specialists (scientists or engineers) has ballooned in recent years. As forecasts predict a major deficit in qualified applicants, new data science degree programs are continually being developed around the world.

While the position of data or business analyst is well established, these employees have been primarily occupied with querying and reporting statistics. They are often responsible for producing reports that are ultimately assessed by managers who call the final shots in the decision-making process. Decisions are often based on these reports, or on some other metric or perception drawn from experience. It is the recent advances in computational power and statistical algorithms that have shifted the collective mindset to rely more on ML and AI-based decisions, thereby boosting the demand for data specialists.

Hiring data specialists is often the first step organizations take while seeking benefits from ML or AI. For many new start-ups, their original business models are already based on automation - such as delivery services, online matchmaking, entertainment, and streaming. Young companies can also relatively easily place data scientists and engineers in key operational areas and quickly benefit from automation, as many processes may still be in a state of flux while the fledgling organization grows. However, some of the larger and well-established corporations are, to their dismay, only sluggishly transforming their working methods to fully benefit from automation, even after the mandate to transform the organization has led to the hiring of data specialists.

There are at least two main barriers preventing established organizations from smoothly incorporating ML and AI, and thereafter consistently benefiting from automation:

1. A ‘don’t fix it if it ain’t broke’ mindset:

           One major barrier within large businesses is the inertia in modifying established workflows that have already produced revenue streams. A ‘don't fix it if it ain't broke’ mindset hinders many decision makers from seeking or embracing alternative workflows, thereby stunting the progress towards full automation. There may be a company-wide mandate to transform itself digitally. However, on the department or team level, the goal of optimization or digitalization may be poorly defined, thereby severely limiting the impact of the hired data specialists.

To maximize the impact of data specialists, they will need to be able to monitor and eventually modify the current working processes. These steps usually require the green light from other teams within the organization, such as IT security or accounting. Without clear instructions from upper management, the initiative to modify workflows spanning multiple departments must be taken by the teams themselves.

Team leaders may be hesitant to begin cross-functional projects, especially if the company-wide goal of optimization is ambiguous on the department level, and current processes already generate profits. This lack of clarity will limit the potential impact of the data specialist, and preserve the status quo.

       2. Diverting Data Specialists for Quick Wins:

           The newly hired data specialists do have the knowledge to transform current working processes - those that might predominantly rely on human choices - towards a system where ML and AI identify optimal decisions. That transformation, however, is not as simple as merely flipping a switch. It often requires extracting relevant information from one or more databases, exploring algorithms for modeling the data, tuning the model parameters for identifying an optimal solution, and then productionizing those steps so that the optimization is automized.

Furthermore, a robust system should be able to account for any foreseeable changes in all stages of the data pipeline. Perhaps new information will become available at a later time, such as new columns in originally structured data. Additionally, the data engineer might build-in an automatic A/B tester, meaning new options and ideas are continuously tested. Depending on the complexity of the pre-existing system in place, constructing and incorporating such a robust pipeline takes time. It is imperative that managers provide the data specialist ample time to explore, build, refine, and monitor the data pipeline. The managers and other colleagues should resist the temptation to divert the new data specialists’ efforts to functions that may provide easy and quick successes on the short term, but hinder the overall goal of transformation towards a more data-driven company.

           A common example of a quick win involves the requests for batch data. The data specialist might be able to efficiently gather data to answer a long-standing question, one that existing data analysts may not have access to. One of the early steps on the way to process automation usually involves querying data. The data specialist may use existing tools to accomplish this, such as a graphical interface available as part of a dashboarding system, or write whole new queries programmatically. Their chosen solution could already provide a more efficient method for obtaining previously hard to get, or even unnoticed, data. After analysts and managers become aware of this, it is only natural to request other data pulls. This data delivery clearly allows colleagues to progress towards another goal, and may generally shed light on valuable information for the company, but at the expense of the high-level goal of automating or optimizing certain aspects of the business.

           Querying data is just one example of a short-term win that a data specialist might provide. Other examples include transforming, aggregating and combining data (perhaps as part of a larger query), or automating data entry into public domains or standardized forms. Requesting any of these stymies the overall effort of digital transformation.

Reaping tangible benefits from ML and AI needs well defined objectives and time

Within larger organizations, achieving the goal of optimization and automation (be it for operational logistics, relationship management with customers, decision making processes, or automation of workflows) requires an investment. When a company decides to transform one or more of its processes to benefit from ML and AI, the investment must not only involve a financial commitment but also just as importantly, time.

Industry efforts to optimize business processes often begin with the hiring of data specialists. Without well-defined company-wide goals, the data specialist may face resistance from inertia and the reluctance to altering pre-existing workflows that have a track record of generating revenue. Furthermore, as the new hires begin their work, their skills might be used for other areas. And those areas do not fall under the original mandate of digital transformation. Such sidetracks will only preserve the status quo and stunt progress towards digitization.

It is therefore imperative that any automation mandate includes well-defined goals on the department levels, and that team leaders give data specialists time to perform their work. These efforts to incorporate AI and ML into key business areas will gradually transform the organization, advancing it towards the overarching goal of digital optimization and automation.

About the author Rahul Shetty

I am a Data Scientist for Qualogy.

More posts by Rahul Shetty