It’s clear that we are living in a data-driven world. Our steady transition toward highly digitalized lives is making data a key asset in the modern economy. When we go online to make purchases, consume content, or share on social media, we are generating valuable data. Many of the largest tech companies are now operating on business models that depend on leveraging data. However none of that is possible without data integration. Data integration is the glue that makes it possible to convert raw data into a valuable asset.

Data integration can be defined as the process of acquiring data from diverse sources and transforming it into a data store or business application so that it can be used more effectively.

Countless business problems are caused by a lack of data integration. Fragmented data silos across companies or departments within companies make it necessary for users to rekey data or duplicate their efforts. Lack of unified data views can make decision making difficult. When individuals or departments have limited visibility to data, they often make decisions that fail to recognize the larger process, and they sub-optimize based on their limited view. Poor data integration causes a large amount of waste in business caused by inefficiencies and poor decision making.

Data integration techniques help to reduce these problems. Data integration techniques can be grouped into three categories:

• Business-to-Business (B2B) integration includes integrations that span across organizational boundaries so that trading partners can more efficiently execute business transactions.

• Application Integration is more specifically targeted at connecting separate business applications into an integrated workflow.

• Database Integration operates at the lowest level which is the data stores themselves. This category involves building pipelines that move raw data from data store to another. This is the type of integration used when building data warehouses and business intelligence solutions.

All three categories of integration are used quite frequently in the modern workplace and are extremely valuable to understand. Successful use of these techniques can make a significant impact on a businesses.

Let’s take a look at the some of the ways data integration can improve business results.

Software Integration = Business Efficiency

In a software-driven world, we all interact with a large number of software applications in the workplace. These applications are purchased with the goal of improving business results. A CRM system aspires to grow an organization’s sales. An ERP system aims to increase productivity and to accurately capture financial results. In recent years we have seen a proliferation of new software delivered with the software-as-a-service model (SaaS). These are usually web applications provided on a subscription basis. There are SaaS applications that target every conceivable niche in every industry. If you have a business problem, there is probably a SaaS application that was designed to solve it.

The challenge becomes what some call “SaaS sprawl”. An organization finds themselves using countless niche software tools for specific business problems. Just in the marketing department, there could be five or more software applications involved with marketing the organization’s products and managing leads. Those applications could include a customer relationship management app, social media marketing tools, a site content management system, and cloud email system, etc. Although it might be possible to achieve many of those functions on one big application like a cloud CRM, that tends not to happen in most organizations. Individual managers like to use the tools they are familiar with that are a good fit for their company’s size, culture and industry. Unfortunately this leads to complexity and data silos.

Let’s say your marketing team has developed a blog to capture leads. Think of all the cloud software that could be involved in that process:

• WordPress to host the blog.
• Spout Social to market the blog on social media.
• Facebook Ads to drive paid traffic to the blog article.
• Mailchimp to capture email list signups.
• Google Analytics to track user activity on the blog and measure the effectiveness of the marketing channels
• Unbounce to capture leads
• Salesforce to manage and cultivate leads
• Twilio to notify the sales team whenever a new lead is captured.

I can stop there at eight applications but there are far more niche tools that could be introduced depending on the industry and size of the marketing department. Maybe salespeople want leads to schedule appointments with a tool like Calendly. There’s probably a separate cloud email system like Office 365. Perhaps marketing content like product sheets or presentations are stored in Dropbox. It goes on and on. Is it theoretically possible to do all of this on a big application like Salesforce or SAP? Maybe, but in reality, it almost never happens.

So there is no lack of great software capabilities out there that can add a great amount of value to organizations. Then what’s the problem? Well there’s a few…

• How do we make all these software apps work together?
• How do you get your customer emails from the Salesforce into Mailchimp?
• How do you get unsubscribes from Mailchimp back into Salesforce?
• How do you get Facebook ads to craft URLs in a way that allow Google Analytics to associate traffic or a conversion back to ad spend?
• How do you get leads from LeadPages into Salesforce?
• How can you assemble these eight different software capabilities into a seamless business process?

Data integration is the answer. Data integration techniques enable data flows between applications and across organizations. Without proper data integration, each business application operates in isolation.

Automate by Integrating

When a company’s business systems are isolated, human labor is required to key data into all these applications. This usually keying data into each app’s user interface or possibly some kind of Excel upload. In most companies, human labor is the most expensive business cost. Any kind of non-value adding human labor should be automated whenever possible. Data integration is often a key component to reducing manual labor. When you consider any kind of business process, it will usually consist of a data flow involving a sequence of data inputs and outputs.

Connecting these inputs and outputs is not always a simple task and can require various data integration techniques. The format and shape of the data for one app will be different from another. Data transformation techniques will need to be applied. Given most apps are now operating on the cloud, tools or programming will be needed to transmit the data to business partners or to call web services. Sometimes the data that comes out of one system has errors which need to be corrected before passing the data to the next system in the workflow. Data cleansing techniques will be necessary. Effective automation of business processes can sometimes require a lot of data engineering to get right. However, these types of projects can have a big impact on the business by reducing costs, improving quality and speeding execution time.

Digitization Requires Data Integration

Recently many companies have been investing in digitization strategies. Digitization involves leveraging software and data to become more competitive and profitable. In the midst of this trend you will often hear CEOs in industries like services, consumer products or manufacturing call their organizations “software companies”. But often the main value from a digitization strategy comes from data. Data is used to operate more efficiently, make better decisions, or even to create new products and services. Often machine learning and artificial intelligence techniques are used to create even more value from the company’s data. Data becomes a key asset to cultivate and manage.

Most companies are trying to use data to make better decisions. As opposed to relying on the gut instinct of executives, data from internal and external data sources can help decision makers evaluate options objectively. Reports and dashboards are used to keep the management team updated on business performance. Often scorecards are used to evaluate individual employee and team performance. Metrics like financials, service response time, net promoter score, quality measurements, and so on are common to see on executive or individual contributor scorecards.

Often the data required to support digitization projects is spread across many different systems and databases. Data integration techniques are necessary to consolidate these disparate data sources into the shape and format these projects require. Perhaps a machine learning algorithm needs to combine sales data from the CRM with weather data from an 3rd party API. Or a dashboard that reports collection risk needs to combine sales data with the accounts receivable system. Data integration is a core practice that will almost always be necessary in these projects. Companies that are good at data integration will have a key competitive advantage in the today’s business environment.

Let’s say a manufacturer of farm equipment wants to use data collected by its tractors to help optimize crop yields. Perhaps sensors on their tractors can collect soil moisture measurements. The manufacturer could collect this data from all their tractors, combine it with external data sources like weather or commodities market prices. The end result might be recommendations to farmers about how to optimize their irrigation systems to optimize yields. Perhaps the data could even be sold to third parties like hedge funds to help them make investment decisions. This could become a new revenue source for the company that could supplement or some day even outperform the company’s original business model.

Create Value with Data Integration

There are so many opportunities to reduce costs and improve sales using data integration techniques. Why aren’t more companies focusing on this? Although many tech workers are focusing on software engineering as a career path, one could argue that a far more valuable skill than creating software is creating the glue that connects existing software into a cohesive process. Data integration just isn’t as sexy as building a web application or a mobile app. I hope to change that mentality. Data integration skills can be an effective way to get started in a tech career, or to find more success in non-technical jobs. Stay tuned for my next post about how data integration is a superpower for modern workers. In the meantime check out my Data Integration Fundamentals course on Udemy.

• DataIntegration

About the Author

John Berry has spent the last 30 years building software and data solutions for some of the world's most well-known supply chains. He believes supply chain and logistics are great career paths for those looking to establish technology careers. He is currently the IT Director for JUSDA Supply Chain Management, a member of the Foxconn Technology Group. In this role he leads a team that develops technology solutions for the global manufacturing supply chain. John is a contributor to the upcoming book The Digital Transformation of Logistics: Demystifying Impacts of the Fourth Industrial Revolution published by IEEE Press.

Want to learn how to use data integration techniques to optimize business results and supercharge your career? Enroll in John's Data Integration Fundamentals course on Udemy.