Data Management

Both data warehouses and data lakes are used for managing analytics data.  Enterprise data warehouses is a more traditional approach, based on a relational or columnar database storing structured data for advanced querying and analytics.  A data warehouse is a repository of data that is already structured and filtered and has been processed for a specific purpose.

Data lakes, on the other hand, is a large set of raw data, which does not yet have a defined purpose.  It is essentially a highly scalable storage repository that holds large volumes of raw data in its native format until it is required for use.  Although most commonly built on Hadoop clusters, data lakes are also done on cloud object storage, or is a combination of different platforms in a distributed data lake environment.

Data integration is the technical and business processes used to combine data from different sources into a unified, single view of the data.  The traditional data integration technique is extract, transform and load (ETL), a batch integration process that run at scheduled intervals. However, data integration platforms now also support a variety of other integration methods.

An alternative data management approach is to do real-time data integration, using methods such as change data capture, and streaming data integration, which integrates streams of real-time data on a continuous basis.  We work with technology like Qlik Data Integration Platform (formerly Attunity) to enable real-time data integration.

Data virtualization uses an abstraction layer to create a virtual logical view of data from disparate systems.  It is a integration option that avoids having to create a whole new integrated physical data platform.  Data virtualization allows you to retrieve and manipulate data while shielding you from the technical details of the data, such as how it is formatted at source, or where it is physically located.  It also unified data for centralized security and governance, and delivers it to business users in real time.

Data virtualization can efficiently integrate and bridge data across data warehouses, data marts, and data lakes without extensive ETL and data storage.  We work with data virtualization technology like Denodo, and DWaaS Snowflake to enable your data virtualization journey.

Data modeling is a part of data management, it is the process of designing and formalising relational rules between both data, and data to business requirements, and building business-useful definitions for the data.  This creates a series of conceptual, logical and physical data models that document data sets and workflows and map them for processing and analytics.

Data modeling imposes upon data a structure that increases consistency in naming, rules, semantics, and business definition of data, while also improving the suitability of data for analytics.

Data quality management is the process for improving the data that’s used for analysis and decision, based on metrics such as accuracy, completeness, consistency, integrity and timeliness.  Data quality improvements leads to better analytics results and decision-making across your organization. The better the quality of data you have, the more confidence you can have in your decisions.

Data quality techniques include data profiling, which scans data sets to identify trends that help in discovering, understanding and exposing inconsistencies in the data, for any corrections and adjustments; data cleansing, also known as data scrubbing, which detects and rectify untrustworthy, inaccurate or outdated data; and data validation, which checks data against preset data standardization and quality rules.