Selecting The Right Master Data Management Product

Kinshuk Dutta
Lisboa, Portugal.

Mastering the Right Data Management Solution

Choose- What's right for you... Not what you are lured with!!

We are often confused by our needs. Sometimes we do not have options and have to trust what we get as the best and some other times we are showered with so many offers that we are not sure whether what we are choosing is actually what we want. Brand names, attractive features, extravagant style often blinds our judgment and we end up with a rather overpriced superficially complex, and high-maintenance seeking product.

Typical MDM Project - Business Value realization

The goal of any MDM implementation is to realize business value. The following diagram shows how a typical MDM solution is implemented.

There are multiple challenges that need to be overcome in order to realize "Mastering the Right Data Management Solution".

First Challenge – Choosing the Right MDM Solution

So how do we choose?

A very innocent question can result in endless nights of a brainstorming session. There is no 1-Fit-All Tool. First of all, we should understand that each enterprise has a unique way of operating. Although we can broadly categorize different types of industries they all in their own way are unique. And that's the beauty as well as the sole reason for their existence. This means that they cannot have the same exact problems to address. 

On one hand, there are organizations who want to streamline the business operation to reduce fraudulent activities and hence keeps on introducing various norms and guidelines which an enterprise has to adhere to without any choice on the other hand there are some critical business-related and revenue-generating operations which will need specific governance and management rules. 

Hence what you need is not a tool but a solution to your problem and an MDM Solution comprises of three main part:

1. The MDM Tool

2. The MDM Architecture

3. The expertise of the Consultants on board.

We are all about Data. So the first and foremost thing we need is an IT-Savvy Business leader who knows his data and is prepared to take ownership.

From an MDM practice point of view, an MDM consultant can guide you with the right architecture or the right practice but you being the business owner should be the ultimate person to make a decision whether or not that is what you want and hence we should follow the following points.

1. Should know your data and its value

2. Always have the big picture in Mind

3. Think Big but start small. Always remember that Rome wasn't built in a day.

4. Before you are blinded by the flashy terms like Multi-Domain, Big-Data, Dynamic-Hierarchy. You must think that do you even need them? Or are they going to generate any extra bucks for you? Or don't you already have an in-house solution that is already delivering the same?

5. How can you make the best use of the already existing resources and still get the most robust and trusty solution

Second Challenge – Choosing the Right MDM Architecture

It is of utmost critical to understand what type of data problem do you have if at all you do have one. It is important because when we speak Mastering the data and creating a "Golden" copy of Information or "The best version of Truth". However, you may put it but the fact is it's not a single team job. It's an amalgamation of multiple task forces. Hence Enterprise architectural integration is also very critical. But let's stick to the issue at hand. Which is MDM. So what type of MDM do we need. In order to know that we first need to understand what are the different implementation patterns available. 

• Consolidation 
• Registry
• Centralization
• Co-existence

Consolidation 

When we are aware that there are silos of the same or similar data in the enterprise and we need to harmonize them to create a golden copy in-fact the aim is to identify the best version among all store information. And in order to do that we need to bring them all to a central location. Then de-dupe it and execute survivorship to generate the Golden record. At times the defined rules in the system are not enough and we will need some expert intervention to resolve duplicities or even to realize whether or not they are duplicates at all like identifying false positives and dealing with them.
Typically large enterprises that have adopted digitalization long back and have collected information using multiple sources like ERP, CRM, Web Apps, Websites, POS, etc... end up with this sort of problem. 

So, if we need consolidation then the MDM solution should be capable of sophisticatedly handle complex matching scenarios and should give the developers the flexibility of choices to pick and select different matching algorithms and different matching techniques.

Registry

Sometimes when they realize that they have data silos but they continue to manage and maintain it in silos because the business demand so. However, it is critical for them to understand the presence of the same information across organizations and also to understand what kind of information is available across which system they go for registry implementation. In this case, the MDM solution is responsible to maintain only a map or linkages of similar/same information stored in various systems which can be used for cross-reference purposes.

Centralization

Typically newer organization starts with this type of implementation. It is the best-case scenario where the MDM solution is in-fact the authoring tool. Hence it is very critical to have a strong governance or business process management capability to master the data in a correct fashion. The MDM solution should have data validation and workflow capabilities to address this implementation strategy and they are responsible to provide the mastered golden record to all the respective applications in the organization.

Co-existence

This is the reality in most of the MDM implementation projects. The aim always is to identify data silos and consolidate them. Once consolidated then use the MDM product also as the authoring tool and phase-wise decommissioning other authoring solutions. 

So, start with the Consolidation pattern and aim for the Centralization pattern. However lucrative this approach seems to be. It is often not possible as there are specialize COTS or custom full-stack apps that address a very specific and business-critical requirement hence cannot be decommissioned hence co-existence is the only option. Where we not only consolidate information coming from other systems but also enrich them using MDM and then publish it to everyone else.

Third Challenge – Building a Business case for MDM

✓Reconciliation – Reducing the direct costs of bad data
✓Reporting – Ensuring accurate reports for high-quality decision making
✓Regulatory compliance – Maintaining the organization’s good standing with the authorities
✓Risk management - Avoiding business risks

 

Reconciliation

Reconciliation is the process of diagnosing and then repairing mismatches between an organization’s accounts/groups/datasets/systems/transactions.
Without an authoritative source of master and reference data, where do your users go to find the right codes? cost centers? product numbers? Or verify the definition of specific reference data elements? Do you trust the sources your enterprise systems are consuming from? 

By not clearly defining a master source, systems and users run the risk of using incorrect or out of date information. All these creating reconciliation problems that need to be fixed later.

People costs
How much time do our employees spend on recon? What impact does this have on employee morale?
# Employees x Average Cost/Employee/Hour x Average Hours Spent† on Issues annually

Costs due to lack of centralization and ambiguity
What’s the cost of independently maintaining master data? Is there value in simplification? What’s the value of a single authoritative source?
# of Systems x Average Time (hours)‡ Required to Update System x Average Cost/Hour to Update


†Employee time should include time to report issues; diagnose, research, and repair errors; and updates to avoid future issues.

‡Other costs to include are the time required to make changes to the synchronization code (everywhere it exists) when there are system updates

Reporting

Enterprise hierarchies, dimensions, and attributes are used to aggregate or subdivide the source system data presented in reports. When reporting across systems, the individual source system dimensions must be standardized (conformed).
Without MDM there is no enterprise source of reporting metadata (conformed dimensions and hierarchies). This means the reporting team prepares for reporting by manually conformation/mapping dimensions. Manual efforts add time to the reporting process and introduce errors and reduce flexibility.


Costs associated with report preparation
How long does it take to prepare the reporting metadata (master data, dimensions, and hierarchies/rollups)? Does anyone review (or approve) the prep? Is this work is duplicated throughout the organization?
# Reports x # Reports per Year x # Hours Spent on Prep† per Report x Weighted Average Cost per Hour

Other questions
How are adjustments (reporting quality issues) managed? Do the same issues recur? Is root cause analysis performed on each issue?
oAre ad-hoc reports easy to create? Are there requirements to report on-demand? Should there be (i.e. For crisis management reasons)?
Does the business want more control (e.g. self-service hierarchy management)? timely/easier to create reports?


†Time spent on prep should include all the time ironing out the inconsistency in the dimensions. This might include creating/merging new codes, comparing/updating hierarchies, and building mappings (or crosswalks) between the different taxonomies used in the organization. Mismatching taxonomies are often found in global companies where each region and country may choose to use different product hierarchies.

Regulatory Compliance

Many organizations need to comply with the region, country, or industry-specific data governance, data quality, and data management regulations. Examples include European privacy rules, US Healthcare (HIPAA, HITECH), Global Banking (BCBS 239, and USA’s DFA).
MDM supports compliance by acting as a place to coordinate policies (governance), their scope (system metadata), implementation (data management), and measure and report on data quality. In addition to traditional data governance policies, some groups may include their privacy policies and their intersection with data domains and security classifications.
Without an integrated foundation, gaps are created between each element of the data governance and data management framework.

Compliance costs
How will the regulator assess compliance? Are there benchmarks for compliance? How much flexibility do we have in terms of compliance?
What are the penalties for non-compliance or poor compliance? Are the penalties civil or criminal in nature? Can employees or executives be found personally culpable?
Does non-compliance create legal standing for customers/suppliers/partners to pursue legal action against us?

Costs due to a lack of integration with data governance
How are our data governance policies created/maintained? Are these policies integrated with data management systems (can we track them)? How do we policy exceptions? Measure KPIs?

How are issues created? managed through to a resolution? 

Risk Management

Master data are an important enterprise data asset because they represent the key elements of your business model: what, to whom, and how, and how to account for what you sell.
Without master data management low-quality data proliferates.
No MDM means few formal controls on the creation, update, and retirement of master data translates into a lack of review and accuracy checks on customer data, product information, and financial accounts. (Incidentally, this is, in and of itself, a form of operational risk).
Also, without MDM, there’s often confusion about which system has the authoritative master data. Confusion means some teams will choose the wrong source. And given that master data is shared, this poor quality data will continue to create more low-quality data as it is shared.

Do we know of situations where poor quality master data affected our organization’s operations? What is the frequency of these problems?
Where were the issues: In-bound logistics? Operations? Distribution? Sales/marketing? Services?
Do we know what the root cause of the errors was? Were they tied to poor operational controls around the master data?
Did the organization suffer any reputational damage in the eyes of our customers, suppliers, distributors, or employees?
Did the failure create any financial issues for the firm? Did supplier terms change? Did customers demand deeper discounts? Did this change customer/supplier/employee retention?
How do our failure rates compare to our competitors? Industry benchmarks? If we improve beyond the benchmark, can it be used as a point of differentiation?

Do we know if situations where poor quality master data has affected or organization's strategic planning/budgeting process?

Fourth Challenge – Choosing the Right MDM Tool

We need to ensure that the following points are realized.

✓MDM/RDM can help an organization avoid costs, reduce risks and create value
✓The 4-Rs (reconciliation, reporting, regulatory compliance … and risk management) are all areas to examine when building business cases
✓Not all value (or costs) are quantifiable—understanding qualitative value often just as important as quantifying cost savings.

Selecting The Right Tool

Modular MDM Solution Vs Unified MDM Solution 

Master Data is the most important data for any enterprise. And as rightly defined in Wikipedia: In business, master data management (MDM) comprises the processes, governance, policies, standards, and tools that consistently define and manage the critical data of an organization to provide a single point of reference.

The concept of MDM has been in the industry for almost a decade now. I remember the initial days of MDM. It was all about Customer Data and Product data as two separate domains. There were two different originating points. Customer Data Integration (CDI) and Product Information Management (PIM) [That's what my older projects were known as]. Just like wine, with time the concept has matured and has been bettered in so many ways. People (read CTO/CFO/COO/Architects/Project Leads) started understanding the MDM ecosystem and the challenges that come along with it.

It didn't take forever to realize that just having a match & merge module or just having workflows for creating a golden record is not going to be enough. Hence the modern-day MDM tools started evolving themselves and started addressing each and every aspect of MDM.
Like.

• Data consolidation via match & merge.
• Granular survivorship capability.
• Excellent UX for stewardship
• Data governance via workflows and identity management.
• Data security at various levels. Data lineage.
• Hierarchical representation of data.
• The capability of customizing UI for rich UX.
• Data distribution patterns using data objects, web services, REST, SQL. Batch etc.
• Reporting capabilities
• Integration capabilities with legacy and modern systems
• Cloud-based
• Scalability and high availability.

The MDM product companies aka product vendors now have this vision. Thanks to either visionary in their team or thanks to the analyst firms who score them based on completeness. The question was how to be the most complete tool. How to have all the checkboxes ticked in order to be the most sought after "Enterprise Master Data Solution".

The race started a few years back and the goal was clear. There was no rule but the message was loud and clear "If you can't evolve you got to lose". Darwin's survival of fittest has now found a new meaning. By this time there were two major groups of MDM product vendors. "The Pioneers" and "The Challengers". In the era of PIM and CDI, pioneers were also the leaders. But the new wave of finding mature, complete, and smart MDM tools for the enterprise has erased the demarcation between the Pioneers and challengers and made the market more competitive. Two types of players emerged out of this competition. One who has the money to acquire or merge in order to overcome its deficiencies. And the other kind who has the intelligence and skill to develop and equip itself with features they used to think was not required.

It is a win-win situation for the consumers or the business enterprise. All they need to decide is whom do they want.
Do they want:
a) Companies that provide complete Enterprise package and have dedicated experts or SMEs for each module.
b) Companies that provide unified solution and have dedicated experts or SME's for the single solution.

The game is quite simple now. If you want to lure the client you should be able to impress the techies or the business users using the tool. The game is now being played in the conference room pilot and not in the golf ground. There are some soft corner and legacy driven decision that still takes place but in the long run, those decisions seldom give up under pressure and expectation.

In order to select the right product, there are two broad criteria.

1. Based on Domain
2. Based on the Implementation Pattern

Based on Domain 

We need to select a product/solution which can seamlessly handle multiple domains either business or technical. 

Business domains are like - 

• Customer
• Vendor
• Asset
• Product
• Location
• Pricing etc.

Technical domains are like - 

• Master Data
• Meta Data
• Reference Data
• Transactional Data

We need to pick a tool that must address multi-domain and not multiple domains. You don't want your IT team to undergo the painful process of implementing separate MDM for each domain. 

I have mentioned the same in one of my previous blogs which I published last year (2012) as to why Multi-Domain MDM is so important.

Based on the Implementation Pattern

As explained in the architectural section above. We must identify a solution that should support all or at least almost all types.

So, keep an eye on a solution that supports a multi-domain and multi-implementation strategy.

Fifth/Final Challenge – Implementation

While implementing an MDM Solution we need to focus on the fact that it should lay a strong foundation for operational excellence by implementing an agile and competitive solution that will ensure that you have quality access to master reference and master data domains.

• Establish master data priorities as part of the larger data and technology strategy
• Embraced the master data standards throughout the organization
• Enabling governance by mandating a stewardship process for maintaining master and reference data using workflows.
• Implement the change process consistently across the enterprise
• Establish roles and responsibilities using data integrity.

Kinshuk Dutta
Kolkata

Why multi domain MDM has become the talk of town?

[Multi - Domain] Master Data management

My blog started with the concept of MDM explained in
Master Data Management (CDI/IR/PIM) published in 2008.

I will try to redefine some of the terms from the previous MDM blog.

Master Data: It is the information that may include data about customers, products, employees, materials, suppliers, etc. often it is non-transactional in nature. In the context of MDM, master data can support transactional processes and operations also. Business-wise the analytics/reporting of an enterprise greatly depends on its master data. Master data is often used by several functional groups and stored in different data systems across an enterprise.

Digitizing all the information has led us to a huge amount of data. The same is true for Master Data. Worldwide organizations are now aware of how easy it is to manage their customers or products if they have that information in digital form. Several data entry solutions are available and also several enterprise-level managing solutions like ERP solutions, CRM solutions, etc. It often so happens that in the same organizations we have different functional group storing and dealing with the same or similar data. It is but obvious that this leads to multiple copies of the master record in an organization.

Master Data Management: In computing, master data management (MDM) comprises a set of processes and tools that consistently defines and manages the master data of an organization (which may include reference data). MDM has the objective of providing processes for collecting, aggregating, matching, consolidating, quality-assuring, persisting, and distributing such data throughout an organization to ensure consistency and control in the ongoing maintenance and application use of this information. Thereby producing a master file which is believed to be the best version of truth or the “Golden Data”

Managing Master Data is of utmost importance because it is used by multiple applications, an error in master data can cause errors in all the applications that use it. E.g. an incorrect address in the customer master might mean orders, bills, and marketing literature is all sent to the wrong address. Similarly, an incorrect price on an item master can be a marketing disaster, and an incorrect account number in an Account Master can lead to huge fines.

DW & MDM: In the IT world we have three main categories of enterprise one is the product company (people who develop and produce the MDM tools), the Service Implementers, and the Client (The organization where MDM will be implemented). Educating the third organization about the importance of MDM was the prior task of the first two. There were a different thought process and lots of challenging questions. Clients always had this question for the MDM vendors, why do we need an MDM solution at all when we already have a Data warehouse. At that point it was very important that all should understand where Data warehouse ends, MDM begins.

In many ways, the processes are similar to those used in populating a data warehouse or datamart, particularly with the consolidation style of MDM. However, despite the close relationship between MDM and data warehousing, a glance at the recent literature reveals that these two important areas tend to be treated as separate:

Volume and Type of Data: MDM systems involve only the master data, not terabytes of transaction data (such as telecommunication companies' call detail records), which may need to be stored at the detail level, in addition to various levels of aggregation for analysis purposes.

Database Design: The master data is typically held in a relatively normalized form, whereas many analytical systems depend on specialized designs, such as star schemas to improve analytical performance.

Use of the Data: MDM reconciles semantic differences to give a single view of master data, usually for operational purposes, which is independent of any single application system. Data warehousing supports BI and analytics but does not usually support transactional activities. MDM systems and data warehouses are complementary, and the introduction of MDM systems into the operational area should reduce the work required to integrate and consolidate master data as it's prepared for loading into the data warehouse.

Two approaches to master data management are widely taken in practice: MDM for operational purposes and MDM for analytical purposes. Operational MDM focuses on ensuring that data in multiple operational systems that should be the same actually is the same. Analytic MDM is usually associated with data warehousing and seeks to leave the operational world alone, instead of focusing on compiling a view of data that can be used for analytic BI and management information purposes. Operational MDM is generally being adopted by organizations with a need to focus on ensuring consistency of master data definitions across the operational systems landscape (e.g., enterprise resource planning, customer relationship management, etc.). Analytic MDM has become established as a style of implementation adopted by businesses needing to effect a significant improvement in the speed and quality of their business reporting, often centered on one or more national, regional, or enterprise data warehouses.

About Multi-Domain MDM: The term multi-domain is very confusing at times. 

For a Client, it looks like different business domains whereas for the vendors it is actually the same business but different nature of data stored or the different types of entities, which the client organization is managing. So do you just manage the customer entity, just the product entity, or manage multiple types of entities. Sometimes this views suppliers and customers as being indifferent “domains” as they might have a different database structure. 

Multi-Domain MDM is actually a technical view of a domain-driven from a product perspective. It should have been Multi-Entity MDM. Frankly speaking, that will make life much easier for the sales reps who want to push a multi-domain MDM implementation. Otherwise from a business perspective, often the clients get confused at the proposal itself and start wondering why we need connectivity between Customer-Centric, Enterprise Centric, and Supply Centric domains.

The MDM product companies are now offering that with a single instance of MDM product one can:

• Manage all domains of Master Data as well as Reference Data.

• Implement both Operational MDM and Analytical MDM solutions.

• Support multiple MDM architecture styles: Transactional, Consolidated, Hybrid.

• Meet both B2B and B2C requirements.

• Manage and harmonize metadata.

• Manage unstructured data (such as documents and images) related to Master Data entities.

--

Kinshuk Dutta 

Kolkata

Master Data Management (CDI/IR/PIM)

What is Master Data Management

It's the process of managing "Master Data". 

Master Data Management is a technology-enabled discipline in which business and Information Technology work together to ensure the uniformity, accuracy, stewardship, semantic consistency, and accountability of the enterprise's official shared master data assets.

Wikipedia

What is Master Data

Enterprize data can be broadly categorized as :

Transactional data are the elements that support the on-going operations of an organization and are included in the application systems that automate key business processes. This can include areas such as sales, service, order management, manufacturing, purchasing, billing, accounts receivable, and accounts payable. Commonly, transactional data refers to the data that is created and updated within the operational systems. Examples of transactional data included the time, place, price, discount, payment methods, etc. used at the point of sale.

Analytical data are the numerical values, metrics, and measurements that provide business intelligence and support organizational decision making. Typically analytical data is stored in Online Analytical Processing (OLAP) repositories optimized for decision support, such as enterprise data warehouses and department data marts. Analytical data is characterized as being the facts and numerical values in a dimensional model. Normally, the data resides in fact tables surrounded by key dimensions such as customer, product, account, location, and date/time. 

Master data is your most important data it plays a critical role in the core operation of a business. Master data refers to the prime entities that are used by several functional groups in your enterprise and are typically stored in different data systems across an organization. 

In other words, it represents the business entities around which the organization’s business transactions are executed and the primary elements around which analytics are conducted. 

Master data is typically persistent, non-transactional data utilized by multiple systems that define the primary business entities. 

Master data may include data about customers, products, employees, inventory, suppliers, and sites.

 

This year we have seen emerging trends in CDI & PIM. Let's understand what are they? Why do enterprises need it or why is it such sought after and how do they really qualify in the MDM ecosphere.

CDI/IR/MDM

Customer data integration (CDI) and master data management (MDM) are getting significant buzz in both information technology (IT) and business circles. 

Master data management represents the data management disciplines and processes, and 

CDI represents the authoritative system for customer data. 

Both concepts are tightly linked.

It all started with CDI and CDI started with CIF or Client Information File. Where we store information for a customer in a digitized file. This information can be accessed by various applications across the enterprise. However, often different applications tend to store the client information in their own way and format which is neither accessible nor sharable with other applications. Hence client data integration is required to share the information across multiple applications.

Identity resolution (IR) is an operational intelligence process, typically powered by an entity resolution engine or middleware, whereby organizations can connect disparate data sources with a view to understanding possible entity matches and non-obvious relationships across multiple data silos. It analyzes all of the information relating to individuals and/or entities from multiple sources of data and then applies likelihood and probability scoring to determine which identities are a match and what, if any, non-obvious relationships exist between those identities.

PIM/MDM

Product information management (PIM) is the process of managing all the information required to market and sell products through distribution channels. This product data is created by an internal organization to support a multichannel marketing strategy. A central hub of product data can be used to distribute information to sales channels such as e-commerce websites, print catalogs, marketplaces such as Amazon and Google Shopping, social media platforms like Facebook and electronic data feeds to trading partners.

PIM solutions are most relevant to business-to-consumer (B2C) and business-to-business (B2B) firms that sell products through a variety of sales channels in a range of industries.

The use of PIM is generally influenced by a company's:

• A wide array of products and/or complex product data set
• Frequently changing product characteristics
• An increasing number of sales channels
• Non-uniform IT infrastructure 
• Online business and electronic ordering
• Various locales and localization requirements

PIM systems manage customer-facing product data needed to support multiple geographic locations, multi-lingual data, and maintenance and modification of product information within a centralized catalog. Product information kept by a business can be scattered throughout departments and held by employees or systems, instead of being available centrally; data may be saved in various formats, or only be available in hard copy form. Information may be needed for detailed product descriptions with prices, or calculating freight costs. PIM represents a solution for centralized, media-independent product data maintenance, as well as efficient data collection, enrichment, data governance, and output

Key concepts of MDM

Let's revisit the basics of Master Data Management & Data Governance! 

MDM Eco-System

The MDM Eco-System as I like to call it mainly evolves around processes like Data Quality, Data Cleansing, De-duplication & Data Stewardship.

Data Quality

There are 7 dimensions of Data Quality:

1. Accuracy: The degree of conformity of a measure to a standard or a true value - see also Accuracy and precision. Accuracy is very hard to achieve through data-cleansing in the general case because it requires accessing an external source of data that contains the true value: such "gold standard" data is often unavailable. Accuracy has been achieved in some cleansing contexts, notably customer contact data, by using external databases that match up zip codes to geographical locations (city and state) and also help verify that street addresses within these zip codes actually exist.
2. Completeness: The degree to which all required measures are known. Incompleteness is almost impossible to fix with data cleansing methodology: one cannot infer facts that were not captured when the data in question was initially recorded. (In some contexts, e.g., interview data, it may be possible to fix incompleteness by going back to the original source of data, i.e. re-interviewing the subject, but even this does not guarantee success because of problems of recall - e.g., in an interview to gather data on food consumption, no one is likely to remember exactly what one ate six months ago. In the case of systems that insist certain columns should not be empty, one may work around the problem by designating a value that indicates "unknown" or "missing", but the supplying of default values does not imply that the data has been made complete.)
3. Consistency: The degree to which a set of measures are equivalent in across systems (see also Consistency). Inconsistency occurs when two data items in the data set contradict each other: e.g., a customer is recorded in two different systems as having two different current addresses, and only one of them can be correct. Fixing inconsistency is not always possible: it requires a variety of strategies - e.g., deciding which data were recorded more recently, which data source is likely to be most reliable (the latter knowledge may be specific to a given organization), or simply trying to find the truth by testing both data items (e.g., calling up the customer).
4. Conformity: Can be ensured by enabling validation constraints which fall into the following categories:
   
• Data-Type Constraints – e.g., values in a particular column must be of a particular data type, e.g., Boolean, numeric (integer or real), date, etc.
• Range Constraints: typically, numbers or dates should fall within a certain range. That is, they have minimum and/or maximum permissible values.
• Mandatory Constraints: Certain columns cannot be empty.
• Unique Constraints: A field, or a combination of fields, must be unique across a dataset. For example, no two persons can have the same social security number.
• Set-Membership constraints: The values for a column come from a set of discrete values or codes. For example, a person's gender may be Female, Male or Unknown (not recorded).
• Foreign-key constraints: This is the more general case of set membership. The set of values in a column is defined in a column of another table that contains unique values. For example, in a US taxpayer database, the "state" column is required to belong to one of the US's defined states or territories: the set of permissible states/territories is recorded in a separate State table. The term foreign key is borrowed from relational database terminology.
• Regular expression patterns: Occasionally, text fields will have to be validated this way. For example, phone numbers may be required to have the pattern (999) 999-9999.
• Cross-field validation: Certain conditions that utilize multiple fields must hold. For example, in laboratory medicine, the sum of the components of the differential white blood cell count must be equal to 100 (since they are all percentages). In a hospital database, a patient's date of discharge from the hospital cannot be earlier than the date of admission.
5. Concurrency
6. Duplication
7. Integrity: The term integrity encompasses accuracy, consistency and some aspects of validation.

Data Cleansing

Data cleansing or data cleaning is the process of detecting and correcting corrupt or inaccurate records from a record set, table, or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data. Wikipedia

Data De-Duplication

Data Survivorship

Data Stewardship

Data Lineage

-

Kinshuk Dutta

Kolkata