Why multi domain MDM has become the talk of town?

[Multi - Domain] Master Data management

My blog started with the concept of MDM explained in
Master Data Management (CDI/IR/PIM) published in 2008.

I will try to redefine some of the terms from the previous MDM blog.

Master Data: It is the information that may include data about customers, products, employees, materials, suppliers, etc. often it is non-transactional in nature. In the context of MDM, master data can support transactional processes and operations also. Business-wise the analytics/reporting of an enterprise greatly depends on its master data. Master data is often used by several functional groups and stored in different data systems across an enterprise.

Digitizing all the information has led us to a huge amount of data. The same is true for Master Data. Worldwide organizations are now aware of how easy it is to manage their customers or products if they have that information in digital form. Several data entry solutions are available and also several enterprise-level managing solutions like ERP solutions, CRM solutions, etc. It often so happens that in the same organizations we have different functional group storing and dealing with the same or similar data. It is but obvious that this leads to multiple copies of the master record in an organization.

Master Data Management: In computing, master data management (MDM) comprises a set of processes and tools that consistently defines and manages the master data of an organization (which may include reference data). MDM has the objective of providing processes for collecting, aggregating, matching, consolidating, quality-assuring, persisting, and distributing such data throughout an organization to ensure consistency and control in the ongoing maintenance and application use of this information. Thereby producing a master file which is believed to be the best version of truth or the “Golden Data”

Managing Master Data is of utmost importance because it is used by multiple applications, an error in master data can cause errors in all the applications that use it. E.g. an incorrect address in the customer master might mean orders, bills, and marketing literature is all sent to the wrong address. Similarly, an incorrect price on an item master can be a marketing disaster, and an incorrect account number in an Account Master can lead to huge fines.

DW & MDM: In the IT world we have three main categories of enterprise one is the product company (people who develop and produce the MDM tools), the Service Implementers, and the Client (The organization where MDM will be implemented). Educating the third organization about the importance of MDM was the prior task of the first two. There were a different thought process and lots of challenging questions. Clients always had this question for the MDM vendors, why do we need an MDM solution at all when we already have a Data warehouse. At that point it was very important that all should understand where Data warehouse ends, MDM begins.

In many ways, the processes are similar to those used in populating a data warehouse or datamart, particularly with the consolidation style of MDM. However, despite the close relationship between MDM and data warehousing, a glance at the recent literature reveals that these two important areas tend to be treated as separate:

Volume and Type of Data: MDM systems involve only the master data, not terabytes of transaction data (such as telecommunication companies' call detail records), which may need to be stored at the detail level, in addition to various levels of aggregation for analysis purposes.

Database Design: The master data is typically held in a relatively normalized form, whereas many analytical systems depend on specialized designs, such as star schemas to improve analytical performance.

Use of the Data: MDM reconciles semantic differences to give a single view of master data, usually for operational purposes, which is independent of any single application system. Data warehousing supports BI and analytics but does not usually support transactional activities. MDM systems and data warehouses are complementary, and the introduction of MDM systems into the operational area should reduce the work required to integrate and consolidate master data as it's prepared for loading into the data warehouse.

Two approaches to master data management are widely taken in practice: MDM for operational purposes and MDM for analytical purposes. Operational MDM focuses on ensuring that data in multiple operational systems that should be the same actually is the same. Analytic MDM is usually associated with data warehousing and seeks to leave the operational world alone, instead of focusing on compiling a view of data that can be used for analytic BI and management information purposes. Operational MDM is generally being adopted by organizations with a need to focus on ensuring consistency of master data definitions across the operational systems landscape (e.g., enterprise resource planning, customer relationship management, etc.). Analytic MDM has become established as a style of implementation adopted by businesses needing to effect a significant improvement in the speed and quality of their business reporting, often centered on one or more national, regional, or enterprise data warehouses.

About Multi-Domain MDM: The term multi-domain is very confusing at times. 

For a Client, it looks like different business domains whereas for the vendors it is actually the same business but different nature of data stored or the different types of entities, which the client organization is managing. So do you just manage the customer entity, just the product entity, or manage multiple types of entities. Sometimes this views suppliers and customers as being indifferent “domains” as they might have a different database structure. 

Multi-Domain MDM is actually a technical view of a domain-driven from a product perspective. It should have been Multi-Entity MDM. Frankly speaking, that will make life much easier for the sales reps who want to push a multi-domain MDM implementation. Otherwise from a business perspective, often the clients get confused at the proposal itself and start wondering why we need connectivity between Customer-Centric, Enterprise Centric, and Supply Centric domains.

The MDM product companies are now offering that with a single instance of MDM product one can:

• Manage all domains of Master Data as well as Reference Data.

• Implement both Operational MDM and Analytical MDM solutions.

• Support multiple MDM architecture styles: Transactional, Consolidated, Hybrid.

• Meet both B2B and B2C requirements.

• Manage and harmonize metadata.

• Manage unstructured data (such as documents and images) related to Master Data entities.

--

Kinshuk Dutta 

Kolkata

Are you a Data Engineer or a Data Scientist?

Are you a Data Engineer or a Data Scientist?

In the recent time two new designation / title is making the headlines in the data world. 🌎
Data Engineer & Data Scientist and it all begun with Data Analytics.

A data is no good unless we derive information out of it and that information should be insightful. The set of tools which lets you do that is known as data analytics products. People working in analytics are often associated with terms like Engineers & Scientists.

Why does it matter at all? 

Well it matters because it determines the career road map for you. So, in order to know who you are or rather which of these broad category define your true nature. You will need to know the difference between them.

Data Engineers 

Decisions backed by data and analytics can provide a competitive advantage and increase ROI.

Hence it is critical that data analytics solutions implemented in your enterprise are fast and efficient.

If you are a data engineer you are expected to design, develop, implement and support product which can deal with data. Be it structural (Master, Reference, Meta, Transactional or Analytics) data or non structural data and curate a data pipeline or move data to production. Data engineers make the appropriate data accessible and available to the right users at the right time by enabling secure, compliant data utilization and democratization across enterprise.

If you have more fun with the technical part of data . Like designing, building and arranging different components of data flow architecture. This is closely related to data architect in-fact when the traditional data architects starts handling and designing Big Data instead of Data Warehouse they are known as data engineers.

Data Scientists give a meaning to your data. Basically if you can read and understand trillions of data to predict a situation or trend by using analytics then you are a data scientist. 

A Data Scientist collects, interprets and publishes data. The data they find can be used for many reasons, but in the business world, it often applies to finances or productivity. For example, a Data Scientist may look at sales figures against business decisions made over a certain time frame to determine how successful those decisions were. 

These professionals provide the forecasting knowledge a business needs to know whether changes will be effective before making a decision. Data Scientists work in a variety of industries, such as IT, healthcare, finance, retail and marketing.

1. In order to do their job efficiently the scientists typically follow the below mention approach:
2. Ask the right questions to begin the discovery process
3. Acquire data
4. Process and clean the data
5. Integrate and store data
6. Initial data investigation and exploratory data analysis
7. Choose one or more potential models and algorithms
8. Apply data science techniques, such as machine learning, statistical modeling, and artificial intelligence
9. Measure and improve results
10. Present final result to stakeholders
11. Make adjustments based on feedback
12. Repeat the process to solve a new problem

--

Kinshuk Dutta

Kolkata

Nov. 2010

I recently found this excellent pictorial representation on Data Camp. Which echoes my thought.

Cloud Computing

Cloud Computing - "Finally, all of you, live in harmony with one another; be sympathetic, love as brothers, be compassionate and humble." 1 Peter 3:8 (NIV)

Come lets share our resources for a much more efficient IT world. Such a nice explanation is available on the following sites that I don need to explain any thing at all.
http://en.wikipedia.org/wiki/Cloud_computing



The major cloud computing implementations are:
SFDC
Oracle Cloud Computing
Windows Azure
Amazon EC2 - Elastic Compute Cloud

Kinshuk Dutta
London

Social CRM - A Cult

Am I Becoming Vulnerable - Social CRM

The more information one puts on the net the more vulnerable he/she is becoming. Earlier crooks were playing with it illegally but now ethically people are using information pertaining to someone available on WWW to analyze, understand, and peruse the person for one’s own gain.

Does it sound weird? Actually not. That’s what Social CRM is all about. A couple of years back everybody was talking about Social CRM. The hype was much more than the promo of Yash Raj Films. At that time things were much mystified and mystical, so what exactly is Social CRM?

The traditional CRM system assists an organization by bringing together data from all areas of an organization, giving a 360-degree view of a customer for marketing and sales to make informed decisions on cross-selling and up-selling opportunities as well as to shape marketing strategies and corporate communications.

Social CRM, on the other hand, is based on your existing CRM that has the ability to leverage the social web and automate the conversion process. The social CRM can be used by marketing and sales teams to listen to conversations, craft appropriate messages, join in immediately with customer conversation, and offer them value in terms of information and solutions.

The ROI for setting up this system will be the relationships and long term loyalty that serve entrepreneurs over time.   Social CRM will help generate marketing intelligence, providing the marketing department with insight that will assist your company to source better leads and reduce customer support costs through self-helping communities.

To begin with, marketers should dedicate time to working with brand advocates, involving them in shaping the product and communications. This is how social CRM is supposed to work; through the integration of customer social networks.

One should find that their existing customers belong to social networks and openly add more information to their profile than what you have in your traditional CRM system. Therefore you should think creatively of ways to integrate your customer’s networks into your current CRM.

One can know completely about a customer from combined information of his LinkedIn, Facebook, and Orkut profiles more insight can be availed from the way the person tweets on Twitter or tracking his conversations on friend feed and Buzz.

Social networks and communities are influencing CRM; resulting in corporate sites and marketing communications being able to recognize social relationships and customers' preferences and deliver customized experiences to them in real-time. We are still some distance from this scenario; however companies such as Appirio and Salesforce are already leading the way in Social CRM.

A couple of years back, Appirio developed a Facebook application that can easily be rebranded by marketers for campaign activity. This application can be distributed at the same time sharing information, recommending information to peers, and used for other purposes such as recruiting, word of mouth, and other typical social network activities. As the information is shared, it can then be passed to a landing page where users can submit information in a web form that is stored on Salesforce.

Social CRM is both exciting and daunting, but by taking tiny steps to understanding social media, and experimenting through integrating social tools into your CRM system now, then testing and improving, you will be very well prepared.

Guess what? there used to be telemarketing, Now we have Twitter marketing.

-

Kinshuk Dutta

London

Introduction to NoSQL | Mongo DB

Introduction to NoSQL

A Brief History of NoSQL

I came across this brief history from a blog

NoSQL was in fact first used by Carlo Strozzi in 1998 as the name of the file-based database he was developing. Ironically it’s a relational database just one without a SQL interface. The term re-surfaced in 2009 when Eric Evans used it to name the current surge in non-relational databases. 

## the 1960s

MultiValue (aka PICK) databases are developed at TRW in 1965.
According to a comment from Scott Jones M[umps] is developed at Mass General Hospital in 1966. It is a programming language that incorporates a hierarchical database with B+ tree storage.
IBM IMS, a hierarchical database, is developed with Rockwell and Caterpillar for the Apollo space program in 1966.

## the 1970s

• InterSystems develops the ISM product family succeeded by the Open M product, all M[umps] implementations. See comment from Scott Jones below.
• M[umps] is approved as an ANSI standard language in 1977.
• in 1979 Ken Thompson creates DBM which is released by AT&T. At its core, it is a file-based hash.

## the 1980’s

Several successors to DBM spring into life.

• TDBM supporting atomic transactions
• NDBM was the Berkeley version of DBM supporting having multiple databases open at the same time.
• SDBM - another clone of DBM mainly for licensing reasons.
• GT.M is the first version of a key-value store with a focus on high-performance transaction processing. It is open-sourced in 2000.
• BerkeleyDB is created at Berkeley in the transition from 4.3BSD to 4.4BSD. Sleepycat software is started as a company in 1996 when Netscape needed new features for BerkeleyDB. Later acquired by Oracle which still sells and maintains BerkeleyDB.
• Lotus Notes or rather the server part, Lotus Domino, which really is a document database has it’s initial release in 1989, now sold by IBM. It has evolved a lot from the early versions and is now a full office and collaboration suite.

## the 1990’s

• GDBM is the Gnu project clone of DBM
• Mnesia is developed by Ericsson as a soft real-time database to be used in telecom. It is relational in nature but does not use SQL as query language but rather Erlang itself.
• InterSystems Caché launched in 1997 and is a hybrid so-called post-relational database. It has object interfaces, SQL, PICK/MultiValue, and direct manipulation of data structures. It is an M[umps] implementation. See Scott Jones comment below for more on the history of InterSystems
• Metakit is started in 1997 and is probably the first document-oriented database. Supports smaller datasets than the ones in vogue nowadays.

##2000-2005

• This is where the NoSQL train really picks up some momentum and a lot is starting to happen.
• Graph database Neo4j is started in 2000.
• db4o an object database for Java and .net is started in 2000
• QDBM is a re-implementation of DBM with better performance by Mikio Hirabayashi.
• Memcached is started in 2003 by Danga to power Livejournal. Memcached isn’t really a database since it’s memory-only but there is soon a version with file storage called memcachedb.
• Infogrid graph database is started as closed source in 2005, open-sourced in 2008
• CouchDB is started in 2005 and provides a document database inspired by Lotus Notes. The project moves to the Apache Foundation in 2008.
• Google BigTable is started in 2004 and the research paper is released in 2006.
• 
  

##2006-2010

• JackRabbit is started in 2006 as an implementation of JSR 170 and 283.
• Tokyo Cabinet is a successor to QDBM by (Mikio Hirabayashi) started in 2006
• The research paper on Amazon Dynamo is released in 2007.
• The document database MongoDB is started in 2007 as a part of an open-source cloud computing stack and first standalone release in 2009.
• Facebooks open sources the Cassandra project in 2008
• Project Voldemort is a replicated database with no single-point-of-failure. Started in 2008.
• Dynomite is a Dynamo clone written in Erlang.
• Terrastore is a scalable elastic document store started in 2009
• Redis is a persistent key-value store that started in 2009
• Riak Another dynamo-inspired database started in 2009.
• HBase is a BigTable clone for the Hadoop project while Hypertable is another BigTable type database also from 2009.
• Vertexdb another graph database is started in 2009
• Eric Evans of Rackspace, a committer on the Cassandra project, introduces the term “NoSQL” often used in the sense of “Not Only SQL” to describe the surge of new projects and products.

-

Kinshuk Dutta

London

Master Data Management (CDI/IR/PIM)

What is Master Data Management

It's the process of managing "Master Data". 

Master Data Management is a technology-enabled discipline in which business and Information Technology work together to ensure the uniformity, accuracy, stewardship, semantic consistency, and accountability of the enterprise's official shared master data assets.

Wikipedia

What is Master Data

Enterprize data can be broadly categorized as :

Transactional data are the elements that support the on-going operations of an organization and are included in the application systems that automate key business processes. This can include areas such as sales, service, order management, manufacturing, purchasing, billing, accounts receivable, and accounts payable. Commonly, transactional data refers to the data that is created and updated within the operational systems. Examples of transactional data included the time, place, price, discount, payment methods, etc. used at the point of sale.

Analytical data are the numerical values, metrics, and measurements that provide business intelligence and support organizational decision making. Typically analytical data is stored in Online Analytical Processing (OLAP) repositories optimized for decision support, such as enterprise data warehouses and department data marts. Analytical data is characterized as being the facts and numerical values in a dimensional model. Normally, the data resides in fact tables surrounded by key dimensions such as customer, product, account, location, and date/time. 

Master data is your most important data it plays a critical role in the core operation of a business. Master data refers to the prime entities that are used by several functional groups in your enterprise and are typically stored in different data systems across an organization. 

In other words, it represents the business entities around which the organization’s business transactions are executed and the primary elements around which analytics are conducted. 

Master data is typically persistent, non-transactional data utilized by multiple systems that define the primary business entities. 

Master data may include data about customers, products, employees, inventory, suppliers, and sites.

 

This year we have seen emerging trends in CDI & PIM. Let's understand what are they? Why do enterprises need it or why is it such sought after and how do they really qualify in the MDM ecosphere.

CDI/IR/MDM

Customer data integration (CDI) and master data management (MDM) are getting significant buzz in both information technology (IT) and business circles. 

Master data management represents the data management disciplines and processes, and 

CDI represents the authoritative system for customer data. 

Both concepts are tightly linked.

It all started with CDI and CDI started with CIF or Client Information File. Where we store information for a customer in a digitized file. This information can be accessed by various applications across the enterprise. However, often different applications tend to store the client information in their own way and format which is neither accessible nor sharable with other applications. Hence client data integration is required to share the information across multiple applications.

Identity resolution (IR) is an operational intelligence process, typically powered by an entity resolution engine or middleware, whereby organizations can connect disparate data sources with a view to understanding possible entity matches and non-obvious relationships across multiple data silos. It analyzes all of the information relating to individuals and/or entities from multiple sources of data and then applies likelihood and probability scoring to determine which identities are a match and what, if any, non-obvious relationships exist between those identities.

PIM/MDM

Product information management (PIM) is the process of managing all the information required to market and sell products through distribution channels. This product data is created by an internal organization to support a multichannel marketing strategy. A central hub of product data can be used to distribute information to sales channels such as e-commerce websites, print catalogs, marketplaces such as Amazon and Google Shopping, social media platforms like Facebook and electronic data feeds to trading partners.

PIM solutions are most relevant to business-to-consumer (B2C) and business-to-business (B2B) firms that sell products through a variety of sales channels in a range of industries.

The use of PIM is generally influenced by a company's:

• A wide array of products and/or complex product data set
• Frequently changing product characteristics
• An increasing number of sales channels
• Non-uniform IT infrastructure 
• Online business and electronic ordering
• Various locales and localization requirements

PIM systems manage customer-facing product data needed to support multiple geographic locations, multi-lingual data, and maintenance and modification of product information within a centralized catalog. Product information kept by a business can be scattered throughout departments and held by employees or systems, instead of being available centrally; data may be saved in various formats, or only be available in hard copy form. Information may be needed for detailed product descriptions with prices, or calculating freight costs. PIM represents a solution for centralized, media-independent product data maintenance, as well as efficient data collection, enrichment, data governance, and output

Key concepts of MDM

Let's revisit the basics of Master Data Management & Data Governance! 

MDM Eco-System

The MDM Eco-System as I like to call it mainly evolves around processes like Data Quality, Data Cleansing, De-duplication & Data Stewardship.

Data Quality

There are 7 dimensions of Data Quality:

1. Accuracy: The degree of conformity of a measure to a standard or a true value - see also Accuracy and precision. Accuracy is very hard to achieve through data-cleansing in the general case because it requires accessing an external source of data that contains the true value: such "gold standard" data is often unavailable. Accuracy has been achieved in some cleansing contexts, notably customer contact data, by using external databases that match up zip codes to geographical locations (city and state) and also help verify that street addresses within these zip codes actually exist.
2. Completeness: The degree to which all required measures are known. Incompleteness is almost impossible to fix with data cleansing methodology: one cannot infer facts that were not captured when the data in question was initially recorded. (In some contexts, e.g., interview data, it may be possible to fix incompleteness by going back to the original source of data, i.e. re-interviewing the subject, but even this does not guarantee success because of problems of recall - e.g., in an interview to gather data on food consumption, no one is likely to remember exactly what one ate six months ago. In the case of systems that insist certain columns should not be empty, one may work around the problem by designating a value that indicates "unknown" or "missing", but the supplying of default values does not imply that the data has been made complete.)
3. Consistency: The degree to which a set of measures are equivalent in across systems (see also Consistency). Inconsistency occurs when two data items in the data set contradict each other: e.g., a customer is recorded in two different systems as having two different current addresses, and only one of them can be correct. Fixing inconsistency is not always possible: it requires a variety of strategies - e.g., deciding which data were recorded more recently, which data source is likely to be most reliable (the latter knowledge may be specific to a given organization), or simply trying to find the truth by testing both data items (e.g., calling up the customer).
4. Conformity: Can be ensured by enabling validation constraints which fall into the following categories:
   
• Data-Type Constraints – e.g., values in a particular column must be of a particular data type, e.g., Boolean, numeric (integer or real), date, etc.
• Range Constraints: typically, numbers or dates should fall within a certain range. That is, they have minimum and/or maximum permissible values.
• Mandatory Constraints: Certain columns cannot be empty.
• Unique Constraints: A field, or a combination of fields, must be unique across a dataset. For example, no two persons can have the same social security number.
• Set-Membership constraints: The values for a column come from a set of discrete values or codes. For example, a person's gender may be Female, Male or Unknown (not recorded).
• Foreign-key constraints: This is the more general case of set membership. The set of values in a column is defined in a column of another table that contains unique values. For example, in a US taxpayer database, the "state" column is required to belong to one of the US's defined states or territories: the set of permissible states/territories is recorded in a separate State table. The term foreign key is borrowed from relational database terminology.
• Regular expression patterns: Occasionally, text fields will have to be validated this way. For example, phone numbers may be required to have the pattern (999) 999-9999.
• Cross-field validation: Certain conditions that utilize multiple fields must hold. For example, in laboratory medicine, the sum of the components of the differential white blood cell count must be equal to 100 (since they are all percentages). In a hospital database, a patient's date of discharge from the hospital cannot be earlier than the date of admission.
5. Concurrency
6. Duplication
7. Integrity: The term integrity encompasses accuracy, consistency and some aspects of validation.

Data Cleansing

Data cleansing or data cleaning is the process of detecting and correcting corrupt or inaccurate records from a record set, table, or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data. Wikipedia

Data De-Duplication

Data Survivorship

Data Stewardship

Data Lineage

-

Kinshuk Dutta

Kolkata