This blog is about different trends in the Data Management ecosphere. Master data (Reference data, Meta data), Big Data, Data Science - R, IoT, Container (Docker), AWS
OpenJDK 64-Bit Server VM (build 25.71-b00, mixed mode)
JAVA_HOME is essentially the full path of the directory that contains a sub-directory named bin which in turn contains the java.
For Mac OSX – it is /Library/Java/Home
Set JAVA_HOME using this command in Terminal: export JAVA_HOME=/Library/Java/Home
echo $JAVA_HOME on Terminal to confirm the path
You should now be able to run your application
Note that this sets JAVA_HOME only for this session. If you want it to persist, you will have to add the command to your ~/.profile file. Below are instructions on how to accomplish this instead:
Open up Terminal.app (Applications >> Utilities >> Terminal)
Type: emacs .profile
add this to the end of the .profile file:
JAVA_HOME=/Library/Java/Home export JAVA_HOME;
Save and exit emacs (ctrl-x, ctrl-s; ctrl-x, ctrl-c)
Open a new Terminal window (cmd-n) and type: $JAVA_HOME/bin/java -version
If you see something like: java version “1.5.0_16″ Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284) Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
Then you’ve successfully set your JAVA_HOME environment variable to the binary stored in /Library/Java/Home/bin/java
Install Home Brew
Open a terminal in MAC. CMD + Space & type Terminal or open terminal from launcher
🍺 /usr/local/Cellar/hadoop/2.7.2: 6,304 files, 310M, built in 1 minute 5 seconds
Configuration
Configuring Hadoop
Edit hadoop-env.sh
The file can be located at /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/hadoop-env.shwhere 2.7.0 is the hadoop version.
One can open and edit the file in the terminal alternatively can also open the file from finder. In order to do so. You need to activate the finder to show all hidden folders.
Show/Hide Hidden Files
Open Terminal found in Finder > Applications > Utilities
In Terminal, paste the following:defaults write com.apple.finder AppleShowAllFiles YES
Press return
Hold the ‘Option/alt’ key, then right click on the Finder icon in the dock and click Relaunch.
Relaunch Finder by right clicking the Finder Icon whilst holding the ‘Option/alt’ key
This will show all hidden files.
To hide them again, follow the same steps but replace the Terminal command with: defaults write com.apple.finder AppleShowAllFiles NO
I used text wrangler to edit the file. Alternatively you can edit it on terminal:
# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are # optional. When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes.
# The java implementation to use. export JAVA_HOME="$(/usr/libexec/java_home)"
# The jsvc implementation to use. Jsvc is required to run secure datanodes # that bind to privileged ports to provide authentication of data transfer # protocol. Jsvc is not required if SASL is configured for authentication of # data transfer protocol using non-privileged ports. #export JSVC_HOME=${JSVC_HOME}
# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler. for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do if [ "$HADOOP_CLASSPATH" ]; then export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f else export HADOOP_CLASSPATH=$f fi done
# The maximum amount of heap to use, in MB. Default is 1000. #export HADOOP_HEAPSIZE= #export HADOOP_NAMENODE_INIT_HEAPSIZE=""
# Extra Java runtime options. Empty by default. export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
# Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" #HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS" # On secure datanodes, user to run the datanode as after dropping privileges. # This **MUST** be uncommented to enable secure HDFS if using privileged ports # to provide authentication of data transfer protocol. This **MUST NOT** be # defined if SASL is configured for authentication of data transfer protocol # using non-privileged ports. export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER} # Where log files are stored. $HADOOP_HOME/logs by default. #export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER # Where log files are stored in the secure data environment. export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
### # HDFS Mover specific parameters ### # Specify the JVM options to be used when starting the HDFS Mover. # These options will be appended to the options specified as HADOOP_OPTS # and therefore may override any similar flags set in HADOOP_OPTS # # export HADOOP_MOVER_OPTS=""
### # Advanced Users Only! ###
# The directory where pid files are stored. /tmp by default. # NOTE: this should be set to a directory that can only be written to by # the user that will run the hadoop daemons. Otherwise there is the # potential for a symlink attack. export HADOOP_PID_DIR=${HADOOP_PID_DIR} export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
# A string representing this instance of hadoop. $USER by default export HADOOP_IDENT_STRING=$USER
The file can be located at/usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/core-site.xml
from:
to:
hadoop.tmp.dir/usr/local/Cellar/hadoop/hdfs/tmpA base for other temporary directories.fs.default.namehdfs://localhost:9000
Edit mapred-site.xml
The file can be located at /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/mapred-site.xml The file doesn't exist by default. Need to create it from mapred-site.xml.template
mapred.job.trackerlocalhost:9010
Edit hdfs-site.xml
The file can be located at /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/hdfs-site.xml.
dfs.replication1
Create and edit your ~/.profile
The .profile file in MacOSX works exactly how you would expect. Simply create the .profile file in your user directory if it doesn't exist.
There is no ~/.profile by default.
When you start your Terminal.app, it loads a shell – in your case Bash. Shells can have configuration files, and .profile is one of these. They are read when the shell starts up, in a specific order. You can read man bash, section "Invocation", for more details.*
touch ~/.profile
open -e !$
Save it, and the next time you start a new instance of Bash (e.g. by opening a new Terminal window), your ~/.profile will be read and all commands therein will be executed.
* Note that once you create a file called ~/.bash_profile, your ~/.profile will not be read anymore.
.profile works with sh or bash. However .bash_profile is bash-specific and will override .profile if present.
In macOS Sierra the .bash_profile exists in the home directory (its empty by default)
Add the following two commands.
alias hstart="/usr/local/Cellar/hadoop/2.6.0/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/2.7.2/sbin/start-yarn.sh"
alias hstop="/usr/local/Cellar/hadoop/2.6.0/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/2.7.2/sbin/stop-dfs.sh"
. ~/.bash_profile to reload the bash profile.
Format the hdfs file before executing Hadoop
$ hdfs namenode -format
16/09/06 13:20:22 INFO namenode.NameNode: STARTUP_MSG:
Command
$ jps
7379 DataNode
7459 SecondaryNameNode
7316 NameNode
7636 NodeManager
7562 ResourceManager
7676 Jps
$ yarn // For resource management more information than the web interface.
$ mapred // Detailed information about jobs
MySQL 5.6.22 is already installed. Versions (2018/03/14) Hadoop 3.0.0 (Since my earlier installed version of Hadoop was 2.7 , I upgraded it to 3.0.0 using brew)
$ tar zxvf mysql-connector-java-5.1.35.tar.gz $ sudo cp mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar /usr/local/Cellar/hive/2.7.1/libexec/lib/
Setup MySQL database
$ mysql mysql> CREATE DATABASE metastore; mysql> USE metastore; mysql> CREATE USER 'hiveuser'@'localhost' IDENTIFIED BY 'password'; mysql> GRANT SELECT,INSERT,UPDATE,DELETE,ALTER,CREATE ON metastore.* TO 'hiveuser'@'localhost';
Copy hive-default-xml to hive-site.xml
$ cd /usr/local/Cellar/hive/2.7.1/libexec/conf $ cp hive-default.xml.template hive-site.xml
Edit following lines in hive-site.xml
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost/metastore</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hiveuser</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>password</value> </property> <property> <name>datanucleus.fixedDatastore</name> <value>false</value> </property> <property> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <value>/tmp/hive</value> <description>Temporary local directory for added resources in the remote file system.</description> </property> <property> <name>hive.querylog.location</name> <value>/tmp/hive</value> <description>Location of Hive run time structured log file</description> </property>
Run hive
$ hive hive > show tables;
Install Apache HBase on Mac
Prerequisite
Zookeeper: brew install zookeeper
To have launchd start zookeeper now and restart at login:
brew services start zookeeper
Or, if you don't want/need a background service you can just run:
==>Downloading from https://d29vzk4ow07wi7.cloudfront.net/c640eade77c3ad69fef4d66872bbccc2e8782fcd5beee84ecb6c5b7dbb28081b?response-content-disposition=attachment%3Bfilename%3D%22openjdk%4011-11.
==>Downloading from https://d29vzk4ow07wi7.cloudfront.net/b18c993c5f4bffe90725bec08d91869878ade5c407d51d93567fbbbd5a1754da?response-content-disposition=attachment%3Bfilename%3D%22hbase-2.3.2.cata
==>Successfully started `hbase` (label: homebrew.mxcl.hbase)
Validating Install
hbase shell
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/3.3.0/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/2.3.2/libexec/lib/client-facing-thirdparty/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.hbase.util.UnsafeAvailChecker (file:/usr/local/Cellar/hbase/2.3.2/libexec/lib/hbase-common-2.3.2.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.hbase.util.UnsafeAvailChecker
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-11-04 15:48:11,821 WARN[main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.3.2, r8dc7de1c75ab615b8992b928150e8b476cb053e3, Sun Sep 20 13:28:26 UTC 2020
Choose- What's right for you... Not what you are lured with!!
We are often confused by our needs. Sometimes we do not have options and have to trust what we get as the best and some other times we are showered with so many offers that we are not sure whether what we are choosing is actually what we want. Brand names, attractive features, extravagant style often blinds our judgment and we end up with a rather overpriced superficially complex, and high-maintenance seeking product.
Typical MDM Project - Business Value realization
The goal of any MDM implementation is to realize business value. The following diagram shows how a typical MDM solution is implemented.
There are multiple challenges that need to be overcome in order to realize "Mastering the Right Data Management Solution".
First Challenge – Choosing the Right MDM Solution
So how do we choose?
A very innocent question can result in endless nights of a brainstorming session. There is no 1-Fit-All Tool. First of all, we should understand that each enterprise has a unique way of operating. Although we can broadly categorize different types of industries they all in their own way are unique. And that's the beauty as well as the sole reason for their existence. This means that they cannot have the same exact problems to address.
On one hand, there are organizations who want to streamline the business operation to reduce fraudulent activities and hence keeps on introducing various norms and guidelines which an enterprise has to adhere to without any choice on the other hand there are some critical business-related and revenue-generating operations which will need specific governance and management rules.
Hence what you need is not a tool but a solution to your problem and an MDM Solution comprises of three main part:
1. The MDM Tool
2. The MDM Architecture
3. The expertise of the Consultants on board.
We are all about Data. So the first and foremost thing we need is an IT-Savvy Business leader who knows his data and is prepared to take ownership.
From an MDM practice point of view, an MDM consultant can guide you with the right architecture or the right practice but you being the business owner should be the ultimate person to make a decision whether or not that is what you want and hence we should follow the following points.
1. Should know your data and its value
2. Always have the big picture in Mind
3. Think Big but start small. Always remember that Rome wasn't built in a day.
4. Before you are blinded by the flashy terms like Multi-Domain, Big-Data, Dynamic-Hierarchy. You must think that do you even need them? Or are they going to generate any extra bucks for you? Or don't you already have an in-house solution that is already delivering the same?
5. How can you make the best use of the already existing resources and still get the most robust and trusty solution
Second Challenge – Choosing the Right MDM Architecture
It is of utmost critical to understand what type of data problem do you have if at all you do have one. It is important because when we speak Mastering the data and creating a "Golden" copy of Information or "The best version of Truth". However, you may put it but the fact is it's not a single team job. It's an amalgamation of multiple task forces. Hence Enterprise architectural integration is also very critical. But let's stick to the issue at hand. Which is MDM. So what type of MDM do we need. In order to know that we first need to understand what are the different implementation patterns available.
Consolidation
Registry
Centralization
Co-existence
Consolidation
When we are aware that there are silos of the same or similar data in the enterprise and we need to harmonize them to create a golden copy in-fact the aim is to identify the best version among all store information. And in order to do that we need to bring them all to a central location. Then de-dupe it and execute survivorship to generate the Golden record. At times the defined rules in the system are not enough and we will need some expert intervention to resolve duplicities or even to realize whether or not they are duplicates at all like identifying false positives and dealing with them. Typically large enterprises that have adopted digitalization long back and have collected information using multiple sources like ERP, CRM, Web Apps, Websites, POS, etc... end up with this sort of problem.
So, if we need consolidation then the MDM solution should be capable of sophisticatedly handle complex matching scenarios and should give the developers the flexibility of choices to pick and select different matching algorithms and different matching techniques.
Registry
Sometimes when they realize that they have data silos but they continue to manage and maintain it in silos because the business demand so. However, it is critical for them to understand the presence of the same information across organizations and also to understand what kind of information is available across which system they go for registry implementation. In this case, the MDM solution is responsible to maintain only a map or linkages of similar/same information stored in various systems which can be used for cross-reference purposes.
Centralization
Typically newer organization starts with this type of implementation. It is the best-case scenario where the MDM solution is in-fact the authoring tool. Hence it is very critical to have a strong governance or business process management capability to master the data in a correct fashion. The MDM solution should have data validation and workflow capabilities to address this implementation strategy and they are responsible to provide the mastered golden record to all the respective applications in the organization.
Co-existence
This is the reality in most of the MDM implementation projects. The aim always is to identify data silos and consolidate them. Once consolidated then use the MDM product also as the authoring tool and phase-wise decommissioning other authoring solutions.
So, start with the Consolidation pattern and aim for the Centralization pattern. However lucrative this approach seems to be. It is often not possible as there are specialize COTS or custom full-stack apps that address a very specific and business-critical requirement hence cannot be decommissioned hence co-existence is the only option. Where we not only consolidate information coming from other systems but also enrich them using MDM and then publish it to everyone else.
Third Challenge – Building a Business case for MDM
✓Reconciliation – Reducing the direct costs of bad data ✓Reporting – Ensuring accurate reports for high-quality decision making ✓Regulatory compliance – Maintaining the organization’s good standing with the authorities ✓Risk management - Avoiding business risks
Reconciliation
Reconciliation is the process of diagnosing and then repairing mismatches between an organization’s accounts/groups/datasets/systems/transactions. Without an authoritative source of master and reference data, where do your users go to find the right codes? cost centers? product numbers? Or verify the definition of specific reference data elements? Do you trust the sources your enterprise systems are consuming from?
By not clearly defining a master source, systems and users run the risk of using incorrect or out of date information. All these creating reconciliation problems that need to be fixed later.
People costs How much time do our employees spend on recon? What impact does this have on employee morale? # Employees x Average Cost/Employee/Hour x Average Hours Spent† on Issues annually
Costs due to lack of centralization and ambiguity What’s the cost of independently maintaining master data? Is there value in simplification? What’s the value of a single authoritative source? # of Systems x Average Time (hours)‡ Required to Update System x Average Cost/Hour to Update
†Employee time should include time to report issues; diagnose, research, and repair errors; and updates to avoid future issues.
‡Other costs to include are the time required to make changes to the synchronization code (everywhere it exists) when there are system updates
Reporting
Enterprise hierarchies, dimensions, and attributes are used to aggregate or subdivide the source system data presented in reports. When reporting across systems, the individual source system dimensions must be standardized (conformed). Without MDM there is no enterprise source of reporting metadata (conformed dimensions and hierarchies). This means the reporting team prepares for reporting by manually conformation/mapping dimensions. Manual efforts add time to the reporting process and introduce errors and reduce flexibility.
Costs associated with report preparation How long does it take to prepare the reporting metadata (master data, dimensions, and hierarchies/rollups)? Does anyone review (or approve) the prep? Is this work is duplicated throughout the organization? # Reports x # Reports per Year x # Hours Spent on Prep† per Report x Weighted Average Cost per Hour
Other questions How are adjustments (reporting quality issues) managed? Do the same issues recur? Is root cause analysis performed on each issue? oAre ad-hoc reports easy to create? Are there requirements to report on-demand? Should there be (i.e. For crisis management reasons)? Does the business want more control (e.g. self-service hierarchy management)? timely/easier to create reports?
†Time spent on prep should include all the time ironing out the inconsistency in the dimensions. This might include creating/merging new codes, comparing/updating hierarchies, and building mappings (or crosswalks) between the different taxonomies used in the organization. Mismatching taxonomies are often found in global companies where each region and country may choose to use different product hierarchies.
Regulatory Compliance
Many organizations need to comply with the region, country, or industry-specific data governance, data quality, and data management regulations. Examples include European privacy rules, US Healthcare (HIPAA, HITECH), Global Banking (BCBS 239, and USA’s DFA). MDM supports compliance by acting as a place to coordinate policies (governance), their scope (system metadata), implementation (data management), and measure and report on data quality. In addition to traditional data governance policies, some groups may include their privacy policies and their intersection with data domains and security classifications. Without an integrated foundation, gaps are created between each element of the data governance and data management framework.
Compliance costs How will the regulator assess compliance? Are there benchmarks for compliance? How much flexibility do we have in terms of compliance? What are the penalties for non-compliance or poor compliance? Are the penalties civil or criminal in nature? Can employees or executives be found personally culpable? Does non-compliance create legal standing for customers/suppliers/partners to pursue legal action against us?
Costs due to a lack of integration with data governance How are our data governance policies created/maintained? Are these policies integrated with data management systems (can we track them)? How do we policy exceptions? Measure KPIs?
How are issues created? managed through to a resolution?
Risk Management
Master data are an important enterprise data asset because they represent the key elements of your business model: what, to whom, and how, and how to account for what you sell. Without master data management low-quality data proliferates. No MDM means few formal controls on the creation, update, and retirement of master data translates into a lack of review and accuracy checks on customer data, product information, and financial accounts. (Incidentally, this is, in and of itself, a form of operational risk). Also, without MDM, there’s often confusion about which system has the authoritative master data. Confusion means some teams will choose the wrong source. And given that master data is shared, this poor quality data will continue to create more low-quality data as it is shared.
Do we know of situations where poor quality master data affected our organization’s operations? What is the frequency of these problems? Where were the issues: In-bound logistics? Operations? Distribution? Sales/marketing? Services? Do we know what the root cause of the errors was? Were they tied to poor operational controls around the master data? Did the organization suffer any reputational damage in the eyes of our customers, suppliers, distributors, or employees? Did the failure create any financial issues for the firm? Did supplier terms change? Did customers demand deeper discounts? Did this change customer/supplier/employee retention? How do our failure rates compare to our competitors? Industry benchmarks? If we improve beyond the benchmark, can it be used as a point of differentiation?
Do we know if situations where poor quality master data has affected or organization's strategic planning/budgeting process?
Fourth Challenge – Choosing the Right MDM Tool
We need to ensure that the following points are realized.
✓MDM/RDM can help an organization avoid costs, reduce risks and create value ✓The 4-Rs (reconciliation, reporting, regulatory compliance … and risk management) are all areas to examine when building business cases ✓Not all value (or costs) are quantifiable—understanding qualitative value often just as important as quantifying cost savings.
Selecting The Right Tool
Modular MDM Solution Vs Unified MDM Solution
Master Data is the most important data for any enterprise. And as rightly defined in Wikipedia: In business, master data management (MDM) comprises the processes, governance, policies, standards, and tools that consistently define and manage the critical data of an organization to provide a single point of reference.
The concept of MDM has been in the industry for almost a decade now. I remember the initial days of MDM. It was all about Customer Data and Product data as two separate domains. There were two different originating points. Customer Data Integration (CDI) and Product Information Management (PIM) [That's what my older projects were known as]. Just like wine, with time the concept has matured and has been bettered in so many ways. People (read CTO/CFO/COO/Architects/Project Leads) started understanding the MDM ecosystem and the challenges that come along with it.
It didn't take forever to realize that just having a match & merge module or just having workflows for creating a golden record is not going to be enough. Hence the modern-day MDM tools started evolving themselves and started addressing each and every aspect of MDM.
Like.
Data consolidation via match & merge.
Granular survivorship capability.
Excellent UX for stewardship
Data governance via workflows and identity management.
Data security at various levels. Data lineage.
Hierarchical representation of data.
The capability of customizing UI for rich UX.
Data distribution patterns using data objects, web services, REST, SQL. Batch etc.
Reporting capabilities
Integration capabilities with legacy and modern systems
Cloud-based
Scalability and high availability.
The MDM product companies aka product vendors now have this vision. Thanks to either visionary in their team or thanks to the analyst firms who score them based on completeness. The question was how to be the most complete tool. How to have all the checkboxes ticked in order to be the most sought after "Enterprise Master Data Solution".
The race started a few years back and the goal was clear. There was no rule but the message was loud and clear "If you can't evolve you got to lose". Darwin's survival of fittest has now found a new meaning. By this time there were two major groups of MDM product vendors. "The Pioneers" and "The Challengers". In the era of PIM and CDI, pioneers were also the leaders. But the new wave of finding mature, complete, and smart MDM tools for the enterprise has erased the demarcation between the Pioneers and challengers and made the market more competitive. Two types of players emerged out of this competition. One who has the money to acquire or merge in order to overcome its deficiencies. And the other kind who has the intelligence and skill to develop and equip itself with features they used to think was not required.
It is a win-win situation for the consumers or the business enterprise. All they need to decide is whom do they want.
Do they want:
a) Companies that provide complete Enterprise package and have dedicated experts or SMEs for each module.
b) Companies that provide unified solution and have dedicated experts or SME's for the single solution.
The game is quite simple now. If you want to lure the client you should be able to impress the techies or the business users using the tool. The game is now being played in the conference room pilot and not in the golf ground. There are some soft corner and legacy driven decision that still takes place but in the long run, those decisions seldom give up under pressure and expectation.
In order to select the right product, there are two broad criteria.
Based on Domain
Based on the Implementation Pattern
Based on Domain
We need to select a product/solution which can seamlessly handle multiple domains either business or technical.
Business domains are like -
Customer
Vendor
Asset
Product
Location
Pricing etc.
Technical domains are like -
Master Data
Meta Data
Reference Data
Transactional Data
We need to pick a tool that must address multi-domain and not multiple domains. You don't want your IT team to undergo the painful process of implementing separate MDM for each domain.
I have mentioned the same in one of my previous blogs which I published last year (2012) as to why Multi-Domain MDM is so important.
Based on the Implementation Pattern
As explained in the architectural section above. We must identify a solution that should support all or at least almost all types.
So, keep an eye on a solution that supports a multi-domain and multi-implementation strategy.
Fifth/Final Challenge – Implementation
While implementing an MDM Solution we need to focus on the fact that it should lay a strong foundation for operational excellence by implementing an agile and competitive solution that will ensure that you have quality access to master reference and master data domains.
Establish master data priorities as part of the larger data and technology strategy
Embraced the master data standards throughout the organization
Enabling governance by mandating a stewardship process for maintaining master and reference data using workflows.
Implement the change process consistently across the enterprise
Establish roles and responsibilities using data integrity.
We
are mere human beings and when we want to find the right match we have so many
tools at our disposal and that too without even our knowledge. That’s God's
power of matching algorithm, you know you are using it but you don't realize
its complexity because you are so much overwhelmed by its simplicity.
Matching
engines are not unknown any more. In computing world when we talk about match
and merge we try to quantify and qualify information under groups.
Information’s no longer remains information they are data and there are
attributes/nature of those data that we try manipulate, compute and calculate. From
1918 till this date, ever since Soundex was patented there are series of
algorithms that mathematicians and computing engineers have tried creating but
the very base remained same. In order to compute the likeness of a word we need
to compute it mainly in two forms
1. Exact, which is very much like stripping
the word into ASCI or binary and comparing them.
2. Fuzzy/Phonetic/Similar,
which is to create codes out of its pronunciation to match, which is indeed
complex but very matured.
Now lets take it to a level above, what happens when
we match sentence to sentence? You cannot break every word in a sentence to
compare. Obviously for group of words the algorithm differs but so far we have
the knowledge of tools matching names, and matching addresses. What if we have
to compute a mechanism to match phrases? Well! The complexity is rising in many
fold and so is the awes of the fact that when we use God's match rules we can
apply that to any thing and any form and most of the time without even thinking
that we are doing it. We can apply it to names, to places, to people and their
types, to smells to colors even to relations. Next time if you ever think,
"What's the Right Match?' try thinking how are you deriving that it is the
right match.. most of the time the answer will be it's that guy whom we all
know as the inner self.
All boils down to the simplest form of matching that we can programmatically obtain and is often known as String Matching.
String Matching involves two major type of algorithms:
In order to match a string you will have to identify the string. Pick it and try searching for that string in a given set of strings.
Kind of String Searching: A common string searching method is to select the candidate to match or search across. This will narrow down the list and also the expense to perform the search.
Another common example involves "normalization". For many purposes, a search for a phrase such as "to be" should succeed even in places where there is something else intervening between the "to" and the "be":
More than one space
Other "whitespace" characters such as tabs, non-breaking spaces, line-breaks, etc.
Less commonly, a hyphen or soft hyphen
In structured texts, tags or even arbitrarily large but "parenthetical" things such as footnotes, list-numbers or other markers, embedded images, and so on.
Many symbol systems include characters that are synonymous (at least for some purposes):
Latin-based alphabets distinguish lower-case from upper-case, but for many purposes string search is expected to ignore the distinction.
Many languages include ligatures, where one composite character is equivalent to two or more other characters.
Many writing systems involve diacritical marks such as accents or vowel points, which may vary in their usage, or be of varying importance in matching.
DNA sequences can involve non-coding segments which may be ignored for some purposes, or polymorphisms that lead to no change in the encoded proteins, which may not count as a true difference for some other purposes.
Some languages have rules where a different character or form of character must be used at the start, middle, or end of words.
Finally, for strings that represent natural language, aspects of the language itself become involved. For example, one might wish to find all occurrences of a "word" despite it having alternate spellings, prefixes or suffixes, etc.
Another more complex type of search is regular expression searching, where the user constructs a pattern of characters or other symbols, and any match to the pattern should fulfill the search.
