Companies of all sorts store their data for years in different data sources. They can face  major challenges with analyzing, managing, and organizing their data. These challenges also include those related to data discovery and data governance, such as those outlined in the General Data Protection Regulation (GDPR).

Finding all similar or duplicate data across large data sources is challenging. Clustering similar columns can facilitate the management of data, and is always helpful during data analysis. Clustered columns can be classified and labeled across the system, allowing convenient management and analysis of the stored data. Moreover, it can assist in removing duplicate (or near duplicate) data, and help improve data storage. The results can be represented in a graph database, allowing fast querying, and can find similar data subsets.

Common approaches usually rely on schema matching (regex) or column name similarity, however this talk presents a method based on the data content of the columns with no need for prior knowledge. The method gives a solution to data pattern matching (pattern-based data similarity) and data content matching in a scalable manner.

Due to the importance of performance and scale, pairwise comparison cannot be conducted. Instead, proxy hash fingerprints, based on the column values and metadata, are used. Comparing the proxy hash fingerprints is a good estimator of similarity tests, such as Cosine and Jaccard similarity applied between vectors representing the original data subsets (e.g., columns). Moreover, it significantly reduces the space search and scales up the search and comparison.

Speaker Meghan Mergui worked on this project from research to productization and a patent as part of her work at BigID. She tested her algorithm on synthetic data. The product is used by all customers and is one of the key and major components of the company. She will present  the hashes used in the fingerprints, and the algorithm itself with approval from her previous employer.