{"id":16205,"date":"2020-07-21T10:07:30","date_gmt":"2020-07-21T17:07:30","guid":{"rendered":"https:\/\/engineering.fb.com\/?p=16205"},"modified":"2020-07-22T10:08:48","modified_gmt":"2020-07-22T17:08:48","slug":"data-classification-system","status":"publish","type":"post","link":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/","title":{"rendered":"Scalable data classification for security and privacy"},"content":{"rendered":"

What the research is:<\/span><\/h2>\n

We\u2019ve built a data classification system that uses multiple data signals, a scalable system architecture, and machine learning to detect semantic types within Facebook at scale. This is important in situations where it\u2019s necessary to detect where an organization\u2019s data is stored in many different formats across various data stores. In these cases, a classification system enables organizations to automatically enforce privacy- and security-related policies, such as access control policies. Our new system scales to both persistent and nonpersistent user data, with no additional constraints on the type or format of data.\u00a0\u00a0<\/span><\/p>\n

Traditional data loss prevention (DLP) systems solve this problem by fingerprinting the data in question and monitoring endpoints for the fingerprinted data. For an organization with a large number of constantly changing data assets, this approach is neither scalable nor effective for discovering which data is where. Instead, the approach described here focuses on scalable infrastructure, multiple data signals, machine learning (ML), and continuous training to solve this challenge.\u00a0\u00a0<\/span><\/p>\n

How it works:<\/span><\/h2>\n

Data often flows through an organization in two ways, so we need two different strategies to detect and classify this data for automated policy enforcement:\u00a0<\/span><\/p>\n

Persisted data:<\/b> For data persisted in offline data stores, our system must first learn about the universe of data assets. The system collects basic metadata for each data store to form a catalog to efficiently retrieve data without overloading clients and other resources used by developers. It reads from our asset metadata, estimates the resources needed to scan each asset, and invokes a job to perform the actual scanning of the asset. <\/span><\/p>\n

Each job is a compiled binary that performs a <\/span>Bernoulli sample<\/span><\/a> on the latest data available. These samples are broken down into individual columns, and features are extracted from each column. The prediction service ingests these features and invokes rule-based and machine learning classification to predict data label(s) for each column. Any downstream processes can either read these predictions from data sets that aggregate all scan job results or from a real-time data catalog API.<\/span><\/p>\n

Online data: <\/b>While the above happens for persisted assets, nonpersisted traffic is part of an organization\u2019s data and also needs to be protected. For this reason, the system provides an online API for generating real-time classification predictions for nonpersisted traffic. This live prediction system is heavily used to classify egress traffic, incoming traffic to ML models, and any real-time data. <\/span><\/p>\n

\"Data
Online and offline prediction flows.<\/figcaption><\/figure>\n

Heuristic rules and ML: <\/b>Using manual heuristics exclusively results in subpar classification accuracy, especially for unstructured data. For this reason, we also built an ML system to deal with classification of unstructured data, such as user-generated content. A modularized PyTorch-based model allows us to quickly and independently change parts of the model \u2014\u00a0without hurting overall performance.<\/span><\/p>\n

\"we
A high level diagram of the ML component of our data classification system.<\/figcaption><\/figure>\n

Why it matters:\u00a0<\/span><\/h2>\n

Organizations generally have a well-defined set of privacy policies aimed at ensuring that people\u2019s privacy is respected. Given these privacy expectations, it is important that organizations have visibility into which metadata is associated with a given piece of information. This visibility allows them to work quickly and make fewer mistakes by providing information to back any given decision.\u00a0<\/span><\/p>\n

Our system provides data type classifications for all data assets across dozens of data sources, which allows us to build enforcement systems to ensure that privacy and security policies are followed. Unlike traditional DLP-like systems with fingerprinted data, the design of our classification system provides the flexibility to easily add detection for additional data types while providing classification results with low latency on a tight memory constraint.\u00a0<\/span><\/p>\n

Looking ahead, we are working on support for unschematized data (files), classifying not only the data type but also the sensitivity level of data, and leveraging self-supervised learning for improving classification accuracy of data classes with few manual labels.<\/span><\/p>\n

Read the full paper:<\/span><\/h2>\n

Secure and scalable data classification<\/span><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

What the research is: We\u2019ve built a data classification system that uses multiple data signals, a scalable system architecture, and machine learning to detect semantic types within Facebook at scale. This is important in situations where it\u2019s necessary to detect where an organization\u2019s data is stored in many different formats across various data stores. In […]<\/p>\n

Read More…<\/a><\/p>\n","protected":false},"author":1,"featured_media":16208,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[65,72,47],"tags":[],"coauthors":[1608,1609,1610,385],"class_list":["post-16205","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-infrastructure","category-ml-applications","category-security","fb_content_type-article"],"yoast_head":"\nScalable data classification for security, privacy - Engineering at Meta<\/title>\n<meta name=\"description\" content=\"We've built a data classification system to detect where an organization\u2019s data is located within different systems in a fast and scalable manner.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nikolay Pavlovich Laptev, Paulo Tanaka, Sameet Sapra, changigi649\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/\"},\"author\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#author\",\"name\":\"\"},\"headline\":\"Scalable data classification for security and privacy\",\"datePublished\":\"2020-07-21T17:07:30+00:00\",\"dateModified\":\"2020-07-22T17:08:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/\"},\"wordCount\":677,\"publisher\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/engineering.fb.com\\\/wp-content\\\/uploads\\\/2020\\\/07\\\/Blue-image.jpg\",\"articleSection\":[\"Data Infrastructure\",\"ML Applications\",\"Security & Privacy\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/\",\"url\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/\",\"name\":\"Scalable data classification for security, privacy - Engineering at Meta\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/engineering.fb.com\\\/wp-content\\\/uploads\\\/2020\\\/07\\\/Blue-image.jpg\",\"datePublished\":\"2020-07-21T17:07:30+00:00\",\"dateModified\":\"2020-07-22T17:08:48+00:00\",\"description\":\"We've built a data classification system to detect where an organization\u2019s data is located within different systems in a fast and scalable manner.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#primaryimage\",\"url\":\"https:\\\/\\\/engineering.fb.com\\\/wp-content\\\/uploads\\\/2020\\\/07\\\/Blue-image.jpg\",\"contentUrl\":\"https:\\\/\\\/engineering.fb.com\\\/wp-content\\\/uploads\\\/2020\\\/07\\\/Blue-image.jpg\",\"width\":1920,\"height\":1080,\"caption\":\"Scalable data classification for security and privacy\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/2020\\\/07\\\/21\\\/security\\\/data-classification-system\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/engineering.fb.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Scalable data classification for security and privacy\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/#website\",\"url\":\"https:\\\/\\\/engineering.fb.com\\\/\",\"name\":\"Engineering at Meta\",\"description\":\"Engineering at Meta Blog\",\"publisher\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/engineering.fb.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/#organization\",\"name\":\"Meta\",\"url\":\"https:\\\/\\\/engineering.fb.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/engineering.fb.com\\\/wp-content\\\/uploads\\\/2023\\\/08\\\/Meta_lockup_positive-primary_RGB.jpg\",\"contentUrl\":\"https:\\\/\\\/engineering.fb.com\\\/wp-content\\\/uploads\\\/2023\\\/08\\\/Meta_lockup_positive-primary_RGB.jpg\",\"width\":29011,\"height\":12501,\"caption\":\"Meta\"},\"image\":{\"@id\":\"https:\\\/\\\/engineering.fb.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Engineering\\\/\",\"https:\\\/\\\/x.com\\\/fb_engineering\"]},[]]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Scalable data classification for security, privacy - Engineering at Meta","description":"We've built a data classification system to detect where an organization\u2019s data is located within different systems in a fast and scalable manner.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/","twitter_misc":{"Written by":"Nikolay Pavlovich Laptev, Paulo Tanaka, Sameet Sapra, changigi649","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#article","isPartOf":{"@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/"},"author":{"@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#author","name":""},"headline":"Scalable data classification for security and privacy","datePublished":"2020-07-21T17:07:30+00:00","dateModified":"2020-07-22T17:08:48+00:00","mainEntityOfPage":{"@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/"},"wordCount":677,"publisher":{"@id":"https:\/\/engineering.fb.com\/#organization"},"image":{"@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#primaryimage"},"thumbnailUrl":"https:\/\/engineering.fb.com\/wp-content\/uploads\/2020\/07\/Blue-image.jpg","articleSection":["Data Infrastructure","ML Applications","Security & Privacy"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/","url":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/","name":"Scalable data classification for security, privacy - Engineering at Meta","isPartOf":{"@id":"https:\/\/engineering.fb.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#primaryimage"},"image":{"@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#primaryimage"},"thumbnailUrl":"https:\/\/engineering.fb.com\/wp-content\/uploads\/2020\/07\/Blue-image.jpg","datePublished":"2020-07-21T17:07:30+00:00","dateModified":"2020-07-22T17:08:48+00:00","description":"We've built a data classification system to detect where an organization\u2019s data is located within different systems in a fast and scalable manner.","breadcrumb":{"@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#primaryimage","url":"https:\/\/engineering.fb.com\/wp-content\/uploads\/2020\/07\/Blue-image.jpg","contentUrl":"https:\/\/engineering.fb.com\/wp-content\/uploads\/2020\/07\/Blue-image.jpg","width":1920,"height":1080,"caption":"Scalable data classification for security and privacy"},{"@type":"BreadcrumbList","@id":"https:\/\/engineering.fb.com\/2020\/07\/21\/security\/data-classification-system\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/engineering.fb.com\/"},{"@type":"ListItem","position":2,"name":"Scalable data classification for security and privacy"}]},{"@type":"WebSite","@id":"https:\/\/engineering.fb.com\/#website","url":"https:\/\/engineering.fb.com\/","name":"Engineering at Meta","description":"Engineering at Meta Blog","publisher":{"@id":"https:\/\/engineering.fb.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/engineering.fb.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/engineering.fb.com\/#organization","name":"Meta","url":"https:\/\/engineering.fb.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/engineering.fb.com\/#\/schema\/logo\/image\/","url":"https:\/\/engineering.fb.com\/wp-content\/uploads\/2023\/08\/Meta_lockup_positive-primary_RGB.jpg","contentUrl":"https:\/\/engineering.fb.com\/wp-content\/uploads\/2023\/08\/Meta_lockup_positive-primary_RGB.jpg","width":29011,"height":12501,"caption":"Meta"},"image":{"@id":"https:\/\/engineering.fb.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Engineering\/","https:\/\/x.com\/fb_engineering"]},[]]}},"jetpack_featured_media_url":"https:\/\/engineering.fb.com\/wp-content\/uploads\/2020\/07\/Blue-image.jpg","jetpack_shortlink":"https:\/\/wp.me\/pa0Lhq-4dn","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/posts\/16205","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/comments?post=16205"}],"version-history":[{"count":7,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/posts\/16205\/revisions"}],"predecessor-version":[{"id":16337,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/posts\/16205\/revisions\/16337"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/media\/16208"}],"wp:attachment":[{"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/media?parent=16205"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/categories?post=16205"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/tags?post=16205"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/engineering.fb.com\/wp-json\/wp\/v2\/coauthors?post=16205"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}