{"id":2687,"date":"2022-04-10T07:53:45","date_gmt":"2022-04-10T02:23:45","guid":{"rendered":"https:\/\/nolowiz.com\/?p=2687"},"modified":"2022-04-10T08:09:08","modified_gmt":"2022-04-10T02:39:08","slug":"python-list-remove-duplicates","status":"publish","type":"post","link":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/","title":{"rendered":"Python List Remove Duplicates"},"content":{"rendered":"\n<p>We may need to work with lists in Python, sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods. <\/p>\n\n\n\n<h2>Naive method <\/h2>\n\n\n\n<p>First, we will look simple naive method to remove list duplicates. The basic logic is :<\/p>\n\n\n\n<ul><li>Define a result list.<\/li><li>Loop the input list.<\/li><li>Check if the input list item is in the result list, if not append it to the result list.<\/li><\/ul>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nnumbers = &#91;10,2,3,1,5,2,10,3,2,5] #input list\n\nresult = &#91;] # To store the result\n\nfor item in numbers:\n  if item not in result:\n    result.append(item)\n\nprint(f&quot;List after duplicates removal : {result}&quot;)\n<\/pre><\/div>\n\n\n<p><strong>Output<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>List after duplicates removal : &#91;10, 2, 3, 1, 5]<\/code><\/pre>\n\n\n\n<script async=\"\" src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js?client=ca-pub-2735334721002354\" crossorigin=\"anonymous\"><\/script>\n<!-- article-horizontal -->\n<ins class=\"adsbygoogle\" style=\"display:block\" data-ad-client=\"ca-pub-2735334721002354\" data-ad-slot=\"8835878737\" data-ad-format=\"auto\" data-full-width-responsive=\"true\"><\/ins>\n<script>\n     (adsbygoogle = window.adsbygoogle || []).push({});\n<\/script>\n\n\n\n<p><\/p>\n\n\n\n<p>Based on the ordering of the output list items we can remove duplicates by :<\/p>\n\n\n\n<ul><li>Using set ( Not keeping the list insertion order)<\/li><li>Using OrderedDict or dict (Keeps the list insertion order)<\/li><\/ul>\n\n\n\n<h2>1. Using set (Not keeping the list insertion order)<\/h2>\n\n\n\n<p>This is a common way to remove list duplicates. We can use <a href=\"https:\/\/docs.python.org\/3\/library\/stdtypes.html#set\" target=\"_blank\" rel=\"noreferrer noopener\">set<\/a> to remove Python list duplicates. Create a new set instance using the input list. The <span class=\"has-inline-color has-vivid-red-color\">set<\/span> does not allow duplicate elements.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nnumbers = &#91;10,2,3,1,5,2,10,3,2,5]\nnumbers = list(set(numbers))\n\nprint(f&quot;List after duplicates removal : {numbers}&quot;)\n<\/pre><\/div>\n\n\n<p><strong>Output<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>List after duplicates removal : &#91;1, 2, 3, 5, 10]<\/code><\/pre>\n\n\n\n<p>This method will not keep the list output order. We got output <strong>[1, 2, 3, 5, 10]<\/strong>  instead of keeping the correct order <strong>[10, 2, 3, 1, 5]<\/strong>. If we are not concerned about the order of the output we can use this method for duplicates removal.<\/p>\n\n\n\n<h2> 2. Using OrderedDict or dict (Keeps the list insertion order)<\/h2>\n\n\n\n<p>If we want to remove list duplicates by keeping the list insertion order we can use the <span class=\"has-inline-color has-vivid-red-color\">OrderedDict.fromkeys<\/span> function.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nfrom collections import OrderedDict\n\nnumbers = &#91;10,2,3,1,5,2,10,3,2,5]\nresult = list(OrderedDict.fromkeys(numbers))\n\nprint(f&quot;List after duplication removal : {result}&quot;)\n<\/pre><\/div>\n\n\n<p><strong>Output<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>List after duplication removal : &#91;10, 2, 3, 1, 5]<\/code><\/pre>\n\n\n\n<p>We can see that this method preserves the list ordering.<\/p>\n\n\n\n<p>Starting with Python 3.7 <a href=\"https:\/\/mail.python.org\/pipermail\/python-dev\/2017-December\/151283.html\" target=\"_blank\" rel=\"noreferrer noopener\">built-in dictionary(dict) keeps the insertion order<\/a>. So if we are on Python 3.7 or later we can use <span class=\"has-inline-color has-vivid-red-color\">dict.fromkeys<\/span> function to remove list duplicates.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nnumbers = &#91;10,2,3,1,5,2,10,3,2,5]\nresult = list(dict.fromkeys(numbers))\n\nprint(f&quot;List after duplication removal : {result}&quot;)\n<\/pre><\/div>\n\n\n<p><strong>Output<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>List after duplication removal : &#91;10, 2, 3, 1, 5]<\/code><\/pre>\n\n\n\n<p><\/p>\n\n\n\n<script async=\"\" src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js?client=ca-pub-2735334721002354\" crossorigin=\"anonymous\"><\/script>\n<!-- article-horizontal -->\n<ins class=\"adsbygoogle\" style=\"display:block\" data-ad-client=\"ca-pub-2735334721002354\" data-ad-slot=\"8835878737\" data-ad-format=\"auto\" data-full-width-responsive=\"true\"><\/ins>\n<script>\n     (adsbygoogle = window.adsbygoogle || []).push({});\n<\/script>\n\n\n\n<p><\/p>\n\n\n\n<h2>Conclusion<\/h2>\n\n\n\n<p>There you have it, different ways to remove Python list duplicates. If you are not concerned about list order it is better to use <span class=\"has-inline-color has-vivid-red-color\">set<\/span> based method. <a href=\"https:\/\/nolowiz.com\/python-list-append-vs-extend-performance-comparison\/\" target=\"_blank\" rel=\"noreferrer noopener\">You can read about list append vs extend performance here<\/a>. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>We may need to work with lists in Python, sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods. Naive method First, we will look simple naive method to remove list duplicates. The basic logic is : Define a result list. Loop the input list. &#8230; <a title=\"Python List Remove Duplicates\" class=\"read-more\" href=\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/\" aria-label=\"More on Python List Remove Duplicates\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[25],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Python List Remove Duplicates - NoloWiz<\/title>\n<meta name=\"description\" content=\"Sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Python List Remove Duplicates - NoloWiz\" \/>\n<meta property=\"og:description\" content=\"Sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/\" \/>\n<meta property=\"og:site_name\" content=\"NoloWiz\" \/>\n<meta property=\"article:published_time\" content=\"2022-04-10T02:23:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-10T02:39:08+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Rupesh Sreeraman\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https:\/\/nolowiz.com\/#organization\",\"name\":\"NoloWiz\",\"url\":\"https:\/\/nolowiz.com\/\",\"sameAs\":[],\"logo\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/nolowiz.com\/#logo\",\"inLanguage\":\"en\",\"url\":\"https:\/\/nolowiz.com\/wp-content\/uploads\/2021\/01\/cropped-android-chrome-512x512-2.png\",\"contentUrl\":\"https:\/\/nolowiz.com\/wp-content\/uploads\/2021\/01\/cropped-android-chrome-512x512-2.png\",\"width\":512,\"height\":512,\"caption\":\"NoloWiz\"},\"image\":{\"@id\":\"https:\/\/nolowiz.com\/#logo\"}},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/nolowiz.com\/#website\",\"url\":\"https:\/\/nolowiz.com\/\",\"name\":\"NoloWiz\",\"description\":\"Technology news, tips and tutorials\",\"publisher\":{\"@id\":\"https:\/\/nolowiz.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/nolowiz.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#webpage\",\"url\":\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/\",\"name\":\"Python List Remove Duplicates - NoloWiz\",\"isPartOf\":{\"@id\":\"https:\/\/nolowiz.com\/#website\"},\"datePublished\":\"2022-04-10T02:23:45+00:00\",\"dateModified\":\"2022-04-10T02:39:08+00:00\",\"description\":\"Sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods.\",\"breadcrumb\":{\"@id\":\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#breadcrumb\"},\"inLanguage\":\"en\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/nolowiz.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Python List Remove Duplicates\"}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#webpage\"},\"author\":{\"@id\":\"https:\/\/nolowiz.com\/#\/schema\/person\/6ef6f57a69193ce0993d74a8b6ac4414\"},\"headline\":\"Python List Remove Duplicates\",\"datePublished\":\"2022-04-10T02:23:45+00:00\",\"dateModified\":\"2022-04-10T02:39:08+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#webpage\"},\"wordCount\":296,\"publisher\":{\"@id\":\"https:\/\/nolowiz.com\/#organization\"},\"articleSection\":[\"Python\"],\"inLanguage\":\"en\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/nolowiz.com\/#\/schema\/person\/6ef6f57a69193ce0993d74a8b6ac4414\",\"name\":\"Rupesh Sreeraman\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/nolowiz.com\/#personlogo\",\"inLanguage\":\"en\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/0b6c0696ed1695a540102a80daa94ad0?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/0b6c0696ed1695a540102a80daa94ad0?s=96&d=mm&r=g\",\"caption\":\"Rupesh Sreeraman\"},\"sameAs\":[\"http:\/\/nolowiz.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Python List Remove Duplicates - NoloWiz","description":"Sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/","og_locale":"en_US","og_type":"article","og_title":"Python List Remove Duplicates - NoloWiz","og_description":"Sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods.","og_url":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/","og_site_name":"NoloWiz","article_published_time":"2022-04-10T02:23:45+00:00","article_modified_time":"2022-04-10T02:39:08+00:00","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Rupesh Sreeraman","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/nolowiz.com\/#organization","name":"NoloWiz","url":"https:\/\/nolowiz.com\/","sameAs":[],"logo":{"@type":"ImageObject","@id":"https:\/\/nolowiz.com\/#logo","inLanguage":"en","url":"https:\/\/nolowiz.com\/wp-content\/uploads\/2021\/01\/cropped-android-chrome-512x512-2.png","contentUrl":"https:\/\/nolowiz.com\/wp-content\/uploads\/2021\/01\/cropped-android-chrome-512x512-2.png","width":512,"height":512,"caption":"NoloWiz"},"image":{"@id":"https:\/\/nolowiz.com\/#logo"}},{"@type":"WebSite","@id":"https:\/\/nolowiz.com\/#website","url":"https:\/\/nolowiz.com\/","name":"NoloWiz","description":"Technology news, tips and tutorials","publisher":{"@id":"https:\/\/nolowiz.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nolowiz.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en"},{"@type":"WebPage","@id":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#webpage","url":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/","name":"Python List Remove Duplicates - NoloWiz","isPartOf":{"@id":"https:\/\/nolowiz.com\/#website"},"datePublished":"2022-04-10T02:23:45+00:00","dateModified":"2022-04-10T02:39:08+00:00","description":"Sometimes we need to remove duplicates from the list. In this article, we will discuss different list duplicate removal methods.","breadcrumb":{"@id":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#breadcrumb"},"inLanguage":"en","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nolowiz.com\/python-list-remove-duplicates\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nolowiz.com\/"},{"@type":"ListItem","position":2,"name":"Python List Remove Duplicates"}]},{"@type":"Article","@id":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#article","isPartOf":{"@id":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#webpage"},"author":{"@id":"https:\/\/nolowiz.com\/#\/schema\/person\/6ef6f57a69193ce0993d74a8b6ac4414"},"headline":"Python List Remove Duplicates","datePublished":"2022-04-10T02:23:45+00:00","dateModified":"2022-04-10T02:39:08+00:00","mainEntityOfPage":{"@id":"https:\/\/nolowiz.com\/python-list-remove-duplicates\/#webpage"},"wordCount":296,"publisher":{"@id":"https:\/\/nolowiz.com\/#organization"},"articleSection":["Python"],"inLanguage":"en"},{"@type":"Person","@id":"https:\/\/nolowiz.com\/#\/schema\/person\/6ef6f57a69193ce0993d74a8b6ac4414","name":"Rupesh Sreeraman","image":{"@type":"ImageObject","@id":"https:\/\/nolowiz.com\/#personlogo","inLanguage":"en","url":"https:\/\/secure.gravatar.com\/avatar\/0b6c0696ed1695a540102a80daa94ad0?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/0b6c0696ed1695a540102a80daa94ad0?s=96&d=mm&r=g","caption":"Rupesh Sreeraman"},"sameAs":["http:\/\/nolowiz.com"]}]}},"_links":{"self":[{"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/posts\/2687"}],"collection":[{"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/comments?post=2687"}],"version-history":[{"count":35,"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/posts\/2687\/revisions"}],"predecessor-version":[{"id":2723,"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/posts\/2687\/revisions\/2723"}],"wp:attachment":[{"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/media?parent=2687"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/categories?post=2687"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nolowiz.com\/wp-json\/wp\/v2\/tags?post=2687"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}