Pandas supports Copy-on-Write, an optimization technique that helps improve memory use, particularly when working with large datasets.<\/p>\n\n\n\n
Starting from version 2.0 of Pandas, the Copy-on-Write (CoW) has taken effect but has not been fully implemented. Most of the optimizations that are possible through Copy-on-Write are supported.<\/p>\n\n\n\n
As the name suggests, the data will be copied when it is modified. What it means?<\/p>\n\n\n\n
When a DataFrame or Series shares the same data as the original, it will initially share the same memory for the data rather than creating a copy. When the data of either the original or new DataFrame is modified, a new copy of the data is created for the DataFrame that is being modified.<\/p>\n\n\n\n
This will efficiently save memory usage and improve performance when working with large datasets.<\/p>\n\n\n\n
It is not enabled by default, so we need to enable it using the You can use any of the options to turn on CoW globally in your environment.<\/p>\n\n\n\n Note: CoW will be enabled by default in Pandas 3.0, so get used to it early on.<\/strong><\/p>\n\n\n\n The CoW will disallow updating the multiple pandas objects at the same time. Here’s how it will happen.<\/p>\n\n\n\n With CoW, the above snippet will not modify Similarly, the We can see that The above operation can be performed in two different ways. One method is to avoid When we modify the DataFrame or Series using multiple indexing operations in a single line of code, this is what we call the chained assignment technique.<\/p>\n\n\n\n The above code snippet is trying to change column Since the CoW is disabled, this operation is allowed, and the original But, this will never work with CoW enabled in pandas.<\/p>\n\n\n\n Instead, with copy-on-write, we can use This will modify column When a Series or DataFrame is accessed as a NumPy array, that array will be read-only if the array shares the same data with the initial DataFrame or Series.<\/p>\n\n\n\n In the above code snippet, Take a look at this case.<\/p>\n\n\n\n The DataFrame When two or more DataFrames share the same data, the copies will not be created immediately.<\/p>\n\n\n\n Both But this is not necessary, if we don’t want initial This same optimization (lazy copy mechanism) is added to the methods that don’t require a copy of the original data.<\/p>\n\n\n\n DataFrame.rename()<\/strong><\/p>\n\n\n\n When CoW is enabled, this method returns the original DataFrame rather than creating an entire copy of the data, unlike the regular execution.<\/p>\n\n\n\n DataFrame.drop() for axis=1<\/strong><\/p>\n\n\n\n Similarly, the same mechanism is implemented for Pandas will by default implement Copy-on-Write (CoW) in version 3.0. All these optimizations that are compliant with CoW will lead to efficient memory and resource management when working with large datasets.<\/p>\n\n\n\n This will reduce unpredictable or inconsistent behavior and greatly maximize performance.<\/p>\n\n\n\n \ud83c\udfc6Other articles you might be interested in if you liked this one<\/strong><\/p>\n\n\n\n \u2705Merge, combine, and concatenate multiple datasets using pandas<\/a>.<\/p>\n\n\n\n \u2705Find and delete duplicate rows from the dataset using pandas<\/a>.<\/p>\n\n\n\n \u2705Find and Delete Mismatched Columns From DataFrames Using pandas<\/a>.<\/p>\n\n\n\n \u2705Create temporary files and directories using tempfile module in Python<\/a>.<\/p>\n\n\n\n \u2705Upload and display images on the frontend using Flask<\/a>.<\/p>\n\n\n\ncopy_on_write<\/code> configuration option in Pandas.<\/p>\n\n\n\nimport pandas as pd\n\n# Option1\npd.options.mode.copy_on_write = True\n# Option2\npd.set_option(\"mode.copy_on_write\" : True)<\/pre><\/div>\n\n\n\n
Impact of CoW in Pandas<\/h2>\n\n\n\n
import pandas as pd\ndf = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6]})\nsubset = df[\"A\"]\nsubset.iloc[0] = 10\ndf<\/pre><\/div>\n\n\n\ndf<\/code> rather it modifies only the data of subset<\/code>.<\/p>\n\n\n\n# df\n\tA\tB\n0\t1\t4\n1\t2\t5\n2\t3\t6\n\n# subset\n\tA\n0\t10\n1\t2\n2\t3<\/pre><\/div>\n\n\n\n
inplace Operations will Not Work<\/h3>\n\n\n\n
inplace<\/code> operations will not work with CoW enabled, which directly modifies the original df<\/code>.<\/p>\n\n\n\ndf = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6]})\ndf[\"A\"].replace(1, 5, inplace=True)\ndf\n--------------------\n\tA\tB\n0\t1\t4\n1\t2\t5\n2\t3\t6<\/pre><\/div>\n\n\n\ndf<\/code> has remained unchanged and additionally, we will see a ChainedAssignmentError<\/code> warning.<\/p>\n\n\n\ninplace<\/code>, and another is to use inplace<\/code> to directly modify the original df<\/code> at the DataFrame<\/code> level.<\/p>\n\n\n\n# Avoid inplace\ndf = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6]})\ndf[\"A\"] = df[\"A\"].replace(1, 5)\ndf\n--------------------\n A\tB\n0\t5\t4\n1\t2\t5\n2\t3\t6<\/pre><\/div>\n\n\n\n# Using inplace at DataFrame level\ndf = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6]})\ndf.replace({\"A\": {2: 34}}, inplace=True)\ndf\n--------------------\n A\tB\n0\t1\t4\n1\t34\t5\n2\t3\t6<\/pre><\/div>\n\n\n\nChained Assignment will Never Work<\/h3>\n\n\n\n
# CoW disabled\nwith pd.option_context(\"mode.copy_on_write\", False):\n df = pd.DataFrame({\"A\": [1, 2, 3, 4], \"B\": [5, 6, 7, 8]})\n df[\"B\"][df['A'] > 2] = 10\ndf<\/pre><\/div>\n\n\n\nB<\/code> from the original df<\/code> where column A<\/code> is greater than 2. It means the value at the 2nd and 3rd index in column B<\/code> will be modified.<\/p>\n\n\n\ndf<\/code> will be modified.<\/p>\n\n\n\n A\tB\n0\t1\t5\n1\t2\t6\n2\t3\t10\n3\t4\t10<\/pre><\/div>\n\n\n\n
# CoW enabled\ndf = pd.DataFrame({\"A\": [1, 2, 3, 4], \"B\": [5, 6, 7, 8]})\ndf[\"B\"][df[\"A\"] > 2] = 10\ndf\n--------------------\n A\tB\n0\t1\t5\n1\t2\t6\n2\t3\t7\n3\t4\t8<\/pre><\/div>\n\n\n\n.loc<\/code> to modify the df<\/code> using multiple indexing conditions.<\/p>\n\n\n\n# CoW enabled\ndf = pd.DataFrame({\"A\": [1, 2, 3, 4], \"B\": [5, 6, 7, 8]})\ndf.loc[(df[\"A\"] == 1) | (df[\"A\"] > 3), \"B\"] = 100\ndf<\/pre><\/div>\n\n\n\nB<\/code> where column A<\/code> is either 1 or greater than 3. The original df<\/code> will look like the following.<\/p>\n\n\n\n\tA\tB\n0\t1\t100\n1\t2\t6\n2\t3\t7\n3\t4\t100<\/pre><\/div>\n\n\n\n
Read-only Arrays<\/h3>\n\n\n\n
df = pd.DataFrame({\"A\": [1, 2, 3, 4], \"B\": ['5', '6', '7', '8']})\narr = df.to_numpy()\narr\n--------------------\narray([[1, '5'],\n [2, '6'],\n [3, '7'],\n [4, '8']], dtype=object)<\/pre><\/div>\n\n\n\narr<\/code> will be a copy because df contains two different types of arrays (int<\/code> and str<\/code>). We can perform modifications on the arr<\/code>.<\/p>\n\n\n\narr[1, 0] = 10\narr\n--------------------\narray([[1, '5'],\n [10, '6'],\n [3, '7'],\n [4, '8']], dtype=object)<\/pre><\/div>\n\n\n\n
df = pd.DataFrame({\"A\": [1, 2, 3, 4], \"B\": [5, 6, 7, 8]})\narr = df.to_numpy()\narr<\/pre><\/div>\n\n\n\ndf<\/code> has only one NumPy array (array of the same data types), so arr<\/code> shares the data with df<\/code>. This means arr<\/code> will be read-only and cannot be modified in place.<\/p>\n\n\n\nprint(arr.flags.writeable)\narr[0,0] = 10\narr\n--------------------\nFalse\nValueError: assignment destination is read-only<\/pre><\/div>\n\n\n\n
Lazy Copy Mechanism<\/h3>\n\n\n\n
df = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6]})\ndf2 = df.reset_index(drop=True)<\/pre><\/div>\n\n\n\ndf<\/code> and df2<\/code> shares the same reference in the memory as both share the same data. The copy mechanism will trigger only when any of the DataFrame is modified.<\/p>\n\n\n\ndf2.iloc[0, 0] = 10\nprint(df2)\nprint(df)\n--------------------\n A B\n0 10 4\n1 2 5\n2 3 6\n A B\n0 1 4\n1 2 5\n2 3 6<\/pre><\/div>\n\n\n\n
df<\/code>, we can simply reassign it to the same variable (df<\/code>) and this process will create a new reference. This will avoid the copy-on-write process.<\/p>\n\n\n\ndf = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6]})\nprint(\"Initial reference: \",id(df))\ndf = df.reset_index(drop=True)\nprint(\"New reference: \",id(df))\ndf.iloc[0, 0] = 10\nprint(df)\n--------------------\nInitial reference: 138400246865760\nNew reference: 138400246860336\n A B\n0 10 4\n1 2 5\n2 3 6<\/pre><\/div>\n\n\n\ndf = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6]})\ndf.rename(columns={\"A\": \"X\", \"B\": \"Y\"})\n--------------------\n\tX\tY\n0\t1\t4\n1\t2\t5\n2\t3\t6<\/pre><\/div>\n\n\n\nDataFrame.drop()<\/code> for axis=1<\/code> (axis='columns'<\/code>).<\/p>\n\n\n\ndf = pd.DataFrame({\"A\": [1, 2, 3], \"B\": [4, 5, 6], \"C\": [7, 8, 9]})\ndf.drop([\"A\"], axis=1)\n--------------------\n\tB\tC\n0\t4\t7\n1\t5\t8\n2\t6\t9<\/pre><\/div>\n\n\n\nConclusion<\/h2>\n\n\n\n
\n\n\n\n