Sometimes you need to send complex data over the network, save the state of the data into a file to keep in the local disk or database, or cache the data of expensive operation, in that case, you need to serialize the data.<\/p>\n\n\n\n
Python has a standard library called In this article, you’ll learn about data serialization and deserialization, the pickle module’s key features and how to serialize and deserialize objects, the types of objects that can and cannot be pickled, and how to modify the pickling behavior in a class.<\/p>\n\n\n\n Well, serialization refers to the process of converting the data into a format that can be easily stored, transmitted, or reconstructed for later use.<\/p>\n\n\n\n Pickling is the name given to the serialization process in Python, where Python objects are converted into a byte stream. Unpickling, also known as deserializing, is the inverse operation in which byte data is converted back to its original state, reconstructing the Python object hierarchy.<\/p>\n\n\n\n Pickling and unpickling are Python-specific operations that require the use of the The The The The The Consider the following scenario: pickling the data and saving it to a file, then unpickling the serialized object from that file to reassemble it in its original form.<\/p>\n\n\n\n The above code serializes the The serialized data is then deserialized from the Take a look at another example in which you have a class that contains multiple operations.<\/p>\n\n\n\n In the above code, an object of the The object Then, the serialized data ( This demonstrates that the deserialization process successfully reconstructed the object.<\/p>\n\n\n\n The However, not all types of objects are picklable. Certain types of objects, for example, file handles, sockets, database connections, and custom classes that lack necessary methods (such as Here’s an example of attempting to pickle a database connection.<\/p>\n\n\n\n When you run this code, you will receive a Similarly, functions that are not defined with the The above code is attempting to pickle the Let’s say you have a class that contains different attributes and some of them are unpicklable. In that case, you can override the If you directly run the above code, the result will be an error due to the To tackle this kind of situation, you can influence the pickling process of the class instance using the In the above example, the When you run the above example, you will get the dictionary containing the results of the attributes.<\/p>\n\n\n\n Now if you want the excluded In the above code, the When you run the above code, you will see the dictionary having the As you know, a variety of objects are unpicklable. Here’s an example that shows how you can pickle the database connection object by modifying the pickling behavior of the class.<\/p>\n\n\n\n The above code defined a class In addition, three new methods are added: Now exclude the database connection from the pickling process using the The The The database connection object is then pickled, which is then unpickled and printed.<\/p>\n\n\n\n As you can see, the dictionary of the object only contains the database name. The connection and cursor objects have been removed.<\/p>\n\n\n\n The Within the To check if the pickling process works, the When you run the whole script, you will obtain the following output.<\/p>\n\n\n\n As you can see, everything went well, and the object’s dictionary now has both a connection and a cursor object along with the database name, demonstrating the successful unpickling of the database connection.<\/p>\n\n\n\n Keep in mind that if the While the ability to customize the So, what can you do to reduce the security risk? You can’t do much, but you can make sure that data from untrustworthy sources isn’t unpickled. Validate the authenticity of the pickled data during unpickling by using cryptographic signatures to ensure that it has not been tampered with, and if possible, sanitize the pickled data by checking for malicious content.<\/p>\n\n\n\n The pickle module lets you serialize and deserialize the data and now you know how to do it using the pickle module. You can now convert the object data into bytes that can be transmitted over a network or saved into disk for the future.<\/p>\n\n\n\n In this article, you’ve learned:<\/p>\n\n\n\n \ud83c\udfc6Other articles you might be interested in if you liked this one<\/strong><\/p>\n\n\n\n \u2705Hash passwords using the bcrypt library in Python<\/a>.<\/p>\n\n\n\n \u2705How to use pytest to test your Python code<\/a>?<\/p>\n\n\n\n \u2705Create a WebSocket server and client in Python<\/a>.<\/p>\n\n\n\n \u2705Create multi-threaded Python programs using a threading module<\/a>.<\/p>\n\n\n\n \u2705Create and integrate MySQL database with Flask app using Python<\/a>.<\/p>\n\n\n\npickle<\/code> that helps you perform the serialization and de-serialization process on the Python objects.<\/p>\n\n\n\nObject Serialization<\/h2>\n\n\n\n
The pickle Module<\/h2>\n\n\n\n
pickle<\/code> module.<\/p>\n\n\n\npickle<\/code> module includes four functions for performing the pickling and unpickling processes on objects:<\/p>\n\n\n\npickle.dump(obj, file)<\/code><\/td>pickle.load(file)<\/code><\/td><\/tr>pickle.dumps(obj)<\/code><\/td>pickle.loads(data)<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\npickle.dump()<\/code> function is used to write the serialized byte representation of the object into a specified file or file-like object.<\/p>\n\n\n\n\n
obj<\/code>: The object to be serialized.<\/li>\n\n\n\nfile<\/code>: The file or file-like object in which the serialized byte representation of the object will be written.<\/li>\n<\/ul>\n\n\n\npickle.dumps()<\/code> function returns the serialized byte representation of the object.<\/p>\n\n\n\n\n
obj<\/code>: The object to be serialized.<\/li>\n<\/ul>\n\n\n\npickle.load()<\/code> function reads the serialized object from the specified file or file-like object and returns the reconstructed object.<\/p>\n\n\n\n\n
file<\/code>: The file or file-like object from which the serialized data is read.<\/li>\n<\/ul>\n\n\n\npickle.loads()<\/code> function returns the reconstructed object from the serialized bytes object.<\/p>\n\n\n\n\n
obj<\/code>: serialized bytes object to reconstruct.<\/li>\n<\/ul>\n\n\n\nHow to Pickle and Unpickle Data<\/h2>\n\n\n\n
import pickle\n\n# Sample data\nmy_data = {\n \"lib\": \"pickle\",\n \"build\": 4.33,\n \"version\": 2.1,\n \"status\": \"Active\"\n}\n\n# Serializing\nwith open(\"lib_info.pickle\", \"wb\") as file:\n pickle.dump(my_data, file)\n\n# De-serializing\nwith open(\"lib_info.pickle\", \"rb\") as file:\n unpickled_data = pickle.load(file)\n\nprint(f\"Unpickled Data: {unpickled_data}\")<\/pre><\/div>\n\n\n\nmy_data<\/code> dictionary and the serialized data is written to a file called lib_info.pickle<\/code> in binary mode (wb<\/code>).<\/p>\n\n\n\nlib_info.pickle<\/code> using the pickle.load()<\/code> function.<\/p>\n\n\n\nUnpickled Data: {'lib': 'pickle', 'build': 4.33, 'version': 2.1, 'status': 'Active'}<\/pre><\/div>\n\n\n\nimport pickle\n\nclass SampleOperation:\n square = 5 ** 2\n addition = 5 + 7\n subtraction = 5 - 7\n division = 14 \/ 2\n\n# Object created\nmy_obj = SampleOperation()\n\n# Serializing\npickled_data = pickle.dumps(my_obj)\nprint(f\"Pickled Data: {pickled_data}\")\n\n# De-serializing\nunpickled_data = pickle.loads(pickled_data)\nprint(f\"Unpickled Data (Division): {unpickled_data.division}\")\nprint(f\"Unpickled Data (Square): {unpickled_data.square}\")\nprint(f\"Unpickled Data (Addition): {unpickled_data.addition}\")\nprint(f\"Unpickled Data (Subtraction): {unpickled_data.subtraction}\")<\/pre><\/div>\n\n\n\nSampleOperation<\/code> class is created and stored in the my_obj<\/code> variable.<\/p>\n\n\n\nmy_obj<\/code> is serialized using the pickle.dumps()<\/code> function and the serialized data is stored in the pickled_data<\/code> variable.<\/p>\n\n\n\npickled_data<\/code>) is deserialized using the pickle.loads()<\/code> function and the attributes of the unpickled object are printed.<\/p>\n\n\n\nPickled Data: b'\\x80\\x04\\x95#\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x08__main__\\x94\\x8c\\x0fSampleOperation\\x94\\x93\\x94)\\x81\\x94.'\nUnpickled Data (Division): 7.0\nUnpickled Data (Square): 25\nUnpickled Data (Addition): 12\nUnpickled Data (Subtraction): -2<\/pre><\/div>\n\n\n\n
What Can be Pickled and Unpickled?<\/h2>\n\n\n\n
pickle<\/code> module can pickle a variety of objects, including strings, integers, floats, tuples, named functions, classes, and others.<\/p>\n\n\n\n__getstate__<\/code> and __setstate__<\/code>), may not be picklable.<\/p>\n\n\n\nimport pickle\nimport sqlite3\n\nconn = sqlite3.connect(\":memory:\")\n# Pickling db connection object\npickle.dumps(conn)<\/pre><\/div>\n\n\n\n
TypeError<\/code> stating that the connection object cannot be pickled.<\/p>\n\n\n\nTypeError: cannot pickle 'sqlite3.Connection' object<\/pre><\/div>\n\n\n\n
def<\/code> keyword, such as the lambda<\/code> function, cannot be pickled using the pickle<\/code> module.<\/p>\n\n\n\nimport pickle\n\nlambda_obj = lambda x: x ** 2\npickle.dumps(lambda_obj)<\/pre><\/div>\n\n\n\n
lambda<\/code> function object, but it will return an error.<\/p>\n\n\n\nTraceback (most recent call last):\n ...\n pickle.dumps(lambda_obj)\n_pickle.PicklingError: Can't pickle <function <lambda> at 0x000001EB55373E20>: attribute lookup <lambda> on __main__ failed<\/pre><\/div>\n\n\n\n
Modify the Pickling Behaviour of the Class<\/h2>\n\n\n\n
__getstate__<\/code> method of the class to choose what you want to pickle during the pickling process.<\/p>\n\n\n\nimport pickle\n\nclass SampleTask:\n def __init__(self):\n self.first = 2**17\n self.second = \"This is a string\".upper()\n self.third = lambda x: x**x\n\nobj = SampleTask()\npickle_instance = pickle.dumps(obj)\nunpickle = pickle.loads(pickle_instance)\nprint(unpickle.__dict__)<\/pre><\/div>\n\n\n\n
lambda<\/code> function defined within the class which is unpicklable.<\/p>\n\n\n\nTraceback (most recent call last):\n ....\n pickle_instance = pickle.dumps(obj)\nAttributeError: Can't pickle local object 'SampleTask.__init__.<locals>.<lambda>'<\/pre><\/div>\n\n\n\n
__getstate__<\/code> method. You can include what to pickle by overriding the __getstate__<\/code> method.<\/p>\n\n\n\nimport pickle\n\nclass SampleTask:\n def __init__(self):\n self.first = 2**17\n self.second = \"This is a string\".upper()\n self.third = lambda x: x**x\n\n def __getstate__(self):\n state = self.__dict__.copy()\n del state['third']\n return state\n\nobj = SampleTask()\npickle_instance = pickle.dumps(obj)\nunpickle = pickle.loads(pickle_instance)\nprint(unpickle.__dict__)<\/pre><\/div>\n\n\n\n
__getstate__<\/code> method is defined, and within this method, a copy of the attributes is made. To exclude the lambda<\/code> function from the pickling process, the attribute named third<\/code> is removed and then the attributes are returned.<\/p>\n\n\n\n{'first': 131072, 'second': 'THIS IS A STRING'}<\/pre><\/div>\n\n\n\nlambda<\/code> expression to appear in the unpickled dictionary above, you can use the __setstate__<\/code> method to restore the state of the class’s object.<\/p>\n\n\n\nimport pickle\n\nclass SampleTask:\n def __init__(self):\n self.first = 2**17\n self.second = \"This is a string\".upper()\n self.third = lambda x: x**x\n\n def __getstate__(self):\n state = self.__dict__.copy()\n del state['third']\n return state\n\n def __setstate__(self, state):\n self.__dict__.update(state)\n self.third = lambda x: x**x\n\nobj = SampleTask()\npickle_instance = pickle.dumps(obj)\nunpickle = pickle.loads(pickle_instance)\nprint(unpickle.__dict__)<\/pre><\/div>\n\n\n\n
__setstate__<\/code> method restores the state of the object. During unpickling, the __setstate__<\/code> method is called to restore the state of the object.<\/p>\n\n\n\nlambda<\/code> function object.<\/p>\n\n\n\n{'first': 131072, 'second': 'THIS IS A STRING', 'third': <function SampleTask.__setstate__.<locals>.<lambda> at 0x000001C54EEB67A0>}<\/pre><\/div>\n\n\n\nCustomizing Pickling: Modifying Class Behavior for Database Connections<\/strong><\/h2>\n\n\n\n
# pickling_db_obj.py\nimport pickle\nimport sqlite3\n\nclass DBConnection:\n def __init__(self, db_name):\n self.db_name = db_name\n self.connection = sqlite3.connect(db_name)\n self.cur = self.connection.cursor()\n \n # Method for creating db table\n def create_table(self):\n self.connection.execute(\"CREATE TABLE IF NOT EXISTS users (name TEXT)\")\n return self.connection\n\n # Method for inserting data into db table\n def create_entry(self):\n self.connection.execute(\"INSERT INTO users (name) VALUES ('Sachin')\")\n res = self.connection.execute(\"SELECT * FROM users\")\n result = res.fetchall()\n print(result)\n return self.connection\n\n # Method for closing db connection\n def close_db_connection(self):\n self.cur.close()\n self.connection.close()<\/pre><\/div>\n\n\n\nDBConnection<\/code>, and the SQLite database connection is initialized within this class.<\/p>\n\n\n\ncreate_table<\/code> (for creating a database table), create_entry<\/code> (for inserting and retrieving data from the table), and close_db_connection<\/code> (for closing the database connection).<\/p>\n\n\n\n__getstate__<\/code> method.<\/p>\n\n\n\n# pickling_db_obj.py\n...\n\n def __getstate__(self):\n state = self.__dict__.copy()\n # Exclude the connection and cursor from pickling\n del state['connection']\n del state['cur']\n return state\n\ndb_conn = DBConnection(\":memory:\")\npickle_db_conn = pickle.dumps(db_conn)\nunpickle_db_conn = pickle.loads(pickle_db_conn)\nprint(unpickle_db_conn.__dict__)<\/pre><\/div>\n\n\n\n
__getstate__<\/code> method creates a copy of the object’s dictionary, then removes the connection (state['connection']<\/code>) and cursor (state['cur']<\/code>) and returns the dictionary (state<\/code>).<\/p>\n\n\n\nDBConnection<\/code> class instance is created and passed the database name (\":memory:\"<\/code>) that will be created in memory.<\/p>\n\n\n\n{'db_name': ':memory:'}<\/pre><\/div>\n\n\n\n__setstate__<\/code> method is now required to restore the object’s original state during unpickling, in which the database connection will be reestablished.<\/p>\n\n\n\n# pickling_db_obj.py\n...\n\n ...\n\n # Restoring the original state of the object\n def __setstate__(self, state):\n self.__dict__.update(state)\n self.connection = sqlite3.connect(self.db_name)\n self.cur = self.connection.cursor()\n\n\ndb_conn = DBConnection(\":memory:\")\npickle_db_conn = pickle.dumps(db_conn)\nunpickle_db_conn = pickle.loads(pickle_db_conn)\n\nunpickle_db_conn.create_table()\nunpickle_db_conn.create_entry()\nunpickle_db_conn.close_db_connection()\n\nprint(unpickle_db_conn.__dict__)<\/pre><\/div>\n\n\n\n
__setstate__<\/code> method, the state<\/code> dictionary is updated and the new database connection and the cursor are created.<\/p>\n\n\n\ncreate_table<\/code>, create_entry<\/code>, and close_db_connection<\/code> methods are called on the unpickled class instance (unpickle_db_conn<\/code>).<\/p>\n\n\n\n[('Sachin',)]\n{'db_name': ':memory:', 'connection': <sqlite3.Connection object at 0x00000240D6F12A40>, 'cur': <sqlite3.Cursor object at 0x00000240D78044C0>}<\/pre><\/div>\n\n\n\n\n
__getstate__<\/code> method returns the false value, the __setstate__<\/code> method will not be called upon unpickling. Source<\/a><\/p>\n<\/blockquote>\n\n\n\n__setstate__<\/code> method during unpickling provides flexibility, it also comes with security considerations. Arbitrary code can be executed during unpickling, which can be a security risk if the pickled data comes from untrusted or malicious sources.<\/p>\n\n\n\nConclusion<\/h2>\n\n\n\n
\n
\n\n\n\n