Reading and Writing JSON Data in Python

JSON (JavaScript Object Notation) has become a ubiquitous data format used for config files, API communication and more. Its simple syntax and native support in most programming languages makes it a popular option for serializing and exchanging data.

As a full-stack Python developer, you‘ll often need to read JSON data from files or APIs and convert that into Python types that can be easily worked with. Similarly, after processing data in Python, you may need to convert it back to JSON format for sending out responses or writing to configuration files.

In this comprehensive guide, we‘ll explore all aspects of handling JSON data in Python.

JSON Format Overview

JSON is a text-based data storage format that uses human-readable syntax to transmit data objects consisting of key-value pairs and list data types (arrays). JSON syntax defines standards for data types like numbers, booleans, null values and is programming language agnostic, meaning data can be easily exchanged between systems written in different languages like Python, JavaScript, Java etc.

Compared to formats like XML, JSON provides a lightweight and compact syntax making it ideal for use in web services, server-side applications and config files. The inbuilt JSON library in Python along with simple syntax makes it one of the most popular choices for handling JSON data in Python applications.

Python Dictionaries and JSON Objects

When JSON data is loaded in Python, it is automatically deserialized and converted into native Python data types. JSON objects become Python dictionary, JSON arrays get converted to Python lists and so on.

This makes it convenient to access and manipulate loaded JSON data using standard Python methods for dictionaries and lists. Once processing is complete, data can be serialized back into standard JSON formatted string.

Here‘s an example demonstrating close similarity between Python dictionaries and JSON objects:

# Python Dictionary 
dict_data = {
  "name": "John",
  "age": 30,
  "married": True,
  "divorced": False,
  "children": ("Ann","Billy"),
  "pets": None,
  "cars": [
    {"model": "BMW 230", "mpg": 27.5},
    {"model": "Ford Edge", "mpg": 24.1}
  ]
}

# Equivalent JSON object
json_str = ‘{"name": "John", "age": 30, "married": true, "divorced": false, "children": ["Ann","Billy"], "pets": null, "cars": [{"model": "BMW 230", "mpg": 27.5}, {"model": "Ford Edge", "mpg": 24.1}]}‘

As we can observe, JSON objects and Python dictionaries have very similar structure with main difference being – former is stored as a text string while latter is a language object.

Reading JSON Files in Python

Let‘s now see how we can load JSON stored in external files into Python. We will use Python‘s built-in json module which provides functions for parsing JSON and converting it into native Python data structures.

Here‘s an example code to load a sample json file data.json:

import json

# Open the JSON file and load data
with open(‘data.json‘) as json_file:
    data = json.load(json_file)

print(data)
print(type(data))

In the above code:

• We first import Python‘s json module
• Using the open() method, we open the json file data.json
• Next, we use json.load() method to parse the file and convert JSON data into a Python dictionary
• Loaded data is printed which lets us inspect converted Python object
• We can see loaded json data is now a dictionary

This is the basic pattern for loading external JSON files in Python. The same technique works for JSON strings as well by replacing file handle with a string variable containing JSON data.

Writing JSON Files in Python

We looked at reading JSON data into Python. Now let us see how we can write Python objects as JSON strings into files.

Python provides a built-in json module for serializing and deserializing Python objects to JSON format.

Here is an example code to encode a Python object as JSON and store it in a file:

import json

# Python dict object
python_dict = {
  "name": "David",
  "age": 35, 
  "married": True,
  "divorced": False,
  "children": ("Ana","Mary"),
  "pets": None,
  "cars": [
    {"model": "BMW 230", "mpg": 27.5},
    {"model": "Ford Edge", "mpg": 24.1}
  ]
}

# Serialize the object
json_object = json.dumps(python_dict)

# Write JSON string to a file
with open(‘data.json‘, ‘w‘) as outfile:
    outfile.write(json_object)

In the above code, we:

• Created a sample Python dictionary object
• Convert dict to JSON string using json.dumps()
• Open a new file called data.json and write the JSON string into it using .write() method

And that‘s it! After running above program, data.json file will contain the Python dict data encoded in JSON format which can be easily loaded and exchanged.

Handling Python and JSON Data Types

When converting between JSON and Python objects, Python‘s JSON decoder and encoder will automatically handle conversion between equivalent data types in two formats.

JSON	Python
object	dict
array	list
string	str
number (int)	int
number (real)	float
true	True
false	False
null	None

This handles conversion for standard JSON and Python data types. But sometimes additional steps maybe needed while dealing with more complex data like custom class instances in Python or datetime objects. We will cover those cases in later sections.

JSON Encode and Decode Options

Python‘s json module provides several optional parameters to tweak encoding and decoding operations:

json.dumps() encode options:

• sort_keys (bool) – Sort keys in output
• indent (int) – Format with indentation and line breaks for human readability
• separators (tuple) – Custom separator strings
• default (function) – Custom encode hooks for non-standard objects

json.loads() decode options

• strict (bool) – Only parse standard JSON objects

Let‘s look at using some of these options to format our encoded JSON:

# Python Object
data = {
  "name": "Sarah",
  "age": 42,
  "married": True
}  

json_str = json.dumps(data, indent=4, sort_keys=True)  

print(json_str)

This will encode our data as a formatted JSON string with indentation:

{
    "age": 42, 
    "married": true, 
    "name": "Sarah"
}

We were able to customize JSON output by passing additional parameters to json.dumps(). This helps create more human-readable JSON strings for logging or debugging purposes.

Command-line JSON Processor

Python provides a useful command-line utility called json.tool for validating and formatting JSON from shell scripts or terminal.

Let‘s use it to pretty print a JSON string:

$ echo ‘{"name": "John", "age": 30}‘ | python -m json.tool

{
   "age": 30,
   "name": "John"
}

We can also use it directly on JSON files:

$ python -m json.tool my_data.json

In addition to printing formatted JSON, json.tool will also validate JSON and detect any errors.

While it doesn‘t provide full functionality like json module, json.tool can be handy for quick tasks or debugging JSON without writing any Python code.

Exceptions and Error Handling

We‘ll briefly cover some basics around error handling while working with JSON data.

Loading invalid JSON or serializing unsupported data types can result in JSONDecode or JSONEncode exceptions being raised.

Here is an example to safely load JSON from a file and handling exception:

import json

try:
    with open(‘data.json‘) as f:
        data = json.load(f)

except json.JSONDecodeError as e: 
    print("Unable to parse JSON", e)

We surround json loading in a try-except block to catch JSONDecodeError and print a friendly error message.

Similarly for invalid data while encoding:

try:
   json_data = json.dumps(my_object)

except TypeError as e:
    print("Unable to serialize", e)   

This ensures our program doesn‘t crash if invalid JSON or data is encountered.

Other json module exceptions like JSONDecodeError or TypeError can be handled similarly for robust Exception handling.

Custom Encoding and Decoding

Python json module provides flexibility to customize encoding/decoding beyond built-in types.

We can implement custom JSONEncoder classes to handle serialization of custom objects.

Here is an example encoder to handle Python datetime by converting them into ISO-8601 strings:

import json
from datetime import datetime

class DateTimeEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        return json.JSONEncoder.default(self, obj)

dt = datetime(2023, 2, 1)  
print(json.dumps(dt, cls=DateTimeEncoder))
# Output: "2023-02-01T00:00:00"  

We can use similar technique to support encoding any custom objects, by overriding the default() method.

For advanced use cases, custom decoder hooks can also be implemented by extending json.JSONDecoder class and overriding object_hook parameter.

Conclusion

In this article we covered all essential aspects of handling JSON data in Python, including:

• Loading external JSON files using json.load()
• Writing python dictionaries as JSON files with json.dump()
• Encoding and decoding customizations for advanced scenarios
• Formatting JSON output and command-line JSON processor
• Robust exception handling and error reporting
• Techniques for customizing JSON encoding/decoding hooks

JSON being a universal data format is heavily used in most Python projects. Mastering Python‘s JSON functionalities allows seamless interchange between JSON data and Python code.

Hope you enjoyed this guide! Please leave any feedback or questions in the comments section below.