book

Python Cookbook, 3rd Edition

by David Beazley, Brian K. Jones

May 2013

Intermediate to advanced

706 pages

14h 42m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Who This Book Is ForWho This Book Is Not ForConventions Used in This BookOnline Code ExamplesUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
1. Data Structures and Algorithms
1.1. Unpacking a Sequence into Separate Variables1.2. Unpacking Elements from Iterables of Arbitrary Length1.3. Keeping the Last N Items1.4. Finding the Largest or Smallest N Items1.5. Implementing a Priority Queue1.6. Mapping Keys to Multiple Values in a Dictionary1.7. Keeping Dictionaries in Order1.8. Calculating with Dictionaries1.9. Finding Commonalities in Two Dictionaries1.10. Removing Duplicates from a Sequence while Maintaining Order1.11. Naming a Slice1.12. Determining the Most Frequently Occurring Items in a Sequence1.13. Sorting a List of Dictionaries by a Common Key1.14. Sorting Objects Without Native Comparison Support1.15. Grouping Records Together Based on a Field1.16. Filtering Sequence Elements1.17. Extracting a Subset of a Dictionary1.18. Mapping Names to Sequence Elements1.19. Transforming and Reducing Data at the Same Time1.20. Combining Multiple Mappings into a Single Mapping
2. Strings and Text
2.1. Splitting Strings on Any of Multiple Delimiters2.2. Matching Text at the Start or End of a String2.3. Matching Strings Using Shell Wildcard Patterns2.4. Matching and Searching for Text Patterns2.5. Searching and Replacing Text2.6. Searching and Replacing Case-Insensitive Text2.7. Specifying a Regular Expression for the Shortest Match2.8. Writing a Regular Expression for Multiline Patterns2.9. Normalizing Unicode Text to a Standard Representation2.10. Working with Unicode Characters in Regular Expressions2.11. Stripping Unwanted Characters from Strings2.12. Sanitizing and Cleaning Up Text2.13. Aligning Text Strings2.14. Combining and Concatenating Strings2.15. Interpolating Variables in Strings2.16. Reformatting Text to a Fixed Number of Columns2.17. Handling HTML and XML Entities in Text2.18. Tokenizing Text2.19. Writing a Simple Recursive Descent Parser2.20. Performing Text Operations on Byte Strings
3. Numbers, Dates, and Times
3.1. Rounding Numerical Values3.2. Performing Accurate Decimal Calculations3.3. Formatting Numbers for Output3.4. Working with Binary, Octal, and Hexadecimal Integers3.5. Packing and Unpacking Large Integers from Bytes3.6. Performing Complex-Valued Math3.7. Working with Infinity and NaNs3.8. Calculating with Fractions3.9. Calculating with Large Numerical Arrays3.10. Performing Matrix and Linear Algebra Calculations3.11. Picking Things at Random3.12. Converting Days to Seconds, and Other Basic Time Conversions3.13. Determining Last Friday’s Date3.14. Finding the Date Range for the Current Month3.15. Converting Strings into Datetimes3.16. Manipulating Dates Involving Time Zones
4. Iterators and Generators
4.1. Manually Consuming an Iterator4.2. Delegating Iteration4.3. Creating New Iteration Patterns with Generators4.4. Implementing the Iterator Protocol4.5. Iterating in Reverse4.6. Defining Generator Functions with Extra State4.7. Taking a Slice of an Iterator4.8. Skipping the First Part of an Iterable4.9. Iterating Over All Possible Combinations or Permutations4.10. Iterating Over the Index-Value Pairs of a Sequence4.11. Iterating Over Multiple Sequences Simultaneously4.12. Iterating on Items in Separate Containers4.13. Creating Data Processing Pipelines4.14. Flattening a Nested Sequence4.15. Iterating in Sorted Order Over Merged Sorted Iterables4.16. Replacing Infinite while Loops with an Iterator
5. Files and I/O
5.1. Reading and Writing Text Data5.2. Printing to a File5.3. Printing with a Different Separator or Line Ending5.4. Reading and Writing Binary Data5.5. Writing to a File That Doesn’t Already Exist5.6. Performing I/O Operations on a String5.7. Reading and Writing Compressed Datafiles5.8. Iterating Over Fixed-Sized Records5.9. Reading Binary Data into a Mutable Buffer5.10. Memory Mapping Binary Files5.11. Manipulating Pathnames5.12. Testing for the Existence of a File5.13. Getting a Directory Listing5.14. Bypassing Filename Encoding5.15. Printing Bad Filenames5.16. Adding or Changing the Encoding of an Already Open File5.17. Writing Bytes to a Text File5.18. Wrapping an Existing File Descriptor As a File Object5.19. Making Temporary Files and Directories5.20. Communicating with Serial Ports5.21. Serializing Python Objects
6. Data Encoding and Processing
6.1. Reading and Writing CSV Data6.2. Reading and Writing JSON Data6.3. Parsing Simple XML Data6.4. Parsing Huge XML Files Incrementally6.5. Turning a Dictionary into XML6.6. Parsing, Modifying, and Rewriting XML6.7. Parsing XML Documents with Namespaces6.8. Interacting with a Relational Database6.9. Decoding and Encoding Hexadecimal Digits6.10. Decoding and Encoding Base646.11. Reading and Writing Binary Arrays of Structures6.12. Reading Nested and Variable-Sized Binary Structures6.13. Summarizing Data and Performing Statistics
7. Functions
7.1. Writing Functions That Accept Any Number of Arguments7.2. Writing Functions That Only Accept Keyword Arguments7.3. Attaching Informational Metadata to Function Arguments7.4. Returning Multiple Values from a Function7.5. Defining Functions with Default Arguments7.6. Defining Anonymous or Inline Functions7.7. Capturing Variables in Anonymous Functions7.8. Making an N-Argument Callable Work As a Callable with Fewer Arguments7.9. Replacing Single Method Classes with Functions7.10. Carrying Extra State with Callback Functions7.11. Inlining Callback Functions7.12. Accessing Variables Defined Inside a Closure
8. Classes and Objects
8.1. Changing the String Representation of Instances8.2. Customizing String Formatting8.3. Making Objects Support the Context-Management Protocol8.4. Saving Memory When Creating a Large Number of Instances8.5. Encapsulating Names in a Class8.6. Creating Managed Attributes8.7. Calling a Method on a Parent Class8.8. Extending a Property in a Subclass8.9. Creating a New Kind of Class or Instance Attribute8.10. Using Lazily Computed Properties8.11. Simplifying the Initialization of Data Structures8.12. Defining an Interface or Abstract Base Class8.13. Implementing a Data Model or Type System8.14. Implementing Custom Containers8.15. Delegating Attribute Access8.16. Defining More Than One Constructor in a Class8.17. Creating an Instance Without Invoking init8.18. Extending Classes with Mixins8.19. Implementing Stateful Objects or State Machines8.20. Calling a Method on an Object Given the Name As a String8.21. Implementing the Visitor Pattern8.22. Implementing the Visitor Pattern Without Recursion8.23. Managing Memory in Cyclic Data Structures8.24. Making Classes Support Comparison Operations8.25. Creating Cached Instances
9. Metaprogramming
9.1. Putting a Wrapper Around a Function9.2. Preserving Function Metadata When Writing Decorators9.3. Unwrapping a Decorator9.4. Defining a Decorator That Takes Arguments9.5. Defining a Decorator with User Adjustable Attributes9.6. Defining a Decorator That Takes an Optional Argument9.7. Enforcing Type Checking on a Function Using a Decorator9.8. Defining Decorators As Part of a Class9.9. Defining Decorators As Classes9.10. Applying Decorators to Class and Static Methods9.11. Writing Decorators That Add Arguments to Wrapped Functions9.12. Using Decorators to Patch Class Definitions9.13. Using a Metaclass to Control Instance Creation9.14. Capturing Class Attribute Definition Order9.15. Defining a Metaclass That Takes Optional Arguments9.16. Enforcing an Argument Signature on *args and **kwargs9.17. Enforcing Coding Conventions in Classes9.18. Defining Classes Programmatically9.19. Initializing Class Members at Definition Time9.20. Implementing Multiple Dispatch with Function Annotations9.21. Avoiding Repetitive Property Methods9.22. Defining Context Managers the Easy Way9.23. Executing Code with Local Side Effects9.24. Parsing and Analyzing Python Source9.25. Disassembling Python Byte Code

10. Modules and Packages
10.1. Making a Hierarchical Package of Modules10.2. Controlling the Import of Everything10.3. Importing Package Submodules Using Relative Names10.4. Splitting a Module into Multiple Files10.5. Making Separate Directories of Code Import Under a Common Namespace10.6. Reloading Modules10.7. Making a Directory or Zip File Runnable As a Main Script10.8. Reading Datafiles Within a Package10.9. Adding Directories to sys.path10.10. Importing Modules Using a Name Given in a String10.11. Loading Modules from a Remote Machine Using Import Hooks10.12. Patching Modules on Import10.13. Installing Packages Just for Yourself10.14. Creating a New Python Environment10.15. Distributing Packages
11. Network and Web Programming
11.1. Interacting with HTTP Services As a Client11.2. Creating a TCP Server11.3. Creating a UDP Server11.4. Generating a Range of IP Addresses from a CIDR Address11.5. Creating a Simple REST-Based Interface11.6. Implementing a Simple Remote Procedure Call with XML-RPC11.7. Communicating Simply Between Interpreters11.8. Implementing Remote Procedure Calls11.9. Authenticating Clients Simply11.10. Adding SSL to Network Services11.11. Passing a Socket File Descriptor Between Processes11.12. Understanding Event-Driven I/O11.13. Sending and Receiving Large Arrays
12. Concurrency
12.1. Starting and Stopping Threads12.2. Determining If a Thread Has Started12.3. Communicating Between Threads12.4. Locking Critical Sections12.5. Locking with Deadlock Avoidance12.6. Storing Thread-Specific State12.7. Creating a Thread Pool12.8. Performing Simple Parallel Programming12.9. Dealing with the GIL (and How to Stop Worrying About It)12.10. Defining an Actor Task12.11. Implementing Publish/Subscribe Messaging12.12. Using Generators As an Alternative to Threads12.13. Polling Multiple Thread Queues12.14. Launching a Daemon Process on Unix
13. Utility Scripting and System Administration
13.1. Accepting Script Input via Redirection, Pipes, or Input Files13.2. Terminating a Program with an Error Message13.3. Parsing Command-Line Options13.4. Prompting for a Password at Runtime13.5. Getting the Terminal Size13.6. Executing an External Command and Getting Its Output13.7. Copying or Moving Files and Directories13.8. Creating and Unpacking Archives13.9. Finding Files by Name13.10. Reading Configuration Files13.11. Adding Logging to Simple Scripts13.12. Adding Logging to Libraries13.13. Making a Stopwatch Timer13.14. Putting Limits on Memory and CPU Usage13.15. Launching a Web Browser
14. Testing, Debugging, and Exceptions
14.1. Testing Output Sent to stdout14.2. Patching Objects in Unit Tests14.3. Testing for Exceptional Conditions in Unit Tests14.4. Logging Test Output to a File14.5. Skipping or Anticipating Test Failures14.6. Handling Multiple Exceptions14.7. Catching All Exceptions14.8. Creating Custom Exceptions14.9. Raising an Exception in Response to Another Exception14.10. Reraising the Last Exception14.11. Issuing Warning Messages14.12. Debugging Basic Program Crashes14.13. Profiling and Timing Your Program14.14. Making Your Programs Run Faster
15. C Extensions
15.1. Accessing C Code Using ctypes15.2. Writing a Simple C Extension Module15.3. Writing an Extension Function That Operates on Arrays15.4. Managing Opaque Pointers in C Extension Modules15.5. Defining and Exporting C APIs from Extension Modules15.6. Calling Python from C15.7. Releasing the GIL in C Extensions15.8. Mixing Threads from C and Python15.9. Wrapping C Code with Swig15.10. Wrapping Existing C Code with Cython15.11. Using Cython to Write High-Performance Array Operations15.12. Turning a Function Pointer into a Callable15.13. Passing NULL-Terminated Strings to C Libraries15.14. Passing Unicode Strings to C Libraries15.15. Converting C Strings to Python15.16. Working with C Strings of Dubious Encoding15.17. Passing Filenames to C Extensions15.18. Passing Open Files to C Extensions15.19. Reading File-Like Objects from C15.20. Consuming an Iterable from C15.21. Diagnosing Segmentation Faults
A. Further Reading
Online ResourcesBooks for Learning PythonAdvanced Books
Index
Colophon
Copyright

Overview

If you need help writing programs in Python 3, or want to update older Python 2 code, this book is just the ticket. Packed with practical recipes written and tested with Python 3.3, this unique cookbook is for experienced Python programmers who want to focus on modern tools and idioms.

Inside, you’ll find complete recipes for more than a dozen topics, covering the core Python language as well as tasks common to a wide variety of application domains. Each recipe contains code samples you can use in your projects right away, along with a discussion about how and why the solution works.

Topics include:

Data Structures and Algorithms
Strings and Text
Numbers, Dates, and Times
Iterators and Generators
Files and I/O
Data Encoding and Processing
Functions
Classes and Objects
Metaprogramming
Modules and Packages
Network and Web Programming
Concurrency
Utility Scripting and System Administration
Testing, Debugging, and Exceptions
C Extensions

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781449357337Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills