Real Python

The Real Python Podcast – Episode #27: Preparing for an Interview With Python Practice Problems

2020-09-18T12:00:00+00:00

What is an effective way to prepare for a Python interview? Would you like a set of problems that increase in difficulty to practice and hone your Python skills? This week on the show, we have Jim Anderson to talk about his new Real Python article, "Python Practice Problems: Get Ready for Your Next Interview." This article provides several problems, which include skeleton code, unit tests, and solutions for you to compare your work.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Numbers in Python

2020-09-16T14:00:00+00:00

You don’t need to be a math whiz to program well. The truth is, few programmers need to know more than basic algebra. Of course, how much math you need to know depends on the application you’re working on. In general, the level of math required to be a programmer is lower than you might expect. Although math and computer programming aren’t as correlated as some people might believe, numbers are an integral part of any programming language, and Python is no exception.

In this tutorial, you’ll learn how to:

Create integers and floating-point numbers
Round numbers to a given number of decimal places
Format and display numbers in strings

Let’s get started!

Note: This tutorial is adapted from the chapter “Numbers and Math” in Python Basics: A Practical Introduction to Python 3.

The book uses Python’s built-in IDLE editor to create and edit Python files and interact with the Python shell, so you will see references to IDLE’s built-in debugging tools throughout this tutorial. However, you should have no problems running the example code from the editor and environment of your choice.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Integers and Floating-Point Numbers#

Python has three built-in numeric data types: integers, floating-point numbers, and complex numbers. In this section, you’ll learn about integers and floating-point numbers, which are the two most commonly used number types. You’ll learn about complex numbers in a later section.

Integers#

An integer is a whole number with no decimal places. For example, 1 is an integer, but 1.0 isn’t. The name for the integer data type is int, which you can see with type():

>>>

>>> type(1)
<class 'int'>

You can create an integer by typing the desired number. For instance, the following assigns the integer 25 to the variable num:

>>>

>>> num = 25

When you create an integer like this, the value 25 is called an integer literal because the integer is literally typed into the code.

You may already be familiar with how to convert a string containing an integer to a number using int(). For example, the following converts the string "25" to the integer 25:

>>>

>>> int("25")
25

int("25") is not an integer literal because the integer value is created from a string.

When you write large numbers by hand, you typically group digits into groups of three separated by a comma or a decimal point. The number 1,000,000 is a lot easier to read than 1000000.

In Python, you can’t use commas to group digits in integer literals, but you can use underscores (_). Both of the following are valid ways to represent the number one million as an integer literal:

>>>

>>> 1000000
1000000

>>> 1_000_000
1000000

There’s no limit to how large an integer can be, which might be surprising considering that computers have a finite amount of memory. Try typing the largest number you can think of into IDLE’s interactive window. Python can handle it with no problem!

Floating-Point Numbers#

A floating-point number, or float for short, is a number with a decimal place. 1.0 is a floating-point number, as is -2.75. The name of the floating-point data type is float:

>>>

>>> type(1.0)
<class 'float'>

Like integers, floats can be created from floating-point literals or by converting a string to a float with float():

Read the full article at https://realpython.com/python-numbers/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Command Line Interfaces in Python

2020-09-15T14:00:00+00:00

Adding the capability of processing Python command line arguments provides a user-friendly interface to your text-based command line program. It’s similar to what a graphical user interface is for a visual application that’s manipulated by graphical elements or widgets.

Python exposes a mechanism to capture and extract your Python command line arguments. These values can be used to modify the behavior of a program. For example, if your program processes data read from a file, then you can pass the name of the file to your program, rather than hard-coding the value in your source code.

By the end of this course, you’ll know:

The origins of Python command line arguments
The underlying support for Python command line arguments
The standards guiding the design of a command line interface
The basics to manually customize and handle Python command line arguments
The libraries available in Python to ease the development of a complex command line interface

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Plot With Pandas: Python Data Visualization for Beginners

2020-09-14T14:00:00+00:00

Whether you’re just getting to know a dataset or preparing to publish your findings, visualization is an essential tool. Python’s popular data analysis library, pandas, provides several different options for visualizing your data with .plot(). Even if you’re at the beginning of your pandas journey, you’ll soon be creating basic plots that will yield valuable insights into your data.

In this tutorial, you’ll learn:

What the different types of pandas plots are and when to use them
How to get an overview of your dataset with a histogram
How to discover correlation with a scatter plot
How to analyze different categories and their ratios

Free Bonus: Click here to get access to a Conda cheat sheet with handy usage examples for managing your Python environment and packages.

Set Up Your Environment#

You can best follow along with the code in this tutorial in a Jupyter Notebook. This way, you’ll immediately see your plots and be able to play around with them.

You’ll also need a working Python environment including pandas. If you don’t have one yet, then you have several options:

If you have more ambitious plans, then download the Anaconda distribution. It’s huge (around 500 MB), but you’ll be equipped for most data science work.
If you prefer a minimalist setup, then check out the section on installing Miniconda in Setting Up Python for Machine Learning on Windows.
If you want to stick to pip, then install the libraries discussed in this tutorial with pip install pandas matplotlib. You can also grab Jupyter Notebook with pip install jupyterlab.
If you don’t want to do any setup, then follow along in an online Jupyter Notebook trial.

Once your environment is set up, you’re ready to download a dataset. In this tutorial, you’re going to analyze data on college majors sourced from the American Community Survey 2010–2012 Public Use Microdata Sample. It served as the basis for the Economic Guide To Picking A College Major featured on the website FiveThirtyEight.

First, download the data by passing the download URL to pandas.read_csv():

>>>

In [1]: import pandas as pd

In [2]: download_url = (
   ...:     "https://raw.githubusercontent.com/fivethirtyeight/"
   ...:     "data/master/college-majors/recent-grads.csv"
   ...: )

In [3]: df = pd.read_csv(download_url)

In [4]: type(df)
Out[4]: pandas.core.frame.DataFrame

By calling read_csv(), you create a DataFrame, which is the main data structure used in pandas.

Note: You can follow along with this tutorial even if you aren’t familiar with DataFrames. But if you’re interested in learning more about working with pandas and DataFrames, then you can check out Using Pandas and Python to Explore Your Dataset and The Pandas DataFrame: Make Working With Data Delightful.

Now that you have a DataFrame, you can take a look at the data. First, you should configure the display.max.columns option to make sure pandas doesn’t hide any columns. Then you can view the first few rows of data with .head():

>>>

In [5]: pd.set_option("display.max.columns", None)

In [6]: df.head()

You’ve just displayed the first five rows of the DataFrame df using .head(). Your output should look like this:

The default number of rows displayed by .head() is five, but you can specify any number of rows as an argument. For example, to display the first ten rows, you would use df.head(10).

Create Your First Pandas Plot#

Your dataset contains some columns related to the earnings of graduates in each major:

"Median" is the median earnings of full-time, year-round workers.
"P25th" is the 25th percentile of earnings.
"P75th" is the 75th percentile of earnings.
"Rank" is the major’s rank by median earnings.

Let’s start with a plot displaying these columns. First, you need to set up your Jupyter Notebook to display plots with the %matplotlib magic command:

>>>

In [7]: %matplotlib
Using matplotlib backend: MacOSX

The %matplotlib magic command sets up your Jupyter Notebook for displaying plots with Matplotlib. The standard Matplotlib graphics backend is used by default, and your plots will be displayed in a separate window.

Note: You can change the Matplotlib backend by passing an argument to the %matplotlib magic command.

For example, the inline backend is popular for Jupyter Notebooks because it displays the plot in the notebook itself, immediately below the cell that creates the plot:

>>>

In [7]: %matplotlib inline

There are a number of other backends available. For more information, check out the Rich Outputs tutorial in the IPython documentation.

Read the full article at https://realpython.com/pandas-plot-python/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #26: 5 Years Podcasting Python with Michael Kennedy: Growth, GIL, Async, and More

2020-09-11T12:00:00+00:00

Why is Python pulling in so many new programmers? Maybe some of that growth is from Python being a full-spectrum language. This week on the show we have Michael Kennedy, the host of the podcast "Talk Python to Me". Michael reflects on five years of podcasting about Python, and many of the changes he has seen in the Python landscape.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Find & Fix Code Bugs in Python: Debug With IDLE

2020-09-09T14:00:00+00:00

Everyone makes mistakes—even seasoned professional developers! Python’s interactive interpreter, IDLE, is pretty good at catching mistakes like syntax errors and runtime errors, but there’s a third type of error that you may have already experienced. Logic errors occur when an otherwise valid program doesn’t do what was intended. Logic errors cause unexpected behaviors called bugs. Removing bugs is called debugging.

A debugger is a tool that helps you hunt down bugs and understand why they’re happening. Knowing how to find and fix bugs in your code is a skill that you’ll use for your entire coding career!

In this tutorial, you’ll:

Learn how to use IDLE’s Debug Control window
Practice debugging on a buggy function
Learn alternative methods for debugging your code

Note: This tutorial is adapted from the chapter “Finding and Fixing Code Bugs” in Python Basics: A Practical Introduction to Python 3.

The book uses Python’s built-in IDLE editor to create and edit Python files and interact with the Python shell, so you will see references to IDLE’s built-in debugging tools throughout this tutorial. However, you should be able to apply the same concepts to the debugger of your choice.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you'll need to take your Python skills to the next level.

Use the Debug Control Window#

The main interface to IDLE’s debugger is the Debug Control window, or the Debug window for short. You can open the Debug window by selecting Debug→Debugger from the menu in the interactive window. Go ahead and open the Debug window.

Note: If the Debug menu is missing from your menu bar, then make sure to bring the interactive window into focus by clicking it.

Whenever the Debug window is open, the interactive window displays [DEBUG ON] next to the prompt to indicate that the debugger is open. Now open a new editor window and arrange the three windows on your screen so that you can see all of them simultaneously.

In this section, you’ll learn how the Debug window is organized, how to step through your code with the debugger one line at a time, and how to set breakpoints to help speed up the debugging process.

The Debug Control Window: An Overview#

To see how the debugger works, you can start by writing a simple program without any bugs. Type the following into the editor window:

 1 for i in range(1, 4):
 2     j = i * 2
 3     print(f"i is {i} and j is {j}")

Save the file, then keep the Debug window open and press F5. You’ll notice that execution doesn’t get very far.

The Debug window will look like this:

Notice that the Stack panel at the top of the window contains the following message:

> '__main__'.<module>(), line 1: for i in range(1, 4):

This tells you that line 1 (which contains the code for i in range(1, 4):) is about to be run but hasn’t started yet. The '__main__'.module() part of the message refers to the fact that you’re currently in the main section of the program, as opposed to being, for example, in a function definition before the main block of code has been reached.

Below the Stack panel is a Locals panel that lists some strange looking stuff like __annotations__, __builtins__, __doc__, and so on. These are internal system variables that you can ignore for now. As your program runs, you’ll see variables declared in the code displayed in this window so that you can keep track of their value.

There are five buttons located at the top left-hand corner of the Debug window: Go, Step, Over, Out, and Quit. These buttons control how the debugger moves through your code.

In the following sections, you’ll explore what each of these buttons does, starting with Step.

The Step Button#

Go ahead and click Step at the top left-hand corner of the Debug window. The Debug window changes a bit to look like this:

Read the full article at https://realpython.com/python-debug-idle/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Exploring HTTPS and Cryptography in Python

2020-09-08T14:00:00+00:00

Have you ever wondered why it’s okay for you to send your credit card information over the Internet? You may have noticed the https:// on URLs in your browser, but what is it, and how does it keep your information safe? Or perhaps you want to create a Python HTTPS application, but you’re not exactly sure what that means.

In this course, you’ll get a working knowledge of the various factors that combine to keep communications over the Internet safe. You’ll see concrete examples of how a Python HTTPS application keeps information secure.

In this course, you’ll learn how to:

Monitor and analyze network traffic
Apply cryptography to keep data safe
Describe the core concepts of Public Key Infrastructure (PKI)
Create your own Certificate Authority
Build a Python HTTPS application
Identify common Python HTTPS warnings and errors

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Video Subtitles & Transcripts Now Available on Real Python

2020-09-07T14:00:00+00:00

Hey there,

I’ve got a big update to share today:

Real Python video courses now have full subtitles and transcripts!

I think this is going to do a lot for accessibility and making your favorite Python learning resources easier to review & more searchable.

Let’s do a quick demo. Video lessons now come with full subtitles that you can turn on and off at your convenience:

Below each video you’ll also find an interactive transcript that animates along with the video to show you what section of the video is currently playing:

Read the full article at https://realpython.com/video-subtitles-transcripts-now-available/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #25: Data Version Control in Python and Real Python Video Transcripts

2020-09-04T12:00:00+00:00

Wouldn't it be nice to a use a form of version control for data? Something that would allow you to track and version your datasets and models. Well, that's what the tool called DVC is designed to do. This week on the show, David Amos is here and he's brought another batch of PyCoder’s Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PyTorch vs Tensorflow for Your Python Deep Learning Project

2020-09-02T14:00:00+00:00

PyTorch vs TensorFlow: What’s the difference? Both are open source Python libraries that use graphs to perform numerical computation on data. Both are used extensively in academic research and commercial code. Both are extended by a variety of APIs, cloud computing platforms, and model repositories.

If they’re so similar, then which one is best for your project?

In this tutorial, you’ll learn:

What the differences are between PyTorch and TensorFlow
What tools and resources are available for each
How to choose the best option for your specific use case

You’ll start by taking a close look at both platforms, beginning with the slightly older TensorFlow, before exploring some considerations that can help you determine which choice is best for your project. Let’s get started!

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

What Is TensorFlow?#

TensorFlow was developed by Google and released as open source in 2015. It grew out of Google’s homegrown machine learning software, which was refactored and optimized for use in production.

The name “TensorFlow” describes how you organize and perform operations on data. The basic data structure for both TensorFlow and PyTorch is a tensor. When you use TensorFlow, you perform operations on the data in these tensors by building a stateful dataflow graph, kind of like a flowchart that remembers past events.

Who Uses TensorFlow?#

TensorFlow has a reputation for being a production-grade deep learning library. It has a large and active user base and a proliferation of official and third-party tools and platforms for training, deploying, and serving models.

After PyTorch was released in 2016, TensorFlow declined in popularity. But in late 2019, Google released TensorFlow 2.0, a major update that simplified the library and made it more user-friendly, leading to renewed interest among the machine learning community.

Code Style and Function#

Before TensorFlow 2.0, TensorFlow required you to manually stitch together an abstract syntax tree—the graph—by making tf.* API calls. It then required you to manually compile the model by passing a set of output tensors and input tensors to a session.run() call.

A Session object is a class for running TensorFlow operations. It contains the environment in which Tensor objects are evaluated and Operation objects are executed, and it can own resources like tf.Variable objects. The most common way to use a Session is as a context manager.

In TensorFlow 2.0, you can still build models this way, but it’s easier to use eager execution, which is the way Python normally works. Eager execution evaluates operations immediately, so you can write your code using Python control flow rather than graph control flow.

To see the difference, let’s look at how you might multiply two tensors using each method. Here’s an example using the old TensorFlow 1.0 method:

>>>

>>> import tensorflow as tf

>>> tf.compat.v1.disable_eager_execution()

>>> x = tf.compat.v1.placeholder(tf.float32, name = "x")
>>> y = tf.compat.v1.placeholder(tf.float32, name = "y")

>>> multiply = tf.multiply(x, y)

>>> with tf.compat.v1.Session() as session:
...     m = session.run(
...         multiply, feed_dict={x: [[2., 4., 6.]], y: [[1.], [3.], [5.]]}
...     )
...     print(m)
[[ 2.  4.  6.]
 [ 6. 12. 18.]
 [10. 20. 30.]]

This code uses TensorFlow 2.x’s tf.compat API to access TensorFlow 1.x methods and disable eager execution.

Read the full article at https://realpython.com/pytorch-vs-tensorflow/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Editing Excel Spreadsheets in Python With openpyxl

2020-09-01T14:00:00+00:00

Excel spreadsheets are one of those things you might have to deal with at some point. Either it’s because your boss loves them or because marketing needs them, you might have to learn how to work with spreadsheets in Python, and that’s when knowing openpyxl comes in handy!

In this course, you’ll learn how to use openpyxl to:

Read Excel spreadsheets and iterate through the data
Manipulate speadsheet data using Python data structures
Create simple or more complex spreadsheets
Format workbooks using styles, filters, and conditional formatting
Enhance spreadsheets by adding images and charts

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python 3 Installation & Setup Guide

2020-08-31T16:15:29+00:00

Installing or updating Python on your computer is the first step to becoming a Python programmer. There are a multitude of installation methods: you can download official Python distributions from Python.org, install from a package manager, and even install specialized distributions for scientific computing, Internet of Things, and embedded systems.

This tutorial focuses on official distributions, as they’re generally the best option for getting started with learning to program in Python.

In this tutorial you’ll learn how to:

Check which version of Python, if any, is installed on your machine
Install or update Python on Windows, macOS, and Linux
Use Python on mobile devices like phones or tablets
Use Python on the Web with online interpreters

No matter what operating system you’re on, this tutorial has you covered. Find your operating system below and dive in!

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

How to Install Python on Windows#

There are three installation methods on Windows:

The Microsoft Store
The full installer
Windows Subsystem for Linux

In this section, you’ll learn how to check which version of Python, if any, is installed on your Windows computer. You’ll also learn which of the three installation methods you should use.

How to Check Your Python Version on Windows#

To check if you already have Python on your Windows machine, first open a command-line application, such as PowerShell.

Tip: Here’s how you open PowerShell:

Press the Win key.
Type PowerShell.
Press Enter.

Alternatively, you can right-click the Start button and select Windows PowerShell or Windows PowerShell (Admin).

You can also use cmd.exe or Windows Terminal.

With the command line open, type in the following command and press Enter:

C:\> python --version
Python 3.8.4

Using the --version switch will show you the version that’s installed. Alternatively, you can use the -V switch:

C:\> python -V
Python 3.8.4

In either case, if you see a version less than 3.8.4, which was the most recent version at the time of writing, then you’ll want to upgrade your installation.

Note: If you don’t have a version of Python on your system, then both of the above commands will launch the Microsoft Store and redirect you to the Python application page. You’ll see how to complete the installation from the Microsoft Store in the next section.

If you’re interested in where the installation is located, then you can use the where.exe command in cmd.exe or PowerShell:

C:\> where.exe python
C:\Users\mertz\AppData\Local\Programs\Python\Python37-32\python.exe

Note that the where.exe command will work only if Python has been installed for your user account.

What Your Options Are#

As mentioned earlier, there are three ways to install the official Python distribution on Windows:

Read the full article at https://realpython.com/installing-python/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #24: Options for Packaging Your Python Application: Wheels, Docker, and More

2020-08-28T12:00:00+00:00

Have you wondered, how should I package my Python code? You've written the application, but now you need to distribute it to the machines it's intended to run on. It depends on what the code is, the libraries it depends on, and with whom do you want to share it. This week on the show we have Itamar Turner-Trauring, creator of the website pythonspeed.com. We discuss his article "Options for Packaging Your Python Code: Wheels, Conda, Docker, and More," covering the how of sharing your code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Common Python Data Structures (Guide)

2020-08-26T14:00:00+00:00

Data structures are the fundamental constructs around which you build your programs. Each data structure provides a particular way of organizing data so it can be accessed efficiently, depending on your use case. Python ships with an extensive set of data structures in its standard library.

However, Python’s naming convention doesn’t provide the same level of clarity that you’ll find in other languages. In Java, a list isn’t just a list—it’s either a LinkedList or an ArrayList. Not so in Python. Even experienced Python developers sometimes wonder whether the built-in list type is implemented as a linked list or a dynamic array.

In this tutorial, you’ll learn:

Which common abstract data types are built into the Python standard library
How the most common abstract data types map to Python’s naming scheme
How to put abstract data types to practical use in various algorithms

Note: This tutorial is adapted from the chapter “Common Data Structures in Python” in Python Tricks: The Book. If you enjoy what you read below, then be sure to check out the rest of the book.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Dictionaries, Maps, and Hash Tables#

In Python, dictionaries (or dicts for short) are a central data structure. Dicts store an arbitrary number of objects, each identified by a unique dictionary key.

Dictionaries are also often called maps, hashmaps, lookup tables, or associative arrays. They allow for the efficient lookup, insertion, and deletion of any object associated with a given key.

Phone books make a decent real-world analog for dictionary objects. They allow you to quickly retrieve the information (phone number) associated with a given key (a person’s name). Instead of having to read a phone book front to back to find someone’s number, you can jump more or less directly to a name and look up the associated information.

This analogy breaks down somewhat when it comes to how the information is organized to allow for fast lookups. But the fundamental performance characteristics hold. Dictionaries allow you to quickly find the information associated with a given key.

Dictionaries are one of the most important and frequently used data structures in computer science. So, how does Python handle dictionaries? Let’s take a tour of the dictionary implementations available in core Python and the Python standard library.

`dict`: Your Go-To Dictionary#

Because dictionaries are so important, Python features a robust dictionary implementation that’s built directly into the core language: the dict data type.

Python also provides some useful syntactic sugar for working with dictionaries in your programs. For example, the curly-brace ({ }) dictionary expression syntax and dictionary comprehensions allow you to conveniently define new dictionary objects:

>>>

>>> phonebook = {
...     "bob": 7387,
...     "alice": 3719,
...     "jack": 7052,
... }

>>> squares = {x: x * x for x in range(6)}

>>> phonebook["alice"]
3719

>>> squares
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

There are some restrictions on which objects can be used as valid keys.

Python’s dictionaries are indexed by keys that can be of any hashable type. A hashable object has a hash value that never changes during its lifetime (see __hash__), and it can be compared to other objects (see __eq__). Hashable objects that compare as equal must have the same hash value.

Immutable types like strings and numbers are hashable and work well as dictionary keys. You can also use tuple objects as dictionary keys as long as they contain only hashable types themselves.

For most use cases, Python’s built-in dictionary implementation will do everything you need. Dictionaries are highly optimized and underlie many parts of the language. For example, class attributes and variables in a stack frame are both stored internally in dictionaries.

Python dictionaries are based on a well-tested and finely tuned hash table implementation that provides the performance characteristics you’d expect: O(1) time complexity for lookup, insert, update, and delete operations in the average case.

There’s little reason not to use the standard dict implementation included with Python. However, specialized third-party dictionary implementations exist, such as skip lists or B-tree–based dictionaries.

Besides plain dict objects, Python’s standard library also includes a number of specialized dictionary implementations. These specialized dictionaries are all based on the built-in dictionary class (and share its performance characteristics) but also include some additional convenience features.

Let’s take a look at them.

`collections.OrderedDict`: Remember the Insertion Order of Keys#

Read the full article at https://realpython.com/python-data-structures/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Django Redirects

2020-08-25T14:00:00+00:00

When you build web applications in Python using the Django framework, you’ll likely need to redirect the user from one URL to another. This course covers what you need to know about redirecting in Django. All the way from the low-level details of the HTTP protocol to the high-level way of dealing with them in Django.

By the end of this course you’ll learn:

How HTTP Redirects work
The difference between temporary and permanent redirects
How to use query strings with a redirect
How to avoid common problems

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python mmap: Improved File I/O With Memory Mapping

2020-08-24T14:00:00+00:00

The Zen of Python has a lot of wisdom to offer. One especially useful idea is that “There should be one—and preferably only one—obvious way to do it.” Yet there are multiple ways to do most things in Python, and often for good reason. For example, there are multiple ways to read a file in Python, including the rarely used mmap module

Python’s mmap provides memory-mapped file input and output (I/O). It allows you to take advantage of lower-level operating system functionality to read files as if they were one large string or array. This can provide significant performance improvements in code that requires a lot of file I/O.

In this tutorial, you’ll learn:

What kinds of computer memory exist
What problems you can solve with mmap
How use memory mapping to read large files faster
How to change a portion of a file without rewriting the entire file
How to use mmap to share information between multiple processes

Free Download: Get a sample chapter from CPython Internals: Your Guide to the Python 3 Interpreter showing you how to unlock the inner workings of the Python language, compile the Python interpreter from source code, and participate in the development of CPython.

Understanding Computer Memory#

Memory mapping is a technique that uses lower-level operating system APIs to load a file directly into computer memory. It can dramatically improve file I/O performance in your program. To better understand how memory mapping improves performance, as well as how and when you can use the mmap module to take advantage of these performance benefits, it’s useful to first learn a bit about computer memory.

Computer memory is a big, complicated topic, but this tutorial focuses only on what you need to know to use the mmap module effectively. For the purposes of this tutorial, the term memory refers to random-access memory, or RAM.

There are several types of computer memory:

Physical
Virtual
Shared

Each type of memory can come into play when you’re using memory mapping, so let’s review each one from a high level.

Physical Memory#

Physical memory is the least complicated type of memory to understand because it’s often part of the marketing associated with your computer. (You might remember that when you bought your computer, it advertised something like 8 gigabytes of RAM.) Physical memory typically comes on cards that are connected to your computer’s motherboard.

Physical memory is the amount of volatile memory that’s available for your programs to use while running. Physical memory should not be confused with storage, such as your hard drive or solid-state disk.

Virtual Memory#

Virtual memory is a way of handling memory management. The operating system uses virtual memory to make it appear that you have more memory than you do, allowing you to worry less about how much memory is available for your programs at any given time. Behind the scenes, your operating system uses parts of your nonvolatile storage, such as your solid-state disk, to simulate additional RAM.

In order to do this, your operating system must maintain a mapping between physical memory and virtual memory. Each operating system uses its own sophisticated algorithm to map virtual memory addresses to physical ones using a data structure called a page table.

Luckily, most of this complication is hidden from your programs. You don’t need to understand page tables or logical-to-physical mapping to write performant I/O code in Python. However, knowing a little bit about memory gives you a better understanding of what the computer and libraries are taking care of for you.

mmap uses virtual memory to make it appear that you’ve loaded a very large file into memory, even if the contents of the file are too big to fit in your physical memory.

Shared Memory#

Shared memory is another technique provided by your operating system that allows multiple programs to access the same data simultaneously. Shared memory can be a very efficient way of handling data in a program that uses concurrency.

Python’s mmap uses shared memory to efficiently share large amounts of data between multiple Python processes, threads, and tasks that are happening concurrently.

Digging Deeper Into File I/O#

Now that you have a high-level view of the different types of memory, it’s time to understand what memory mapping is and what problems it solves. Memory mapping is another way to perform file I/O that can result in better performance and memory efficiency.

In order to fully appreciate what memory mapping does, it’s useful to consider regular file I/O from a lower-level perspective. A lot of things happen behind the scenes when reading a file:

Read the full article at https://realpython.com/python-mmap/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #23: Python Wheels and Pass by Reference in Python

2020-08-21T12:00:00+00:00

Have you wondered what are Python wheels? How are they used to package Python code? Does Python use pass by value or pass by reference? This week on the show, David Amos is here to help answer these questions, and he has brought another batch of PyCoder’s Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Data Version Control With Python and DVC

2020-08-19T14:00:00+00:00

Machine learning and data science come with a set of problems that are different from what you’ll find in traditional software engineering. Version control systems help developers manage changes to source code. But data version control, managing changes to models and datasets, isn’t so well established.

It’s not easy to keep track of all the data you use for experiments and the models you produce. Accurately reproducing experiments that you or others have done is a challenge. Many teams are actively developing tools and frameworks to solve these problems.

In this tutorial, you’ll learn how to:

Use a tool called DVC to tackle some of these challenges
Track and version your datasets and models
Share a single development computer between teammates
Create reproducible machine learning experiments

This tutorial includes several examples of data version control techniques in action. To follow along, you can get the repository with the sample code by clicking the following link:

Get the Source Code: Click here to get the source code you'll use to learn about data version control with DVC in this tutorial.

What Is Data Version Control?#

In standard software engineering, many people need to work on a shared codebase and handle multiple versions of the same code. This can quickly lead to confusion and costly mistakes.

To address this problem, developers use version control systems, such as Git, that help keep team members organized.

In a version control system, there’s a central repository of code that represents the current, official state of the project. A developer can make a copy of that project, make some changes, and request that their new version become the official one. Their code is then reviewed and tested before it’s deployed to production.

These quick feedback cycles can happen many times per day in traditional development projects. But similar conventions and standards are largely missing from commercial data science and machine learning. Data version control is a set of tools and processes that tries to adapt the version control process to the data world.

Having systems in place that allow people to work quickly and pick up where others have left off would increase the speed and quality of delivered results. It would enable people to manage data transparently, run experiments effectively, and collaborate with others.

Note: An experiment in this context means either training a model or running operations on a dataset to learn something from it.

One tool that helps researchers govern their data and models and run reproducible experiments is DVC, which stands for Data Version Control.

What Is DVC?#

DVC is a command-line tool written in Python. It mimics Git commands and workflows to ensure that users can quickly incorporate it into their regular Git practice. If you haven’t worked with Git before, then be sure to check out Introduction to Git and GitHub for Python Developers. If you’re familiar with Git but would like to take your skills to the next level, then check out Advanced Git Tips for Python Developers.

DVC is meant to be run alongside Git. In fact, the git and dvc commands will often be used in tandem, one after the other. While Git is used to store and version code, DVC does the same for data and model files.

Git can store code locally and also on a hosting service like GitHub, Bitbucket, or GitLab. Likewise, DVC uses a remote repository to store all your data and models. This is the single source of truth, and it can be shared amongst the whole team. You can get a local copy of the remote repository, modify the files, then upload your changes to share with team members.

The remote repository can be on the same computer you’re working on, or it can be in the cloud. DVC supports most major cloud providers, including AWS, GCP, and Azure. But you can set up a DVC remote repository on any server and connect it to your laptop. There are safeguards to keep members from corrupting or deleting the remote data.

When you store your data and models in the remote repository, a .dvc file is created. A .dvc file is a small text file that points to your actual data files in remote storage.

The .dvc file is lightweight and meant to be stored with your code in GitHub. When you download a Git repository, you also get the .dvc files. You can then use those files to get the data associated with that repository. Large data and model files go in your DVC remote storage, and small .dvc files that point to your data go in GitHub.

The best way to understand DVC is to use it, so let’s dive in. You’ll explore the most important features by working through several examples. Before you start, you’ll need to set up an environment to work in and then get some data.

Set Up Your Working Environment#

In this tutorial, you’ll learn how to use DVC by practicing on examples that work with image data. You’ll play around with lots of image files and train a machine learning model that recognizes what an image contains.

To work through the examples, you’ll need to have Python and Git installed on your system. You can follow the Python 3 Installation and Setup Guide to install Python on your system. To install Git, you can read through Installing Git.

Read the full article at https://realpython.com/python-data-version-control/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python Office Hours

2020-08-18T14:00:00+00:00

The Real Python Office Hours is a weekly hangout where members of Real Python get the chance to meet fellow Pythonistas to chat about your learning progress, ask questions, and discuss Python tips & tricks via screen sharing.

In this video course, you can:

View previous office hours meetings
Download the chat transcript from each meeting
Explore resources and bonus materials discussed during each meeting

Office Hours Schedule & Registration: Join us live for the next members-only Q&A session with the Real Python Team: View upcoming Office Hours events »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A Practical Introduction to Web Scraping in Python

2020-08-17T16:11:05+00:00

Web scraping is the process of collecting and parsing raw data from the Web, and the Python community has come up with some pretty powerful web scraping tools.

The Internet hosts perhaps the greatest source of information—and misinformation—on the planet. Many disciplines, such as data science, business intelligence, and investigative reporting, can benefit enormously from collecting and analyzing data from websites.

In this tutorial, you’ll learn how to:

Parse website data using string methods and regular expressions
Parse website data using an HTML parser
Interact with forms and other website components

Note: This tutorial is adapted from the chapter “Interacting With the Web” in Python Basics: A Practical Introduction to Python 3.

The book uses Python’s built-in IDLE editor to create and edit Python files and interact with the Python shell, so you will see occasional references to IDLE throughout this tutorial. However, you should have no problems running the example code from the editor and environment of your choice.

Free Bonus: Click here to get our free Python Cheat Sheet that shows you the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Scrape and Parse Text From Websites#

Collecting data from websites using an automated process is known as web scraping. Some websites explicitly forbid users from scraping their data with automated tools like the ones you’ll create in this tutorial. Websites do this for two possible reasons:

The site has a good reason to protect its data. For instance, Google Maps doesn’t let you request too many results too quickly.
Making many repeated requests to a website’s server may use up bandwidth, slowing down the website for other users and potentially overloading the server such that the website stops responding entirely.

Important: Before using your Python skills for web scraping, you should always check your target website’s acceptable use policy to see if accessing the website with automated tools is a violation of its terms of use. Legally, web scraping against the wishes of a website is very much a gray area.

Please be aware that the following techniques may be illegal when used on websites that prohibit web scraping.

Let’s start by grabbing all the HTML code from a single web page. You’ll use a page on Real Python that’s been set up for use with this tutorial.

Your First Web Scraper#

One useful package for web scraping that you can find in Python’s standard library is urllib, which contains tools for working with URLs. In particular, the urllib.request module contains a function called urlopen() that can be used to open a URL within a program.

In IDLE’s interactive window, type the following to import urlopen():

>>>

>>> from urllib.request import urlopen

The web page that we’ll open is at the following URL:

>>>

>>> url = "http://olympus.realpython.org/profiles/aphrodite"

To open the web page, pass url to urlopen():

>>>

>>> page = urlopen(url)

urlopen() returns an HTTPResponse object:

>>>

>>> page
<http.client.HTTPResponse object at 0x105fef820>

To extract the HTML from the page, first use the HTTPResponse object’s .read() method, which returns a sequence of bytes. Then use .decode() to decode the bytes to a string using UTF-8:

>>>

>>> html_bytes = page.read()
>>> html = html_bytes.decode("utf-8")

Now you can print the HTML to see the contents of the web page:

>>>

>>> print(html)
<html>
<head>
<title>Profile: Aphrodite</title>
</head>
<body bgcolor="yellow">
<center>
<br><br>
<img src="/static/aphrodite.gif" />
<h2>Name: Aphrodite</h2>
<br><br>
Favorite animal: Dove
<br><br>
Favorite color: Red
<br><br>
Hometown: Mount Olympus
</center>
</body>
</html>

Read the full article at https://realpython.com/python-web-scraping-practical-introduction/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #22: Create Cross-Platform Python GUI Apps With BeeWare

2020-08-14T12:00:00+00:00

Do you want to distribute your Python applications to other users who don't have or even use Python? Maybe you're interested in seeing your Python application run on iOS or Android mobile devices. This week on the show we have Russell Keith-Magee, the founder and maintainer of the BeeWare project. Russell talks about Briefcase, a tool that converts a Python application into native installers on macOS, Windows, Linux, and mobile devices.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Identify Invalid Python Syntax

2020-08-11T14:00:00+00:00

Python is known for its simple syntax. However, when you’re learning Python for the first time or when you’ve come to Python with a solid background in another programming language, you may run into some things that Python doesn’t allow. If you’ve ever received a SyntaxError when trying to run your Python code, then this guide can help you. Throughout this course, you’ll see common examples of invalid syntax in Python and learn how to resolve the issue.

By the end of this course, you’ll be able to:

Identify invalid syntax in Python
Make sense of SyntaxError tracebacks
Resolve invalid syntax or prevent it altogether

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #21: Exploring K-means Clustering and Building a Gradebook With Pandas

2020-08-07T12:00:00+00:00

Do you want to learn the how and when of implementing K-means clustering in Python? Would you like to practice your pandas skills with a real-world project? This week on the show, David Amos is back with another batch of PyCoder’s Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Practical Recipes for Working With Files in Python

2020-08-04T14:00:00+00:00

Python has several built-in modules and functions for handling files. These functions are spread out over several modules such as os, os.path, shutil, and pathlib, to name a few. This course gathers in one place many of the functions you need to know in order to perform the most common operations on files in Python.

In this course, you’ll learn how to:

Retrieve file properties
Create directories
Match patterns in filenames
Traverse directory trees
Make temporary files and directories
Delete files and directories
Copy, move, or rename files and directories
Create and extract ZIP archives

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #20: Building PDFs in Python with ReportLab

2020-07-31T12:00:00+00:00

Have you wanted to generate advanced reports as PDFs using Python? Maybe you want to build documents with tables, images, or fillable forms. This week on the show we have Mike Driscoll to talk about his book "ReportLab - PDF Processing with Python."

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python's None: Null in Python

2020-07-28T14:00:00+00:00

If you have experience with other programming languages, like C or Java, then you’ve probably heard of the concept of null. Many languages use this to represent a pointer that doesn’t point to anything, to denote when a variable is empty, or to mark default parameters that you haven’t yet supplied. null is often defined to be 0 in those languages, but null in Python is different.

Python uses the keyword None to define null objects and variables. While None does serve some of the same purposes as null in other languages, it’s another beast entirely. As the null in Python, None is not defined to be 0 or any other value. In Python, None is an object and a first-class citizen!

In this course, you’ll learn:

What None is and how to test for it
When and why to use None as a default parameter
What None and NoneType mean in your traceback
How to use None in type checking
How null in Python works under the hood

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #19: Advanced Python Import Techniques and Managing Users in Django

2020-07-24T12:00:00+00:00

Would you like to clearly understand what's happening when you use the Python import keyword? Do you want to use modules more effectively to structure your code? Or maybe you're ready to move to the next level with your Django project by adding user management. This week on the show, David Amos is back with another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Mastering Python's Built-in time Module

2020-07-21T14:00:00+00:00

The Python time module provides many ways of representing time in code, such as objects, numbers, and strings. It also provides functionality other than representing time, like waiting during code execution and measuring the efficiency of your code.

This course will walk you through the most commonly used functions and objects in time.

By the end of this course, you’ll be able to:

Understand core concepts at the heart of working with dates and times, such as epochs, time zones, and daylight savings time
Represent time in code using floats, tuples, and struct_time
Convert between different time representations
Suspend thread execution
Measure code performance using perf_counter()

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #18: Ten Years of Flask: Conversation With Creator Armin Ronacher

2020-07-17T12:00:00+00:00

This week on the show we have Armin Ronacher to talk about the first 10 years of Flask. Armin talks about the origins of Flask and the components that make up the framework. He talks about what goes into documenting a framework or API. He also talks about the community working on the ongoing development of Flask.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Grow Your Python Portfolio With 13 Intermediate Project Ideas

2020-07-14T14:00:00+00:00

Now that you know the basics of Python, you can put that knowledge to use by building projects to put in your portfolio. The trick is finding project ideas that are just right for your level. Creating a variety of applications is a way to demonstrate your knowledge and share it with others.

In this course, you’ll get 13 project ideas that you can work on as an intermediate Python developer. The project ideas cover a wide range of application type and can be built using a command line, web based, or graphical user interface. You’ll see examples of existing applications for inspiration, and get links to Python libraries and resources to assist in your application development.

In this course, you’ll learn:

The importance of building projects
The major platforms you can build projects for
Thirteen project ideas you can work on
Some tips for working on projects

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

The Real Python Podcast – Episode #27: Preparing for an Interview With Python Practice Problems

Numbers in Python

Integers and Floating-Point Numbers#

Integers#

Floating-Point Numbers#

Command Line Interfaces in Python

Plot With Pandas: Python Data Visualization for Beginners

Set Up Your Environment#

Create Your First Pandas Plot#

The Real Python Podcast – Episode #26: 5 Years Podcasting Python with Michael Kennedy: Growth, GIL, Async, and More

Find & Fix Code Bugs in Python: Debug With IDLE

Use the Debug Control Window#

The Debug Control Window: An Overview#

The Step Button#

Exploring HTTPS and Cryptography in Python

Video Subtitles & Transcripts Now Available on Real Python

The Real Python Podcast – Episode #25: Data Version Control in Python and Real Python Video Transcripts

PyTorch vs Tensorflow for Your Python Deep Learning Project

What Is TensorFlow?#

Who Uses TensorFlow?#

Code Style and Function#

Editing Excel Spreadsheets in Python With openpyxl

Python 3 Installation & Setup Guide

How to Install Python on Windows#

How to Check Your Python Version on Windows#

What Your Options Are#

The Real Python Podcast – Episode #24: Options for Packaging Your Python Application: Wheels, Docker, and More

Common Python Data Structures (Guide)

Dictionaries, Maps, and Hash Tables#

dict: Your Go-To Dictionary#

collections.OrderedDict: Remember the Insertion Order of Keys#

Django Redirects

Python mmap: Improved File I/O With Memory Mapping

Understanding Computer Memory#

Physical Memory#

Virtual Memory#

Shared Memory#

Digging Deeper Into File I/O#

The Real Python Podcast – Episode #23: Python Wheels and Pass by Reference in Python

Data Version Control With Python and DVC

What Is Data Version Control?#

What Is DVC?#

Set Up Your Working Environment#

Real Python Office Hours

A Practical Introduction to Web Scraping in Python

Scrape and Parse Text From Websites#

Your First Web Scraper#

The Real Python Podcast – Episode #22: Create Cross-Platform Python GUI Apps With BeeWare

Identify Invalid Python Syntax

The Real Python Podcast – Episode #21: Exploring K-means Clustering and Building a Gradebook With Pandas

Practical Recipes for Working With Files in Python

The Real Python Podcast – Episode #20: Building PDFs in Python with ReportLab

Python's None: Null in Python

The Real Python Podcast – Episode #19: Advanced Python Import Techniques and Managing Users in Django

Mastering Python's Built-in time Module

The Real Python Podcast – Episode #18: Ten Years of Flask: Conversation With Creator Armin Ronacher

Grow Your Python Portfolio With 13 Intermediate Project Ideas

`dict`: Your Go-To Dictionary#

`collections.OrderedDict`: Remember the Insertion Order of Keys#