Real Python

The Pandas DataFrame: Working With Data Efficiently

2021-07-27T14:00:00+00:00

The Pandas DataFrame is a structure that contains two-dimensional data and its corresponding labels. DataFrames are widely used in data science, machine learning, scientific computing, and many other data-intensive fields.

DataFrames are similar to SQL tables or the spreadsheets that you work with in Excel or Calc. In many cases, DataFrames are faster, easier to use, and more powerful than tables or spreadsheets because they’re an integral part of the Python and NumPy ecosystems.

In this course, you’ll learn:

What a Pandas DataFrame is and how to create one
How to access, modify, add, sort, filter, and delete data
How to handle missing values
How to work with time-series data
How to quickly visualize data

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python's collections: A Buffet of Specialized Data Types

2021-07-26T14:00:00+00:00

Python’s collections module provides a rich set of specialized container data types carefully designed to approach specific programming problems in a Pythonic and efficient way. The module also provides wrapper classes that make it safer to create custom classes that behave similar to the built-in types dict, list, and str.

Learning about the data types and classes in collections will allow you to grow your programming tool kit with a valuable set of reliable and efficient tools.

In this tutorial, you’ll learn how to:

Write readable and explicit code with namedtuple
Build efficient queues and stacks with deque
Count objects quickly with Counter
Handle missing dictionary keys with defaultdict
Guarantee the insertion order of keys with OrderedDict
Manage multiple dictionaries as a single unit with ChainMap

To better understand the data types and classes in collections, you should know the basics of working with Python’s built-in data types, such as lists, tuples, and dictionaries. Additionally, the last part of the article requires some basic knowledge about object-oriented programming in Python.

Free Download: Get a sample chapter from Python Tricks: The Book that shows you Python’s best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

Getting Started With Python’s `collections`

Back in Python 2.4, Raymond Hettinger contributed a new module called collections to the standard library. The goal was to provide various specialized collection data types to approach specific programming problems.

At that time, collections only included one data structure, deque, which was specially designed as a double-ended queue that supports efficient append and pop operations on either end of the sequence. From this point on, several modules in the standard library took advantage of deque to improve the performance of their classes and structures. Some outstanding examples are queue and threading.

With time, a handful of specialized container data types populated the module:

Data type	Python version	Description
`deque`	2.4	A sequence-like collection that supports efficient addition and removal of items from either end of the sequence
`defaultdict`	2.5	A dictionary subclass for constructing default values for missing keys and automatically adding them to the dictionary
`namedtuple()`	2.6	A factory function for creating subclasses of `tuple` that provides named fields that allow accessing items by name while keeping the ability to access items by index
`OrderedDict`	2.7, 3.1	A dictionary subclass that keeps the key-value pairs ordered according to when the keys are inserted
`Counter`	2.7, 3.1	A dictionary subclass that supports convenient counting of unique items in a sequence or iterable
`ChainMap`	3.3	A dictionary-like class that allows treating a number of mappings as a single dictionary object

Besides these specialized data types, collections also provides three base classes that facilitate the creations of custom lists, dictionaries, and strings:

Class	Description
`UserDict`	A wrapper class around a dictionary object that facilitates subclassing `dict`
`UserList`	A wrapper class around a list object that facilitates subclassing `list`
`UserString`	A wrapper class around a string object that facilitates subclassing `string`

The need for these wrapper classes was partially eclipsed by the ability to subclass the corresponding standard built-in data types. However, sometimes using these classes is safer and less error-prone than using standard data types.

With this brief introduction to collections and the specific use cases that the data structures and classes in this module can solve, it’s time to take a closer look at them. Before that, it’s important to point out that this tutorial is an introduction to collections as a whole. In most of the following sections, you’ll find a blue alert box that’ll guide you to a dedicated article on the class or function at hand.

Improving Code Readability: `namedtuple()`

Python’s namedtuple() is a factory function that allows you to create tuple subclasses with named fields. These fields give you direct access to the values in a given named tuple using the dot notation, like in obj.attr.

The need for this feature arose because using indices to access the values in a regular tuple is annoying, difficult to read, and error-prone. This is especially true if the tuple you’re working with has several items and is constructed far away from where you’re using it.

Note: Check out Write Pythonic and Clean Code With namedtuple for a deeper dive into how to use namedtuple in Python.

A tuple subclass with named fields that developers can access with the dot notation seemed like a desirable feature back in Python 2.6. That’s the origin of namedtuple(). The tuple subclasses you can build with this function are a big win in code readability if you compare them with regular tuples.

To put the code readability problem in perspective, consider divmod(). This built-in function takes two (non-complex) numbers and returns a tuple with the quotient and remainder that result from the integer division of the input values:

>>>

>>> divmod(12, 5)
(2, 2)

It works nicely. However, is this result readable? Can you tell what the meaning of each number in the output is? Fortunately, Python offers a way to improve this. You can code a custom version of divmod() with an explicit result using namedtuple:

>>>

>>> from collections import namedtuple

>>> def custom_divmod(x, y):
...     DivMod = namedtuple("DivMod", "quotient remainder")
...     return DivMod(*divmod(x, y))
...

>>> result = custom_divmod(12, 5)
>>> result
DivMod(quotient=2, remainder=2)

>>> result.quotient
2
>>> result.remainder
2

Now you know the meaning of each value in the result. You can also access each independent value using the dot notation and a descriptive field name.

Read the full article at https://realpython.com/python-collections-module/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #70: What Can You Do With Python and Counting Objects Using "Counter"

2021-07-23T12:00:00+00:00

How is Python being used today, and what can you do with the language? Do you want to develop software, dive into data science and math, automate parts of your job and digital life, or work with electronics? This week on the show, David Amos is back, and he's brought another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Your First Steps With Django: Set Up a Django Project

2021-07-21T14:05:12+00:00

Before you can start to build the individual functionality of a new Django web application, you always need to complete a couple of setup steps. This tutorial gives you a reference for the necessary steps to set up a Django project.

The tutorial focuses on the initial steps you’ll need to take to start a new web application. To complete it, you’ll need to have Python installed and understand how to work with virtual environments and Python’s package manager, pip. While you won’t need much programming knowledge to complete this setup, you’ll need to know Python to do anything interesting with the resulting project scaffolding.

By the end of this tutorial, you’ll know how to:

Set up a virtual environment
Install Django
Pin your project dependencies
Set up a Django project
Start a Django app

Use this tutorial as your go-to reference until you’ve built so many projects that the necessary commands become second nature. Until then, follow the steps outlined below. There are also a few exercises throughout the tutorial to help reinforce what you’ve learned.

Free Bonus: Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.

Prepare Your Environment

When you’re ready to start your new Django web application, create a new folder and navigate into it. In this folder, you’ll set up a new virtual environment using your command line:

$ python3 -m venv env

This command sets up a new virtual environment named env in your current working directory. Once the process is complete, you also need to activate the virtual environment:

$ source env/bin/activate

If the activation was successful, then you’ll see the name of your virtual environment, (env), at the beginning of your command prompt. This means that your environment setup is complete.

You can learn more about how to work with virtual environments in Python, and how to perfect your Python development setup, but for your Django setup, you have all you need. You can continue with installing the django package.

Install Django and Pin Your Dependencies

Once you’ve created and activated your Python virtual environment, you can install Django into this dedicated development workspace:

(env) $ python -m pip install django

This command fetches the django package from the Python Package Index (PyPI) using pip. After the installation has completed, you can pin your dependencies to make sure that you’re keeping track of which Django version you installed:

Read the full article at https://realpython.com/django-setup/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Speech Recognition With Python

2021-07-20T14:00:00+00:00

Have you ever wondered how to add speech recognition to your Python project? It’s more straightforward than you might think. In this course, you’ll find out how.

In this course, you’ll learn:

How speech recognition works
What speech recognition packages are available on PyPI
How to install and use the SpeechRecognition package—a full-featured and straightforward Python speech recognition library

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Community Interview With Dustin Ingram

2021-07-19T14:00:00+00:00

Today I’m joined by Dustin Ingram, a developer advocate at Google focused on supporting the Python community on Google Cloud. He is also a director of the Python Software Foundation (PSF) and a maintainer of PyPI. In this interview, we discuss how Google’s use of Python might differ from your own, what it takes to maintain PyPI, his goals as a PSF director, his love of PyCons, and more.

Ricky: Thanks for joining me, Dustin. I’d like to start with my usual questions: how did you get into programming, and when did you start using Python?

Dustin: Thanks for having me!

I like to say that my first programming experience was when someone decided it was allowed—if not encouraged—for me to carry around a TI-83 calculator throughout high school. I spent far less time in geometry class learning geometry and far more time writing text-based games in BASIC and, later, programs that would do my physics homework for me.

I even released my first open source program: a little animation that made your calculator screen look like the flowing, garbled characters in The Matrix (which I was obsessed with) and published it to Texas Instruments’ third-party software repository.

But my first actual exposure to programming was probably when I was first introduced to Logo, the graphical programming language with the turtle as a cursor. I think it was elementary school, and the task was to try and reproduce some simple shapes, but I distinctly remember being blown away by the endless possibilities. I wasn’t confined to a predetermined path like a lot of the point-and-click computer games I had played before—I could make that turtle do anything.

My dad is a tool designer, and when I was in high school, he ran a machine shop. The shop had a lot of the classic metalworking tools: lathes and milling machines that were operated mostly by hand. But they also had some newer computer numerical control (CNC) machines that were programmed with a language called G-code.

If you’ve never seen G-code before, the best I can describe it is as a mashup of BASIC and Logo, but instead of a cute little turtle moving around, you have a drill bit spinning at several thousand rotations per minute attached to a half-ton machine. All the oldheads who were used to operating the classic, manual machines couldn’t quite wrap their head around this new technology, but it made total sense to me!

Later in high school, I took some computer science classes that taught me some Java and convinced me that the field was worth pursuing. I went to college for computer science and encountered Scheme and other Lisps, some C and C++, but mostly more Java.

Around that time, I started working at a research lab for the university, and while that was all Java too, some of the older, wiser grad students there were really into this new language called Python and were basically trying to sneak it into as many of their projects as they could. Once I was exposed to it, I was hooked too and started trying to write as much of it as I could.

Read the full article at https://realpython.com/interview-dustin-ingram/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #69: Planning a Faster Future at the Python Language Summit

2021-07-16T12:00:00+00:00

Do you wonder what the future may hold for the Python language? Are there speed improvements coming soon? What if you could be in the room while the core developers discuss Python's future? This week on the show, we have Joanna Jablonski, who was invited to the Python Language Summit 2021 as a journalist to summarize and document the event.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Using FastAPI to Build Python Web APIs

2021-07-14T14:00:00+00:00

Creating APIs, or application programming interfaces, is an important part of making your software accessible to a broad range of users. In this tutorial, you will learn the main concepts of FastAPI and how to use it to quickly create web APIs that implement best practices by default.

By the end of it, you will be able to start creating production-ready web APIs, and you will have the understanding needed to go deeper and learn more for your specific use cases.

In this tutorial, you’ll learn how to:

Use path parameters to get a unique URL path per item
Receive JSON data in your requests using pydantic
Use API best practices, including validation, serialization, and documentation
Continue learning about FastAPI for your use cases

This tutorial is written by the author of FastAPI. It contains a careful selection of fragments from the official documentation, avoiding getting lost in technical details while helping you get up to speed as fast as possible.

To get the most out of this tutorial, it would be helpful for you to know the basics of what HTTP is and how it works, what JSON is, and Python type hints. You will also benefit from using a virtual environment, as is the case for any Python project.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you’ll need to take your Python skills to the next level.

What Is FastAPI?

FastAPI is a modern, high-performance web framework for building APIs with Python based on standard type hints. It has the following key features:

Fast to run: It offers very high performance, on par with NodeJS and Go, thanks to Starlette and pydantic.
Fast to code: It allows for significant increases in development speed.
Reduced number of bugs: It reduces the possibility for human-induced errors.
Intuitive: It offers great editor support, with completion everywhere and less time debugging.
Straightforward: It’s designed to be uncomplicated to use and learn, so you can spend less time reading documentation.
Short: It minimizes code duplication.
Robust: It provides production-ready code with automatic interactive documentation.
Standards-based: It’s based on the open standards for APIs, OpenAPI and JSON Schema.

The framework is designed to optimize your developer experience so that you can write simple code to build production-ready APIs with best practices by default.

Install FastAPI

As with any other Python project, it would be best to start by creating a virtual environment. If you are not familiar with how to do that, then you can check out the Primer on Virtual Environments.

The first step is to install FastAPI and Uvicorn using pip:

$ python -m pip install fastapi uvicorn[standard]

With that, you have FastAPI and Uvicorn installed and are ready to learn how to use them. FastAPI is the framework you’ll use to build your API, and Uvicorn is the server that will use the API you build to serve requests.

First Steps

To get started, in this section, you will create a minimal FastAPI app, run it with a server using Uvicorn, and then learn all the interacting parts. This will give you a very quick overview of how everything works.

Create a First API

A basic FastAPI file looks like this:

# main.py

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def root():
    return {"message": "Hello World"}

Copy the code above to a file named main.py, and just like that, you have a fully functional API application with some best practices like automatic documentation and serialization built in. You will learn more about those features next.

This code defines your application, but it won’t run on itself if you call it with python directly. To run it, you need a server program. In the steps above, you already installed Uvicorn. That will be your server.

Run the First API App With Uvicorn

Run the live server using Uvicorn:

Read the full article at https://realpython.com/fastapi-python-web-apis/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Square Root Function in Python

2021-07-13T14:00:00+00:00

Are you trying to solve a quadratic equation? Maybe you need to calculate the length of one side of a right triangle. You can use the math module’s sqrt() method for determining the square root of a number. This course covers the use of math.sqrt() as well as related methods.

In this course, you’ll learn:

About square roots and related mathematical operations
How to use the Python square root function, sqrt()
Where sqrt() can be useful in real-world examples

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python News: What's New From June 2021?

2021-07-12T14:00:00+00:00

If you want to get up to speed on what happened in the world of Python in June 2021, then you’ve come to the right place to get your news!

June was a month of change. Ewa Jodlowska, the Executive Director of the Python Software Foundation (PSF), announced her departure after serving for ten years, and the PSF Board of Directors gained three new directors.

Let’s dive into the biggest Python news from the past month!

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you’ll need to take your Python skills to the next level.

Ewa Jodlowska Steps Down as PSF Executive Director

On June 16, the Python Software Foundation (PSF) announced that Executive Director Ewa Jodlowska has decided to leave the foundation at the end of 2021.

Even if you haven’t heard of Ewa, you’ve undoubtedly been impacted by her work. As Executive Director, Ewa is responsible for ensuring that the PSF achieves its mission “to promote, protect, and advance the Python programming language, and to support and facilitate the growth of a diverse and international community of Python programmers.”

The PSF is responsible for managing and maintaining the python.org website as well as the Python Package Index. The PSF also supports and funds events and workshops for organizations such as Django Girls and PyLadies as well as other Python conferences and meetups worldwide.

In addition to managing community outreach, the PSF supports and funds Python software development. This includes organizing, funding, and supporting CPython sprints. As you’ll learn later in this article, the PSF’s funding and support go beyond support for the core Python language. The PSF also helps Python libraries with various grants and fiscal sponsorship programs.

Read the full article at https://realpython.com/python-news-june-2021/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #68: Exploring the functools Module and Complex Numbers in Python

2021-07-09T12:00:00+00:00

Are you ready to expand your Python knowledge into the intermediate to advanced territory? What tools are awaiting your discovery inside Python's functools module? This week on the show, David Amos is back, and he's brought another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

What Can I Do With Python?

2021-07-07T14:00:00+00:00

You’ve finished a course or finally made it to the end of a book that teaches you the basics of programming with Python. You’ve learned about variables, lists, tuples, dictionaries, for and while loops, conditional statements, object-oriented concepts, and more. So, what’s next? What can you do with Python nowadays?

Python is a versatile programming language with many use cases in a variety of different fields. If you’ve grasped the basics of Python and are itching to build something with the language, then it’s time to figure out what your next step should be.

In this article, you’ll see how you can use Python for:

Doing general software development
Diving into data science and math
Speeding up and automating your workflow
Building embedded systems and robots

You’ll also find ideas for practical projects, resources, and tutorials that you can use to start building things with Python right away.

Free Bonus: Get a sample chapter from Python Basics: A Practical Introduction to Python 3 to see how you can go from beginner to intermediate in Python with a complete curriculum, up to date for Python 3.9.

Python in the Real World

Python is a high-level and general-purpose programming language. As this definition implies, you can use Python for several purposes, from web development to data science, machine learning, and robotics. Python’s real-world use cases are limitless.

You’re probably wondering what people are successfully building with Python. If you take a quick look at companies using the language, then you’ll find world-class companies, such as Google, YouTube, Facebook, Instagram, Spotify, Netflix, and more.

Google has used Python from the start, and it’s gained a place as one of the tech giant’s main server-side languages. Guido van Rossum, Python’s creator, worked there for several years, overseeing the language’s development.

Instagram likes Python for its simplicity. The service is known for running “the world’s largest deployment of the Django web framework, which is written entirely in Python.”

Spotify uses the language for data analysis and back-end services. According to its team, Python’s ease of use leads to a lightning-fast development pipeline. Spotify performs a ton of analysis to give recommendations to its users, so it needs a productive tool that works well. Python to the rescue!

You’ll also find that Python has been vital for science and space exploration, with a lot of exciting use cases in robotics and hardware control.

In this article, you’ll see how you can use your Python skills in a wide range of areas.

Develop Cool Software

Python’s ecosystem provides a rich set of frameworks, tools, and libraries that allow you to write almost any kind of application. You can use Python to build applications for the Web as well as desktop and mobile platforms. You can even use Python to create video games.

Web Development

Developing web applications with Python is one of the most in-demand skills, with a lot of opportunities for you out there. In this field, you’ll find several useful Python frameworks, libraries, and tools for developing cool web applications, APIs, and more. Here are some of the most popular Python web frameworks:

Framework	Description
Django	Django is a high-level framework that encourages rapid web application development with a clean and pragmatic design. It allows you to focus on writing your applications without having to reinvent the wheel.
FastAPI	FastAPI is a fast and performant web framework for building web APIs. It’s built on top of modern Python type hint features and enables asynchronous programming.
Flask	Flask is a lightweight framework for creating WSGI web applications. It allows you to get started quickly and to scale up to complex applications if needed.
Tornado	Tornado is a web framework and asynchronous networking library. It uses non-blocking network I/O, so you can write applications that can scale to tens of thousands of open connections.

To get started with web development, check out:

If you want some practical project ideas for applying your web development skills right away, then you can build a portfolio web application with Django. With so many jobs and career opportunities out there, it’s a great idea to have a personal portfolio these days, so go ahead and give it a try. You don’t need to know anything about Django to get started with this step-by-step tutorial. It’s perfect if you’re itching to get your hands dirty with web development in Python.

CLI Development

Another field in which Python shines is command-line interface (CLI) application development. CLI applications are everywhere and allow you to automate repetitive and boring tasks in your day-to-day work by creating small and large tools for your command line.

In Python, you have an impressive set of CLI libraries and frameworks that can make your life more pleasant and help you build command-line tools quickly:

Read the full article at https://realpython.com/what-can-i-do-with-python/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Defining and Calling Python Functions

2021-07-06T14:00:00+00:00

A function is a self-contained block of code that encapsulates a specific task or related group of tasks. This course will show you how to define your own Python function. You’ll learn when to divide your program into separate user-defined functions and what tools you’ll need to do this.

You’ll also learn the various ways to pass data into a function when calling it, which allows for different behavior from one invocation to the next.

In this course, you’ll learn:

How functions work in Python and why they’re beneficial
How to define and call your own Python function
Mechanisms for passing arguments to your function
Some differences between how to work with functions in Python vs C++
How to return data from your function back to the calling environment

This course will be the most helpful for you if you’re already familiar with the fundamental concepts of Python, including basic data types, lists and tuples, dictionaries, the import statement, conditional statements, and for loops.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python's Counter: The Pythonic Way to Count Objects

2021-07-05T14:00:00+00:00

Counting several repeated objects at once is a common problem in programming. Python offers a bunch of tools and techniques you can use to approach this problem. However, Python’s Counter from collections provides a clean, efficient, and Pythonic solution.

This dictionary subclass provides efficient counting capabilities out of the box. Understanding Counter and how to use it efficiently is a convenient skill to have as a Python developer.

In this tutorial, you’ll learn how to:

Count several repeated objects at once
Create counters with Python’s Counter
Retrieve the most common objects in a counter
Update object counts
Use Counter to facilitate further computations

You’ll also learn about the basics of using Counter as a multiset, which is an additional feature of this class in Python.

Free Bonus: 5 Thoughts On Python Mastery, a free course for Python developers that shows you the roadmap and the mindset you’ll need to take your Python skills to the next level.

Counting Objects in Python

Sometimes you need to count the objects in a given data source to know how often they occur. In other words, you need to determine their frequency. For example, you might want to know how often a specific item appears in a list or sequence of values. When your list is short, counting the items can be straightforward and quick. However, when you have a long list, counting things can be more challenging.

To count objects, you typically use a counter, which is an integer variable with an initial value of zero. Then you increment the counter to reflect the number of times a given object appears in the input data source.

When you’re counting the occurrences of a single object, you can use a single counter. However, when you need to count several different objects, you have to create as many counters as unique objects you have.

To count several different objects at once, you can use a Python dictionary. The dictionary keys will store the objects you want to count. The dictionary values will hold the number of repetitions of a given object, or the object’s count.

For example, to count the objects in a sequence using a dictionary, you can loop over the sequence, check if the current object isn’t in the dictionary to initialize the counter (key-value pair), and then increment its count accordingly.

Here’s an example that counts the letters in the word “Mississippi”:

>>>

>>> word = "mississippi"
>>> counter = {}

>>> for letter in word:
...     if letter not in counter:
...         counter[letter] = 0
...     counter[letter] += 1
...

>>> counter
{'m': 1, 'i': 4, 's': 4, 'p': 2}

The for loop iterates over the letters in word. In each iteration, the conditional statement checks if the letter at hand isn’t already a key in the dictionary you’re using as counter. If so, it creates a new key with the letter and initializes its count to zero. The final step is to increment the count by one. When you access counter, you see that the letters work as keys and the values as counts.

Note: When you’re counting several repeated objects with Python dictionaries, keep in mind that they must be hashable because they’ll work as dictionary keys. Being hashable means that your objects must have a hash value that never changes during their lifetime. In Python, immutable objects are also hashable.

Another way to count objects with a dictionary is to use dict.get() with 0 as a default value:

>>>

>>> word = "mississippi"
>>> counter = {}

>>> for letter in word:
...     counter[letter] = counter.get(letter, 0) + 1
...

>>> counter
{'m': 1, 'i': 4, 's': 4, 'p': 2}

When you call .get() this way, you get the current count of a given letter, or 0 (the default) if the letter is missing. Then you increment the count by 1 and store it under the corresponding letter in the dictionary.

You can also use defaultdict from collections to count objects within a loop:

>>>

>>> from collections import defaultdict

>>> word = "mississippi"
>>> counter = defaultdict(int)

>>> for letter in word:
...     counter[letter] += 1
...

>>> counter
defaultdict(<class 'int'>, {'m': 1, 'i': 4, 's': 4, 'p': 2})

This solution is more concise and readable. You first initialize the counter using a defaultdict with int() as a default factory function. This way, when you access a key that doesn’t exist in the underlying defaultdict, the dictionary automatically creates the key and initializes it with the value that the factory function returns.

In this example, since you’re using int() as a factory function, the initial value is 0, which results from calling int() without arguments.

Like with many other frequent tasks in programming, Python provides a better way to approach the counting problem. In collections, you’ll find a class specially designed to count several different objects in one go. This class is conveniently called Counter.

Getting Started With Python’s `Counter`

Read the full article at https://realpython.com/python-counter/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #67: Securing Your Python Software Supply Chain With Dustin Ingram

2021-07-02T12:00:00+00:00

How well do you know your software supply chain? When you PIP install a package, what steps can you take to minimize the risk of installing something malicious? This week on the show, we have Dustin Ingram, a director of the Python Software Foundation (PSF) and a maintainer of the Python Package Index (PyPI).

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Beautiful Soup: Build a Web Scraper With Python

2021-06-30T12:58:06+00:00

The incredible amount of data on the Internet is a rich resource for any field of research or personal interest. To effectively harvest that data, you’ll need to become skilled at web scraping. The Python libraries requests and Beautiful Soup are powerful tools for the job. If you like to learn with hands-on examples and have a basic understanding of Python and HTML, then this tutorial is for you.

In this tutorial, you’ll learn how to:

Inspect the HTML structure of your target site with your browser’s developer tools
Decipher data encoded in URLs
Use requests and Beautiful Soup for scraping and parsing data from the Web
Step through a web scraping pipeline from start to finish
Build a script that fetches job offers from the Web and displays relevant information in your console

Working through this project will give you the knowledge of the process and tools you need to scrape any static website out there on the World Wide Web. You can download the project source code by clicking on the link below:

Get Sample Code: Click here to get the sample code you’ll use for the project and examples in this tutorial.

Let’s get started!

What Is Web Scraping?

Web scraping is the process of gathering information from the Internet. Even copying and pasting the lyrics of your favorite song is a form of web scraping! However, the words “web scraping” usually refer to a process that involves automation. Some websites don’t like it when automatic scrapers gather their data, while others don’t mind.

If you’re scraping a page respectfully for educational purposes, then you’re unlikely to have any problems. Still, it’s a good idea to do some research on your own and make sure that you’re not violating any Terms of Service before you start a large-scale project.

Reasons for Web Scraping

Say you’re a surfer, both online and in real life, and you’re looking for employment. However, you’re not looking for just any job. With a surfer’s mindset, you’re waiting for the perfect opportunity to roll your way!

There’s a job site that offers precisely the kinds of jobs you want. Unfortunately, a new position only pops up once in a blue moon, and the site doesn’t provide an email notification service. You think about checking up on it every day, but that doesn’t sound like the most fun and productive way to spend your time.

Thankfully, the world offers other ways to apply that surfer’s mindset! Instead of looking at the job site every day, you can use Python to help automate your job search’s repetitive parts. Automated web scraping can be a solution to speed up the data collection process. You write your code once, and it will get the information you want many times and from many pages.

In contrast, when you try to get the information you want manually, you might spend a lot of time clicking, scrolling, and searching, especially if you need large amounts of data from websites that are regularly updated with new content. Manual web scraping can take a lot of time and repetition.

There’s so much information on the Web, and new information is constantly added. You’ll probably be interested in at least some of that data, and much of it is just out there for the taking. Whether you’re actually on the job hunt or you want to download all the lyrics of your favorite artist, automated web scraping can help you accomplish your goals.

Challenges of Web Scraping

The Web has grown organically out of many sources. It combines many different technologies, styles, and personalities, and it continues to grow to this day. In other words, the Web is a hot mess! Because of this, you’ll run into some challenges when scraping the Web:

Variety: Every website is different. While you’ll encounter general structures that repeat themselves, each website is unique and will need personal treatment if you want to extract the relevant information.
Durability: Websites constantly change. Say you’ve built a shiny new web scraper that automatically cherry-picks what you want from your resource of interest. The first time you run your script, it works flawlessly. But when you run the same script only a short while later, you run into a discouraging and lengthy stack of tracebacks!

Unstable scripts are a realistic scenario, as many websites are in active development. Once the site’s structure has changed, your scraper might not be able to navigate the sitemap correctly or find the relevant information. The good news is that many changes to websites are small and incremental, so you’ll likely be able to update your scraper with only minimal adjustments.

However, keep in mind that because the Internet is dynamic, the scrapers you’ll build will probably require constant maintenance. You can set up continuous integration to run scraping tests periodically to ensure that your main script doesn’t break without your knowledge.

An Alternative to Web Scraping: APIs

Some website providers offer application programming interfaces (APIs) that allow you to access their data in a predefined manner. With APIs, you can avoid parsing HTML. Instead, you can access the data directly using formats like JSON and XML. HTML is primarily a way to present content to users visually.

When you use an API, the process is generally more stable than gathering the data through web scraping. That’s because developers create APIs to be consumed by programs rather than by human eyes.

The front-end presentation of a site might change often, but such a change in the website’s design doesn’t affect its API structure. The structure of an API is usually more permanent, which means it’s a more reliable source of the site’s data.

However, APIs can change as well. The challenges of both variety and durability apply to APIs just as they do to websites. Additionally, it’s much harder to inspect the structure of an API by yourself if the provided documentation lacks quality.

Read the full article at https://realpython.com/beautiful-soup-web-scraper-python/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Inner Functions

2021-06-29T14:00:00+00:00

Python allows the declaration of functions inside of other functions. Inner functions, also known as nested functions, are defined within a function. This type of function has direct access to variables and names defined in the enclosing function in Python. Inner functions have many uses, most notably as closure factories and decorator functions.

In this course, you’ll learn how to:

Define inner functions
Use inner functions as helper functions
Build function closures
Use captured variables in a closure
Use captured data functions in a closure
Work with decorators

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Reverse Python Lists: Beyond .reverse() and reversed()

2021-06-28T14:00:00+00:00

Are you diving deeper into Python lists and wanting to learn about different ways to reverse them? If so, then this tutorial is for you. Here, you’ll learn about a few Python tools and techniques that are handy when it comes to reversing lists or manipulating them in reverse order. This knowledge will complement and improve your list-related skills and make you more proficient with them.

In this tutorial, you’ll learn how to:

Reverse existing lists in place using .reverse() and other techniques
Create reversed copies of existing lists using reversed() and slicing
Use iteration, comprehensions, and recursion to create reversed lists
Iterate over your lists in reverse order
Sort your lists in reverse order using .sort() and sorted()

To get the most out of this tutorial, it would be helpful to know the basics of iterables, for loops, lists, list comprehensions, generator expressions, and recursion.

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Reversing Python Lists

Sometimes you need to process Python lists starting from the last element down to the first—in other words, in reverse order. In general, there are two main challenges related to working with lists in reverse:

Reversing a list in place
Creating reversed copies of an existing list

To meet the first challenge, you can use either .reverse() or a loop that swaps items by index. For the second, you can use reversed() or a slicing operation. In the next sections, you’ll learn about different ways to accomplish both in your code.

Reversing Lists in Place

Like other mutable sequence types, Python lists implement .reverse(). This method reverses the underlying list in place for memory efficiency when you’re reversing large list objects. Here’s how you can use .reverse():

>>>

>>> digits = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> digits.reverse()
>>> digits
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

When you call .reverse() on an existing list, the method reverses it in place. This way, when you access the list again, you get it in reverse order. Note that .reverse() doesn’t return a new list but None:

>>>

>>> digits = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> reversed_digits = digits.reverse()
>>> reversed_digits is None
True

Trying to assign the return value of .reverse() to a variable is a common mistake related to using this method. The intent of returning None is to remind its users that .reverse() operates by side effect, changing the underlying list.

Note: Most of the examples in this tutorial use a list of numbers as input. However, the same tools and techniques apply to lists of any type of Python objects, such as lists of strings.

Okay! That was quick and straightforward! Now, how can you reverse a list in place by hand? A common technique is to loop through the first half of it while swapping each element with its mirror counterpart on the second half of the list.

Python provides zero-based positive indices to walk sequences from left to right. It also allows you to navigate sequences from right to left using negative indices:

This diagram shows that you can access the first element of the list (or sequence) using either 0 or -5 with the indexing operator, like in sequence[0] and sequence[-5], respectively. You can use this Python feature to reverse the underlying sequence in place.

For example, to reverse the list represented in the diagram, you can loop over the first half of the list and swap the element at index 0 with its mirror at index -1 in the first iteration. Then you can switch the element at index 1 with its mirror at index -2 and so on until you get the list reversed.

Here’s a representation of the whole process:

To translate this process into code, you can use a for loop with a range object over the first half of the list, which you can get with len(digits) // 2. Then you can use a parallel assignment statement to swap the elements, like this:

>>>

>>> digits = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> for i in range(len(digits) // 2):
...     digits[i], digits[-1 - i] = digits[-1 - i], digits[i]
...

>>> digits
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

Read the full article at https://realpython.com/python-reverse-list/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #66: Practicing Python With CSV Files and Extracting Values With "filter()"

2021-06-25T12:00:00+00:00

Are you ready to practice your Python skills some more? There is a new set of practice problems prepared for you to tackle, and this time they're based on working with CSV files. This week on the show, David Amos is back, and he's brought another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

CPython Internals: Paperback Now Available!

2021-06-23T14:00:00+00:00

After almost two years of writing, reviewing, and testing, we’re delighted to announce that CPython Internals: Your Guide to the Python 3 Interpreter is now available in paperback!

Are there certain parts of Python that just seem like magic? Once you see how Python works at the interpreter level, you’ll be able to optimize your applications and fully leverage the power of Python.

In CPython Internals, you’ll unlock the inner workings of the Python language, learn how to compile the Python interpreter from source code, and cover what you’ll need to know to confidently start contributing to CPython yourself!

In this article, you’ll see:

How you can take your Python skills to the next level with CPython Internals
How other Pythonistas have already been doing it

To skip right to the good stuff and get your hands on the book, click the link below:

Buy on Amazon »

You Can Get CPython Internals in Paperback!

It’s finally here! Readers have been showing us their copies and letting us know what they think:

This book is a sort of “missing manual” for CPython’s internals, and we’re looking forward to the Python community getting tons of knowledge and skills from this unique resource. We’re delighted with the final product, so here’s the author, Anthony Shaw, to walk you through the paperback version of the book:

Read the full article at https://realpython.com/cpython-internals-paperback/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python vs JavaScript for Python Developers

2021-06-22T14:00:00+00:00

Python isn’t the only language out there, and one of the other languages frequently fighting Python for the top of the “most popular” lists is JavaScript. JavaScript is the de facto language on the web but also has a robust toolset on the server side. This course explores JavaScript from a Python programmer’s perspective.

If you’ve never used JavaScript before or have felt overwhelmed by the quick pace of its evolution in recent years, then this course will set you on the right path. You should already know the basics of Python to benefit fully from the comparisons made between the two languages.

In this course, you will learn:

Where JavaScript comes from and where it is used
How JavaScript’s type system is different from Python’s
How to write functions in JavaScript
The two ways of creating objects in JavaScript
General language syntax in JavaScript
Surprises and behaviors in JavaScript that Python programmers wouldn’t expect

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #65: Expanding the International Python Community With the PSF

2021-06-18T12:00:00+00:00

The popularity of Python is continuing to grow Developers across the globe are embracing the language. How is Python being used in all of these different countries? How does an organization like the Python Software Foundation (PSF) work toward the goals in its mission statement for supporting and growing this international community? This week on the show, we have Marlene Mhangami, a PSF board member and part of the Diversity and Inclusion Work Group.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Using Pandas to Make a Gradebook in Python

2021-06-15T14:00:00+00:00

One of the jobs that all teachers have in common is evaluating students. Whether you use exams, homework assignments, quizzes, or projects, you usually have to turn students’ scores into a letter grade at the end of the term. This often involves a bunch of calculations that you might do in a spreadsheet. Instead, you can consider using Python and pandas.

In this course, you’ll learn how to:

Load and merge data from multiple sources with pandas
Filter and group data in a pandas DataFrame
Calculate and plot grades in a pandas DataFrame

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #64: Detecting Deforestation With Python & Using GraphQL With Django and Vue

2021-06-11T12:00:00+00:00

Are you looking for an in-depth data science project to practice your skills on? Perhaps you would like to add new tools to your Python web development projects instead? This week on the show, David Amos is back, and he's brought another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python Basics: Setting Up Python

2021-06-08T14:00:00+00:00

Setting up Python is the first step to becoming a Python programmer. In this course, you’ll learn how to download and install Python for Windows, macOS, and Ubuntu Linux and how to open Python’s Integrated Development and Learning Environment, IDLE.

There are many ways to install Python. You can download official Python distributions from Python.org, install from a package manager, and even install specialized distributions for scientific computing, Internet of Things, and embedded systems. This course focuses on official distributions, as they’re generally the best option for getting started with learning to program in Python.

This course can be enjoyed alone or as an accompaniment to Python Basics: A Practical Introduction to Python 3.

In this course, you’ll learn how to:

Install Python on Windows, macOS, and Linux
Open IDLE, Python’s integrated development and learning environment

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #63: Create Web Applications Using Only Python With Anvil

2021-06-04T12:00:00+00:00

What if you could create an application and deploy it to the web with just Python? Wouldn't it be nice to skip the additional full-stack development steps of learning three different languages in addition to Python? That's the idea behind Anvil. This week on the show, we have Meredydd Luff, co-founder of Anvil.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Explore Your Dataset With Pandas

2021-06-01T14:00:00+00:00

Do you have a large dataset that’s full of interesting insights, but you’re not sure where to start exploring it? Has your boss asked you to generate some statistics from it, but they’re not so easy to extract? These are precisely the use cases where Pandas and Python can help you! With these tools, you’ll be able to slice a large dataset down into manageable parts and glean insight from that information.

In this course, you’ll learn how to:

Calculate metrics about your data
Perform basic queries and aggregations
Discover and handle incorrect data, inconsistencies, and missing values
Visualize your data with plots

You’ll also learn about the differences between the main data structures that Pandas and Python use.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #62: Selecting the Ideal Data Structure & Unravelling Python's "pass" and "with"

2021-05-28T12:00:00+00:00

How do you know you're using the correct data structure for your Python project? There are so many built into Python and even more that are importable from the collections module. This week on the show, David Amos is back, and he's brought another batch of PyCoder's Weekly articles and projects. We discuss a recent three-part video course on selecting the ideal data structure.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

How to Set Up a Django Project

2021-05-25T14:00:00+00:00

When you start building any new Django web application, there’s a basic setup you need to tackle first. This course outlines the necessary steps to set up a Django project.

During this course, you’ll focus on the initial steps you’ll need to take to start a new web application. You should first have Python installed and you should know how to work with virtual environments and Python’s package manager, pip. You won’t need much programming knowledge to follow along with this setup, though you’ll need to know some Python to develop your project further afterwards.

After finishing this course, you’ll know how to:

Set up a virtual environment
Install Django
Pin your project dependencies
Set up a Django project
Start a Django app

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Real Python Podcast – Episode #61: Scaling Data Science and Machine Learning Infrastructure Like Netflix

2021-05-21T12:00:00+00:00

Would you move your data science project from a laptop to the cloud? Would you also like to have snapshots of your project saved along the way so that you can go back in time or share the state of your project with another team member? This week on the show, we have Savin Goyal from Netflix. Savin is the technical lead for machine learning infrastructure at Netflix. He joins us to talk about Metaflow, an open-source tool to simplify building, managing, and scaling data science projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Real Python

The Pandas DataFrame: Working With Data Efficiently

Python's collections: A Buffet of Specialized Data Types

Getting Started With Python’s collections

Improving Code Readability: namedtuple()

The Real Python Podcast – Episode #70: What Can You Do With Python and Counting Objects Using "Counter"

Your First Steps With Django: Set Up a Django Project

Prepare Your Environment

Install Django and Pin Your Dependencies

Speech Recognition With Python

Python Community Interview With Dustin Ingram

The Real Python Podcast – Episode #69: Planning a Faster Future at the Python Language Summit

Using FastAPI to Build Python Web APIs

What Is FastAPI?

Install FastAPI

First Steps

Create a First API

Run the First API App With Uvicorn

The Square Root Function in Python

Python News: What's New From June 2021?

Ewa Jodlowska Steps Down as PSF Executive Director

The Real Python Podcast – Episode #68: Exploring the functools Module and Complex Numbers in Python

What Can I Do With Python?

Python in the Real World

Develop Cool Software

Web Development

CLI Development

Defining and Calling Python Functions

Python's Counter: The Pythonic Way to Count Objects

Counting Objects in Python

Getting Started With Python’s Counter

The Real Python Podcast – Episode #67: Securing Your Python Software Supply Chain With Dustin Ingram

Beautiful Soup: Build a Web Scraper With Python

What Is Web Scraping?

Reasons for Web Scraping

Challenges of Web Scraping

An Alternative to Web Scraping: APIs

Python Inner Functions

Reverse Python Lists: Beyond .reverse() and reversed()

Reversing Python Lists

Reversing Lists in Place

The Real Python Podcast – Episode #66: Practicing Python With CSV Files and Extracting Values With "filter()"

CPython Internals: Paperback Now Available!

You Can Get CPython Internals in Paperback!

Python vs JavaScript for Python Developers

The Real Python Podcast – Episode #65: Expanding the International Python Community With the PSF

Using Pandas to Make a Gradebook in Python

The Real Python Podcast – Episode #64: Detecting Deforestation With Python & Using GraphQL With Django and Vue

Python Basics: Setting Up Python

The Real Python Podcast – Episode #63: Create Web Applications Using Only Python With Anvil

Explore Your Dataset With Pandas

The Real Python Podcast – Episode #62: Selecting the Ideal Data Structure & Unravelling Python's "pass" and "with"

How to Set Up a Django Project

The Real Python Podcast – Episode #61: Scaling Data Science and Machine Learning Infrastructure Like Netflix

Getting Started With Python’s `collections`

Improving Code Readability: `namedtuple()`

Getting Started With Python’s `Counter`