RustPython

How to contribute to RustPython by CPython unittest

2020-04-04T16:45:00+00:00

At the very end of 2019, we finally reached one of the short-term goals: CPython unittest support. Due to this enhancement, finding CPython compatibility is easier than before. Probably this will be the major source of contribution spots for the new contributors this year. Here is a simple guideline.

Fix known compatibility bugs

Let’s find an incompatibility issue and fix it.

See Lib/test directory of the project. There are many test_ prefixed files like test_unicode.py.
Try to open one of them. It might look just fine at a glance - but search for TODO: RUSTPYTHON in the files. There are tons of skipped, marked as an expected failure or commented out tests.
Alternatively, try looking at the regression tests results to find skipped or expected failure tests; some of them have notes for a way to resolve the issue.
Choose one or two interesting bugs. Remove the test blocker - skip, expectedFailure or comments.
Try to fix them.

Here is a quick tip to run single unittest file.

$ RUSTPYTHONPATH=Lib cargo run --release Lib/test/test_unicode.py

Add a new unittest file

Because CPython unittest is not perfectly working in RustPython, we are doing this one by one with editings.

Download CPython source code.
Check out a specific version of CPython. For now, 3.8.2 is recommended. (We are testing against CPython 3.8 and 3.8.2 is the most recent version for now)
Copy a file from CPython Lib/test
Commit the file without editing. Specify copied CPython version to commit message.
Try to edit it until it runs without a crash or failure.
Commit the changes to make it run. This is the core contribution.

Because RustPython is not perfect, “try to edit it until it runs” doesn’t mean to make 100% successful running. The common editing methods here:

At least it must be able to start to run the test. Fix the test code or bug until it runs at least a single unit of the test. Typically, unimplemented stdlib or missing files of unittest can make issues. Sometimes RustPython bugs make issues too.
If any test is not loadable by SyntaxError, that part is required to be commented out.
If any test leads to a crash of RustPython, this code is not possible to run. Mark the test to skip.
If any test is run but fails, this is an incompatibility issue. Mark the test as an expected failure.

We prefer the reversed order of upper methods. The later the more strict so easy to detect any progress or regression. When we temporarily disable parts of unittest due to RustPython caveats, we mark them to find it out easily later. Please check the examples below or search for TODO: RUSTPYTHON in Lib/test directory to check actual usage.

Comment out:

# TODO: RUSTPYTHON
#
# def ...  # commented out tests

skip:

@unittest.skip("TODO: RUSTPYTHON")
def ...  # skipped tests

expectedFailure:

# TODO: RUSTPYTHON
@unittest.expectedFailure
def ...  # failed tests

Development guide

For the general source of the development, please visit the RustPython development guide

Introduction to the RustPython parser

2020-04-02T15:34:01+00:00

This post goes over the RustPython parser. You can see the source code at RustPython/parser/.

When you write code in python and run it, an interpreter, such as the RustPython interpreter, acts as the translator between you and your machine.

The interpreter has the job of turning your human code into byte code that a python virtual machine can run. Bytecode is an intermediate code between source code and machine code. This makes it portable across multiple hardware and operating systems. Bytecode “works” as long as you implement a virtual machine(vm) that can run it. There is a performance penalty for this flexibility. RustPython has a vm under RustPython/vm/. Other posts, will go into the details of that vm but now let’s figure out how to turn code into bytecode.

How does bytecode look like

Seeing is believing. To see what bytecode looks like, you can use a Python module called dis. dis stands for disassembler. You can write source code then see how its bytecode looks like. Here is an example:

How RustPython turns your code to bytecode

Here are the main steps that RustPython currently does:

parse the line of source code into tokens
determine if the tokens have a valid syntax
create an Abstract Syntax Tree (AST)
compile the AST into bytecode

This list of steps introduces some new concepts like: tokens and abstract syntax trees. We’ll explain and demistify those.

Step 1: parsing source code into tokens

The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a tokenizer. Here is what happen if I run the tokenizer on the function that I created earlier.
$ python -m tokenize file.py

file.py has the function that I used in the previous example.

def add(x,y):
    return x+y

Tokenize output:

A picture IS worth a thousand word 😛 Those are the tokens. They are the basic “units” in the programming language. They are the keywords and operators that you typed. Even new lines and identations count.

If you want to sound fancy:

The tokens are the basic “lexical components”
The parsing process is called “Lexical Analysis”
The thing that does the process is a “lexer”

Here is the link to the RustPython lexer.

RustPython/parser/lexer.rs » source code

If you want dive into the details of lexical analysis, check out Python in a nutshell / Lexical structure

Step 2 : determine if the tokens are valid syntax

In the previous step, if you add random stuff to your function and tokenize it, it will work and still tokenize.

So don’t hate on the whole interpreter when you get error messages! or at least don’t hate on the tokenizer!

To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined “grammar” or set of rules. The official reference is on this link. There, you will find a machine readable file. You may read a book to know the rules of python, but words are too “fluffy”, the machine needs a very strict set of rules encoded in a file. This video explains the notation and the Python grammar. As the presenter puts it, this is the spirit of the beast (python) and it is only ~10KB 😭 (compare that to the size of python books you had to read!)

So, we have the rules or grammar of a programming language in a machine encoded format… now we need to write something that verifies that those rules were followed… This sounds like something that other people could use and like something that should exist as an open source project! 🤔

Sure enough, there is a whole Rust framework called LALRPOP. It takes the tokens generated by the lexer, verifies the syntax and turns the tokens into an AST (Abstract Syntax Tree). More information and a tutorial can be found in the LALRPOP book.

RustPython does one nice extra thing on top of LALRPOP . It masks the errors and provides you with safer, nicer errors. You can see the code for this in RustPython/parser/src/error.rs

Using RustPython to generate an AST

You can do:

 use rustpython_parser::{parser, ast};  
 let python_source = "print('Hello world')";  
 let python_ast = parser::parse_expression(python_source).unwrap();

Recap 🥴 🥵

As a recap, when you write a line of python code and “run it”, here is what happens:

your code (in file.py or interactive shell)
⭣ parse the line of source code into tokens
⭣ determine if the tokens are valid syntax
⭣ create an Abstract Syntax Tree (AST)
⭣ compile the AST into bytecode
bytecode (in __pycache__/file.pyc or in memory)

The compiler is under RustPython/compiler we’ll dive into the details in a next post. In the meantime, check out the parser source code in RustPython/parser/.