memcpy.io

Git pull request fetching & searching

2023-07-03T19:32:00+02:00

Adam Jackson created the script add-gitlab-merge-requests.sh which is the basis of this workflow.

git pr

The idea is to provide local access to all of the PRs that exist upstream. This both provides a better general overview of which PRs that have been pulled into the branch you're working on, but also enables you to search the contents of all PRs.

This function automagically detects if your remote is hosted on GitHub or GitLab and makes the necessary adjustments to work on either platform.

[alias]
  mr = pr
  pr = "!f() { \
    REMOTES=$(git remote); \
    REMOTE=\"origin\"; \
    case \"$REMOTES\" in \
    *upstream*) \
      REMOTE=\"upstream\"; \
    ;; \
    esac; \
    \
    ORIGIN=${1:-${REMOTE}}; \
    URL=$(git remote get-url ${ORIGIN}); \
    \
    case \"$URL\" in \
    *gitlab*) \
      FORGE_EXT="mr" \
      FORGE_PATH="merge-requests" \
    ;; \
    *github*) \
      FORGE_EXT="pr" \
      FORGE_PATH="pull" \
    ;; \
    esac; \
    ORIGIN_NAME=${ORIGIN}-${FORGE_EXT}; \
    git remote add ${ORIGIN_NAME} ${URL}; \
    git config remote.${ORIGIN_NAME}.fetch "+refs/${FORGE_PATH}/*/head:refs/remotes/${ORIGIN_NAME}/*"; \
    git fetch ${ORIGIN_NAME}; \
  }; f"

The syntax is git pr [remote].

[remote] will default to the upstream remote if available or origin if it isn't available.

$ git pr
remote: Enumerating objects: 886, done.
remote: Counting objects: 100% (827/827), done.
remote: Compressing objects: 100% (273/273), done.
remote: Total 886 (delta 637), reused 692 (delta 551), pack-reused 59
Receiving objects: 100% (886/886), 313.79 KiB | 918.00 KiB/s, done.
Resolving deltas: 100% (647/647), completed with 228 local objects.
From https://gitlab.freedesktop.org/wayland/wayland
 * [new ref]         refs/merge-requests/102/head -> upstream-mr/102
 * [new ref]         refs/merge-requests/104/head -> upstream-mr/104
 * [new ref]         refs/merge-requests/105/head -> upstream-mr/105
 * [new ref]         refs/merge-requests/106/head -> upstream-mr/106
 * [new ref]         refs/merge-requests/107/head -> upstream-mr/107

git search-all

git search-all or git sa is used for search all commits in a repository. When used in conjunction with git pr this enables searching through all PRs to look for previous implementations of functionality for example.

[alias]
    search-all = sa
    sa = "!f() { git log --all --color --pretty=log --abbrev-commit -S\"$1\" $2; }; f"

[pretty]
    log = %Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset

With both of these additions you'll be able to search PR commits for strings and have the relevant PR id listed in the search output.

$ git sa "~/.local/share"
91a92e5 - (upstream-mr/192) cursor: add XDG_DATA_DIRS to xcursor search path (10 months ago) <catsout>
0aebb5b - (upstream-mr/112) cursor: add one more directory to XCURSORPATH (2 years, 2 months ago) <Alexander Dunaev>

Setting up a Linux userspace graphics dev environment

2023-04-26T10:32:00+02:00

Set up alternative install location

These build instructions are based on the Wayland instructions from freedesktop.org.

You probably don't want to install experimental builds of software among the usual software of your operating system, so let's define a prefix for where to install our builds.

# Change ALT_LOCAL to any location you like
export WORK_DIR="/opt"
export PROJECT_NAME="feature_x"
export PROJECT_PATH="$WORK_DIR/$PROJECT_NAME"
export ALT_LOCAL="$PROJECT_PATH/local"
export LD_LIBRARY_PATH="$ALT_LOCAL/lib"
export PKG_CONFIG_PATH="$ALT_LOCAL/lib/pkgconfig/:$ALT_LOCAL/share/pkgconfig/"
export PATH="$ALT_LOCAL/bin:$PATH"
export ACLOCAL_PATH="$ALT_LOCAL/share/aclocal"
export ACLOCAL="aclocal -I $ACLOCAL_PATH"

# Needed by autotools
mkdir -p "$ALT_LOCAL/share/aclocal"

export XDG_RUNTIME_DIR="$ALT_LOCAL/xdg"
export XDG_CONFIG_HOME="$ALT_LOCAL/home"
export MESA_LOADER_DRIVER_OVERRIDE=zink

Installing dependencies

Start by installing the build dependencies of mesa, weston and wayland.

# Enable source packages
sudo sed -e "s/#\sdeb-src/deb-src/g" -i /etc/apt/sources.list
sudo apt update

The above step can alternatively be completed using the GUI of your package manager, by enabling source packages.

# Install build dependencies of mesa
sudo apt build-dep gtk3 gtk4 libinp/opt/zink/local/home/weston.iniut mesa wayland weston xserver

sudo apt install xcb-util-cursor-dev

# Install a recent version of meson
pip3 install --user meson

Building Wayland

git clone https://gitlab.freedesktop.org/wayland/wayland.git \
cd wayland \
meson build/ --prefix="$ALT_LOCAL" \
ninja -C build/ install \
cd ..

Building Wayland Protocols

git clone https://gitlab.freedesktop.org/wayland/wayland-protocols.git \
cd wayland-protocols \
meson build/ --prefix="$ALT_LOCAL" \
ninja -C build/ install \
cd ..

libglvnd OpenGL Vendor-Neutral Dispatch library

git clone https://github.com/NVIDIA/libglvnd.git \
cd libglvnd \
meson setup build/ --prefix="$ALT_LOCAL" \
ninja -C build/ install \
cd ..

libinput

libinput is a dependency of Weston, handles input devices like keyboards, touchpads and mice.

git clone https://gitlab.freedesktop.org/libinput/libinput \
cd libinput \
meson setup build/ --prefix="$ALT_LOCAL" \
ninja -C build/ install \
cd ..

Building Mesa

This build is targeting using swrast for Vulkan and Zink for OpenGL support.

git clone https://gitlab.freedesktop.org/mesa/mesa.git \
cd mesa
meson setup build/ -Dprefix="$ALT_LOCAL" \
  -Degl=x11,wayland,drm \
  -Degl=enabled \
  -Dgallium-drivers=swrast,zink,r600 \
ninja -C build/ install

# On some systems the lib paths need to be symlinked
ln -s "$ALT_LOCAL/lib64" "$ALT_LOCAL/lib"

Weston

Finally we've built all of the dependencies of Weston and can now build it.

git clone git://anongit.freedesktop.org/wayland/weston \
cd weston \
meson setup build/ --prefix="$ALT_LOCAL" \
ninja -C build/ install \
cd ..

x server

git clone git://git.freedesktop.org/git/xorg/xserver \
cd xserver \
meson setup build/ -Dprefix="$ALT_LOCAL" \
  -Dxkb_bin_dir=/usr/bin \
ninja -C build/ install \
cd ..

Running Weston

That wasn't so bad, it took a little while, but now we're ready to start Weston. Now, let's fire up a (virtual) terminal. Make sure that you're not running an X terminal, ssh terminal or serial terminal.

mkdir -p $XDG_RUNTIME_DIR
mkdir -p $XDG_CONFIG_HOME/.config
cp $PROJECT_PATH/weston/build/compositor/weston.ini $XDG_CONFIG_HOME/
export WESTON_CONFIG_FILE="$XDG_CONFIG_HOME/.config/weston.ini"
chmod -R 0700 $XDG_RUNTIME_DIR
chmod -R 0700 $XDG_CONFIG_HOME



# Make sure that $DISPLAY is unset.
unset DISPLAY

# And that $XDG_RUNTIME_DIR has been set and created.
if test -z "${XDG_RUNTIME_DIR}"; then
  export XDG_RUNTIME_DIR=/tmp/${UID}-runtime-dir
  if ! test -d "${XDG_RUNTIME_DIR}"; then
    mkdir "${XDG_RUNTIME_DIR}"
    chmod 0700 "${XDG_RUNTIME_DIR}"
  fi
fi

# Run weston:
weston

Try weston applications

Now that we're running weston, let's try some applications. They're located in the top level directory of weston.

weston-terminal
weston-flower
weston-gears
weston-smoke
weston-image
weston-view
weston-resizor
weston-eventdemo

When you've started all of your favorite applications you can grab a screenshot by pressing Super + s, which will save wayland-screenshot.png in your home directory.

Configuring Linux VLANs

2022-07-20T10:48:00+02:00

$ vlan
Device not provided

    vlan $DEV $VLAN $SUBNET

    vlan eth0 42 10.31.155.1/27

This is a achieved by pasting the below function into your .bashrc / .zshrc and issuing a source .bashrc or source .zshrc correspondingly.

function vlan {
    DEV=$1
    VLAN=$2
    ADDR=$3

    HELP="

    vlan \$DEV \$VLAN \$SUBNET

    vlan eth0 42 10.31.155.1/27
"

    if [ -z "$DEV" ]; then
        echo "Device not provided"
        echo "$HELP"
        return 1
    fi

    ip link | grep "${DEV}: " >/dev/null 2>&1
    if [ $? -ne 0 ]; then
        echo "\"$DEV\" is not a valid device"
        echo "$HELP"
        return 1
    fi

    if [ -z "$VLAN" ]; then
        echo "VLAN not provided"
        echo "$HELP"
        exit 1
    fi
    REGEX='^[0-9]+$'
    if ! [[ $VLAN =~ $REGEX ]] ; then
        echo "\"$VLAN\" is not a number" >&2; exit 1
        echo "$HELP"
        return 1
    fi

    if [ -z "$ADDR" ]; then
        echo "IP address not provided"
        echo "$HELP"
        return 1
    fi
    ipcalc -cs $ADDR >/dev/null 2>&1
    if [ $? -ne 0 ]; then
        echo "\"$$ADDR\" is not a valid address"
        echo "$HELP"
        return 1
    fi

    VLAN_DEV=vlan_${VLAN}
    SUBNET="$(ipcalc $ADDR | grep Network: | cut -d' ' -f4)"

    sudo ip link add link $DEV name $VLAN_DEV type vlan id $VLAN
    sudo ip addr add $ADDR dev $VLAN_DEV metric 20
    sudo ip link set dev $VLAN_DEV up
}

If adding a full shell function is a bit too much for you, then the actual commands that need to be issued are the following.

sudo ip link add link wlp0s20f3 name vlan_42 type vlan id 42
sudo ip addr add 10.31.155.1/27 dev vlan_42 metric 20
sudo ip link set dev vlan_42 up

Git Alias function syntax

2021-11-04T20:10:00+01:00

A basic example of the git alias function syntax looks like this.

[alias]
    shortcut = "!f() \
    {\
        echo Hello world!; \
    }; f"

This syntax defines a function f and then calls it. These aliases are executed in a sh shell, which means there's no access to Bash / Zsh specific functionality.

Every command is ended with a ; and each line ended with a \. This is easy enough to grok. But when we try to clean up the above snippet and add some quotes to "Hello world!", we hit this obtuse error message.

}; f: 1: Syntax error: end of file unexpected (expecting "}")

This syntax error is caused by quotes needing to be escaped. The reason for this comes down to how git tokenizes and executes these functions. If you're curious about how it's managed internally, have a look at git/run-commands.c.

Tom Hale wrote this script for doing automatic escaping.

quote() {
    printf %s "$1" | sed -r 's/(["\\])/\\\1/g';
}

IFS=$(printf '\n')
printf '\n!"'
read -r previous

while read -r line; do
    quote "$previous"
    printf ' \\n\\\n'
    previous="$line"
done

quote "$previous"
printf " #\"\n";

Applying mailing list patches with 'git b4'

2021-03-08T12:23:00+01:00

It was created by Konstantin Ryabitsev and has become a very frequently used tool for me.

It supports a lot of different ways for interacting with the Linux Kernel mailing lists. Of these the b4 am subcommand is what I primarily use. This subcommand downloads all of the patches belonging to a patch series and drops them into a .mbox file. But! It doesn't apply them to the repository we're currently in, and herein lies the itch that I would like to scratch.

The inspiration for this post is the script that @stellarhopper authored and @widawsky pointed out to me.

The Good, the Bad & the Ugly

After first publishing this post, people on the twittersphere suggested some alternative approaches, and it would seem that there are three different approaches to creating an alias like this. Naturally my original idea is the ugly.

The Good

@gregkh suggested a really short and to the point approach, where b4 am simply pipes the .mbox file to git am. I think it is ideal in this specific case since it avoids most the complexity of writing git alias functions but at the same time doesn't require any external script files.

As you can see the verbosity is really nice, and none of the b4 output is thrown out.

$ cat ~/.gitconfig
... snip ...
[alias]
    b4 = "!f() { b4 am -t -o - $1 | git am -s; }; f"
... snip ...

The Bad

When @widawsky first linked the ~/bin/git-b4am script by @stellarhopper, it looked like a standalone shell script. This is not the case however, it is automatically integrated as a subcommand by git. As explained by @EnJens, git will present any executable accessible through $PATH/git-XXX as a subcommand git XXX.

Using Zsh, I'm seeing some of the output being written after the command has returned. Using bash, this was less of an issue. I'd think this is due to the really neat way that processes substitution + tee is used, tee >(find_apply_mbx).

For the general case of running a script as a part of git, I think this is the way to go. The only downside to me is that the script is an external file.

$ cat ~/.local/bin/git-b4am
#!/bin/bash -eE

find_apply_mbx()
{
    mbx=$(awk '/^Writing .*\.mbx/{ print $2 }')
    if [[ $mbx ]]; then
        git am "$mbx"
    fi
}

b4 am -cls "$@" 2>&1 | tee >(find_apply_mbx)

The Ugly

Wanting to avoid the external scripts (and not knowing about $PATH/git-XXX functionality) I wrote a cursed version of the ~/bin/git-b4am script.

As you can see, the helpful output from b4 is lost. This is due to git alias functions being executed in a sh shell, which doesn't support the really neat processes substitution approach that @stellarhopper had used.

I apologize in advance for the escaped hellscape that is this snippet. A hierarchy of escapes is required to conform to the git function syntax when using both multiple lines and quoted strings.

$ cat ~/.gitconfig
... snip ...
[alias]
    b4 = "!f() \
    {\
        b4 am $1 -l -o /tmp/ 2>&1 | \n\
        $( \n\
            mbx=$( \n\
                awk '/^Writing .*\\.mbx/{ print $2 }' \n\
            );  \n\
            [ -z \"$mbx\" ] ||  \n\
                git am \"$mbx\" 1>&2; \n\
        );  \n\
    }; f"
... snip ...

A git syntax escape script written by Tom Hale made the escaping much easier.

Example

After picking your favorite approach, go find the Message-Id for the LKML patch or series you want to apply.

$ cd linux
linux$ git b4 1607651182-12307-1-git-send-email-victor.liu@nxp.com

Upstream camera support for Qualcomm platforms

2021-02-23T14:54:00+01:00

Linaro has been working together with Qualcomm to enable camera support on their platformssince 2017. The Open Source CAMSS driver was written to support the ISP IP-block with the same name that is present on Qualcomm SoCs coming from the smartphone space.

The first development board targeted by this work was the DragonBoard 410C, which was followed in 2018 by DragonBoard 820C support. Recently support for the Snapdragon 660 SoC was added to the driver, which will be part of the v5.11 Linux Kernel release. These SoCs all contain the CAMSS (Camera SubSystem) version of the ISP architecture.

Currently, support for the ISP found in the Snapdragon 845 SoC and the DragonBoard 845C is in the process of being upstreamed to the mailinglists. Having seen major changes, the ISP is no longer referred to as CAMSS, but is instead known as Titan.

The Titan architecture offers improvements in resolution, framerates and most other dimensions of the ISP, and is the latest architecture shipped on modern Qualcomm chipsets.

Overview

CAMSS is a V4L2 (Video for Linux 2) Linux driver which focuses on supporting the basic use cases of the ISP, such as receiving the MIPI CSI-2 signals from the sensors, decoding them, and then writing them to memory. This leaves a lot of functionality typically provided by an ISP unimplemented, but that is intentional as the development priority has been to enable the data path from camera sensor to userspace.

The sub-components that the CAMSS drivers supports are the following:

CSIPHY - the CSI PHYsical layer manages the physical electrical signals sent by camera sensors.
CSID - the CSI Decoder decodes the CSI-2 encoded data transmitted by the sensors
VFE - the Video Front End formats received data and exposes it through to further hardware blocks. The VFE block was renamed to IFE (Image Front End) in the Titan architecture.
RDI - the Raw Dump Interface is exposed by the VFE, and is used to write the raw decoded CSI output directly to memory.
PIX - the PIXel interface is exposed by the VFE, and is used to transmit data prepared for advanced processing by more specialized hardware blocks.
ISPIF - the Image Signal Processor InterFace ties together a lot of specialized hardware blocks into a data pipeline which can provide various levels of additionalprocessing.

Qualcomm ISP Gen 1 - CAMSS

The first generation of the ISP hardware block supported by the (aptly named) CAMSS driver is called CAMSS by Qualcomm.

The CSIPHY, CSID & VFE IP-blocks are relatively fully featured, but some functionality like Virtual Channels are not implemented. However, a more substantial limitation of the ISPIF support is that only basic cropping and rotation is currently implemented.

An obstacle caused by this limited ISPIF functionality is that it's not able to do format conversions using the ISP. So whatever format the camera sensor is outputting, is directly output to userspace. This can be a problem for Bayer pixel format sensors, because they're poorly supported by userspace applications and require at least debayering to be done in a post-processing step before the output can be viewed.

Qualcomm ISP Gen 2 - Titan

The next iteration of the Qualcomm ISP architecture is called Titan. The changes from Gen1 to Gen2/Titan can be summarized in two parts.

The Titan frontend blocks (CSIPHY, CSID & VFE) are almost identical between Gen1 and Gen2 with only minor structural changes and improvements.

However, the image processing pipeline has been revamped, and the ISPIF no longer exists. What replaces it is an embedded CPU which is fed commands and in turn configures the data processing blocks. Adding support for this CPU is beyond the current scope of CAMSS due to the amount of work it would take to enable and the lack of documentation for the CPU command stream. However, userspace applications are able to manage some of this post processing. This is enough to support the most basic use cases, but many will be CPU/GPU intensive and likely not have the same quality as an ISP based implementation would.

Future

Linaro will continue to maintain this driver, and is likely to extend it to support additional hardware platforms. However, contributions are very welcome and platforms like the SDM630 and SSDM660 have already had support contributed to the CAMSS driver.

Currently libcamera is a very useful development tool for working with CAMSS. It is more flexible than most V4L2 applications, and using the libcamera/qcam application makes it possible to view live output of even Bayer camera sensors without the ISP doing any debayering.

Recent Linaro contributions towards libcamera have enabled GPU accelerated format conversion and debayering for the libcamera/qcam test application, you can read more about that here. Linaro has also contributed the Open Embedded libcamera recipe for developers building their products with Yocto Project based Linux distributions.

Other Linaro contributions to the Linux camera landscape include work in both userspace with libcamera and in the kernel with camera sensor drivers, ISP drivers and V4L2 API work.

Further developments regarding software and GPU debayering are expected for libcamera, and hopefully the community will see libcamera debayering enabled for all of its data paths soon enough.

Acknowledgments

A lot of different contributors enabled this work, both directly and indirectly.

Todor Tomov for creating the camss driver.
Jonathan Marek for trail-blazing Titan driver work.
Andrey Konovalov for testing, finding bugs & being a great sounding board.
Qualcomm for sponsoring and providing hardware documentation.

This post has been a part of work undertaken by my employer Linaro.

Speed up `git log --graph` 18x times

2020-04-07T15:06:00+02:00

$ time git lg
git lg  13.34s user 0.87s system 84% cpu 16.845 total

# True by default as of git v2.24
git config --global core.commitGraph true
git config --global gc.writeCommitGraph true

# Added in git v2.29, pre-compute file paths, so that git log commands
# that are scoped to files also benefit from this cache.
git commit-graph write --reachable --changed-paths

$ time git lg
git lg  0.72s user 0.14s system 74% cpu 1.154 total

This is a speed up of ~18x, compared to the older versions.

The way this works is that the commit-graph file stores the commit graph structure along with some extra metadata to speed up graph in the .git/objects/info directory.

Panfrost: Liberating ARM GPUs @ Linux Conf Au

2020-01-15T00:00:00+01:00

Downloads

If you're curious about the slides, you can download the PDF or the ODP.

Diagrams

Overview

Kernel

Userspace

Mesa overview

Mesa Gallium

Mesa Winsys

Mesa Compiler

License

All of the material you can find on this page is licensed under the MIT license, and you are free to use, modify and re-distribute these materials however you like.

Thanks

I would like to thank the wonderful organizers of Linux Conf Au for hosting the event. This was my first LCA, and I think it may be the best Linux conference out of all of them.

I would also like to thank the organizers for paying for my flights, without it I would not have been able to give this talk.

Panfrost the Open Source Arm GPU Driver @ ELC NA

2019-09-22T00:00:00+02:00

Downloads

If you're curious about the slides, you can download the PDF or the ODP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of ELC NA for hosting the event.

Status of the Embedded GPU Space @ ELC NA

2019-09-21T00:00:00+02:00

Downloads

If you're curious about the slides, you can download the PDF or the ODP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of ELC NA for hosting the event.

Running Android and Wayland on Embedded Devices

2019-05-03T10:39:00+02:00

A previous post introduced the SPURV Android compatibility layer for Wayland based Linux environment.
In this post we're going to dig into how you can run an Android application on the very common i.MX6 based Nitrogen6_MAX board from Boundary Devices.

Install dependencies

sudo apt install \
    apt-transport-https \
    bmap-tools \
    ca-certificates \
    curl \
    git \
    gnupg2 \
    repo \
    software-properties-common \
    u-boot-tools \
    qemu-kvm

Set up Docker container for building

# Install Docker
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt update
sudo apt install docker-ce

# Set up privileges for Docker
sudo usermod -aG docker ${USER}
su - ${USER}

# Fetch Docker image
docker pull godebos/debos:latest

Build

Build Android

mkdir android; cd android
repo init -u https://android.googlesource.com/platform/manifest -b android-9.0.0_r10
git clone https://gitlab.collabora.com/spurv/android_manifest.git .repo/local_manifests/
repo sync -j15
. build/envsetup.sh
lunch spurv-eng
make -j12
cd ..

Build Linux Kernel

git clone https://gitlab.collabora.com/spurv/linux.git -b android-container_v5.1-rc5
cd linux
sh ../android/device/freedesktop/spurv/build-kernel.sh
cd ..

Create root filesystem

Just a kernel does not make an OS, so we're using Debian as a base.
The way we're going to create the root filesystem is using debos, which is a tool for creating Debian based OS images.

Create & flash image

Now we're ready to integrate all of the above into one coherent image. This is where the Nitrogen6_MAX devboard targeting comes in.

git clone https://gitlab.collabora.com/spurv/debos.git
sudo debos/build_image.sh -b /dev/mmcblk0

The platform specific parts are contained in uboot_nitrogen6qp-max.scr and build_image.sh.

Boot!

Pop the flashed SD-card into your device and restart it, and then log in as root/root.

In order to start Android, run one of these two commands:

# Launch just and Android application
/home/aosp/run.sh

Starting the Android application might take a minute or two, but Weston should start immediately.

Acknowledgments

A lot of different contributors enabled this work, both directly and indirectly.

Boundary Devices
Pengutronix
Zodiac

This post has been a part of work undertaken by my employer Collabora.

Running Android next to Wayland

2019-04-01T18:07:00+02:00

Running Android has some advantages compared to native Linux applications, for example with regard to the availability of applications and application developers.

For current non-Android systems, this work enables a path forward to running Android applications in the same graphical environment as traditional non-Android applications are run.

What is SPURV?

SPURV is our experimental containerized Android environment, and this is a quick overview of what it is.

It's aptly named after the first robotic fish since a common Android naming scheme is fish-themed names. Much like its spiritual ancestor Goldfish, the Android emulator.

Other Android Compatibility Layers

This means that Anbox which is LXC based, is different from SPURV in terms of how hardware is accessed. The hardware access that Anbox provides in indirect, and through the Qemu Pipes functionality, which is something it adopted from the Android (goldfish) emulator.

Shashlik and Genimobile are Android on Linux integration layers both based on Qemu, which means even better security properties than Anbox and certainly SPURV, but at the cost of an even larger performance penalty.

Direct Hardware Access

SPURV is different from other Linux desktop integrations for Android since it offers direct hardware access to the Android application. This is a choice we made for performance reasons. But has drawbacks, especially when it comes to security.
Using direct hardware access does however grant us increased GPU and CPU performance, which is important since we're targeting embedded platforms which can have very limited resources.

Components

SPURV consists of a few different parts, all living in the same project.

Android target device

This component integrates SPURV into Android, and it does so by using the device infrastructure that the Android codebase provides.

Devices are normally used to customize an Android build to the specific needs of a given hardware platform, like a new smartphone SOC. In the case of SPURV, we're targeting being run inside of a systemd-nspawn container.

SPURV Audio

This component bridges the Android Audio Hardware Abtraction Layer (HAL) to the host PulseAudio stack.

SPURV HWComposer

Integrates Android windows into Wayland. It does so by implementing a HWC-to-Wayland bridge.

HWC is the Android API for implementing display & buffer management, and what it essentially does in interpret all of the different display buffers that Android applications produce, and organizes them into one cohesive Desktop.

This protocol is conceptually not unlike the Wayland protocol, which allows for the HWC to be translated into Wayland. This is essentially what the SPURV HWComposer does.

Additionally it deals with input, like touch screen events and passes them along from Wayland to Android, this however is unrelated to the HWC API.

How does it work?

The SPURV Android target device behaves as a faux Android device, and tailors the Android build to our requirements.

Functions SPURV performs:

Customizes defaults.
Configures network.
Enables an audio bridge from Android to PulseAudio.
Enables a graphics bridge from Android to Wayland.

How can I use it?

Full build instructions as can be found on our GitLab for the SPURV project.

An overview of setting up:

Fetch Android (AOSP) and the Linux kernel,
Integrate SPURV into Android,
Build Android & Linux Kernel,
Build a debootstrap based root filesystem, and
Flash Kernel, Android and root filesystem to the device of your choice.

What comes next?

The next few steps will be adding support for more hardware platforms in our build scripts, but also optimizing the experience.

In no particular order, this is what we would like to look at next:

Bring-up on the i.MX8M with the etnaviv graphics driver.
Slimming things down so it takes less time to start an app and consumes less RAM for the case where the goal is to just to run a single app.
Bring-up on x86 with Ubuntu, publishing runtime binaries.

Caveats

The way SPURV is implemented means that a full OS is being run in a container, which has implications both positive and negative.

One of the positive effects is increased isolation of Android applications, which means improved security and privacy for potentially untrusted applications.

Additionally, this approach allows for Android applications to be run next to Wayland based applications in a desktop environment.

The downsides relate to hardware access and performance. All hardware access that is needed by Android has to be passed into the container. Besides manually having to configure such access using systemd-nspawn, there are also performance costs associated with running a container. One part of this is the static cost of having to load an entire OS on top of the base OS, but there are also additional runtime performance penalties for applications in the container.

Acknowledgements

Pengutronix
Zodiac
Boundary Devices

This post has been a part of work undertaken by my employer Collabora.

An Overview of the Panfrost driver

2019-03-13T16:25:00+01:00

The process of reverse engineering Arm GPUs has been going on for a long time, starting with Luc Verhaegens work on the low-end Mali 2/3/400 series of GPUs based on the Arm Utgard family of GPUs.
This driver has recently seen a lot new attention and is itself progressing quickly, which means it will likely be accepted into the kernel soon.
A piece of trivia is that this GPU architecture was what Arm received when they purchased the Norwegian GPU IP vendor Falanx Microsystems.

The Mali T and G-series of GPUs are based on the Midgard and Bifrost architectures respectively, both of which are quite different from the 2/3/400 series. However the T and G-series are somewhat similar at least when it comes to the way a driver can be built for them. This is why the Panfrost driver is aiming to support both architectures with one driver.

At Embedded World 2019 Collabora demoed the Panfrost driver running kmscube (pictured to the right). The singleboard computer used was a Radxa Rock Pi 4, which was generously sent to us by Tom Cubie.

Panfrost currently runs simple 3D applications like kmscube, the Wayland based Weston desktop and even more complex 3D benchmarks like glmark2.

This is still a new driver and it is in heavy development currently.

Current status

There are two semi-parallel parts under development currently; the new kernel driver and the Mesa userspace driver.

The new kernel driver is intended to replace the Open Source driver that Arm provides for its Mali GPUs (mali_kbase). Up until recently the Mesa Panfrost driver has been used with a shim between the Arm kernel driver and the userspace driver. While the Arm kernel driver exists, it cannot be accepted into the upstream Linux kernel project for multiple reasons, but most importantly it doesn't expose the DRM API that userspace expects of modern GPU drivers in the kernel.

As for the Panfrost Mesa driver, this driver is under heavy development and is seeing fixes, improvements to the compiler and new features added at a rapid pace. This driver is being built on top of the common Gallium driver framework in Mesa, which means that it will be relatively easy to move features from other drivers to the Panfrost driver.
Additionally the Panfrost driver uses the NIR intermediate representation (IR) for its compiler, which is the most common and most modern IR that Mesa implements. This again means that new and upcoming features like OpenCL for example, will be portable from the other Gallium/NIR drivers to Panfrost.

Thanks

These drivers are community drivers, but have been spearheaded by Alyssa Rosenzweig, Lyude Paul, Connor Abott, Rob Herring and Collabora's very own Tomeu Vizoso.

I would also like to thank Tom Cubie for sending out Rock Pi 4 boards to not just me, but the wider Panfrost development community.

This post has been a part of work undertaken by my employer Collabora.

Status of the Embedded GPU Space @ Embedded World

2019-02-27T00:00:00+01:00

Downloads

If you're curious about the slides, you can download the PDF or the ODP and of course the whitepaper.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of Embedded World for hosting a great event.

Running Docker privileged inside of LXC / LXD

2018-12-03T19:00:00+01:00

The architecture is a bit of container matroska, but what we're trying to achieve is running Docker privileged inside of a LXC container on a baremetal host.

Setup container on LXC Host

In order to give Docker in the guest privileges, the guest container itself has to be given privileges.

There is no simple switch for doing this in LXC unfortunately, but a few config options will do the trick.

lxc launch images:ubuntu/bionic container

lxc config set container security.nesting true
lxc config set container security.privileged true
cat <<EOT | lxc config set container raw.lxc -
lxc.cgroup.devices.allow = a
lxc.cap.drop =
EOT

lxc restart container

Setup docker on container

Just to verify that this works, start a privileged Docker container inside of the LXC container.

$ lxc exec container bash
$ docker run --privileged hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
d1725b59e92d: Pull complete
Digest: sha256:0add3ace90ecb4adbf7777e9aacf18357296e799f81cabc9fde470971e499788
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/

Thanks

This write-up is based info provided by Stéphane Graber.

Android on Mainline Graphics @ All Systems Go

2018-09-29T00:00:00+02:00

Downloads

If you're curious about the slides, you can download the PDF or the ODP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of All Systems Go!, the @ASGConf for hosting a great event.

Status of the Open Source GPU Ecosystem @ OSSummit NA

2018-08-31T00:00:00+02:00

Downloads

If you're curious about the slides, you can download the PDF or the ODP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of OSSummit NA, the Linux Foundation for hosting a great event.

git reset upstream

2018-08-24T12:32:00+02:00

robertfoss@xps9570 ~/work/libdrm $ git ru
remote: Counting objects: 234, done.
remote: Compressing objects: 100% (233/233), done.
remote: Total 234 (delta 177), reused 0 (delta 0)
Receiving objects: 100% (234/234), 53.20 KiB | 939.00 KiB/s, done.
Resolving deltas: 100% (177/177), completed with 36 local objects.
From git://anongit.freedesktop.org/mesa/drm
   cb592ac8166e..bcb9d976cd91  master     -> upstream/master
 * [new tag]                   libdrm-2.4.93 -> libdrm-2.4.93
 * [new tag]                   libdrm-2.4.94 -> libdrm-2.4.94

The idea here is that we by issuing a single short command can fetch the latest master branch from the upstream repository of the codebase we're working on and set our local master branch to point to the most recent upstream/master one.

This works by looking for a remote called upstream (or falling back to origin if it isn't found). And resetting the local master branch to point at the upstream/master branch.

~/.gitconfig

Add this snippet under the [alias] section of your ~/.gitconfig file.

[alias]
    ru = "!f() { \
        REMOTES=$(git remote); \
        REMOTE=\"origin\"; \
        case \"$REMOTES\" in \
            *upstream*) \
                REMOTE=\"upstream\"; \
                ;; \
        esac; \
        git fetch $REMOTE; \
        git update-ref refs/heads/master refs/remotes/$REMOTE/master; \
        git checkout master >/dev/null 2>&1; \
        git reset --hard $REMOTE/master >/dev/null 2>&1; \
        git checkout - >/dev/null 2>&1; \
    }; f

If you have a closer look, you'll notice that the upstream remote is used if has been added, otherwise the origin remote is used. This selection is done using git running a shell script.

Example

This is what git ru might look like when used.

robertfoss@xps9570 ~/work/libdrm $ git remote -v
origin  git@gitlab.collabora.com:robertfoss/libdrm.git (fetch)
origin  git@gitlab.collabora.com:robertfoss/libdrm.git (push)
upstream    git://anongit.freedesktop.org/mesa/drm (fetch)
upstream    git://anongit.freedesktop.org/mesa/drm (push)

robertfoss@xps9570 ~/work/libdrm $ git log --pretty=log --abbrev-commit
cb592ac8166e - (HEAD -> master, upstream/master, tag: libdrm-2.4.92) bump version for release (4 months ago) <Rob Clark>
c5a656818492 - freedreno: add fd_pipe refcounting (4 months ago) <Rob Clark>
1ac3ecde2f2c - intel: add support for ICL 11 (4 months ago) <Paulo Zanoni>
bc9c789073c8 - amdgpu: Deinitialize vamgr_high{,_32} (4 months ago) <Michel Dänzer>
[snip]

robertfoss@xps9570 ~/work/libdrm $ git ru
remote: Counting objects: 234, done.
remote: Compressing objects: 100% (233/233), done.
remote: Total 234 (delta 177), reused 0 (delta 0)
Receiving objects: 100% (234/234), 53.20 KiB | 939.00 KiB/s, done.
Resolving deltas: 100% (177/177), completed with 36 local objects.
From git://anongit.freedesktop.org/mesa/drm
   cb592ac8166e..bcb9d976cd91  master     -> upstream/master
 * [new tag]                   libdrm-2.4.93 -> libdrm-2.4.93
 * [new tag]                   libdrm-2.4.94 -> libdrm-2.4.94

robertfoss@xps9570 ~/work/libdrm $ git log --pretty=log --abbrev-commit
cb9d976cd91 - (HEAD -> master, upstream/master) xf86drm: fallback to normal path when realpath fails (4 hours ago) <Emil Velikov>
8389c5454804 - (tag: libdrm-2.4.94) Bump to version 2.4.94 (19 hours ago) <Kristian H. Kristensen>
f0c642e8df41 - libdrm: add msm drm uapi header (25 hours ago) <Tanmay Shah>
[snip]

Thanks

This post has been a part of work undertaken by my employer Collabora.

And thanks @widawsky for pointing out some improvements.

kms_swrast: A hardware-backed graphics driver

2018-07-31T09:14:00+02:00

Stack overview

Let's start with having a look at a high level overview of what the graphics stack looks like.

Before digging too much further into this, lets cover some terminology.

DRM - Direct Rendering Manager - is the Linux kernel graphics subsystem, which contains all of the graphics drivers and does all of the interfacing with hardware.
The DRM subsystem implements the KMS - kernel mode setting - API.

Mode setting is essentially configuring output settings like resolution for the displays that are being used. And doing it using the kernel means that userspace doesn't need access to setting these things directly.

The DRM subsystem talks to the hardware and Mesa is used by applications through the APIs it implements. APIs like OpenGL, OpenGL ES, Vulkan, etc. All of Mesa is built ontop of DRM and libdrm.

libdrm is a userspace library that wraps the DRM subsystem in order to simplify talking to drivers and avoiding common bugs in every user of DRM.

Looking inside Mesa we find the Gallium driver framework. It is what most of the Mesa drivers are built using, with the Intel i965 driver being the major exception.

kms_swrast is built using Gallium, with the intention of re-using as much of the infrastructure provided by Gallium and KMS as possible instead.

kms_swrast itself is backed by a backend, like softpipe or the faster llvmpipe, which actually implements the 3D primitives and functionality needed in order to reach OpenGL and OpenGL ES compliance.

Softpipe is the older and less complicated of the two implementations, whereas is llvmpipe is newer and relies on LLVM as an external dependency.
But as a result llvmpipe support JIT-compilation for example, which makes it a lot faster.

Why is this a good idea?

Re-using the Gallium framework gives you a lot of things for free. And the driver can remain relatively lightweight.

Apart from the features that Gallium provides today, you'll also get free access to new features in the future, without having to write them yourself.
And since Gallium is shared between many drivers, it will be better tested and have fewer bugs than any one driver.

kms_swrast is built using DRM and actual kernel drivers, but no rendering hardware is actually used. Which may seem a bit odd.

So why are the kernel drivers used for a software renderer? The answer is two-fold.

It is what Gallium expects, and there is a kernel driver called VGEM (Virtual GEM) which was created specifically for this usecase. In order to not have to make invasive changes to it or make the switch to VGEM right away, just providing it with access to some driver is the simplest possible solution. Since the actual hardware is mostly unused, it doesn't really matter what hardware you use.

The DRM driver is actually only used for a single thing, to allocate a slice of memory which can be used to render pixels to and then be sent to the display.

Thanks

This post has been a part of work undertaken by my employer Collabora.

Configuring QNAP LXC containers for VPNs using TUN

2018-06-24T22:16:00+02:00

Configuring the QNAP device

The first step is to SSH into your QNAP device using the admin account.

ssh admin@NAS_IP
cat >> /share/Container/container-station-data/lib/lxc/CONTAINER_NAME/config << EOF
lxc.cgroup.devices.allow = c 10:200 rwm
EOF

Configuring the container guest

The second step is open the QNAP web-ui, open the Container Station application and enter the console of your container.

sed -i '/exit 0/d' /etc/rc.local
cat >> /etc/rc.local << EOF
if ! [ -c /dev/net/tun ]; then
    mkdir -p /dev/net
    mknod -m 666 /dev/net/tun c 10 200
fi

exit 0
EOF

Lastly the container needs to be restarted, and then your VPN application should be able to access TUN devices and work normally.

Twistyplexing: A Charlieplexing variety

2018-06-22T12:26:00+02:00

The above layout has N = 7, yielding 42 LEDs.

Apart from the symmetry being visually pleasing compared to the normal row & column Charlieplexing layouts, it's relatively easy to spot errors in the schematic.

Avoiding lookup tables

The major advantage of twistyplexing is the ability to avoid lookup tables and replace them with some relatively straight forward arithmetic.

row = led_number / (N - 1)
column = led_number % (N - 1)
anode = (row + column + 1) % N

Of course the cathode still has to be controlled, but its pin id already defined by the row variable above.

Thanks

The Twistyplexing concept was created by Tom Yu, and defined in this blog post.

APA102 1515 LEDs

2018-06-04T22:26:00+02:00

The quiescent current of the APA102 2020 LEDs is about 0.7-1.0mA. Which is rather high for some usecases. But recently some new option have been made available.

The APA102 1515 and the APA104 1515, both of which come in a 1.5x1.5mm package. Additionally the APA104 1515 IC has a quiescent current of 0.3mA, which is good step in the right direction.

APA102 1515 datasheet

APA104 1515 datasheet

Unfortunately only the APA104 datasheet specifies quiescent current, so the quiescent current of the APA102 15155 is still an unknown.

Status of the Open Source Graphics Ecosystem @ OpenTechSummit

2018-05-10T00:00:00+02:00

Downloads

If you're curious about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of OpenTechSummit, specifically @hpdang and @mariobehling for hosting a great event.

Running Android on the Mainline Graphics Stack @ FossNorth

2018-04-23T00:00:00+02:00

Downloads

If you're curious about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of FossNorth, specifically @e8johan for hosting a great event.

Upstream Linux support for the new NXP i.MX 8M

2018-04-13T11:39:00+02:00

The i.MX6 platform has for the past few years enjoyed a large effort to add upstream support to Linux and surrounding projects. Now it is at the point where nothing is really missing any more. Improvements are still being made, to the graphics driver for i.MX 6, but functionally it is complete.

The i.MX8 is a different story. The newly introduced platform, with hardware still difficult to get access to, is seeing lots of work being done, but much still remains to be done.

That being said, initial support for the GPU, the Vivante GC7000, is in place and is able to successfully run Wayland/Weston, glmark, etc. This should also mean that running Android ontop of the currently not-quite-upstream stack is possible using drm_hwcomposer.

An upstream display/scanout driver does currently not exist, since the display IP in the i.MX 8 is different and more capable than the IP in the i.MX 6 platform, the current imx-drm driver is not capable of supporting it. A driver is provided by the NXP base support package. This BSP driver is based on KMS Atomic and supports most of the bells and whistles one would hope for, but is currently not in an upstreamable shape.

But patches for the gpio, clk, netdev & arm-kernel subsystems have been submitted to their respective mailing lists by Lucas Stach.

The direct support for the i.MX8 that has landed in the kernel at this point is mostly done by NXP engineers.

But there are lots of components that currently have no support. The Video Processor Unit IP, the Hantro G1/G2, does not have any upstream support.

Looking at bootloader support, U-Boot has good support for the i.MX 8M platform since early 2018, and can be expected to just work.

Looking forward

While lot's of support is still missing for the i.MX 8, the platform is under active development, with many new pieces of the hardware seeing attention.

Purism is one of the vendors who currently is actively working towards full Open Source support of the i.MX8 platform.

Devboards

WandPi 8M is a series of 3 different boards based on the i.MX 8M platform.

Nitrogen 8M is another i.MX 8M based option, made by Boundary Devices who also made the popular Sabre Lite series of boards for the i.MX 6.

Thanks

This post has been a part of work undertaken by my employer Collabora.

Status of the Embedded GPU Space @ Embedded Linux Conference NA

2018-03-12T00:00:00+01:00

A recording of the talk is available here.

Downloads

If you're curious about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of Embedded Linux Conference NA, for hosting a great event.

APA102 LED Current Usage

2018-02-20T22:26:00+01:00

What we're seeing here is a the LED being fully off (albeit with floating clock and data inputs), drawing somewhere between 0.7-1 mA.

I was quite surprised to see such a high quiescent current.

For the APA102 2020 which has a 20x20mm footprint this is somewhat disappointing, not because it is worse than the normal 50x50 APA102 variants, but rather because the small footprint begs for the IC to be used in wearables and other power consumption sensitive applications.

Setup

So this is the very simple setup I was using. It's nothing fancy; a multimeter set to the mA range, connected between the power supply and the APA102 breakout board I happened to have laying around.

Virtualizing GPU Access

2018-02-09T11:17:00+01:00

For the past few years a clear trend of containerization of applications and services has emerged. Having processes containerized is beneficial in a number of ways. It both improves portability and strengthens security, and if done properly the performance penalty can be low.

In order to further improve security containers are commonly run in virtualized environments. This provides some new challenges in terms of supporting the accelerated graphics usecase.

OpenGL ES implementation

Currently Collabora and Google are implementing OpenGL ES 2.0 support. OpenGL ES 2.0 is the lowest common denominator for many mobile platforms and as such is a requirement for Virgil3D to be viable on the those platforms.

That is is the motivation for making Virgil3D work on OpenGL ES hosts.

How does this work?

This stack is commonly referred to as Virgil3D, since all of the parts originated from a project with that name.

There are a few parts to this implementation. QEMU, virglrenderer and virtio-gpu. They way it works is by letting the guest applications speak unmodified OpenGL to the Mesa. But instead of Mesa handing commands over to the hardware it is channeled through virtio-gpu on the guest to QEMU on the host.

QEMU then receives the raw graphics stack state (Gallium state) and interprets it using virglrenderer from the raw state into an OpenGL form, which can be executed as entirely normal OpenGL on the host machine.

The host OpenGL stack does not even have to be Mesa, and could for example be the proprietary nvidia stack.

Trying it out

Environment

First of all, let's have a look at the development environment. When doing graphical development I find it quite helpful to set up a parallel graphics stack in order to not pollute or depend on the stack of the host machine more than we have to.

function concatenate_colon {
  local IFS=':'
  echo "$*"
}

function add_export_env {
  local VAR="$1"
  shift
  local VAL=$(eval echo "\$$VAR")
  if [ "$VAL" ]; then
    VAL=$(concatenate_colon "$@" "$VAL");
  else
    VAL=$(concatenate_colon "$@");
  fi
  eval "export $VAR=\"$VAL\""
}

function prefix_setup {
  local PREFIX="$1"

  add_export_env PATH "$PREFIX/bin"
  add_export_env LD_LIBRARY_PATH "$PREFIX/lib"
  add_export_env PKG_CONFIG_PATH "$PREFIX/lib/pkgconfig/" "$PREFIX/share/pkgconfig/"
  add_export_env MANPATH "$PREFIX/share/man"
  export ACLOCAL_PATH="$PREFIX/share/aclocal"
  mkdir -p "$ACLOCAL_PATH"
  export ACLOCAL="aclocal -I $ACLOCAL_PATH"
}

function projectshell {
  case "$1" in
    virgl | virglrenderer)
        export ALT_LOCAL="/opt/local/virgl"
        mkdir -p "$ALT_LOCAL"
        prefix_setup "$ALT_LOCAL"
        ;;
  esac
}

The above snippet is something that I would put in my .bashrc or .zshrc. Don't forget so run source ~/.bashrc or the equivalent after making changes.

To enter the environment I simply type projectshell virgl.

Build libepoxy

libepoxy is a library for managing OpenGL function pointers for you. And it is a dependency of virglrenderer, which we'll get to below.

git clone https://github.com/anholt/libepoxy.git
cd libepoxy
./autogen.sh --prefix=$ALT_LOCAL
make -j$(nproc --ignore=1)
make install

Build virglrenderer

Virgilrenderer is the component that QEMU uses to provide accelerated rendering. It receives Gallium states from the guest kernel via its virtio-gpu interface, which are then translated into OpenGL on the host. It also translates shaders from the TGSI format used by Gallium into the GLSL format used by OpenGL.

git clone git://anongit.freedesktop.org/virglrenderer
cd virglrenderer
./autogen.sh --prefix=$ALT_LOCAL
make -j$(nproc --ignore=1)
make install

Build Mesa

# Fetch dependencies
sudo sed -i 's/\#[ ]*deb-src/deb-src/' /etc/apt/sources.list
sudo apt update
sudo apt-get build-dep mesa

# Actually build Mesa
git clone https://anongit.freedesktop.org/git/mesa/mesa.git
cd mesa
./autogen.sh \
    --prefix=$ALT_LOCAL \
    --enable-driglx-direct \
    --enable-gles1 \
    --enable-gles2 \
    --enable-glx-tls \
    --enable-texture-float \
    --with-platforms=drm,x11,wayland \
    --with-dri-drivers=i915,i965,nouveau \
    --with-gallium-drivers=nouveau,swrast,radeonsi,virgl \
    --without-vulkan-drivers
make -j$(nproc --ignore=1)
make install

Build QEMU

git clone git://git.qemu.org/qemu.git
cd qemu
./configure \
    --prefix=$ALT_LOCAL \
    --target-list=x86_64-softmmu \
    --enable-gtk \
    --with-gtkabi=3.0 \
    --enable-kvm \
    --enable-spice \
    --enable-usb-redir \
    --enable-libusb \
    --enable-opengl \
    --enable-virglrenderer
make -j$(nproc --ignore=1)
make install

Set up a VM

As a guest we're going to use Ubuntu 17.10, but just use the latest release of whatever distro you like. The kernel has to have been built with the appropriate virtio-gpu Kconfig options though.

wget http://releases.ubuntu.com/17.10/ubuntu-17.10.1-server-amd64.iso
qemu-img create -f qcow2 ubuntu.qcow2 35G
qemu-system-x86_64 \
    -enable-kvm -M q35 -smp 2 -m 4G \
    -hda ubuntu.qcow2 \
    -net nic,model=virtio \
    -net user,hostfwd=tcp::2222-:22 \
    -vga virtio \
    -display sdl,gl=on \
    -boot d -cdrom ubuntu-17.10.1-server-amd64.iso

Run VM

qemu-system-x86_64 \
    -enable-kvm -M q35 -smp 2 -m 4G \
    -hda ubuntu.qcow2 \
    -net nic,model=virtio \
    -net user,hostfwd=tcp::2222-:22 \
    -vga virtio \
    -display sdl,gl=on

Et Voila! Your guest should now have GPU acceleration!

Conclusion

Hopefully this guide will have helped you to build all of the software needed to set up your very own virglrenderer enabled graphics stack.

This post has been a part of work undertaken by my employer Collabora.

Android on Mainline Graphics @ FOSDEM

2018-02-04T00:00:00+01:00

A recording of the talk can be found here.

Downloads

If you're curious about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers and volunteers of FOSDEM, for hosting a great community event.

Building ChromiumOS for Qemu

2017-11-28T11:32:00+01:00

So let's start off by covering how ChromiumOS relates to ChromeOS. The ChromiumOS project is essentially ChromeOS minus branding and some packages for things like the media digital restrictions management.

But on the whole, almost everything is there, and the pieces that aren't, you don't need.

ChromiumOS

Depot tools

In order to check out ChromiumOS and other large Google projects, you'll need depot tools.

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export PATH=$PATH:$(PWD)/depot_tools

Maybe you'd want to add the PATH export to your .bashrc.

Building ChromiumOS

mkdir chromiumos
cd chromiumos
repo init -u https://chromium.googlesource.com/chromiumos/manifest.git --repo-url https://chromium.googlesource.com/external/repo.git [-g minilayout]
repo sync -j75
cros_sdk
export BOARD=amd64-generic
./setup_board --board=${BOARD}
./build_packages --board=${BOARD}
./build_image --board=${BOARD} --boot_args "earlyprintk=serial,keep console=tty0" --noenable_rootfs_verification test
./image_to_vm.sh --board=${BOARD} --test_image

How to (not) boot ChromiumOS

So, this is a command baked into ChromiumOS using the cros_start_vm command, but at least on my machine it does not seem to boot properly. I have as of yet not been able to get any graphical output (over VNC).

cros_sdk
./bin/cros_start_vm --image_path=../build/images/${BOARD}/latest/chromiumos_qemu_image.bin --board=${BOARD}

Running Qemu ourselves

So if the intended tools don't work, we'll just have to roll up our sleeves and do it ourselves. This is how I got ChromiumOS booting.

Install build dependencies

These dependencies were available on Ubuntu 17.10, some alternative packages might be needed for your distributions.

sudo apt install autoconf libaio-dev libbluetooth-dev libbrlapi-dev libbz2-dev libcap-dev libcap-ng-dev libcurl4-gnutls-dev libepoxy-dev libfdt-dev libgbm-dev libgles2-mesa-dev libglib2.0-dev libgtk-3-dev libibverbs-dev libjpeg8-dev liblzo2-dev libncurses5-dev libnuma-dev librbd-dev librdmacm-dev libsasl2-dev libsdl1.2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh2-1-dev libspice-server-dev libspice-server1 libtool libusb-1.0-0 libusb-1.0-0-dev libvde-dev libvdeplug-dev libvte-dev libxen-dev valgrind xfslibs-dev xutils-dev zlib1g-dev libusbredirhost-dev usbredirserver

Virglrenderer

Virglrenderer creates a virtual 3D GPU, that allows the Qemu guest to use the graphics capabilities of the host machine.

This step is optional, but allows for hardware accelerated OpenGL support on the guest system. If you don't want to use Virgl, remove it from the Qemu configure step and the Qemu runtime flags.

git clone git://git.freedesktop.org/git/virglrenderer
cd virglrenderer
./autogen.sh
make -j7
sudo make install

Qemu

Qemu is a full system emulator, and supports a multitude of machine architectures. We're going to to use x86_64.

git clone git://git.qemu-project.org/qemu.git
mkdir -p qemu/build
cd qemu/build
../configure --target-list=x86_64-softmmu --enable-gtk --with-gtkabi=3.0 --enable-kvm --enable-spice --enable-usb-redir --enable-libusb --enable-virglrenderer --enable-opengl
make -j7
sudo make install

Run image

Now you can boot the image using Qemu.

Note that running Qemu with the virtio options requires that your host machine is running a Linux kernel which was built with the kconfig options CONFIG_DRM_VIRTIO, CONFIG_VIRT_DRIVERS and CONFIG_VIRTIO_XXXX.

cd chromiumos
/usr/local/bin/qemu-system-x86_64 \
    -enable-kvm \
    -m 2G \
    -smp 4 \
    -hda src/build/images/amd64-generic/latest/chromiumos_qemu_image.bin \
    -vga virtio \
    -net nic,model=virtio \
    -net user,hostfwd=tcp:127.0.0.1:9222-:22 \
    -usb -usbdevice keyboard \
    -usbdevice mouse \
    -device virtio-gpu-pci,virgl \
    -display gtk,gl=on

Conclusion

Hopefully this guide will have helped you to build all of the software needed to boot your very own ChromiumOS.

This post has been a part of work undertaken by my employer Collabora.

Fixing bluetooth on the XPS 15 9550 on Ubuntu

2017-10-28T00:00:00+02:00

Why doesn't this work automatically?

The firmware blob that is needed by Broadcom devices is not supplied by default, and it has to be supplied manually.

How To

Download BCM-0a5c-6410.hcd and copy it into /lib/firmware/brcm/ and then reboot your device.

wget https://memcpy.io/files/2017-10-28/BCM-0a5c-6410.hcd
sudo cp BCM-0a5c-6410.hcd /lib/firmware/brcm/
sudo chmod 0644 /lib/firmware/brcm/BCM-0a5c-6410.hcd
sudo reboot

Android on Mainline Graphics @ Embedded Linux Conference EU

2017-10-25T00:00:00+02:00

A recording of the talk can be found here.

Downloads

If you're curious about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of Embedded Linux Conference EU, for hosting a great community event.

drm_hwcomposer: A brief update @ XDC 20117

2017-09-22T00:00:00+02:00

Downloads

If you're curios about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of XDC, and Google for hosting a great community event.

Android on Mainline Graphics @ Open Source Summit NA

2017-09-11T00:00:00+02:00

Downloads

If you're curious about the slides, you can download the PDF or the OTP.

Thanks

This post has been a part of work undertaken by my employer Collabora.

I would like to thank the wonderful organizers of Open Source Summit NA, for hosting a great community event.

Android on Mainline Graphics @ GDG Berlin Android

2017-09-01T00:00:00+02:00

I would like to thank the wonderful organizers, GDG Berlin Android, for hosting a great community event.

Downloads

If you're curios about the slides, you can download the PDF or the OTP.

Android: NXP i.MX6 on Etnaviv Update

2017-07-21T00:00:00+02:00

Since the last post a lot work has gone into upstreaming and stabilizing the etnaviv on Android ecosystem. This has involved Android, kernel and Mesa changes. Many of which are available upstream now. A How-To for getting you up and running on an iMX6 dev board is available here.

Improvements

Modifiers support

Modifiers support has been accepted into Mesa, GBM and gbm_gralloc. Modifiers were mentioned in a previous post.

Etnaviv driver support for Android

Patches enabling the etnaviv Mesa driver being built for Android have now landed upstream.

Stability on Android

A number for small stability issues present while running Android on i.MX6 hardware have now been fixed, and the platform is now relatively stable.

Performance diagnostics

We have a decent understanding that the platform is slow when running the desktop and other apps that have multiple surfaces due to rendering using CPU instead of GPU.

Etnaviv improvements

Etnaviv performance and feature set have both been increased since Mesa v17.1.

EGL support

A number of games using EGL have been successfully run on Android, some minor graphical issues still remain, but overall games run well and fast.

Thanks

This work is built on efforts by a lot people:

Aleksander Morgado - Independent
Daniel Stone - Collabora
Christian Gmeiner - Independent
Emil Velikov - Collabora
Lucas Stach - Pengutronix
Rob Herring - Linaro
Varad Gautam - Collabora
Wladimir Van Der Laan - Independent

This post has been a part of work undertaken by my employer Collabora, and has been funded by Zodiac Inflight Innovations.

GALLIUM_HUD: Debug Mesa Graphics Performance

2017-06-28T00:00:00+02:00

GALLIUM_HUD

GALLIUM_HUD is a feature that adds performance graphs to applications that describe various aspects like FPS, CPU usage, etc in realtime.

It is enabled using an environment variable, GALLIUM_HUD, that can be set for GL/EGL/etc applications. It only works for Mesa drivers that are Gallium based, which means that the most drivers (with the notable exception of some Intel drivers) support GALLIUM_HUD.

See GALLIUM_HUD options:

export GALLIUM_HUD=help
glxgears

Android

If you're building Android, you can supply system-wide environment values by doing an export in the init.rc file of the device you are using, like this.

# Go to android source code checkout
cd android

# Add export to init.rc (linaro/generic is the device I use)
nano device/linaro/generic/init.rc
export GALLIUM_HUD cpu,cpu0+cpu1+cpu2+cpu3;pixels-rendered,fps,primitives-generated

Linux

If you're using one of the usual Linux distros, GALLIUM_HUD can be enabled by setting the environtment variable in a place that it loaded early.

# Add export to /etc/environment
nano /etc/environment 
export GALLIUM_HUD cpu,cpu0+cpu1+cpu2+cpu3;pixels-rendered,fps,primitives-generated

Thanks

This post has been a part of work undertaken by my employer Collabora.

Android: NXP i.MX6 Buffer Modifier Support

2017-06-02T00:00:00+02:00

With modifier support added to Mesa and gbm_gralloc, it is now possible to boot Android on iMX6 platforms using no proprietary blobs at all. This makes iMX6 one of the very few embedded SOCs that needs no blobs at all to run a full graphics stack.

Not only is that a great win for Open Source in general, but it also makes the iMX6 more attractive as a platform. A further positive point is that this lays the groundwork for the iMX8 platform, and supporting it will come much easier.

What are modifiers used for?

Modifiers are used to represent different properties of buffers. These properties can cover a range of different information about a buffer, for example compression and tiling.

For the case of the iMX6 and the Vivante GPU which it is equipped with, the modifiers are related to tiling. The reason being that buffers can be tiled in different ways (Tiled, Super Tiled, etc.) or not at all (Linear). Before sending buffers out to a display, they need to have the associated tiling information made available, so that the actual image that is being sent out is not tiled.

How was support added?

Support was added in two places; Mesa and gbm_gralloc. Mesa has had support added to many of the buffer allocation functions and to GBM (which is the API provided by Mesa, that gbm_gralloc uses).

gbm_gralloc in turn had support added for using a new GBM API call, GBM_BO_IMPORT_FD_MODIFIER, which imports a buffer object as well as accompanying information like modifier used by the buffer object in question.

Getting up and running

Currently the modifiers work is in the process of being upstreamed, but in the meantime it can be found here. If you'd like to test this out yourself a How-To can be found here.

Thanks

This work is built on the efforts of a lot people:

Varad Gautam - Collabora
Lucas Stach - Pengutronix
Tomeu Vizoso - Collabora
Rob Herring - Linaro
Emil Velikov - Collabora
Christian Gmeiner - Independent
Wladimir Van Der Laan - Independent

This post has been a part of work undertaken by my employer Collabora, and has been funded by Zodiac Inflight Innovations.

Android: Getting up and running on the iMX6

2017-04-27T00:00:00+02:00

Since the hardware very much matters this is going to be divided into a few parts, the common steps and the hardware specific ones.

This post is a bit of a living document and will be changed over time, and if you have any questions about it, please reach out through email (robert.foss at collabora.com) or irc (tomeu or robertfoss on #dri-devel on freenode).

Changelog:
2017-04-27: build_android.sh, setup_sdcard.sh: Added -b [device] support to build_android.sh and setup_sdcard.sh
2017-05-02: build_android.sh: Don't write to SD-card without -b option
2017-05-04: Switch git repo urls to shared repository
2017-05-09: Add compiler installation to apt-get
2017-05-09: Re-ordered some instructions
2017-05-10: Add java installation to apt-get
2017-05-19: Change paths to android-etnaviv instead of client
2017-05-19: Change kernel branch
2017-05-19: Add -d, device flag and info about it
2017-05-19: Add sabrelite board info
2017-06-03: Add more apt-get packages
2017-06-10: Add lzop apt-get packages

Common steps

# Install dependencies
sudo apt install u-boot-tools gcc-arm-linux-gnueabihf openjdk-8-jdk android-tools-fsutils python-mako git-core gnupg flex bison gperf build-essential zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache libgl1-mesa-dev libxml2-utils xsltproc unzip lzop

mkdir /opt/android
cd /opt/android
repo init -u https://android.googlesource.com/platform/manifest -b android-7.1.1_r28
cd /opt/android/.repo
git clone https://customer-git.collabora.com/git/android-etnaviv/android_manifest.git local_manifests -b android-etnaviv
repo sync -j10

mkdir /opt/imx6_android
cd /opt/imx6_android

# Fetch Kconfig, bootloaders and some scripts
git clone https://customer-git.collabora.com/git/android-etnaviv/android-etnaviv.git .

# Fetch the Linux Kernel
git clone https://customer-git.collabora.com/git/android-etnaviv/linux.git -b android-etnaviv

# This will destroy all data on /dev/mmcblk0 and
# create boot/system/cache/data partitions
./setup_sdcard.sh -b /dev/mmcblk0

Hardware: iMX6 Sabre

Build Android and Linux

# Build android, the kernel, and flash it onto an SD-card
# Run build_android with the correct -d flag
./build_android.sh -b /dev/mmcblk0 -d imx6q-sabre
./build_android.sh -b /dev/mmcblk0 -d imx6qp-sabre

Start Android

The SD-card can now be put into the SD3 slot and the device can be restarted.

Hardware: iMX6 Sabrelite

Build Android and Linux

# Build android, the kernel, and flash it onto an SD-card
# Run build_android with the correct -d flag
./build_android.sh -b /dev/mmcblk0 -d imx6q-sabrelite

Start Android

The micro-sd card can now be put into the micro-sd slot and the device can be restarted.

Hardware: RDU2

Build Android and Linux

# Build android, the kernel, and flash it onto an SD-card
# Run build_android with the correct -d flag
./build_android.sh -b /dev/mmcblk0 -d imx6q-zii-rdu2
./build_android.sh -b /dev/mmcblk0 -d imx6qp-zii-rdu2

Install the bootloader

# Depending if you have a >=13" version of the RDU2
# use the imx6qp, if <13" then use the imx6q

IMX6_TYPE=imx6q
IMX6_TYPE=imx6qp
BAREBOX="zodiac/barebox-zii-${IMX6_TYPE}-rdu2.img"

# Flash bootloader to SD-card
dd if=${BAREBOX} of=/dev/mmcblk0 bs=1k
sync

# Put SD-card in the middle-most slot on the RDU2

# Install lrzsz, since it is used for a ymodem upload
sudo apt install lrzsz

# Connect to serial device /dev/ttyUSB2 and
# /dev/ttyUSB3 with minicom
# The numbering assumes the RDU2 is the only serial
# serial device connected
sudo minicom -s
    +------------------------------------------+
    | A -    Serial Device      : /dev/ttyUSB3
    | B - Lockfile Location     : /var/lock
    | C -   Callin Program      :
    | D -  Callout Program      :
    | E -    Bps/Par/Bits       : 115200 8N1
    | F - Hardware Flow Control : No
    | G - Software Flow Control : No
    |
    |    Change which setting?
    +------------------------------------------+

# Connect to Quark console on /dev/ttyUSB3
# Set boot SD-card as boot source 
#HostBoot s 0
reset

# Restart device, connect to barebox loaded just loaded
# from the SD-card on /dev/ttyUSB2
pic_setwdt 0 60
loady

# Using the minicom quickly initiate a ymodem file
# of the same barebox image you wrote to the SD-card
# Be quick, the upload will timeout after a few seconds

# Write the bootloader to SPI NOR
erase /dev/m25p0.barebox
# Depending on your RDU2 type flash one of the following
cp barebox-zii-imx6q-rdu2.img /dev/m25p0.barebox
# Or
cp barebox-zii-imx6qp-rdu2.img /dev/m25p0.barebox

# Connect to the Quark console on /dev/ttyUSB3 again
# Set SPI NOR as the boot source
#HostBoot s 2
reset

# Connect to the barebox console on /dev/ttyUSB2 again
# Edit configuration to automatically boot from mmc:
sedit /env/config
export global.boot.default=/env/boot/mmc
export global.bootm.image=/mnt/mmc1.0/android_zImage
export global.bootm.initrd=/mnt/mmc1.0/android_ramdisk.img.gz
export global.bootm.oftree=/mnt/mmc1.0/imx6qp-zii-rdu2.dtb
export global.linux.bootargs.base="console=ttymxc0,115200 console=tty0 rw rootwait ip=dhcp buildvariant=userdebug debug ignore_loglevel root=/dev/mmcblk0p2 rootfstype=ext4 rootwait init=/init printk.devkmsg=on verbose enforcing=0 androidboot.selinux=permissive drm.debug=0x00"

sedit /env/boot/mmc
#!/bin/sh
detect mmc1
mkdir -p /mnt/mmc1.0
automount -d /mnt/mmc1.0 'mount /dev/mmc1.0 /mnt/mmc1.0'
bootm

pic_setwdt 0 60     # Disable watchdog

exit

Start Android

The SD-card created in the common steps can now be put into the middlemost slot and the device can be restarted.

Thanks

This work is built on efforts by a lot people:

Pengutronix who's been doing i.MX6 platform work.
Christian Gmeiner, Wladimir Van Der Laan, and the other etanviv developers.
Rob Herring at Linaro for getting the ball rolling with AOSP for Zii.
Andrey Smirnov for driver support for the RDU2 such as i.MX6 PCI, ARM PL310 L2 Cache controller, RTC, and other i.MX6qp driver fixups.

This post has been a part of work undertaken by my employer Collabora.

Android: Changing the bootanimation

2017-04-20T00:00:00+02:00

There exists official documentation for how to create a custom boot animation, but unfortunately it is lacking in actual examples.

So this guide is a bit more hands on.

Structure of bootanimation.zip

Without covering too much of the same gound as the documentation, let's have a quick look at what is in a simple bootanimation.zip.

$ ls -la bootanimation
total 28
drwxr-xr-x 4 hottuna hottuna 4096 Apr 19 22:39 .
drwxr-xr-x 8 hottuna hottuna 4096 Apr 19 22:39 ..
-rw-r--r-- 1 hottuna hottuna   92 Apr 19 15:21 desc.txt
drwxr-xr-x 2 hottuna hottuna 4096 Apr 19 12:44 part0
drwxr-xr-x 2 hottuna hottuna 4096 Apr 19 12:45 part1

$ cat bootanimation/desc.txt 
1920 1080 30         # WIDTH HEIGHT FPS
c 5 15 part0 FFFFFF  # TYPE COUNT PAUSE PATH [#RGBHEX] [CLOCK]
c 5 15 part1 FFFFFF  # TYPE COUNT PAUSE PATH [#RGBHEX] [CLOCK]

ls -la bootanimation/part0 
total 28
drwxr-xr-x 2 hottuna hottuna  4096 Apr 19 12:44 .
drwxr-xr-x 6 hottuna hottuna  4096 Apr 19 15:40 ..
-rw-r--r-- 1 hottuna hottuna 10688 Apr 19 12:31 0000.png
-rw-r--r-- 1 hottuna hottuna 10688 Apr 19 12:31 0001.png
-rw-r--r-- 1 hottuna hottuna 10688 Apr 19 12:31 0002.png
-rw-r--r-- 1 hottuna hottuna 10688 Apr 19 12:31 0003.png
-rw-r--r-- 1 hottuna hottuna 10688 Apr 19 12:31 XXXX.png

Note that the "#" comments are mine and not actually present in the files.

An important thing to note with the zip file is needs to have compression turned off.

Switch bootanimation

Download bootanimation.zip.

unzip bootanimation.zip
cd bootanimation
# Edit desc.txt and partN folders to your needs
zip -0qry -i \*.txt \*.png \*.wav @ ../bootanimation.zip *.txt part*

# Option 1, use adb to send bootanimation.zip
adb root
adb remount
adb push bootanimation.zip /system/media/bootanimation.zip

# Option 2, bake bootanimation.zip into your AOSP build
cp bootanimation.zip /opt/aosp/out/target/product/linaro_arm/system/bootanimation.zip
./your_favorite_buildscript_here.sh

References

bootanimation documentation

Thanks

This post has been a part of work undertaken by my employer Collabora.

Audio editing images

2017-04-09T21:35:00+02:00

Audio Shop is a simple script that I cobbled together that gets you started with mangling image data as if it was audio data.

The script wraps 3 individually excellent tools; ffmpeg, ImageMagick and SoX.

The way it works is by first converting an image to a raw format like rgb or yuv. This is done to prevent the audio editor from destroying the structure of (relatively) complex formats like jpg, png or gif.

If converting to a raw format is the first step, the second step is importing the raw image data into the audio editor. To do this in way you can expect good results from, the raw format should use a bit-depth that your image editor can use. For example RGB, where each color channel is 8 bits. And the audio editor imports the raw data as 8-bit unsigned data.

Additionally when using a format like RGB the Red/Green/Blue color bits of the channels are located next to eachother like RRRRRRRGGGGGGGGBBBBBBBB which causes issues when doing things like echoes and the echo of of the Red 8-bit field bleeds into the Green or Blue field.

The solution to this is using a planar format where the RGB or YUV channels are not interleaved in the RGBRGBRGBRGB fashion but rather RRRRGGGGBBBB. A format like that is YUV444P.

I hope you find this tool somewhat useful and/or fun!

Android: Enabling mainline graphics

2017-03-28T15:18:00+02:00

The HWC (Hardware Composer) API is used by SurfaceFlinger for compositing layers to the screen. The HWC abstracts objects such as overlays and 2D blitters and helps offload some work that would normally be done with OpenGL. SurfaceFlinger on the other hand accepts buffers from multiple sources, composites them, and sends them to the display.

The above graphic depicts the traditional Android graphics stack.

This is where drm_hwcomposer comes into play. Since the mainline kernel graphics stack doesn't offer the HWC API, drm_hwcomposer is introduced to interface with the mainline graphics stack through mesa and libdrm. Before this work drm_hwcomposer only offered the HWC1 API. Since Android 7.0 version 2 of the HWC API is used by SurfaceFlinger. HWC2 differs in a few ways from the previous version, for example the semantics of fence support were changed and the GPU can now be used as a fallback when compositing layers.

Up until recently the mainline kernel lacked the fence primitive offered by Android used in HWC1 and HWC2. But after my fellow Collaboran Gustavo Padovan's work on adding fence support to the mainline kernel was upstreamed in v4.10, the mainline kernel now has fence support equivalent to that of Android.

The new fence support enabled work on drm_hwcomposer to add HWC2 support. And with it we are now able to boot Android on the db410c running the freedreno driver. But in theory it should work on any mainline kernel graphics stack enabled GPU.

Currently the work is being upstreamed to the ChromiumOS repo which is the official upstream for drm_hwcomposer.

A number of projects have seen contributions 8in order to enable this work:

kernel - sync_file, in-fence and out-fence support added.
libdrm - fence support added.
mesa - support for passing fences added.
intel-gpu-tools - sync and fence tests added.
drm_hwcomposer - HWC2 and fence support added.

Thanks

This drm_hwcomposer work is part of a long-standing collaboration between Google's ChromeOS team and Collabora.

A number of people have played an important role in this work: Gustavo Padovan, Rob Clark, Sean Paul, Zach Reizner and Rob Herring.

This post has been a part of work undertaken by my employer Collabora.

Removing the Chromebook Write-Protect screw

2017-02-27T16:55:00+01:00

This post will look specifically at removing the WP screw from a Chell (HP Chromebook 13 G1) device, and verifying that it has been successfully removed.

To actually flash firmware to Chromebook machines, a device called a Servo is needed. While these devices aren't available publicly, they can be produced freely or possibly requested from Google if you are contributing code to the ChromiumOS project.

Removing the Write-Protect screw

So this is what the WP screw looks like on a Chell Chromebook. This may or may not be what you will find in other devices. But if you take a close look, you will notice that the copper pad that the the screw attaches against is split into parts that are bridged by a screw being inserted.

Disable Write-Protect

So this is the part that requires a Servo device. And a ChromiumOS checkout, for some help setting one up, have a look at my previous post.

# Go to your ChromiumOS checkout
cd /opt/chromiumos

# Enter dev environment
cros_sdk

# Set device variable
export BOARD=chell

# Connect to Chromebook using a Servo device
sudo servod -b $BOARD &

# Disable WP
# This step may vary depending on the hardware of your actual Chromebook
dut-control fw_wp:off
sudo /usr/sbin/flashrom -p ft2232_spi:type=servo-v2 --wp-disable
sudo /usr/sbin/flashrom -p ec --wp-disable

References

ChromiumOS Servo
Setting up a ChromiumOS dev environment

Thanks

This post has been a part of work undertaken by my employer Collabora.

Precompiling APK files during Android AOSP build

2017-02-22T16:55:00+01:00

Enable precompilation

cd /opt/aosp_checkout/

find . -name BoardConfig.mk
./device/huawei/angler/BoardConfig.mk
./device/generic/mini-emulator-x86/BoardConfig.mk
./device/generic/arm64/BoardConfig.mk
./device/generic/mini-emulator-x86_64/BoardConfig.mk
./device/generic/mini-emulator-armv7-a-neon/BoardConfig.mk
./device/generic/mips/BoardConfig.mk
./device/generic/mips64/BoardConfig.mk
./device/generic/x86_64/BoardConfig.mk
./device/generic/mini-emulator-arm64/BoardConfig.mk
./device/generic/mini-emulator-mips/BoardConfig.mk
./device/generic/x86/BoardConfig.mk
./device/generic/armv7-a-neon/BoardConfig.mk
./device/generic/mini-emulator-mips64/BoardConfig.mk
./device/lge/bullhead/BoardConfig.mk
./device/linaro/generic/linaro_x86_64_only/BoardConfig.mk
./device/linaro/generic/linaro_arm64/BoardConfig.mk
./device/linaro/generic/linaro_arm64_only/BoardConfig.mk
./device/linaro/generic/linaro_x86_64/BoardConfig.mk
./device/linaro/generic/BoardConfig.mk
./device/linaro/generic/linaro_arm/BoardConfig.mk
./device/linaro/hikey/hikey/BoardConfig.mk

# Edit the BoardConfig.mk that you are using for your build
cd device/linaro/generic/
nano BoardConfig.mk

# Add the config option
WITH_DEXPREOPT := true

# Propagate the new settings
make defconfig
make all

Increase system partition size

Depending on the previous system partition size and how many APKs that are built, you may need to increase the system partition size.

nano configs/defconfig

# Change the below variable to something big enough to house all of you binaries
CONFIG_BOARD_SYSTEMIMAGE_PARTITION_SIZE=1100000000

# Propagate the new settings
make defconfig
make all

Thanks

This post has been a part of work undertaken by my employer Collabora.

Setting up a ChromiumOS dev environment

2017-02-16T10:31:00+01:00

Set up environment

export DEV_DIR="/opt"
mkdir -p $DEV_DIR
export CHROMIUM_DIR="$DEV_DIR/chromiumos"
mkdir -p $CHROMIUM_DIR
export PATH="$DEV_DIR/depot_tools:$PATH"

# The BOARD variable used here is specific for the Chromebook that is
# being targeted, a more generic target like "amd64-generic" could
# be more useful for you needs.
export BOARD=chell

# The USB_DEVICE variable refers to the USB device that will be used
# for flashing ChromiumOS onto a Chromebook.
# Make sure that this device does not contain anything important!
export USB_DEVICE="/dev/sda"

Install dependencies

sudo apt install git-core gitk git-gui subversion curl
cd $DEV_DIR
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

Get ChromiumOS source

cd ${CHROMIUM_DIR}
repo init -u https://chromium.googlesource.com/chromiumos/manifest.git
repo sync -j25

Build ChromiumOS

cros_sdk -- ./build_packages --board=${BOARD}
cros_sdk -- ./build_image --board=${BOARD}

Flash ChromiumOS to storage medium

cros_sdk -- cros flash --board=${BOARD} usb:/$USB_DEVICE

Install ChromiumOS on Chromebook

Enter Chromebook into dev-mode

This part is highly device specific, and depends on how the manufacturer of your device has chosen to implement the dev-mode switch.

A partial list of devices and how to enter them into dev-mode can be found here.

Flash ChromiumOS to Chromebook

Fire up your Chrombook device and hit Ctrl+Alt+Back, followed by 'chronos' and hit enter. Followed by the below command to install the ChromiumOS build that was just flashed.

/usr/sbin/chromeos-install

Debug an application

On Chromebook

Again fire up your Chrombook device and hit Ctrl+Alt+Back, followed by 'chronos' and hit enter.

# Remount the root drive read / write
sudo mount -o remount,rw /

# Open port so that gdbserver can be reached
sudo /sbin/iptables -A INPUT  -p tcp --dport 1234 -j ACCEPT

# Run gdb server, listening on port 1234 (opened in iptables command above)
sudo gdbserver :1234 chrome

On dev machine

# On x86
cros_sdk -- sudo USE=expat emerge cross-i686-pc-linux-gnu/gdb
# On ARMv7
cros_sdk -- sudo USE=expat emerge cross-armv7a-cros-linux-gnueabi/gdb

cros_sdk -- i686-pc-linux-gnu-gdb "/build/$BOARD/opt/google/chrome/chrome"
(gdb) set sysroot /build/$BOARD/
(gdb) target remote IP_ADDR_CHROMEBOOK:1234
(gdb) continue

Conclusion

This is a bit of a rough outline, and is only suitable for Chromebook devices that already are in dev-mode.

Thanks

This post has been a part of work undertaken by my employer Collabora.

References

ChromiumOS Depo Tools
ChromiumOS Quick Start
ChromiumOS Dev Mode
ChromiumOS Debug

Running an Android Unit Test

2016-12-07T20:23:00+01:00

A similar approach can be used for any Android module.

cd aosp
bash
source build/envsetup.sh && \
lunch linaro_arm64-userdebug && \
mmm system/core/libsync/tests && \
adb root && \
adb remount && \
adb sync && \
adb shell /data/nativetest64/sync-unit-tests/sync-unit-tests

XDC 2016

2016-09-23T20:20:00+02:00

XDC 2016 was hosted in Helsinki at Haaga-Helia. The full program was filmed and is archived here.

[Slides] FastUIDraw - High Performance 2D renderer for GPUs

Kevin Rogovin gave an excellent talk about FastUIDraw, which is a highly optimiser 2d renderer for GPUs.

By agressively targetting GPUs only and limiting the feature set to what is required by a browser FastUIDraw performs >9.3x faster than Cairo-CPU and >4.8x times the previous GPU state of the art SKIA-GL.

Hopefully FastUIDraw can be incorporated into upstream of ChromiumOS and Android.

[Slides] 2D Performance

Martin Perez gave a talk about 2D performance and power consumption of the Xserver.

The xf86-video-intel driver showed quite mixed performance numbers of 1.51-32.6x times the CPU performance. The FPS/Watt measurements showed 0.73-15.1x times the CPU efficiency.

When looking at Cairo traces the power efficiency is actually lower using GPU accelaration than with CPU acceleration. This is somewhat expected in that the Cairo workload is not very high throughput, so the overhead of doing 2D operations is relatively high compared to the actual work.

Overall toolkits are moving away from letting the Xserver do 2D rendering, for reasons of portability and performance.

[Slides] libglvnd Status Update

Andy Ritger gave a talk about the current libglvnd status. The goal of libglvnd is to allow different graphics libraries from potentially different vendors to coexist on a filesystem and in a process. GLX/EGl/OpenGL/OpenGLES/GL are all supported by libglvnd.

Currently mesa supports libglvnd for OpenGL and GLX, with EGL support being in the pipeline.

[Slides] drm_hwcomposer

Sean Paul and Zach Reizner gave an exceptionally well timed talk about the Android && Chromium drm_hwcomposer project. drm_hwcomposer is an implementation of the hwcomposer (HWC) API ontop of the Linux DRM/KMS.

The talk detailed the implications of HWC2 and explicit fencing. A large part of the HWC1 implementation can be removed from drm_hwcomposer since it's made redundant by the fencing support in HWC2. So, for example the worker threads (DrmCompositorWorker and FrameWorker) are no longer necessary and can be removed.

[Slides] Status update of Nouveau

Samuel Pitoiset, Karol Herbst, Pierre Moreau and Martin Perez gave a talk about what has happened in Nouveau land the last year.

Hardware support is as always taking steps forward, with Fermi support is scheduled for Linux v4.9+.

A call to arms for Nouveau compiler optimizations was issued by Martin Perez. There is a lot of low hanging fruit for optimizations is available, and the compiler is overall in a good shape. So feel free to contact Martin or the Nouveau project if you are interested or curious.

As of the Maxwell generation of GPUs the firmware needed now has to be signed, GM20x/GP100 firmware has been released, but support for loading firmware on Tegra has not been provided by NVidia.

Martin made a rather clear point of needing to see some cooperation from NVidia in order for the Nouveau project to be able to make progress with new and upcoming NVidia hardware.

Conclusion

Thanks to the X.Org Foundation and the board of directors for arranging XDC 2016. This post has been a part of work undertaken by my employer Collabora.

Building Android for Qemu with Mesa and Virgil3D

2016-08-30T15:22:00+02:00

Developing Linux for Android on Qemu allows you to do some things that are not necessarily possible using the stock emulator. For my purposes I need access to a GPU and be able to modify the driver, which is where Virgilrenderer and Qemu comes in handy.

The guide below helps you compile Android and run it on top of Qemu with Mesa/Virgilrenderer supplying a virtual GPU. Because of this, the following guide is aimed at Linux hosts.

This guide is based on Rob Herrings fantastic guide, but has been slightly streamlined and had physical hardware support stripped out.

Install dependencies

These dependencies were available on Ubuntu 16.04, some alternative packages might be needed for other distributions.

sudo apt install autoconf gcc-aarch64-linux-gnu libaio-dev libbluetooth-dev libbrlapi-dev libbz2-dev libcap-dev libcap-ng-dev libcurl4-gnutls-dev libepoxy-dev libfdt-dev libgbm-dev libgles2-mesa-dev libglib2.0-dev libgtk-3-dev libibverbs-dev libjpeg8-dev liblzo2-dev libncurses5-dev libnuma-dev librbd-dev librdmacm-dev libsasl2-dev libsdl1.2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh2-1-dev libspice-server-dev libspice-server1 libtool libusb-1.0-0 libusb-1.0-0-dev libvde-dev libvdeplug-dev libvte-dev libxen-dev valgrind xfslibs-dev xutils-dev zlib1g-dev libusbredirhost-dev usbredirserver

Set up paths

Naturally all of the paths below are configurable, this is just what I used.

export PROJECT_PATH="/opt/qemu_android"
export VIRGLRENDERER_PATH="${PROJECT_PATH}/virglrenderer"
export QEMU_PATH="${PROJECT_PATH}/qemu"
export LINUX_PATH="${PROJECT_PATH}/linux"
export ANDROID_PATH="${PROJECT_PATH}/android"
export ANDROID_TOOLS_PATH="${PROJECT_PATH}/android-tools"

Virglrenderer

Virglrenderer creates a virtual 3D GPU, that allows the Qemu guest to use the graphics capabilities of the host machine.

git clone git://git.freedesktop.org/git/virglrenderer ${VIRGLRENDERER_PATH}
cd ${VIRGLRENDERER_PATH}
./autogen.sh
make -j7
sudo make install

Qemu

Qemu is a full system emulator, and supports a multitude of machine architectures. We're going to to use x86_64 but also build support for arm64/aarch64.

git clone git://git.qemu-project.org/qemu.git ${QEMU_PATH}
mkdir ${QEMU_PATH}/build
cd ${QEMU_PATH}/build
../configure --target-list=aarch64-softmmu,x86_64-softmmu --enable-gtk --with-gtkabi=3.0 --enable-kvm --enable-spice --enable-usb-redir --enable-libusb
make -j7

Linux kernel

Build trunk of mainline linux kernel.

Important: The below instructions use upstream/master but during testing of this guide, https://git.kernel.org/pub/scm/linux/kernel/git/padovan/linux.git and the fences branch was used due to SW_SYNC not yet being included in upstream. Inclusion is targeted for v4.9.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ${LINUX_PATH}
cd ${LINUX_PATH}
wget http://memcpy.io/files/2016-08-30/Kconfig -O ${LINUX_PATH}/.config
make oldconfig
make -j7

Important: If you decide not to use the .config linked in this step, a few Kconfig options need to be set:

CONFIG_ANDROID=y
CONFIG_ANDROID_BINDER_IPC=y
CONFIG_AUDIT=y
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_DRM=y
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=0
CONFIG_DEFAULT_SECURITY_SELINUX=y
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_VIRTIO_BLK=y
CONFIG_SCSI_VIRTIO=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_DRM_VIRTIO_GPU=y
CONFIG_VIRT_DRIVERS=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_SYNC=y
CONFIG_SW_SYNC=y
CONFIG_SYNC_FILE=y

Android

Build the Android Open Source Project.

Important: When running source build/envsetup.sh make sure that you are using bash. I had issues running lunch using zsh.

mkdir ${ANDROID_PATH}
cd ${ANDROID_PATH}
repo init -u https://android.googlesource.com/platform/manifest -b master
cd .repo
git clone https://github.com/robherring/android_manifest.git -b android-6.0 local_manifests
cd ..
repo sync -j20
cd device/linaro/generic
make defconfig
make all
cd ../../..
# The following snippet must be run in bash
bash
source build/envsetup.sh
# Select linaro_x86_64-userdebug
lunch
make -j7
# We don't need to use bash any longer
exit

As of this writing DRM fences related patches by Gustavo Padovan have yet to be included into AOSP, and therefore have to be included manually until it is upstreamed. After switching to this branch, the AOSP project has to be rebuilt again.

cd $ANDROID_PATH/system/core/
git remote add padovan git://git.collabora.com/git/user/padovan/android-system-core.git
git fetch padovan
git checkout padovan/master

mkbootimg

Fetch the make boot image script. This script later assembles the boot image, boot.img.

git clone https://android.googlesource.com/platform/system/core.git $ANDROID_TOOLS_PATH

Run Qemu machine

When running the below script, make sure that the all of the paths from step two have been exported.

wget http://memcpy.io/files/2016-08-30/boot_android_qemu.sh -O ${PROJECT_PATH}/boot_android_qemu.sh
chmod +x ${PROJECT_PATH}/boot_android_qemu.sh
${PROJECT_PATH}/boot_android_qemu.sh x86_64

Conclusion

Hopefully this guide will have enabled you build the required software and run Android on Qemu with a virtual GPU. This post has been a part of work undertaken by my employer Collabora.

Ethernet device stress testing

2016-08-25T13:05:00+02:00

During testing of power management patches for usb ethernet dongles, a script was needed to stress test connecting/disconnecting/reconnecting these devices.

Luckily a script like that already exists as a part of the chromiumos project, and can be found here.

That script does however not run standalone and requires a remote device (chromebook) to execute on. So I took the liberty of changing it to support local testing. The modified version can be found here.

This might come in handy for someone, if not, the script will at least be archived on this site.

Example

$ sudo pip2.7 install autotest
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DORMANT group default qlen 1000
    link/ether 48:e2:44:f6:e8:5b brd ff:ff:ff:ff:ff:ff
27: enx000ec689ab9e: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:0e:c6:89:ab:9e brd ff:ff:ff:ff:ff:ff
$ export INTERFACE=enx000ec689ab9e
$ export NUM_ITERATIONS=10
$ sudo python network_EthernetStressPlug.py $INTERFACE $NUM_ITERATIONS

Linux kernel development shell scripts

2016-07-26T10:32:00+02:00

While upstreaming kernel patches scripts/checkpatch.pl and scripts/get_maintainer.pl often come in handy. But to me the interface they provide is slightly bulky and rely on using patch files instead of git commits, which to me is a bit inconvenient.

These scripts are all meant to be included in .bashrc or .zshrc

scripts/checkpatch.pl helper

#!/bin/bash
function checkpatch {
  if [ -z ${1+x} ]; then
    exec git diff | scripts/checkpatch.pl --no-signoff -q -
  elif [[ $1 == *"cache"* ]]; then
    exec git diff --cached | scripts/checkpatch.pl --no-signoff -q -
  else
    NUM_COMMITS=$1
    exec git diff HEAD~$NUM_COMMITS..HEAD | scripts/checkpatch.pl --no-signoff -q -
  fi
}

The checkpatch script simply wraps the patch creation process and allows you to right away specify which

Example

~/work/linux $ checkpatch 15
WARNING: ENOSYS means 'invalid syscall nr' and nothing else
#349: FILE: drivers/tty/serial/sh-sci.c:3026:
+   if (IS_ERR(sciport->gpios) && PTR_ERR(sciport->gpios) != -ENOSYS)

total: 0 errors, 1 warnings, 385 lines checked

In this example the 15 last commits are checked against scripts/checkpatch.pl for correctness.

scripts/get_maintainer.pl helper

#!/bin/bash
function get_maintainer {
  NUM_COMMITS=$1

  MAINTAINERS=$(git format-patch HEAD~$NUM_COMMITS..HEAD --stdout | scripts/get_maintainer.pl)

  # Remove extraneous stats
  MAINTAINERS=$(echo "$MAINTAINERS" | sed 's/(.*//g')

  # Remove names from email addresses
  MAINTAINERS=$(echo "$MAINTAINERS" | sed 's/.*<//g')

  # Remove left over character
  MAINTAINERS=$(echo "$MAINTAINERS" | sed 's/>//g')

  echo "$MAINTAINERS" | while read email; do
    echo -n "--to=${email}  ";
  done
}

Example

~/work/linux $ get_maintainer 1
--to=gregkh@linuxfoundation.org  --to=jslaby@suse.com  --to=linux-serial@vger.kernel.org  --to=linux-kernel@vger.kernel.org

~/work/linux $ git send-email -1 $(get_maintainer 1)

Running Weston on a Raspbian

2016-06-03T10:32:00+02:00

Progress in the VC4 graphics camp and the Wayland camp now enables us to run Weston on top of the drm backend for VC4 platforms. Previously software acceleration using pixman was needed, but this is no longer the case.

Additionally the rpi backend for weston is now being removed since it has been obsoleted by the improved drm layer.

Let's explore running hardware accelerated Weston on the Raspberry Pi.

Building Linux kernel

A comprehensive guide for building a recent Linux kernel for Raspberry Pi boards has been written by the Raspberry Pi foundation and is available here.

As of this writing the guide helps you build a v4.4 kernel which is good enough for our purposes.

Set up alternative install location

These build instructions are based on the Wayland instructions from freedesktop.org, but altered to target VC4 and Raspbian.

You probably don't want to install experimental builds of software among the usual software of your operating system, so let's define a prefix for where to install our builds.

# Change WLD to any location you like
export WLD=~/local
export LD_LIBRARY_PATH=$WLD/lib
export PKG_CONFIG_PATH=$WLD/lib/pkgconfig/:$WLD/share/pkgconfig/
export PATH=$WLD/bin:$PATH
export ACLOCAL_PATH=$WLD/share/aclocal
export ACLOCAL="aclocal -I $ACLOCAL_PATH"

# Needed by autotools
mkdir -p $WLD/share/aclocal

Installing dependencies

Start by installing the build dependencies of mesa, weston and wayland.

# Enable source packages
sudo sed -e "s/#\sdeb-src/deb-src/g" -i /etc/apt/sources.list
sudo apt update

The above step can alternatively be completed using the GUI of your package manager, by enabling source packages.

# Install build dependencies of mesa
sudo apt-get build-dep mesa

# Install build dependencies of wayland/weston
sudo apt-get install \
  libevdev libevdev-dev \
  libwacom libwacom-dev \
  libxkbcommon libxkbcommon-dev

Building Mesa

Configure and compile mesa with vc4, wayland and EGL support.

git clone git://anongit.freedesktop.org/mesa/mesa
cd mesa
./autogen.sh --prefix=$WLD \
  --enable-gles2 \
  --with-egl-platforms=x11,wayland,drm \
  --enable-gbm --enable-shared-glapi \
  --with-gallium-drivers=vc4 \
  --without-dri-drivers \
  --disable-va \
  --disable-vdpau \
  --disable-xvmc \
  --disable-omx
make -j4 && make install

Building Weston and dependencies

Weston and Wayland have a number of dependencies that also need to be fetched and built.

Wayland

Weston is a Wayland compositor, so we're going to have to build Wayland.

git clone git://anongit.freedesktop.org/wayland/wayland
cd wayland
./autogen.sh --prefix=$WLD
make -j4 && make install
cd ..

git clone git://anongit.freedesktop.org/wayland/wayland-protocols
cd wayland-protocols
./autogen.sh --prefix=$WLD
make install
cd ..

libinput

libinput is a dependency of Weston, handles input devices like keyboards, touchpads and mice.

git clone git://anongit.freedesktop.org/wayland/libinput
cd libinput
./autogen.sh --prefix=$WLD
make -j4 && make install
cd ..

Weston

Finally we've built all of the dependencies of Weston and can now build it.

git clone git://anongit.freedesktop.org/wayland/weston
cd weston
./autogen.sh --prefix=$WLD \
  --disable-libunwind
make -j4 &&
sudo make install
cd ..

Running Weston

Running weston in this way depends on logind.

# Make sure that $DISPLAY is unset.
unset DISPLAY

# And that $XDG_RUNTIME_DIR has been set and created.
if test -z "${XDG_RUNTIME_DIR}"; then
  export XDG_RUNTIME_DIR=/tmp/${UID}-runtime-dir
  if ! test -d "${XDG_RUNTIME_DIR}"; then
    mkdir "${XDG_RUNTIME_DIR}"
    chmod 0700 "${XDG_RUNTIME_DIR}"1
  fi
fi

# Run weston:
weston

Try weston applications

Now that we're running weston, let's try some applications. They're located in the top level directory of weston.

weston-terminal
weston-flower
weston-gears
weston-smoke
weston-image
weston-view
weston-resizor
weston-eventdemo

When you've started all of your favorite applications you can grab a screenshot by pressing Super + s, which will save wayland-screenshot.png in your home directory.

Coverpage template

2016-03-27T00:34:00+01:00

Coverpage is a single-page landing page built to showcase an idea or a product. To allow interested parties to get notified of updates, the template has mailchimp subscription integration.

A live version of the site can be found at coverpage.memcpy.io.

Sources

git clone https://github.com/robertfoss/coverpage.git

GitHub hosting

This template was built with the explicit intention of having it be hosted at GitHub in a gh-pages branch. Therefore it includes a Makefile for pushing copy of the current design to a gh-pages branch.

ESP8266 APA102 Bulb

2016-02-07T22:46:00+01:00

The product of this project is a WiFi connected LED bulb. Every LED on this bulb is individually programmable over the WiFi, by simply sending UDP packets to the bulb.

Software and hardware sources

git clone https://github.com/robertfoss/esp8266_apa102_bulb.git

This project consists of 3 parts: the software running on the led bulb, the software running on some host computer and the hardware.

Firmware

The firmare is based on the NodeMCU firwmare for the ESP8266. It's running the APA102 LED driver and the enduser setup module, which I've written about previously.

Additionally it's running 3 lua scripts that deal with different aspects.

There's init.lua which makes sure we're connected to a WiFi.

udp_listener.lua receives UDP packets and then sends forwards that data to the APA102 strips.

And lastly udb_broadcast.lua which periodically broadcasts a heartbeat for this LED bulb to signal that it is alive and well.

Host application

The current (as of the publish date of this post) incarnation of the host application listens for bulbs that are alive on the hosts network. If a bulb is found is will be added to the list of bulbs to be animated. All animations are simple and sinusoidal and only use the time a bulb has been 'alive' as an input for the animation.

Hardware

The hardware is based around the ESP8266 WiFi IC and the APA102 SPI LED IC.

The flavor of ESP8266 used in this project is the ESP12-F module, since it the latest module available with the integrated antenna form factor.

APA102 was chosen instead of the much more common WS2812B chip, since it uses a SPI like protocol which isn't timing sensitive and also does not require external capacitors at next to each LED.

v3.1 Schematic

v2 3D Model

Assembled v2 hardware

WS2812 LED Matrix

2015-10-12T16:51:00+02:00

The LED Matrix is constructed out of a WS2812B 8x8 panel, a 3D printed case, an ESP8266, a LiPo battery and some software.

Code

git clone https://github.com/robertfoss/esp8266_ws2812_matrix.git

The software is divided into two parts. The receiving end is run on the ESP8266 that has been flashed with NodeMCU and is a simple lua script that receives UDP packets and forwards them to the excellent WS2812 module created by the the equally excellent Konrad Beckmann.

The second part is run on a host pc. It's a simple python script that generates a framebuffer from a sinusoidal animation and transmits it over UDP to the IP of the ESP8266. This relies on the host pc being able to connect to the ESP8266. Ideally they're on the same LAN to minimise UDP packets being lost or delayed, which causes the animation to lag or skip. This part was also written by Konrad Beckmann as part of a larger fancier animation, but repurposed here to be used as a proof of concept.

3D Model

The 3D model was created using google sketchup. The case is a made out of 2 parts. A front and a lid. The front has grooves that fit the WS2812 LEDs and make sure that the matrix is firmly stuck in place but also make sure that the light from the LEDs is diffused the proper amount.

Bill of materials

18$ WS2812B 8x8 Matrix
2$ ESP8266 ESP-01
1$ AMS1117-3.3 Voltage regulator
4$ LiPo 3.7V 300mAh - A higher capacity battery is better as long as it fits inside of the case.
$3 LiPo USB Charger
$6 Battery connector cable - Only a few will be needed.

A thing to be aware of is the fact that all orders from AliExpress will take at least 30 days to arrive to your door. Sometimes more.

Hardware construction

You'll have to excuse the lack of a proper schematic, but this has been a fairly organic project.

The connectors of for the battery (B_VCC and B_GND), the WS2812 matrix (LED_VCC, LED_GND and LED_DATA), the ESP8266 (ESP_3V3, ESP_CH_PD, ESP_GND, ESP_GPIO2) and the AMS1117-3.3V (LDO_VCC, LDO_GND and LDO_3V3)

B_VCC + LED_VCC + LDO_VCC
B_GND + LED_GND + LDO_GND + ESP_GND
LDO_3V3 + ESP_3V3 + ESP_CH_PD
LED_DATA + ESP_GPIO2

Issues and TODOs

The AMS1117-3.3 is not ideal. The voltage drop is too high. Ideally it should be replaced with a Step-UP/DOWN converter. When the battery is running low the ESP8266 is the first part to fail. It does so while there is quite a lot of power left in the LiPo.
This setup does not work at all without a WiFi or a host computer feeding it animations. Nothing is preventing is from working without a connection, but the connection is one of the most interesting aspects of the project.

WS2812 Welding Goggles

2015-10-11T13:16:00+02:00

The goggles were built using an AtTiny85 module and a WS2812B strip and some bits and bobs that were laying around. A more complete BOM can be found below.

Code

git clone https://github.com/robertfoss/digispark_ws2812_waves.git

The animation is built ontop of the excellent light_ws2812 library. It supports the AtTiny85 better than other competing libraries like FastLED.

The animation is built around an non-float sin function and a HSV function that generates the actual colors.

Video

Your browser does not support the video tag.

Bill of materials

6€ Weldorado STROOF DIN4 Welding Goggles
2$ Digispark
23$ WS2812B Strip 144 LEDs/m - Only about a 5th of the length needed.
2x$4 LiPo 3.7V 300mAh
$3 LiPo USB Charger
$6 Battery connector cable - Only a few will be needed.
$4 Rainbow cable - Any cable would do, but this keeps the cables bunched.
$3 Black 20mm velcro - Only a small part will be needed for attaching to batteries and the inside of the goggles.

Things to keep in mind when ordering these parts are that Weldorado won't accept an order for a single pair of goggles. You will have to order 3-4 pairs. And additionally pay a decent chunk of change for the shipping. So the more goggles you can find a use for the cheaper it will get. They are very nice looking though. Especially the rubber ones. A thing to note about the goggles is that the headstrap that ships with them is failry useless and probably should be replaced.

Another thing to be aware of is the fact that all orders from AliExpress will take at least 30 days to arrive to your door. Sometimes more.

Hardware construction

You'll have to excuse the lack of a proper schematic, but this has been a fairly organic project.

The connectors of for the batteries (B_vcc and B_gnd), both WS2812B strips (Led_vcc, Led_gnd and Led_data) and the digispark (Dig_5v, Dig_gnd and Dig_pin#0) are soldered as follows:

B_vcc + Led_vcc + Dig_5v
B_gnd + Led_gnd + Dig_gnd
Led_data + D_pin#0

Issues and TODOs

There is complete lack of an off-switch, and plugging and unplugging the battery is slightly messy. Especially in the unlit situations where these goggles would be fun to wear.
The brightness is static and can only be changed by reflashing the AtTiny85. A potentiometer or button that toggles between different intensities (and possibly an OFF of sleep mode) would be very useful.
The current head strap is made out of leather and is slightly hard to put on yourself.
The glass is black which does make it very hard to see when it's dark. Since that is when the goggles are most fun to use, maybe the glass should be replaced.

User friendly setup of ESP8266 gadgets

2015-07-30T15:10:00+02:00

The ESP8266 lacks a simple end-user compatible way to be configured with the login details of the local WiFi. Running the ESP8266 in a 'captive portal' mode where every http request is hijacked and replaced with form allows the end user to submit the credentials of the local WiFi hotspot by simply connecting to an ESP8266 hosted access point.

esp8266_nodemcu_wifi_setup is designed to provide a simple interface for users to configure their ESP8266/NodeMCU based devices through.

Start ESP.
Connect to the "SetupGadget" WiFi through your internet enabled thing of choice.
Submit credentials of your local WiFi.
The ESP reboots and connects to your local WiFi.

What is this magic! How could a mere mortal like me summon features like this?!

Let me tell you..

By default DNS is not announced in the DHCP offer message, because a DNS server does not ship with NodeMCU. In order to announce that the ESP8266 is running a DNS server, NodeMCU has to be recompiled to support that. A guide to building NodeMCU can be found here.

Thanks to the wonderful work of Andy Reischles on Captive Portal, a lua implementation of a domain hijacking DNS server exists. Which will allow an ESP8266 to redirect all traffic to itself.

Detailed guide

In dhcpserver.h, #define USE_DNS 1
Build NodeMCU.
Flash NodeMCU.
Upload all .lua files and index.html.
Reboot ESP.

What you need to do

After a user has connected throught the portal and sucessfully configured the the ESP8266 with the credentials of the local WiFi, something lua service should be started so that the ESP8266 actually does something.

I would suggest adding something like this to init.lua:

dofile("init_connected.lua")

Where init_connected.lua is where your script resides.

Building NodeMCU for the ESP8266

2015-05-14T11:31:00+02:00

This is a simple step-by-step guide to building and flashing NodeMCU for the ESP8266.

Build ESP8266 SDK

sudo apt-get install make unrar autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python sed
cd /opt
git clone https://github.com/pfalcon/esp-open-sdk.git
cd esp-open-sdk
git submodule sync
git submodule update
make
export PATH=/opt/esp-open-sdk/xtensa-lx106-elf/bin:$PATH

Build NodeMCU

git clone https://github.com/nodemcu/nodemcu-firmware.git
cd nodemcu-firmware
# Switch to dev branch, since it's the most up to date
git checkout --track origin/dev
# Make INTEGER version, since it uses less memory/space
make EXTRA_CCFLAGS="-DLUA_NUMBER_INTEGRAL"

Flash NodeMCU

COMPORT=/dev/ttyUSB0 make flash

Modifying NodeMCU to fit your needs

app/include/user_modules.h

Contains a list of which modules to include in the build.

...
#define LUA_USE_MODULES_NODE
#define LUA_USE_MODULES_FILE
#define LUA_USE_MODULES_GPIO
#define LUA_USE_MODULES_WIFI
#define LUA_USE_MODULES_NET
...

app/include/user_config.h

Contains NodeMCU developer options, but could be of use.

...
#define DEVELOP_VERSION
#define NODE_ERROR
...