Abstract

For a long time, using Optimus systems with multiple GPUs is guaranteed to have a poor experience. Multiple GPUs were supposed to bring many benefits to Laptops:

Images are composited and displayed via the integrated GPU, while resource-intensive can render on the discrete GPU.
Both GPUs can be used to render at the same time.
The integrated GPU can be used to accelerate video encoding / decoding
The discrete GPU should be able to power down to preserve battery life and generate less heat % noise

But… There were multiple critical issues blocking this vision:

Many applications can wake up the discrete GPU, introducing delays while increasing power draw.
There isn’t a uniform way of specifying which GPU an application should use. For example EGL runs on iGPU while Vulkan may run on the discrete GPU (this happens a lot)

So, as the sandbox framework for the user, we introduced many useful workarounds for the hybrid GPU system configuration.

DRM / DRI nodes

Each graphics device may have some so-called “device files” under Direct Rendering Manager and Direct Rendering Infrastructure. They are typically named as renderD12* and card*. For example, /dev/dri on my device has the following content:

drwxr-xr-x   2 root root        120 Nov 17 22:54 by-path/
crw-rw----+  1 root video  226,   0 Nov 17 22:55 card0
crw-rw----+  1 root video  226,   1 Nov 17 22:55 card1
crw-rw-rw-   1 root render 226, 128 Nov 17 22:54 renderD128
crw-rw-rw-   1 root render 226, 129 Nov 17 22:54 renderD129

While the drm class has the following:

1	card0@ card0-eDP-1@ card1@ card1-DP-1@ card1-DP-2@ card1-eDP-2@ card1-HDMI-A-1@ renderD128@ renderD129@ version

You can see multiple devices creates many connector nodes. But wait… which one do we actually need?

Discovery

DRM connector entries (under sysfs/drm) list connectors such as card1-eDP-1, card1-DP-1, etc. Portable probes these connectors to determine which GPU(s) actually drive displays and then classifies the system into one of three modes:

Single GPU system
Game mode activated
Hybrid GPU with game mode disabled

Behaviour by mode

For the first two modes, things are simple. We just do the following:

Bind the whole PCI root (which will be explained further below)
Leave the drm class and /dev/dri untouched
Bind any NVIDIA devices under /dev since they are not included in the devtmpfs created by bubblewrap
Sets PRIME offload environment variables to guide applications to the dGPU in game mode
- Note that we also ignore some OS environments since they may cause issues in the sandbox, like the VK_LOADER_DRIVERS_DISABLE variable
- We also take driver differences in to account. Portable will set different environment variables based on the driver in use.

For hybrid systems with game mode disabled, Portable isolates and hides the unwanted dGPU:

First, we mount tmpfs over DRM/DRI interfaces in the sandboxed root to ensure clean disconnection from the host. A list of cards active is initialized. Portable then loops through interfaces under the DRM system to find out which connector from which GPU is active, e.g. card1-eDP-1 is representing an active connection being made from GPU 1 -> eDP 1. When a list of active GPUs has been generated. It then builds a list of active GPUs and maps cards to the corresponding renderD indexes. The generated bubblewrap commandline arguments are later passed to execApp() function via runtime directory for sandbox use.

Portable uses the above techniques to hide the unwanted discrete GPU when needed, and saves power during the process. Note that all of the compute part is done in a dedicated process to ensure speed. But we aren’t done yet.

Tracing sysfs entries

Some apps (looking at you, Chromium) enumerate PCI devices on startup, which may also cause a wake up. You might wonder: why can’t we hide all PCIe devices in the new root? Well, turns out you can. But that causes GPU acceleration to fail. Chromium also complains when it cannot read from /sys/devices and the GPU process exits early. Leaving only software rendering. Which is not we want.

Instead, we then try to find out the PCI address via link lookups, and bind the interface under /sys/devices into the sandbox for application access, and hide unrelated entries under sysfs. This happens in the same time while binding DRM/DRI nodes and is resolved using a function inline. Which ensures that only the specific graphics device is available under the devices folder of sysfs, with the side benefit of improving privacy since applications cannot determine what PCI hardware you have. This covers the PCI sysfs side of things.

Kernel modules

Portable has been using the ProtectKernelModules flag for a long period of time. This ensures that apps running inside could not tamper with kernel modules in the extremely rare situation where a module may allow regular users write things or someone is running Portable as root (we want to deny that ofc). However, kernel module sysfs entries (e.g., /sys/module/nvidia*, driver info under /proc) can still reveal or wake dGPUs. All of them seem to contain similar interfaces and structure. Yet again, they are another source of waking up the discrete GPU. The driver interface under procfs also contains many information about the driver and the GPU, it’s best to cut it out too.

With ProtectKernelModules alone, information can still be queried. To solve this issue, Portable mounts tmpfs on top of those interfaces when they exist. Note that hiding the entire module/ sysfs interface is not possible because i915 and other drivers will need them to function. With these measures, Portable addresses:

Many applications can wake up the discrete GPU, introducing delays while increasing power draw.

Afterthoughts

The upcoming gpu-daemon proposed may help to check out the following issue:

There isn’t a uniform way of specifying which GPU an application should use. For example EGL runs on iGPU while Vulkan may run on the discrete GPU (this happens a lot)

Portable will try to implement support for such daemon to provide a uniform experience for multi-GPU systems. Stay tuned!