The last few days have been a deep dive into V\OS’s graphics pipeline. What started as “make the DRM backend boot in QEMU” turned into building out per-window GPU compositing, fixing Haiku’s menu rendering, writing a gears demo, and learning more about virtio-gpu’s limitations than I ever wanted to know.

where we started#

V\OS had a DRM display backend from x512’s Haiku work. It could put pixels on screen via dumb buffers and page flips. But there was no cursor, windows flickered when dragged, the GBM/EGL code had never been tested on a running system, and the whole thing only worked with -vga std in QEMU.

what we built#

GBM display backend#

The GBM backend now handles buffer allocation, DRM mode setting, and display output. Key details:

  • GBM buffers created with LINEAR|WRITE flags for cross-driver compatibility. A Flush() method calls gbm_bo_write to sync CPU writes to GPU memory, which turned out to be essential for virtio-gpu where mapped buffers use shadow copies.
  • libseat for unprivileged DRM access. The app_server doesn’t need to run as root.
  • drmModeSetCrtc instead of async drmModePageFlip. Simpler, no EBUSY races, works everywhere. We tried page flips first and spent hours debugging event draining and timing issues before realizing synchronous mode setting is just better for a software-rendered desktop.

software cursor#

Both the DRM and GBM backends had no mouse cursor. The root cause: they overrode CopyBackToFront with full page flips, bypassing the base class HWInterface::CopyBackToFront which handles partial region copy and software cursor compositing. The fix was to delegate to the base class for all region/cursor work and only add the display update step (SetCrtc) after.

per-window compositing#

The biggest piece of work. In the original Haiku model, all windows render into one shared back buffer via AGG (a CPU software renderer). When you drag a window, everything behind it redraws. Per-window compositing gives each window its own buffer.

The architecture: each window gets a WindowBuffer (a MallocBuffer wrapper). Drawing happens via AGG with a virtual base pointer that maps screen coordinates to window-local buffer positions. The Desktop::_CompositeAllWindows assembles all window buffers onto the shared back buffer, then the normal CopyBackToFront pipeline handles cursor and display.

The virtual base pointer trick is interesting. Instead of translating coordinates everywhere (which would require patching every Painter method), we shift the AGG rendering buffer’s base pointer backward so that screen-coordinate access at (frame.left, frame.top) maps to buffer position (0, 0). This makes all existing AGG drawing code work without modification. The tradeoff: if a window is dragged off-screen, the virtual base points before the buffer allocation, causing segfaults in AGG’s own rendering methods. We handle this by clamping all drawing regions to screen bounds.

GL compositor#

We built a full GLES2 compositor (GLCompositor) that renders window textures as positioned quads into an FBO, then reads pixels back via glReadPixels. The EGL context uses a pbuffer surface (or surfaceless) with deferred creation on the render thread to avoid cross-thread EGL issues.

On virtio-gpu with virgl, the compositor initializes and renders correctly. But glReadPixels from FBO returns zeros intermittently. This is a known virtio-gpu limitation. The compositor is wired up and ready for real hardware where readback works reliably.

BMenu fixes#

Haiku’s BMenuItem::Draw() only drew backgrounds for selected (highlighted) items. On initial menu show, no items are selected, so backgrounds were empty. Combined with per-window compositing’s deferred display, this made menus appear as grey rectangles until hovered.

The fix: fill unselected item backgrounds with the view’s low color (B_SOLID_LOW). Also, BMenu::Draw() deferred drawing after relayout via Invalidate() and returned without rendering. We now draw immediately with full Bounds() after relayout.

demos#

es2gears: A software-rendered spinning gears demo that displays in a BWindow via BBitmap. Three interlocking gears with flat shading, z-buffer, and perspective projection. Shows FPS in the window title. We tried GPU-rendered gears first (EGL/GLES2 via virgl) but virgl’s glReadPixels returns zeros for offscreen surfaces, making readback impossible in QEMU.

setres: A display resolution tool that queries DRM connector modes and lets you switch by number or WxH. Uses BScreen::GetModeList and BScreen::SetMode which we implemented in the GBM backend.

hard-won lessons#

Do not bypass the base class. Haiku’s HWInterface::CopyBackToFront has decades of battle-tested cursor compositing and partial region copy logic. Every attempt to replace it with a “simpler” full page flip broke something.

Synchronous beats clever. drmModeSetCrtc vs drmModePageFlip. The async path caused EBUSY errors, required event draining, and had race conditions. SetCrtc just works.

GBM mapped buffers are not always direct. On virtio-gpu, gbm_bo_map gives you a shadow copy even for LINEAR buffers. CPU writes are invisible to the GPU until explicitly flushed with gbm_bo_write.

EGL contexts have strict thread affinity. Creating a context on one thread and using it on another fails with EGL_BAD_ACCESS. All EGL resources must be created on the thread that uses them.

Never use unmap/remap for buffer flushing. We crashed in _CopyToFront because Flush() used gbm_bo_unmap + gbm_bo_map which invalidated the pointer while another thread was writing to it. gbm_bo_write copies from the existing mapped pointer without invalidating it.

The menu timing problem is architectural. Haiku menus draw items via a client update round-trip. The server-side expose only fills the view background. In per-window mode, the compositor shows the buffer before the client draws items. Fixing this properly requires either synchronous client drawing or a “don’t show until first commit” policy like Wayland compositors use.

what is next#

The software display path is solid and looks correct. Per-window compositing infrastructure is complete and crash-free but disabled by default due to cosmetic artifacts (stale content on drag, timing issues with menus). The GL compositor is wired and tested, waiting for real hardware.

Near-term priorities:

  • Test GPU compositor on real hardware (the virgl readback issue doesn’t exist on real GPUs)
  • Desktop icon rendering investigation (pre-existing xattr/Tracker issue)
  • BGLView implementation for GPU-rendered applications
  • GLTeapot port

The code is at github.com/VitruvianOS.