Learning WebGL (GPU Architecture and Shaders)

Earlier this past week, I decided it was time to break some new ground (for myself), and make the dive into learning WebGL. This is less about specifically learning WebGL, and more about introducing myself to GPU-based rendering and programmable shaders.

Motivation

For the past several years, I've been building up the Exploring Winnipeg Parks website. Last summer, I decided to really up the ante by building a series of Python scripts to automatically generate the entire website using Jinja2 templates. It includes several Gimp Python-fu scripts to perform image processing for all of the website's photo galleries:

• Levels and Sharpen for wallpaper photos.
• Rotate, Crop, and Resize.
• Drawing captions.
• Processing and attaching meta-data, including GPS co-ordinates.

I grew up in the 80s, and took my formal education in the 90s, and I've long had a CPU rendering mindset. About 20-25 years ago, the graphics world was experiencing an upheaval with the proliferation of programmable GPUs. I finally decided to explore how modern GPU rendering pipelines actually work.

So I took the Gimp Python-fu scripts I'd written, and worked on a proof-of-concept to see if I could replicate what those scripts are doing with WebGL. The results have been extremely promising.

I wrote a set of WebGL shaders that mimic Gimp's Levels and Sharpen (Unsharp Mask) operations, as well as a general-purpose matrix transform shader. For the captions, I turned to the 2D Canvas API. I'm able to replicate the results of the Gimp Python-fu scripts, with only very subtle differences. The only real challenge is the technical difficulty -- WebGL is definitely a low-level API, meant for the technically-inclined. The tangible results only come away with positives:

• Dramatically faster image processing. The bottleneck is no longer expensive image processing operations, but instead shifts to loading and saving image files.
• Equivalent in terms of deterministic results.
• A pathway to a more streamlined workflow.

My Game Development Future

Another important benefit to this weekend-long project is the impact it will hopefully have on my future in game development. I consistently come across discussions about graphics and rendering that suggest I've been behind the curve for quite some time. This should open the door to a whole new set of tools. I hope to take this knowledge and be able to apply it to future game development projects.

GPU Architecture Concepts

Vertex Arrays describe the geometry that will be rendered. For 2D image processing, this is typically just a quad (two triangles) covering the entire image or screen.

Framebuffers are the target surface that the GPU renders into. This is often an off-screen texture. On a webpage, it can also be a canvas element. When chaining together multiple image processing operations, it's common to have several "scratch" framebuffers for intermediate results.

Vertex Shaders are small GPU programs that run for every vertex being rendered. It essentially maps each vertex to its position and appearance in the framebuffer. For 2D image processing operations, I only needed a single simple vertex shader.

Fragment Shaders are GPU programs that run for every pixel being drawn. For 2D image processing operations, this is where the majority of the work gets done. If you're coming at this from the mindset of software-based image processing, you can think of this as being the work done in the inner loop, for each pixel. On the GPU, it can be done in parallel across many execution units (CUDA cores or stream processors).

When porting my Gimp Python-fu scripts to WebGL, each image processing operation I wanted to support became its own custom fragment shader:

• Levels (input and output colors, gamma correction).
• Generalized 1D convolution (blurs).
• Unsharp Mask.
• Generalized matrix transform.

While building my jigsaw puzzle game earlier this year, I wrote a software rasterization and image processing library. It included operations like blurs and glows, pixel erosion, and image compositing. All of these concepts are transferrable to GPU shaders.

Peg City Parks Jigsaw

And speaking of my jigsaw puzzle game, it has been live for about two months now! There are now over 50 puzzles, with new bundles being added regularly throughout the rest of the summer. I think it's quite good for an online jigsaw puzzle game, so go check it out if you haven't yet!