Porting Android live wallpaper to WebGL

Now WebGL works in almost any browser, including mobile ones so it was tempting to try it out. We have experience in creating Android apps using OpenGL ES 2.0. We have released quite a lot of 3D live wallpapers with rich 3D graphics. They are implemented in Java + OpenGL ES 2.0 without using third-party game engines (such as Unity) or high-level frameworks (such as libGDX). This makes our apps lightweight and well-optimized. And since WebGL is based on OpenGL ES 2.0 process of porting live wallpaper to run in browser is quite straightforward.

Basic renderer code

Similar to original Java code, I’ve implemented BaseRenderer and BaseShader classes. I decided to use ECMAScript 2015 JS classes because they would improve readability of the code and the only browser not supporting it is IE11. BaseRenderer handles WebGL context creation and initialization, viewport resize, etc. Also it has empty stubs for all necessary functions. BaseShader has code to compile and use shader. Actual implementation of renderer with loading data and drawing stuff is in BitcoinRenderer.

Loading raw binary data

Most WebGL engines and demos load data from either JSON, OBJ or other formats. This approach requires some additional processing of data to create buffers which will be consumed by GPU. In our Java engine we use binary data which is ready to be put to OpenGL buffers, and it is implemented in demo the same way. Luckily, XMLHttpRequest Level 2, which is a part of HTML5 specification, supports loading of binary data in JavaScript. For convenience, I’ve created a simple BinaryDataLoader wrapper for loading binary data using XHR 2.

FullModel class handles work with meshes. Its load() method loads two buffers for mesh - strides buffer with data about vertices, UVs and other data and indices buffer which defines actual triangles of a mesh. These buffers contain binary data ready to be provided to GPU. This class also has bindBuffers() method to bind buffers before actual glDrawElements() call.

Using ETC1 compressed textures

To save video memory usage we use various compressed textures. Our Java framework supports ETC1, ETC2, PVRTC and ASTC compressed textures and it uses the one which is most suitable for given Android device. In WebGL port I’ve implemented only ETC1 and uncompressed RGB textures.

In OpenGL ES 2.0 ETC1 is an obligatory part of specification and is supported on all devices with no exceptions. However, in WebGL this compression is optional, and you have to check for presence of WEBGL_compressed_texture_etc1 extension. All desktop browsers except IE11 and Edge support ETC1 compression. For Microsoft browsers we simply fall back to uncompressed textures.

You can check which texture compressions your browser supports using this handy page - http://toji.github.io/texture-tester/

By using ETC1 textures we achieved reduced memory usage and faster loading times. Uncompressed textures have some loading overhead. They have to be decoded from original image format (PNG, JPEG, WebP, GIF, etc) to bitmap (either RGBA or RGB, with or without alpha) and then uploaded to GPU. On the other hand, ETC1 textures are ready to be uploaded to GPU and therefore load significantly faster. Speaking of a memory footprint, 512x512 uncompressed RGB texture uses 768 kb and the same textures uses only 128 kb in ETC1 compressed format. However, ETC1 is not perfect and introduces some noticeable visual artifacts. These artifacts are not visible on diffuse textures and lightmaps but can be clearly seen on normal maps (blocky, distorted normals) and spherical reflection maps (colors are not accurate). So we use both compressed and uncompressed textures depending on requirements to texture quality.

In Android, loading of ETC1 is very simple - you can use ETC1Util to load texture stored in PKM format. Since texture compression is not a required part of WebGL, it doesn’t provide any utilities for loading compressed textures from any known file format. I had to create my own utility to handle this.

PKM is a quite simple file format, it has a 16 bytes header followed by raw data ready to be passed to GPU. You can get more information about its header here and here. However, there is one tricky part in retrieving texture dimensions from header in JavaScript. These values are stored as 16 bit big-endian integers. We cannot use Int16Array to get these values because there is no way to specify endianness of buffer and we have to read bytes using Uint8Array and calculate 16-bit value manually from its high and low bytes.

Shaders

Application makes use of only two GLSL shaders: one for table surface and another one for coins.

Table surface shader is a basic lightmapping shader - it uses two texture samplers: one for diffuse color and another one for lightmap texture. Their values are simply multiplied and that is the final output of the shader.

Coins shader is more complicated, it incorporates following features:

1. Sphere mapping (a.k.a. matcap material);
2. Lightmapping with color boosting;
3. Normal mapping.

Sphere mapping is a variation of reflection mapping technique. It stores information about reflection in a texture which looks like a photo of chrome sphere:

Sphere mapping has following pros and cons reflection techniques:

• minimal shader code, great efficiency and performance;
• overhead of only one 2D texture sample;
• reflection texture is simple to understand and manipulate in Photoshop;

• it does not work on flat surfaces;
• texture space usage is sub-optimal (corners are not used).

The best thing about sphere mapping is the ease of implementation: once you have your screen-space normals calculated, use its (x,y) part as UV coordinates to read reflection and that’s all. Below is an excerpt from our shader:

  vec4 sphereColor = texture2D(sphereMap, vec2(vNormal2.x, vNormal2.y));

However, sphere mapping has one disadvantage: it is calculated based on screen-space normal only, thus, it would fill large flat polygons with a constant color (as they have constant normals). And coin model consists mostly of two flat surfaces. To overcome this problem, we’ve added normal maps to our coin models and made them as busy as possible: with noise, numbers and text added. This disrupted the flatness of the model enough, and sphere mapping worked just fine.

One more trick was used for lighting coins. Coins use baked lightmap texture as well, but with one trick in shader. Usual multiplication with lightmap texture would result in quite correct, but dull result: coin colors would become only darker in shadowed places. Instead, we multiply coins color by itself, using pow() function. The exponent is bigger for darker lightmap areas. This simulates the behavior of pile of shiny coins when light gets ‘trapped’ in tight places and intensifies itself because of multiple reflections from the same metal. These techniques result in in more realistic metallic surface:

Conclusion

You can watch final demo on this page and compare it to original Bitcoins 3D Live Wallpaper. It works in latest Chrome, Firefox, Safari and MS Edge browsers. You can get sources from GitHub here and reuse it in your projects under terms of MIT license.