The Plan - Emscripten - Makefile - Making a Splash - Handling Input - Audio - Fullscreen

The Plan

April 29, 2017

So here's the plan: I'm going to make a demo of Funky Karts using WebAssembly that runs on the open web. And, I'm going to document my journey on this page to share what I learn along the way.

What is WebAssembly?

From the official website:

"WebAssembly or wasm is a new portable, size- and load-time-efficient format suitable for compilation to the web."

It has now shipped in Firefox 52 and Chrome 57 with more browser support arriving in the near future. We are on the bleeding edge of web technology here, so expect a few bumps on the road ahead.

WebAssembly has many interesting use cases, but this is going to be about games. Specifically, one little game called Funky Karts.

Why Funky Karts?

Here's why I think Funky Karts and WebAssembly are a match made in heaven:

Tell me more.

Okay! Let the games begin. In the next article I'll dive into setting up my development environment and I'll compile something to WebAssembly.

Emscripten

April 30, 2017

Today will be the first time that I use Emscripten. From its website:

"Emscripten is an LLVM-based project that compiles C and C++ into highly-optimizable JavaScript in asm.js format. This lets you run C and C++ on the web at near-native speed, without plugins."

The above description hasn't been updated yet to include wasm, but the WebAssembly project was able to adapt Emscripten to target their new format and quickly provide a working toolchain for early adopters. Unlike WebAssembly itself, Emscripten has a mailing list that is definitely worth following to stay up-to-date with this evolving technology.

Before I dive into the Emscripten setup, I should mention that I will be doing this work on a Mac mini (Late 2014) running OSX 10.12.2 and I'll use Chrome 58 and Firefox 53 to test the output.

Alright, now let's get setup. A natural starting point is the official WebAssembly Developer's Guide. Looking at their list of prerequisites, I think that I've already got what I need on my system. I don't have CMake, but I'm not sure I'll need it so I am going to ignore that for now. I have a hunch that CMake is required to compile Emscripten itself, but a recent thread on the mailing list mentions new autocompiled builds so let's see if we can get by without needing to compile the compiler.

Instead of doing a git clone, my first step is to download the emsdk-portable for OSX and extract it to a freshly created ~/emsdk directory.

Now we have a minimal working C program compiled to WebAssembly and running in a browser! That went pretty smooth, and I'm eager to keep going. Next time I will create a Makefile and build our demo from multiple compilation units.

Makefile

May 2, 2017

Last time we learned how to build a minimal WebAssembly program from a single file, or compilation unit. Today, I'll use that knowledge to write a complete Makefile for Funky Karts on the web.

Why a Makefile?

It sounds pretty old school, doesn't it? In fact, that is one of the things that I like about Makefiles - they are tried and true. But also:

What's the structure?

In order to write a Makefile I'll first need to tell you a bit about the structure of the Funky Karts project. The code that we'll build into the web demo lives on the file system in the following directories: 

funky_karts
    |--- engine
        |--- brite
        |--- web
    |--- platform
        |--- web

The engine directory contains the vast majority of code in the game. The brite directory contains a custom cross-platform game engine, and we won't need to change a single line of that code for this project. It's a dream come true! But one of the classes in there, brite::Platform, is abstract and we'll need to fully implement it to allow the engine to interact with the web platform. We'll write that C++ code in the engine/web folder.

The platform directory contains the entry-point code and related files for each supported platform. The platform/web folder will therefore contain all of our html, css, JavaScript and also the Makefile that we'll work on today.

Now let's add some files to these directories: 

funky_karts
    |--- engine
        |--- brite
        |--- web
            |--- web_main.cpp
            |--- web_main.h
            |--- web_platform.cpp
            |--- web_platform.h
    |--- platform
        |--- web
            |--- main.cpp
            |--- Makefile

We see our Makefile now in the platform/web directory along with main.cpp which is the program entry-point from my last article (renamed from main.c). We've also added two compilation units, web_main.cpp and web_platform.cpp, to the engine/web directory. The difference between web_main.cpp and main.cpp is that the former is for engine logic specific to the web platform (but game agnostic) whereas the latter will contain code specific to Funky Karts.

Let's write it

I'm not going to explain every little detail in this Makefile. Writing a Makefile is one of those things you learn by doing, and there are some great resources out there to help. I'll start this one by defining a BUILD_DIR to dump all of our output: 

BUILD_DIR := build

This build directory will be a subdirectory of platform/web, i.e. the location of our Makefile. I have a rule in my .gitignore to keep this out of revision control.

Next, we'll define a few variables for each group of compilation units in our project structure: 

FK_SOURCES := main.cpp
FK_OBJECTS := $(addprefix $(BUILD_DIR)/, $(notdir $(FK_SOURCES:.cpp=.o)))
FK_CFLAGS := $(CFLAGS) -Wall -I../../engine/

WEB_SOURCES := $(wildcard ../../engine/web/*.cpp)
WEB_OBJECTS := $(addprefix $(BUILD_DIR)/web/, $(notdir $(WEB_SOURCES:.cpp=.o)))
WEB_CFLAGS := $(CFLAGS) -Wall -I../../engine/

BRITE_SOURCES := $(wildcard ../../engine/brite/*.cpp)
BRITE_OBJECTS := \
  $(addprefix $(BUILD_DIR)/brite/, $(notdir $(BRITE_SOURCES:.cpp=.o)))
BRITE_CFLAGS := $(CFLAGS) -Wall -I../../engine/

A few things worth noting:

Next, we'll define the two primary targets in this Makefile: 

all: $(BUILD_DIR)/fk

clean:
	@ rm -rf $(BUILD_DIR)

The `all` target is the default target which gets built when we call make without specifying a target. We see above that it depends on another target `$(BUILD_DIR)/fk` that we'll examine later. The other target here is `clean` which deletes our temporary build directory when we invoke it from the command line: 

emmake make clean

Our next set of targets use pattern rules to build object files from our sources. 

$(BUILD_DIR)/%.o: %.cpp
	@printf "%-40s %s\n" $@ "$(FK_CFLAGS)"
	@mkdir -p $(BUILD_DIR)
	@$(CC) -c $(FK_CFLAGS) -o $@ $<

$(BUILD_DIR)/web/%.o: ../../engine/web/%.cpp
	@printf "%-40s %s\n" $@ "$(WEB_CFLAGS)"
	@mkdir -p $(BUILD_DIR)/web
	@$(CC) -c $(WEB_CFLAGS) -o $@ $<

$(BUILD_DIR)/brite/%.o: ../../engine/brite/%.cpp
	@printf "%-40s %s\n" $@ "$(BRITE_CFLAGS)"
	@mkdir -p $(BUILD_DIR)/brite
	@$(CC) -c $(BRITE_CFLAGS) -o $@ $<

From our object files, we can write a target to generate linked LLVM bitcode: 

$(BUILD_DIR)/fk.bc: \
	$(FK_OBJECTS) \
	$(WEB_OBJECTS) \
	$(BRITE_OBJECTS)
	@printf "%s\n" $@
	@$(CC) -o $@ $^

The final step in the build process is to compile the linked bitcode to WebAssembly, along with the generated JavaScript runtime from Emscripten: 

$(BUILD_DIR)/fk: $(BUILD_DIR)/fk.bc
	@printf "%s\n" $@
	@$(CC) -O2 -s WASM=1 -o $(BUILD_DIR)/fk.js $<

This final step is also where we apply optimizations to the code. For now I'll pass just the -O2 flag to the compiler. It is likely that we'll add some more flags to this command line as we progress.

Looking back at our `all` target we remember that it depends on the `$(BUILD_DIR)/fk` target, meaning this is what will execute by default from the command line: 

emmake make

Let's have a final look inside of our build directory: 

build
    |--- brite
    |--- web
        |--- web_main.o
        |--- web_platform.o
    |--- fk.bc
    |--- fk.js
    |--- fk.wasm
    |--- main.o

Notice how we don't have any html in our build directory. We've told Emscripten to just generate the JavaScript and WebAssembly for us. The Emscripten-generated html is convenient for tinkering around, but for our final product we'll want full control over the html page that loads the game.

Today I compiled Funky Karts' cross-platform engine to LLVM bitcode using Emscripten, and put the platform-specific parts into place. Things are moving along nicely! The fk.bc linked bitcode is only 1.6 MB which is really promising since that contains the full engine without any optimizations applied. Next time I'll work on getting a main loop going for the game. This will also involve some graphics setup, since the two go hand in hand. If things go well we might even get our splash screen to render!

Making a Splash

May 5, 2017

Today our goal is to get Funky Karts' splash screen to render on the web. That may seem like a far stretch from where we left off last time, but we're closer than you might think!

Our Own Shell

Remember that last time we configured our Makefile to emit only JavaScript and WebAssembly with the intent to use our own html page. I ran into a problem with that plan, because loading the platform/web/build/fk.js from my platform/web/main.html results in an error. The error in this case is that the JavaScript runtime tries to load fk.wasm from platform/web instead of platform/web/build.

A bit of searching led me to the `--shell-file` option in `emcc --help`: 

"--shell-file <path>"
   The path name to a skeleton HTML file used when generating HTML
   output. The shell file used needs to have this token inside it:
   "{{{ SCRIPT }}}".

Following these instructions, I added {{{ SCRIPT }}} just before my closing </body> tag in platform/web/main.html and modified my Makefile: 

$(BUILD_DIR)/fk: $(BUILD_DIR)/fk.bc
	@printf "%s\n" $@
	@$(CC) -O2 -s WASM=1 --shell-file main.html -o $(BUILD_DIR)/demo.html $<

Building with these changes, the generated `$(BUILD_DIR)/demo.html` loads and runs without error. Now we have our own html shell that we can customize as we see fit.

Our Own Canvas

The <canvas> element is the most important element for our game, so once again we'll want to have full control. This means that we won't be using Emscripten's builtin canvas features. I'll need to set the id attribute of our <canvas> to a value other than #canvas, what Emscripten's runtime uses, to avoid crossing the streams. I'll call ours #display because that aligns with the engine code that we'll look at shortly: 

<body>
  <canvas id="display"></canvas>
  {{{ SCRIPT }}}
</body>

Above we see that the <body> of our main.html shell now includes our <canvas> and our magical {{{ SCRIPT }}} placeholder for the generated Emscripten runtime. I'll also add an inline <style> to our <head>: 

<style>
html, body
{
  height: 100%;
  margin: 0;
}
body
{
  display: flex;
  display: -webkit-flex;
  -webkit-justify-content: center;
          justify-content: center;
  -webkit-align-items: center;
          align-items: center;  
}
canvas#display
{
  display: block;
  outline: none;
  height: 100%;
  max-height: 75vw;
  width: 100%;
  max-width: 180vh;
}
</style>

The first grouping of selectors in our style just makes sure that we've got the entire page to work with. The second selector sets up our body as a flex container, so that we can center our canvas (the flex item) along both axes. The final selector for our #display is a bit more interesting.

Since this is a web page, the user is able to resize it to just about any dimensions that she wishes. Funky Karts' engine can actually handle that rather well, but with extreme aspect ratios the player can get confused and even cheat a bit by seeing more of the world than they should. For the best game experience, we want to keep a reasonable aspect ratio while at the same time offering some flexibility. That is where this last selector comes in, by limiting the canvas aspect ratio to between 4:3 and 16:9. It does this by using viewport-percentage lengths, to constrain the `max-height` and `max-width` of the canvas in terms of a percentage of the other dimension.

These style selectors are likely to evolve as the demo advances towards launch. But for now they work well enough that we can shift our focus to the engine code.

Graphics

Funky Karts uses the OpenGL ES 2.0 API for drawing its graphics. This makes it highly portable across a large variety of present day platforms, including the web. Thanks to OpenGL Support in Emscripten I won't need to change a single draw call in our engine. What I will need to do, however, is to create a graphics context and to handle certain display events such as resize, focus, and blur. I'll start by adding a new compilation unit, web_display to our set of platform-specific engine files: 

funky_karts
    |--- engine
        |--- brite
        |--- web
            |--- web_display.cpp
            |--- web_display.h
            |--- web_main.cpp
            |--- web_main.h
            |--- web_platform.cpp
            |--- web_platform.h

Next, inside this new header file we'll declare the following class for managing our graphics context: 

class WebDisplay : public brite::Display
{
  public:

    WebDisplay()
    {
    }

    virtual ~WebDisplay()
    {
    }

    void Initialize();
    void Update();

};

There won't be much logic in this WebDisplay class. As a subclass of brite::Display it is primarily responsible for reporting information about the graphics context back to the engine. I'll create the graphics context in the WebDisplay::Initialize function: 

void WebDisplay::Initialize()
{
  EmscriptenWebGLContextAttributes attribs;
  emscripten_webgl_init_context_attributes(&attribs);
  attribs.alpha = false;
  attribs.enableExtensionsByDefault = false;
  EMSCRIPTEN_WEBGL_CONTEXT_HANDLE context = 
      emscripten_webgl_create_context("display", &attribs);
  emscripten_webgl_make_context_current(context);
  emscripten_set_resize_callback(0 /* Window */, this, false, OnResize);
  emscripten_set_blur_callback("#window", NULL, false, OnBlur);
  emscripten_set_focus_callback("#window", NULL, false, OnFocus);
  Update();
  ContextRestored();
  brite::Platform::instance->Focus();
}

We begin this function by filling in the EmscriptenWebGLContextAttributes structure. We call emscripten_webgl_init_context_attributes to ensure forward compatibility in the case that new fields are added to this structure in the future. Then we override a couple of the default values:

Now that we've described the graphics context we'd like to receive from the browser, we call emscripten_webgl_create_context to create that context. We pass in "display" as the first argument which you'll remember is the id we chose previously for our <canvas> element. With the context in hand, we call emscripten_webgl_make_context_current to activate it for rendering.

Next we register callbacks with Emscripten to receive resize, focus and blur events from the Window object. Then we call our Update method: 

void WebDisplay::Update()
{
  double w = 0.0;
  double h = 0.0;
  emscripten_get_element_css_size("display", &w, &h);
  widthInt_ = (int32_t)brite::Roundf(w);
  heightInt_ = (int32_t)brite::Roundf(h);
  width_ = widthInt_ * 1.0f;
  height_ = heightInt_ * 1.0f;
  halfWidth_ = widthInt_ * 0.5f;
  halfHeight_ = heightInt_ * 0.5f;
  EM_ASM_({
    const display = document.querySelector("#display");
    display.width = $0;
    display.height = $1;
    display.tabIndex = 0;
  }, widthInt_, heightInt_);
  Resized();
}

The Update method is where we poll the browser for the current size of our <canvas>. We call emscripten_get_element_css_size to get the width and height values that we'll use to render.

We store redundant copies of the width and height as integers and as single-precision floating point values. And we also store the half-value of each dimension. These different values are used frequently in the engine, so calculating them ahead-of-time provides an optimization.

EM_ASM_ is a useful macro provided by Emscripten. It allows us to write JavaScript code `inline` in our C++ code. We use it here to set the logical `width` and `height` attributes of our <canvas> element.

The last call in Update notifies the engine of the resize by calling the Resized method which is defined in the brite::Display base class.

That's the entire WebDisplay class! We are ready to try it out, but first we'll need one more important piece of the puzzle - a main loop.

A Main Loop

Getting the details of a main loop right can be tricky. We want it to be buttery smooth and to draw in harmony, or vsync, with our monitor's refresh cycle. We also want to offload as much work as we can from this main drawing thread, to reduce dropped frames and take advantage of multi-core processors.

Funky Karts' engine is designed to update its simulation in a separate thread to address this last point. However, as of today WebAssembly does not yet support multiple threads, so we will need to make some adjustments. The good news is that there are plans to add support for threads in the future.

Before we kick off our main loop, we'll need to perform some initialization work. We'll add this code to our engine/web/web_main.cpp source file. The first thing I'll do is to define some static variables: 

static WebPlatform platform_;
static brite::Engine engine_;
static brite::Allocator allocator_;

These three instances are the main components of Funky Karts and we've placed them in static memory to keep them in scope for the entire game. The WebPlatform contains our platform-specific functionality and interacts with the cross-platform brite::Engine. Lastly, the brite::Allocator manages memory in the engine. We use these instances to write our entry-point function: 

void MainThread()
{
  brite::Platform::instance = &platform_;
  brite::Engine::instance = &engine_;
  brite::Allocator::instance = &allocator_;
  engine_.Initialize(&platform_);
  platform_.Initialize();
  platform_.webDisplay.Initialize();
  brite::Splash* splash = (brite::Splash*)brite::Data::Decode(
      SPLASH, SPLASH_LENGTH);
  engine_.LoadSplash(splash);
  emscripten_set_main_loop(MainLoop, 0 /* RAF */, 0 /* no shenanigans */);
}

The first three lines assign pointers to these instances so that they are available internally. Then, we call Initialize methods on the engine, the platform, and the WebDisplay that we made earlier in this article. We must explicitly initialize the display because the timing of graphics context creation varies between platforms. On the web we can do this as soon as our <canvas> is on the page.

The next couple of lines are responsible for loading the splash screen data into the engine. SPLASH is a string constant containing a serialized snapshot of the entire splash, and SPLASH_LENGTH is simply the length of that string.

At this point, everything is loaded and ready to render! We call emscripten_set_main_loop to kick of the first frame of our periodic function. Passing `0` for the second argument, fps, indicates that we'd like our main loop to use requestAnimationFrame. This is important for that buttery smooth rendering I talked about earlier.

Here is the code that we'll execute for each iteration of our loop: 

static void MainLoop()
{
  if (platform_.IsFocused())
  {
    engine_.SwapInputBuffers();
    engine_.Update();
    engine_.SwapFrames();
  }
  engine_.Draw();
}

We call the engine's Update method each frame if and only if the canvas has focus. This means that if the player clicks to activate another window, for example, our game simulation will pause until the player moves focus back to the canvas. However, we do call the Draw method every frame regardless of focus. The reason for this is because we create our graphics context using the default `false` value of preserveDrawingBuffer. This default is good for performance, but it also means that if we don't draw every frame we can end up with undefined graphical content. Even though the player's focus has moved away from the canvas, it may still be visible on their screen. If the canvas goes fully invisible, the browser will stop calling requestAnimationFrame and no code will execute.

The Swap methods in our MainLoop relate to Funky Karts' multi-threaded design. Ideally, we would Update on a separate thread and the Swap functions manage the sharing of data such as input and display lists between the Update and Draw threads. We only have one thread right now, but we still need to call these methods to move the data into the right place.

All of the pieces are now in place to render Funky Karts' splash screen. I proudly present to you King Grolar on the web. Behold his glory.

Handling Input

May 9, 2017

In this article I will add code to the engine that handles keyboard, mouse, and touch input from the web platform. At the end, we'll be able to play the first level of Funky Karts in a browser!

Keyboard

The keyboard is probably my favorite way to play Funky Karts, so I'm going to implement support for it first. I'll start by registering callbacks on the Window object from my entry-point MainThread function: 

emscripten_set_keydown_callback("#window", NULL, false, OnKeyDown);
emscripten_set_keyup_callback("#window", NULL, false, OnKeyUp);

The OnKeyDown callback function is defined as: 

static EM_BOOL OnKeyDown(int eventType, const EmscriptenKeyboardEvent* keyEvent,
    void* userData)
{
  brite::Key key;
  if (WebPlatform::TranslateKey(keyEvent->keyCode, key))
  {
    engine_.drawInputBuffer->OnKeyDown(key);
    return true;
  }
  return false;
}

The first line of the function allocates a brite::Key on the stack. Then, we pass it by reference to the WebPlatform::TranslateKey along with the keyCode from the EmscriptenKeyboardEvent. Since keyCode is an integer the implementation of TranslateKey is one big switch statement. If TranslateKey returns `true` it has handled the event, and we pass the filled in key to the engine's drawInputBuffer. In the next iteration of our MainLoop, when we call SwapInputBuffers this data will move into place for processing.

That's all there is to handling keyboard input. The implementation of OnKeyUp follows the same logic.

Mouse

Even when I play on the keyboard, I sometimes like to navigate the menu system with my mouse. To receive mouse input from the web platform, we will again start by registering callbacks, this time on our display <canvas>: 

emscripten_set_mousedown_callback("display", NULL, false, OnMouseDown);
emscripten_set_mouseup_callback("display", NULL, false, OnMouseUp);

Let's look at the OnMouseDown callback function: 

static EM_BOOL OnMouseDown(int eventType, 
    const EmscriptenMouseEvent* mouseEvent, void* userData)
{
  if (mouseEvent->button == 0 /* left */)
  {
    brite::Finger f;
    f.id = 0;
    f.x = mouseEvent->targetX;
    f.y = (platform_.display->GetHeight() - mouseEvent->targetY);
    engine_.drawInputBuffer->OnTouchStart(f);
    return true;
  }
  return false;
}

The first thing to notice is that we're only going to handle mouse input from the left mouse button. If the event is from the left mouse button, then we allocate a brite::Finger on the stack. Funky Karts has very simple controls, so we are able to treat mouse input the same as touch input inside the engine.

We use the targetX and targetY fields of the mouseEvent to set the coordinates of our finger. Html uses a coordinate space where the top-left of the canvas has coordinates 0,0. Funky Karts' engine differs from this 2D norm and uses a space where the bottom-left has coordinates 0,0. So, we need to subtract the targetY from our display's height to make the conversion.

Lastly, we pass the populated finger data to the engine's drawInputBuffer for processing. That wraps up mouse input, and as you might expect handling touch will be similar.

Touch

On mobile devices touch provides the primary input to the game. Funky Karts supports multi-touch, although you only really need one finger at a time to play the game. Let's again register callback functions on our canvas: 

emscripten_set_touchstart_callback("display", NULL, false, OnTouchStart);
emscripten_set_touchend_callback("display", NULL, false, OnTouchEnd);
emscripten_set_touchcancel_callback("display", NULL, false, OnTouchEnd);

Notice here how we use the same OnTouchEnd function to handle both the touchend and touchcancel events from the canvas. In either case we want to let the engine know that the finger has been removed from the canvas. Let's examine the logic in OnTouchStart: 

static EM_BOOL OnTouchStart(int eventType, 
    const EmscriptenTouchEvent* touchEvent, void* userData)
{
  for (uint32_t i = 0; i < touchEvent->numTouches; ++i)
  {
    const EmscriptenTouchPoint& p = touchEvent->touches[i];
    if (p.isChanged)
    {
      brite::Finger f;
      f.id = p.identifier;
      f.x = p.targetX;
      f.y = (platform_.display->GetHeight() - p.targetY);
      engine_.drawInputBuffer->OnTouchStart(f);
    }
  }
  return true;
}

Here we iterate over all of the touches in the touchEvent, but we only take action on those that have changed. When a given touch has changed, we convert it to a brite::Finger and once again pass it to the drawInputBuffer for processing next frame.

That is all of the user input that we'll handle for now. I may add support for gamepads in the future. In order to test these changes, I'll need to load a game into the engine.

Out of Memory

The code to load game data into the engine is nearly identical to the splash loading logic we wrote last time, so I'll skip past that. I'm going to remove the splash because the game loads so quickly. After loading the game and playing a few levels, it crashes with the following console output: 

Cannot enlarge memory arrays. Either (1) compile with  -s TOTAL_MEMORY=X  
with X higher than the current value 16777216, (2) compile with  
-s ALLOW_MEMORY_GROWTH=1  which adjusts the size at runtime but prevents 
some optimizations, (3) set Module.TOTAL_MEMORY to a higher value before 
the program runs, or if you want malloc to return NULL (0) instead of this 
abort, compile with  -s ABORTING_MALLOC=0

Funky Karts is pretty well-behaved in terms of memory allocation. It allocates the majority of memory it needs up front when the game loads. However, it does sometimes request a bit more between levels when necessary. With this crash, I learned that by default Emscripten gives our WebAssembly module 16 MB of memory. Surprisingly, it was nearly enough! I'm going to choose resolution option (1) provided by the error message. The game probably only needs around 8 MB more, but let's play it safe and request a total of 32 MB in our Makefile: 

$(BUILD_DIR)/fk: $(BUILD_DIR)/fk.bc
	@printf "%s\n" $@
	@$(CC) -O2 -s WASM=1 -s TOTAL_MEMORY=33554432 --shell-file main.html \
		-o $(BUILD_DIR)/demo.html $<

With a spacious heap to allocate from, the game is now running safely in the browser. I've published the first level of the game as our initial demo. Give it a try!

Audio

May 14, 2017

Today I'll get the sound effects and music in Funky Karts to play on the web. Audio can be one of the tougher engineering challenges in a game engine. Luckily for us, Funky Karts' engine is quite mature at this point and we should only need to ask a few favors of the web platform to make things work.

Sound FX

Let's talk about the sound effects first. The sound effects in Funky Karts are kept short, and are designed to be "fire and forget". To begin the implementation, I'll add a new web_audio compilation unit to the engine: 

funky_karts
    |--- engine
        |--- brite
        |--- web
            |--- web_audio.cpp
            |--- web_audio.h
            |--- web_display.cpp
            |--- web_display.h
            |--- web_main.cpp
            |--- web_main.h
            |--- web_platform.cpp
            |--- web_platform.h

Inside of this new header file I'll declare the following class: 

class WebAudio : public brite::Audio
{
  public:

    WebAudio()
    : fxVolume_(1.0f)
    , midiVolume_(1.0f)
    {
    }

    virtual ~WebAudio()
    {
    }

    virtual void Initialize();
    virtual void Shutdown();
    virtual void Pause();
    virtual void Resume();

    virtual float GetFXVolume() const
    {
      return fxVolume_;
    }

    virtual void SetFXVolume(float volume);
    virtual void LoadAndStartFX(brite::Sound* fx);
    virtual void StartFX(brite::Sound* fx);

    virtual float GetMidiVolume() const
    {
      return midiVolume_;
    }
    
    virtual void SetMidiVolume(float volume);
    virtual void LoadAndStartMidi(brite::Sound* midi);
    virtual void StartMidi(brite::Sound* midi);
  
  private:

    float fxVolume_;
    float midiVolume_;

};

This is the biggest class that we've looked at so far in this series of articles. But don't worry, I'm going to walk you through it and hopefully it'll make sense by the end. This class is a subclass of brite::Audio, and all of the methods we see above are pure virtual methods declared in that base class. So we've got to implement them!

This class has two fields, fxVolume_ and midiVolume_ and defines getters for both values. They are also both initialized to `1.0f`, and they'll be kept in the range from `0.0f` (mute) to `1.0f` (max) by the engine. The audio lifecycle begins with a call from the engine to the Initialize method, so let's look at that next: 

void WebAudio::Initialize()
{
  AudioInitialize();
  Midi_Initialize();
}

The first thing that this method does is to call the AudioInitialize function, which is where things start to get interesting. In current WebAssembly, we cannot directly call the web (html5) APIs. Instead, all calls into the web platform need to pass through JavaScript. Emscripten abstracts some of these APIs for us with its html5.h, but that does not include Web Audio. So I'll need to write a similar layer myself for our audio engine.

After reading the documentation I declare the following block of functions: 

extern "C"
{
  extern void AudioInitialize();
  extern void AudioPause();
  extern void AudioResume();
  extern void AudioSetFXVolume(float volume);
  extern void AudioSetMidiVolume(float volume);
  extern void AudioLoadAndStartFX(uint32_t id, uint8_t* bytes, 
      uint32_t byteLength);
  extern void AudioStartFX(uint32_t id);
  EMSCRIPTEN_KEEPALIVE
  void Midi_FillBuffers(float* leftChannel, float* rightChannel);
}

The first function declared in this block is the AudioInitialize function we are currently investigating. The function definition will be in JavaScript, so I'll add a couple new files to the project: 

funky_karts
    |--- platform
        |--- web
            |--- audio_lib.js
            |--- audio.js
            |--- main.cpp
            |--- main.html
            |--- Makefile

The audio_lib.js file is what Emscripten calls a library file, and it holds the function definitions for the extern functions we declared above. This library file has some limitations, so I've added the audio.js file also which gives me a place to write regular JavaScript code.

I need to modify our Makefile to tell Emscripten about these two files: 

$(BUILD_DIR)/fk: $(BUILD_DIR)/fk.bc
	@printf "%s\n" $@
	@$(CC) -O3 --llvm-lto 1 -s WASM=1 -s TOTAL_MEMORY=33554432 \
		--js-library audio_lib.js --pre-js audio.js --shell-file main.html \
		-o $(BUILD_DIR)/demo.html $<

The --js-library option indicates our library file and the --pre-js option prepends our regular JavaScript file to the generated Emscripten runtime. Notice that I've also bumped our optimization level to -O3 and added --llvm-lto 1 since my last article. While working on audio I did some performance tuning, and I found that I was able to safely make these optimizations.

Let's look now at the library file that defines our audio engine API: 

mergeInto(LibraryManager.library,
{
  AudioInitialize : function()
  {
    window.setTimeout(AudioInitialize, 0);
  },
  AudioPause : function()
  {
    window.setTimeout(AudioPause, 0);
  },
  AudioResume : function()
  {
    window.setTimeout(AudioResume, 0);
  },
  AudioSetFXVolume : function(volume)
  {
    window.setTimeout(AudioSetFXVolume, 0, volume);
  },
  AudioSetMidiVolume : function(volume)
  {
    window.setTimeout(AudioSetMidiVolume, 0, volume);
  },
  AudioLoadAndStartFX : function(id, bytes, byteLength)
  {
    window.setTimeout(AudioLoadAndStartFX, 0, id, bytes, byteLength);
  },
  AudioStartFX : function(id)
  {
    window.setTimeout(AudioStartFX, 0, id);
  }
});

Consider the definition for our AudioInitialize function above. All that it does is schedule another call for a top-level function named AudioInitialize using window.setTimeout. Basically, we are simply re-routing the call to our regular audio.js file. But the use of setTimeout here is important. Some of the audio work we'll be doing is expensive, and we don't want to risk a glitch inside of our requestAnimationFrame callback. So we instead schedule all of the audio calls to happen as separate tasks in the event loop.

Next we define the top-level AudioInitialize function in audio.js: 

function AudioInitialize()
{
  this.fxBuffers_ = new Map();
  try {
    this.audioContext_ = new AudioContext({ latencyHint: "balanced" });
  } catch (_) {
    this.audioContext_ = new AudioContext();
  }
  this.dynamicsCompressor_ = this.audioContext_.createDynamicsCompressor();
  this.dynamicsCompressor_.connect(this.audioContext_.destination);
  this.fxGain_ = this.audioContext_.createGain();
  this.fxGain_.connect(this.dynamicsCompressor_);

  // More code here for the music, but I'll show you that a bit later!
}

Here we'll be working with the Web Audio API. Our initialization function is primarily concerned with using an AudioContext to construct an audio graph. The AudioContext constructor takes an optional latencyHint according to the specification, and this works in Chrome. The values it takes are pretty handwavy, but experimenting on my Mac leads me to choose "balanced". We don't require super low latency, and the default "interactive" latency seems to cause some glitches in the game's main loop and graphics. This is something I want to evaluate on more systems to make sure that I've chosen the optimal setting. Using this constructor throws an exception in Firefox 53, so the try{} catch{} is there to fall back on the no-arg constructor in this case.

The following diagram shows the flow of sound effect data from the game engine through the nodes of the audio graph:

We connect a Dynamics Compressor to the final Audio Destination, or speakers. The documentation on MDN describes this node well:

"The DynamicsCompressorNode interface provides a compression effect, which lowers the volume of the loudest parts of the signal in order to help prevent clipping and distortion that can occur when multiple sounds are played and multiplexed together at once. This is often used in musical production and game audio."

It is easy to wire one up to our audio graph and take advantage of these benefits. I'll keep all of the default settings, as they do a fine job.

Next we connect an FX Gain node to the Dynamics Compressor to provide volume control for our sound effects. The engine calls our top-level AudioSetFXVolume function to modify this gain: 

function AudioSetFXVolume(volume)
{
  this.fxGain_.gain.value = volume;
}

The final nodes of our audio graph are the FX Buffers themselves. In our AudioInitialize function we created a Map named fxBuffers_ that will serve as a cache for AudioBuffer data. The first time the engine plays a given sound effect, it needs to be decoded to an AudioBuffer and placed into this cache. Later, on subsequent playback of the same effect, we will reuse this buffer to play the sound again.

The engine knows when it is playing a sound effect for the first time, and it calls the AudioLoadAndStartFX function in this case: 

function AudioLoadAndStartFX(id, bytes, byteLength)
{
  const data = Module.HEAPU8.slice(bytes, bytes + byteLength).buffer;
  this.audioContext_.decodeAudioData(data, function(buffer)
  {
    this.fxBuffers_.set(id, buffer);
    AudioStartFX(id);
  }.bind(this));
}

Each sound effect has a unique integer id assigned by the engine, and this is the first argument of the function. The bytes argument is a pointer, or address of a contiguous array of bytes in the module's heap memory. We use this address and byteLength to copy a slice of the data out of the heap and into a new ArrayBuffer. We then call decodeAudioData and receive a callback with the decoded AudioBuffer when it is ready. After placing this audio buffer in our cache we call AudioStartFX to begin playback: 

function AudioStartFX(id)
{
  if (this.fxBuffers_.has(id))
  {
    const source = this.audioContext_.createBufferSource();
    source.buffer = this.fxBuffers_.get(id);
    source.connect(this.fxGain_);
    source.start();
  }
}

The AudioStartFX function performs the "fire and forget" playback of sound effects. It checks that the requested sound effect id is present in the buffer cache before creating a new AudioBufferSourceNode to play the audio. It connects this new node to the FX Gain node in our audio graph and calls its start method. This node will be automatically garbage collected after the sound finishes playing, so we don't keep any reference to it.

That wraps up the logic for our sound effects. Next, I'll connect more nodes to the audio graph to process the game's music.

Music

The music in Funky Karts uses the MIDI protocol and is synthesized at runtime. This allows the game to pack a bunch of cool songs into a really small download. However, it also means that we'll need to do some extra processing of audio at a constant frequency during gameplay. Funky Karts' engine is capable of synthesizing music at a frequency, or sampling rate, of either 22050 Hz or 44100 Hz.

Before I can choose a sampling rate, I must investigate how to get an audio callback from the web platform. My research brings me to the ScriptProcessorNode, which is marked as deprecated. Reading about this node, I learn that its callback occurs on the main thread of the application. This is not good, as normally such audio callbacks are run on a dedicated audio thread. I read that there is a new API called Audio Workers that will solve this, but it is not yet implemented by any browser so we will need to use the ScriptProcessorNode for now.

Let's go back to our AudioInitialize function and add more nodes to the audio graph: 

// TODO: Configure the AudioContext sampleRate: https://crbug.com/432248
if (this.audioContext_.sampleRate != 44100)
{
  return;
}
this.midiGain_ = this.audioContext_.createGain();
this.midiGain_.connect(this.dynamicsCompressor_);
this.scriptProcessor_ = this.audioContext_.createScriptProcessor(4096, 2, 2);
this.scriptProcessor_.connect(this.midiGain_);

The first thing to notice above is the conditional test for the sampleRate of the AudioContext. It turns out that there is currently no way to specify the sampling rate of the AudioContext or the ScriptProcessorNode. I find this remarkable, and investigate more to discover that this was recently added to the Web Audio specification but is not yet implemented in browsers. So for now I've chosen to simply bail out if the platform-provided sampleRate is not 44100 Hz. It won't ever be 22050, which is the rate I would prefer now that I know these callbacks will happen on my main thread!

It would be possible for me to perform sample rate conversion and handle any sampling rate the web throws at me. But, this would add even more load to the callback on my main thread. The browser is in a better position to perform this operation in an optimized manner. Hopefully my article will raise awareness that this feature is desired.

Okay, once we pass the sampling rate test we proceed to connect two additional nodes to our audio graph:

The MIDI Gain node works the same as the FX Gain node. Funky Karts provides separate volume controls for the music and the sound effects.

The Script Processor is configured with a bufferSize of 4096 samples and 2 channels (stereo). It pulls frames of audio from the engine with the Midi_FillBuffers function. Let's look at the implementation of the audio callback to understand the flow of sample frames: 

const leftPtr = Module._malloc(4096 * 4);
const rightPtr = Module._malloc(4096 * 4);
const leftChannel = new Float32Array(Module.HEAPF32.buffer, leftPtr, 4096);
const rightChannel = new Float32Array(Module.HEAPF32.buffer, rightPtr, 4096);
this.scriptProcessor_.onaudioprocess = function(audioProcessingEvent)
{
  if (midiGain_.gain.value > 0.01)
  {
    _Midi_FillBuffers(leftPtr, rightPtr);
    _Midi_FillBuffers(leftPtr + (512 * 4 * 1), rightPtr + (512 * 4 * 1));
    _Midi_FillBuffers(leftPtr + (512 * 4 * 2), rightPtr + (512 * 4 * 2));
    _Midi_FillBuffers(leftPtr + (512 * 4 * 3), rightPtr + (512 * 4 * 3));
    _Midi_FillBuffers(leftPtr + (512 * 4 * 4), rightPtr + (512 * 4 * 4));
    _Midi_FillBuffers(leftPtr + (512 * 4 * 5), rightPtr + (512 * 4 * 5));
    _Midi_FillBuffers(leftPtr + (512 * 4 * 6), rightPtr + (512 * 4 * 6));
    _Midi_FillBuffers(leftPtr + (512 * 4 * 7), rightPtr + (512 * 4 * 7));
    audioProcessingEvent.outputBuffer.copyToChannel(leftChannel, 0);
    audioProcessingEvent.outputBuffer.copyToChannel(rightChannel, 1);
  }
};

The first two lines in the above code allocate space in the module's heap to share audio samples between JavaScript and C++. The leftPtr and rightPtr variables contain the starting byte offsets for these two arrays. The variables leftChannel and rightChannel are set to typed array views for these two blocks of memory. This memory will be re-used by every invocation of the audio callback.

Next, we assign an anonymous function to the onaudioprocess callback. This function is called with an AudioProcessingEvent each time that a new buffer of samples is requested. If we don't do anything in this callback, then silence will be output by the node. We check that the gain is significant before we do any processing as an optimization for the case where the user has turned off the music volume.

The next thing the callback does is to call _Midi_FillBuffers eight times. The engine is designed to fill 512 samples per frame, but we've configured the Script Processor to process a buffer of 4096 samples. I chose this buffer size of 4096 based on experimentation; my tests in Chrome 58 on a Galaxy S4 (Android 4.4.2) presented much audio glitching at smaller buffer sizes. To fill up this buffer we do a little pointer math and ask the engine to fill samples multiple times.

The last thing to do in the audio callback is to copy the samples to the outputBuffer of the AudioProcessingEvent. We call the copyToChannel method for both the left and right channels, passing in our constant typed array views.

One final consideration of our audio engine is to define the AudioPause and AudioResume functions. We want to be courteous and pause our music if the user switches to another tab or focuses on another window. These functions are trivial to implement thanks to the suspend and resume methods available on the AudioContext: 

function AudioPause()
{
  this.audioContext_.suspend();
}
function AudioResume()
{
  this.audioContext_.resume();
}

That concludes the audio implementation for Funky Karts on the web! Hopefully, you'll be able to enjoy the music as well as the sound effects. If not, please star the issue and put on your favorite album in the meantime. Give it a listen!

Fullscreen

May 20, 2017

Funky Karts is best enjoyed while fully immersed. In this article I will implement the game's fullscreen feature for the web platform.

Fullscreen is a bit tricky on the web where it is considered a 'powerful' feature due to concerns about security and user experience. Therefore, applications may only request to enter fullscreen from within an event handler for a user gesture such as a key press or touch. So Funky Karts won't be able to launch as fullscreen except under certain circumstances that I'll discuss at the end of this article.

It is also considered bad practice to automatically request fullscreen on the user's first input to the application. Instead, the user should make this decision through an action in the user interface. Funky Karts has a settings menu where we'll put a fullscreen toggle. With this design in mind, let's implement the toggle.

Fullscreen Toggle

The code for our fullscreen toggle will go in the WebPlatform class: 

class WebPlatform : public brite::Platform
{
  public:

    virtual bool GetIsFullscreen();
    virtual bool GetFullscreenToggleable() const;
    virtual void ToggleFullscreen();
    void ProcessPendingFullscreen();

    // There are other methods in this class
};

The virtual methods above are inherited from the brite::Platform base class and provide a bridge from the engine to the web platform. The last method, ProcessPendingFullscreen, is specific to this implementation.

Emscripten has a fullscreen API in html5.h that offers similair functions to the underlying web API. For my initial implementation I used Emscripten's API and things worked for the most part. However, I encountered a couple of corner cases and limitations that led me to rewrite the code using only the web API. I'll provide details on those issues and how I resolved them as we examine each function. Let's look first at the definition of GetIsFullscreen: 

bool WebPlatform::GetIsFullscreen()
{
  return EM_ASM_INT_V({
    const d = window.document;
    return !!(d.fullscreenElement
        || d.mozFullScreenElement
        || d.webkitFullscreenElement
        || d.msFullscreenElement);
  });
}

This method uses the EM_ASM_INT_V macro to wrap some JavaScript code that returns a boolean value. It tests whether or not the fullscreenElement is set to a defined value that is not null. The checks for various browser prefixes are an unfortunate but necessary part of this logic.

Next we define the GetFullscreenToggleable method: 

bool WebPlatform::GetFullscreenToggleable() const
{
  return EM_ASM_INT_V({
    const d = window.document;
    const fullscreenEnabled = d.fullscreenEnabled
        || d.mozFullScreenEnabled
        || d.webkitFullscreenEnabled
        || d.msFullscreenEnabled;
    const isFullscreen = !!(d.fullscreenElement
        || d.mozFullScreenElement
        || d.webkitFullscreenElement
        || d.msFullscreenElement);
    const isFullSize = (window.innerWidth === screen.width)
        && (window.innerHeight === screen.height);
    return fullscreenEnabled && (isFullscreen || !isFullSize);
  });
}

GetFullscreenToggleable is called by the engine to show or hide the fullscreen toggle in the user interface. We want to disable the toggle (by returning `false`) whenever it is not possible to programmatically change the fullscreen state. The logic here is more complex as it works around one of the corner cases that I mentioned earlier.

The issues arise when we take the browser into fullscreen mode from outside of the fullscreen API. For example, by pressing the F11 key on Windows. In this case the fullscreenElement is not set and fullscreenEnabled will return `true` even though we are unable to exit fullscreen with an API call. Extra logic is needed to consider this case. If we detect that the browser window is using the entire width and height of the screen but the fullscreenElement is not set, then we return `false` to disable our toggle control. The user must press the F11 key again, or the ESC key, to leave this mode.

Let's look now at the implementation of ToggleFullscreen: 

void WebPlatform::ToggleFullscreen()
{
  if (GetIsFullscreen())
  {
    EM_ASM({
      const d = window.document;
      const exitFullscreen = d.exitFullscreen
          || d.mozCancelFullScreen
          || d.webkitExitFullscreen
          || d.msExitFullscreen;
      exitFullscreen.call(d);
    });
  }
  else
  {
    isPendingFullscreen_ = true;
  }
}

This function first checks if the document is fullscreen and if so calls the exitFullscreen function. In the other case we want to toggle into fullscreen mode, but instead set a variable isPendingFullscreen_ to `true`. We are not able to call requestFullscreen here according to the specified rules. Calls to ToggleFullscreen are made during engine updates and not from any event handlers. By setting this flag we defer the actual request until the next appropriate event callback. This is where ProcessPendingFullscreen comes in: 

void WebPlatform::ProcessPendingFullscreen()
{
  if (isPendingFullscreen_)
  {
    isPendingFullscreen_ = false;
    EM_ASM({
      const d = window.document.documentElement;
      const requestFullscreen = d.requestFullscreen 
          || d.mozRequestFullScreen 
          || d.webkitRequestFullScreen 
          || d.msRequestFullscreen;
      requestFullscreen.call(d);
    });
  }
}

I've added calls to ProcessPendingFullscreen from the engine's OnKeyUp, OnMouseUp and OnTouchEnd callbacks that I introduced in the Handling Input article. This works out pretty nicely, because the game's toggle control initiates the action from OnKeyDown, OnMouseDown and OnTouchStart handlers. So the call to ToggleFullscreen will happen in the next engine update and by the time of the matching OnKeyUp, OnMouseUp or OnTouchEnd callback the flag should be set and ready to be processed into a fullscreen request.

Another important detail in this function is that we call requestFullscreen on the document.documentElement. This is another change that I made in the second iteration of the code. I discovered on Chrome 58 for Android that a call to requestFullscreen on my canvas element did not have the desired effect. I learned from reading this article that I should instead make the call on the document.documentElement. However, I was unable to do this with the current Emscripten API so I opened an issue to let others know.

That concludes the implementation of the game's fullscreen toggle. There were a couple of tricky cases to consider but in the end the code is quite terse. Before I call it a day, let's see what we can do about launching the game directly into fullscreen mode.

Launching Fullscreen

As I mentioned at the beginning of this article, the rules of the web prevent us from directly launching as fullscreen in the common case. However, there are some ways that we can launch fullscreen on mobile devices.

There is great information in this article about 'installing' web apps to the home screen on both iOS and Android, so I won't repeat all of that here. After reading that article I added the following tags to the <head> element of the game's main.html template: 

<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="mobile-web-app-capable" content="yes">

I tested this on Chrome 58 for Android, and it is a nicer experience. The status bar is still shown, which I was not expecting. But the fullscreen toggle continues to function in this case and can be used to hide the status bar. I'm unable to test on iPhone currently, as it does not yet support WebAssembly.

The next thing that I did was to add a web app manifest. Following their instructions, I added a link to the <head> element: 

<link rel="manifest" href="/manifest.json">

And I created a manifest.json file with these minimal contents: 

{
  "short_name": "Funky Karts",
  "name": "Funky Karts",
  "start_url": "/demo.html",
  "display": "fullscreen",
  "orientation": "landscape"
}

Testing this on Chrome 58 for Android, it works really well! The game now launches from the home screen directly into true fullscreen mode, and the fullscreen toggle is disabled keeping the user immersed for the entire experience. I plan to add icons and more to this manifest as things progress.

That wraps up the fullscreen feature for Funky Karts on the web. We are getting pretty close to a polished demo, so this may be my last article. I am pretty excited by how well it works, and the future looks bright with WebAssembly. Thank you to everyone involved.

Enjoy the demo!


Copyright (c) 2017 Ross Smith. All rights reserved.