Fugue Devlog 19: Character Generation (Part 2)

12.02.2022

After much chipping away the character generation system ("chargen") is mostly complete!

As mentioned in the previous post the character generation system is driven by MakeHuman for the model generation, along with a custom texturing system for generating skin-level details.

This more or less reduces character creation to three distinct production tasks:

  1. Creating clothes (and hair)
  2. Creating new texture layers (e.g. tattoos, eyebrows, etc)
  3. Creating animations

The process works like this:

  1. Generate the skin-level details as a single texture.
  2. Generate the model using MakeHuman
  3. Dress the model using MakeHuman
  4. Retarget and remap animations to the model
  5. Export as a GLTF
  6. Generate preview renders

For documentation purposes I'll go through each part in detail below...but first, some screenshots (see the end for a video demo):

Top-level parameters, randomization, in-game previews, and texture view

Left: color parameters; right: a preview of the resulting GLTF model

Body parameters derived from the top-level parameters, used for MakeHuman

Left: some texture parameters

Generating the texture

This texture is composed of different layers to represent different features: there's a base skintone layer, then additional layers of shading, e.g. some red in the cheeks and other high blood-flow areas, then other features like eyes, eyebrows, facial hair, close-cut hair, tattoos, and accessories/clothing items that don't need to have separate geometry (e.g. earrings, socks).

These layers are added using different blend modes—e.g. some layers are simply overlaid and others are multiplied—so they mix in better with the base layers.

Texture layering

Some layers are used as masks (rather than being used directly) so that they can be filled in with different colors. For example eyes (irises), hair, and socks. In these cases the texture uses a base color (usually something close to black or a very dark grey) that is used as the "anchor" or "reference" color, and then the image is colorized to maintain the relative changes in color throughout. For example, a mask texture might have the base color #181818 but also include blacks and greys that are slightly darker/lighter, and want to change it to a red, say #ff0000. The image will be recolored such that pixels that were #181818 become #ff0000 and other pixels become a shade of red as well, maintaining the same relative lightness/darkness to the base color.

Generating the base skintone texture was tricky at first. Something that is probably obvious to digital/visual artists but that I had to learn is that solid colors don't really look good or interesting (unless you're going all-in on solid colors, like those blocky start-up illustration styles), so the original characters looked kind of bland with their perfectly uniform skintones. Creating a color field with interesting texture and variation is easy enough to do by hand in a digital painting application but took a bit of work to do procedurally. I considered trying to use Perlin noise and its ilk but I don't have any intuition how to control noise well...so it would have taken me forever to get results I'd be happy with.

Instead the algorithm just roughly emulates how a person would produce such a texture...just using a paintbrush texture, painting a base layer of color, and then varying the color a bit and painting low-opacity random strokes on top:

Color painting output

Color painting output

I found that using LAB color space provided the best color variations that "vibrate" or "resonate" with the base color (I love the subtle pinks and greens and blues in the examples above).

This approach not only produces nice textures for the base skintone but also for hair.

Creating the other texture layers is straightforward—just paint onto a transparent image, using this reference image to know where things will show up:

UV reference image for texturing

The last bit worth mentioning here is performance/speed. There are a few very slow points in the character generation process and texture generation is one of them...sometimes. The image manipulation is happening in Rust, which is very fast if compiled in release mode. Otherwise it's on the order of 10x slower. Because I'm still developing this content management tool I'm not often building in release mode since you lose access to the browser dev tools when doing so.

Generating the model

This step is the most straightforward since the bulk of the work is handled by MakeHuman. MakeHuman provides a number of parameters to control the body model, e.g. how much muscle there is in the upper arms, the shape of the face, etc. What I do is take a few higher-level parameters—gender, fat, muscle, height, age, and a few parameters that MakeHuman uses for more racialized features—and translate them into the more fine-grained parameters. These finer parameters are all random but parameterized by the higher-level ones, with the exception of some parameters such as facial parameters, which are totally random.

For example there's a parameter called arms_{L,R}_lowerarm_muscle_decr_incr. The sampling distribution for this might find most of its density in (-1, 0) if the muscle slider is turned all the way down, and would be in (0, 1) if turned all the way up.

This step also applies the MakeHuman "game engine" rig to the model.

Dressing the model

The dressing of the model (applying clothes and hair) is also very straightforward since it's again handled by MakeHuman. The part that's more complicated is the production of the actual clothes and hair assets to be used.

This process is tricky in part because I'm just not a very experienced modeler, but the lowish-resolution look is more forgiving of that. The main reason it's difficult is that MakeHuman/MakeClothes has a couple requirements that can be very hard to satisfy. The mesh must consist either entirely of quads or triangles—no mix allowed. In practice this would not be too bad—you can always convert a mesh entirely to triangles (using Blender's Triangulate Faces), which as far as I know is what a game engine typically does anyways—were it not for a second MakeHuman/MakeClothes requirement that limits the maximum edges allowed for each vertex. It's something like 8 edges max for any vertex (called "poles"). Going through and figuring out how to reconstruct your geometry to get rid of these poles is hard...some are easy, but others become weird geometry puzzles.

A pole

I haven't been able to come up with a good foolproof method to avoid this problem. I try to stick to quads when modeling clothes/hair but sometimes you just forget and end up introducing a triangle or n-gon somewhere. Or I want to reduce the poly count so I use Blender's Decimate modifier, which works amazing for reducing polys while preserving the general shape, but then introduces triangles and poles.

I did come up with one procedure for removing poles that has so far worked maybe 2/3 of the time, and is fortunately easy to script:

import bpy

bpy.ops.mesh.select_more()
bpy.ops.mesh.hide(unselected=True)
bpy.ops.mesh.select_less()
bpy.ops.mesh.delete(type='VERT')
bpy.ops.mesh.select_all(action='SELECT')
bpy.ops.mesh.edge_face_add()
bpy.ops.mesh.quads_convert_to_tris(quad_method='BEAUTY', ngon_method='BEAUTY')
bpy.ops.mesh.reveal()

What this does is it deletes the problematic vertex/pole, builds an n-gon face out of its surrounding vertices, and then converts that into triangles. Usually this creates edges that get rid of the pole completely, but it can also just move the problem to a different vertex.

I anticipate spending a lot of time making clothes/hair because that's probably where the bulk of character variation will come from, so I hope I can figure out a more reliable solution soon.

The last thing to note here is that the grammar system I mentioned in the previous post is also implemented into the tool. Almost all the parameters depicted above can be randomized, including the character's outfit, which can be randomized using an outfit grammar.

One funny thing is that I spent a bunch of time modeling hair and it's given me a greater appreciation of the variety of hair styles and how much they change someone's look. I'm sure it's obvious to many people but not something I spent much time thinking about. Now I catch myself really studying the silhouettes of people's hair lol.

Mapping animations onto the model

There isn't a whole lot to say here since I haven't figured out a animation/mocap process, so I'm just relying on Mixamo for now. The process of retargeting/remapping these animations to the character model is fortunately very easy thanks to a Blender addon called "Auto-Rig Pro". I did have to create a custom remap config to go from Mixamo to MakeHuman's game engine rig though, and it was a bit tricky to make sure that all animations were correctly exported (due to me not understanding how Blender's actions and NLA editor work).

This step is also a hit for performance. The retargeting/remapping process just takes a lot of time: a few minutes for 3 animations, but this will vary on the keyframe length of each animation. Not sure that I can really do anything about that though.

Exporting the model to GLTF

This step is also pretty simple, just exporting the GLTF from Blender. The GLTF format is what Godot uses for importing the model.

The only thing I need to look into here is performance. It takes a ridiculous amount of time to export and absolutely shreds my CPU. I can barely do anything else while it's exporting. I'm not sure what's going on as it's just a fairly low poly model that's being exported.

Generating preview renders

The final step is generating preview renders to see what the model will actually look like. A couple of these renders are done in Blender but they aren't as useful as the in-game renders, which let me see how the character looks with the game's lighting conditions and screen shaders. This was also surprisingly easy to setup as Godot provides a convenient way to run the engine headlessly and take screenshots.

In-game preview

I was also able to use the same approach for previewing things in-game directly from Blender:

Quickly view an in-game preview from Blender

Fears and Steps for the Future

There's still more work to be done. I haven't fully finished integrating characters into Godot, so there may be other things that need to be fixed there. And I don't have a great intuition about what poly counts I should be going for, so a great anxiety is that my poly counts will be too high and I'll have to go back and redo the base character mesh and all the clothes/hair meshes. I don't really anticipate this being a problem because poly count is (what I think is) low across the board and from what I've read poly counts are less likely to be bottlenecks nowadays. But you never know with these things...and I could probably safely lower the poly count of the base mesh, so I should probably do that earlier rather than later...

Update: Lowering the mesh poly count

After writing this I investigated lowering the poly count for the mesh. The original human mesh is around 3k tris—for reference, the Final Fantasy VIII Kiros model I was using for testing is around 750 tris, which was for the original PlayStation. Individual items of clothing so far ranging from 250-800 tris, though I could probably reduce those a fair amount more. Some of these clothing items delete the parts of the human mesh they cover, so they add less to the overall poly count.

Lowering the poly count was a difficult process: after applying Blender's Decimate modifier there were several poles (the same issue mentioned above) that took a very, very long time to eliminate, but also because the poly reduction process and subsequent pole-editing messed up parts of the UV map which took awhile to fix (UV map editing in Blender is awful). I had to try and match the original UV map as much as possible or I'd need to re-create all the texture system layers, which I really did not want to do.

But I managed to produce a few lower-poly human meshes. The biggest reduction was down to about 750 tris. This lost too much detail, especially around the face. I tried a less extreme version that was about 1300-1400 tris, which was better, but having fewer vertices reduces the effectiveness of MakeHuman's parameters though and made some parts too angular. Then I tried another version targeting the Decimate to the hands (which is the most detailed region after the face) and feet (which usually aren't going to be visible and don't need much detail), getting things down to about 2k tris while entirely preserving the face and preserving the original UV map as much as possible. A 33% reduction in polys isn't bad!

In the higher resolution previews the hands are noticeably blockier:

Lower poly in high res

But this is less apparent in-game:

In-game preview

For all I know this is a premature optimization and ~3k poly character models are fine. In theory if I ever need to switch the base human model I can just do so by batch re-generating all of the characters, but as the game develops it may become more complicated to do that. Hopefully I'll have a better sense of what poly count works before that.

Update: Improving GLTF export times

The GLTF export times were around 10 minutes with 3 animations, which was way too slow. I was not exporting much—only the renderable/visible objects, which was just the lowish-poly human mesh and clothes. Everything else was hidden. But for some reason (which may be related to this issue) the GLTF exporter still processes hidden meshes, even if they aren't to be included in the final GLTF. So the base MakeHuman mesh, which is pretty detailed, was taking up the bulk of the processing time. After deleting that mesh and some other objects (rather than just hiding them) the export time went down to about a minute. The character generation system does this exporting through a headless Blender script, but when exporting from the Blender UI it takes only ~15-16 seconds. No idea what could be the cause of that discrepancy.

Now the main bottleneck is the retargeting/remapping of animations, but again, I don't think I can do much to speed that up.


Video demo of the tool


Fugue Devlog 18: Character Generation

10.21.2022

The character generation system is starting to come together. There are still a few things to figure out, but I think I have the bulk of it figured out.

The character generation system is roughly broken down into four sub-problems:

  1. Mesh generation: how will the actual geometry of a character be generated?
  2. Texturing: how will the skin-level features (skin tone, eyes, eyebrows, nose, mouth, close-cut hair, facial hair, tattoos, scars, etc) of a character be generated?
  3. Clothing/hair: how will additional geometry, like clothing and hair, be generated and mapped to the human mesh?
  4. Rigging: how will the character's skeleton be configured?

(A fifth sub-problem could be "animation" but I'll handle that separately.)

A good system will encompass these sub-problems and also make it easy to:

  • generate characters according to specific constraints through a single UI (eventually as a tab in the CMS)
  • add or modify clothing, hair, etc
  • generate a large amount of variation across characters

In summary, the whole system should be represented by a single configuration screen, and with a single press of a button I can produce a fully rigged and clothes character model. I'm not quite at that point yet but making good progress.

The system so far is all built around the open source MakeHuman, which makes the whole process much simpler. It provides a way to generate parameterized human meshes that supports the easy adding of components like clothing, and it has an addon for working directly within Blender. MakeHuman works by generating a "base mesh" which then can be used with other meshes ("proxies") that map to the vertices of the base mesh. When the base mesh is transformed—either through animation or through a variety of different body shape/proportion parameters—these proxies are transformed too. Clothes are proxies, but so are "topologies" which replace the base mesh as the human mesh. This allows me to use a custom lower-poly mesh instead of the higher-resolution default mesh.

So MakeHuman takes care of mesh generation, and it provides a way to attach clothing and hair. The clothing and hair still need to be modeled individually, but this is less daunting a task as I figure I'll only need to create a relatively small amount of clothing forms that each have several different textures. It may be easier to automate the generation of some of these textures, e.g. color variations. In the end this is not too different than your run-of-the-mill modeling and texturing; there are just a couple of extra steps to ensure that the clothes map on to the human mesh correctly.

MakeHuman also generates a rig for the mesh, so that sub-problem may be taken care of too. But because I haven't figured out the animation process, I don't know exactly if/how I'll integrate the auto-generated mesh. For my test characters I've just been using Mixamo's auto-rigger...so for now this one needs more work.

So that leaves texturing, or what I called "skin-level features". These are features that don't really have any volume to them, such as tattoos, scars, and undergarments and socks. This isn't too difficult in theory: you just need to generate an image texture. The approach is to work in layers, and assemble a final texture by sampling/generating different layers for different features. So the bottom-most layer is the skintone, and on top of that you'd paste on layers of underwear, socks, eyes, nose, mouth, etc.

The base skin UV map

The face is of course very important here, and it's the hardest to get right. I don't yet have a good way of generating facial features. While the other parts (socks, undergarments, etc) can be generated by hand because they don't require a ton of variation (e.g. I could probably get away with like 20 different pairs of socks), faces should be unique per character (PCs and NPCs alike). I would rather not have to create all of these by hand.

I've had some success using Stable Diffusion to generate faces to work from but it's not consistent enough to automate (faces may not be head-on and so require some manual adjusting, for example). I think a parameterized generator might make the most sense here, where, for example, facial features are defined by bezier curves with constrained parameter ranges, and each face is just a sample of that parameter space. There could then be a pool of textures (for skin tones, lip color, eye color, etc) that are also sampled from to fill in the details.

For testing I just created the skin-level texture by hand, just so I could place a character into the game and see if it works visually:

And here is a comparison with the screen effects, without the dithering, and without the resolution downsampling:

The face definitely needs work but I feel ok—but not thrilled—about everything else. It does feel somewhere between the graphics from the original FF8 and its remaster (except for the face), which is sort of what I was going for. I think I need to sit with it for awhile, see how it feels as the rest of the game's environments develop, and try different character models, clothing, etc. It's at least a starting point—I feel a bit reassured that I have some method for generating decent-looking characters, one that could be modified if needed.

FF8 remaster comparison, from Rock Paper Shotgun

On this point: I'm kind of hoping that all the models and characters and so on just kind of work together visually, visually but not expecting that to be the case. I'm trying to design this character generation system so that I can make adjustments to e.g. textures, models and have those adjustments propagate through all existing character models. That gives me more latitude to experiment with the game's visual aesthetic and makes me feel less like I'm committing to a particular one so early on.

This brings me to the actual generation system—everything up to this point is more about producing the assets that are then mix-and-matched to generate the actual characters. I don't want to allow for totally random character generation because there are combinations that are atypical or implausible. With clothes, for example, people generally don't wear a dress and pants at the same time, so I want to prevent this particular outfit from being generated (apologies if you do dress this way regularly). A context-free grammar (CFG) makes the most sense to me because it allows you to define set configurations that have variation, thus avoiding these problems of complete randomness.

With a CFG you will essentially define different "outfits", where each component of the outfit can be drawn from a pre-defined list of options. Say for example I need to generate a lot of road workers. A simple CFG might look like:

RoadWorker:
    - HardHat
    - TShirt
    - HighVisVest
    - WorkPants
    - WorkBoots
HighVisVest:
    - YellowHighVisVest
    - OrangeHighVisVest
HardHat:
    - YellowHardHat
    - WhiteHardHat
    - HardHatWithLight
HardHatWithLight:
    - WhiteHardHatWithLight
    - YellowHardHatWithLight
TShirt:
    - RedShirt
    - BlueShirt
    - BlackShirt
WorkPants:
    - CarpenterPants
    - Jeans
WorkBoots:
    - BrownWorkBoots
    - BlackWorkBoots

A CFG is recursive in the sense that, if I want to create a RoadWorker, the program will see that HardHat itself can be expanded into different options. And then the program will see that one of those options, HardHatWithLight, can also be expanded into more options. So it will do this until it's got all options, and sample from those.

Another feature to add is the ability to draw from the CFG with some fixed options. Like say I'm generating an audience for a group where everyone has to wear an orange beret; I can fix that option the program would only generate characters in an outfit which is allowed to include an orange beret.

Finally, every time a character is generated with the CFG, the resulting model will be saved with the combination of terms used to generate that character (in the case of a RoadWorker that might be YellowHardHat,BlueShirt,OrangeHighVisVest,...). This makes it easy to "regenerate" or update existing characters if one of the underlying models change. That way I can feel free to tweak the textures and models of clothing and other components without needing to manually update every character that I've generated so far.

In the near term this will probably all work via a Python script, but it would be amazing to be able to see changes in a character real-time. So a character might be generated through the CFG, but can be hand-adjusted afterwards, e.g. by swapping out that BlueShirt for a BlackShirt, or, if I go the bezier curve route for face features, by adjusting the eye shape for this character, etc. This might be feasible by calling MakeHuman and Blender via their Python interfaces, rendering the model, and then displaying the rendered image, but it sounds really complicated and janky. I'll have to think on it some more.


Fugue Devlog 17: Content Management System

10.14.2022

It's been awhile but I have been working on Fugue here and there. The biggest accomplishment is a more cohesive content management system, making it easier to keep most game-related content in one place and in a consistent structure. That way I don't have to worry about misplacing things and I can build more automated processing tools without the whole development process becoming a total mess (I hope).

World Wiki

One of the CMS tools is a simplified wiki for the game world and mechanics. I'm working on copying over all my random scattered notes but this is so much easier to think with already.

World Wiki entry

Sequence Editor

I've also ported over the sequence editor. I'm not 100% certain that I'll stick with Godot and so I want to be able to write all the action/dialogue sequences in a portable format. This saves sequences as a .seq file (which is really just a JSON file). Godot supports import plugins which are able to import custom filetypes into Godot resources. I'm doing the same with items (not pictured here), which as saved as .item (again, just a JSON file).

The actual integration with Godot is very hands-off. I just symlink the relevant CMS folders into my Godot project, and then the importer plugins handle the rest. That way I don't have to mess around with copying over files, worrying if they're the latest/correct version, etc. Whatever the CMS shows is what Godot sees.

Having a totally custom sequence editor gives a lot more control than the Godot addon I created. The downside is that validating actors/agents, scene objects, etc is difficult because that data is in Godot scene files (.tscn). These are plaintext but still requires parsing, so I wrote a .tscn parser which will pull out all the relevant entities that might be available to a sequence. I decided to use these extracted entities as suggestions for a sequence, not for validation. I thought that perhaps the workflow would be: write sequences in the CMS and then setup the Godot scene to match it, in which case validation is probably best handled from within Godot.

The rest is less interesting. There's a visual reference manager, which is really just a fancy image viewer:

Reference Manager

And I've ported over the texture manager/editor from before, with some snazzier features, including texture/image synthesis:

Texture Manager

Texture Editor

Outside of this CMS tool I've been experimenting with character modeling/generation, which is one of the last major (and most intimidating) content production questions (along with character animation—I'm waiting for the next release of FreeMoCap, and I've kind of accepted that object modeling will just happen by hand). I feel that I'm getting close to a good process. Hopefully I'll figure it out soon.


Fugue Devlog 16: Motion Capture Tests

08.12.2022

An example rendering from the CMU MoCap database.

After a few busy weeks I've found a bit of time to continue work on Fugue. One of the last major questions content production, after writing, textures, and music, is character animation (this still leaves object and character modeling as the other two problem areas). While I believe I can get away with lower-poly models and crunchier photo-textures, I don't think that's the case with low-quality animation—it's too jarring. So I want to figure out a way to produce fairly good, realistic motion on the cheap.

There are a number of deep learning-based projects available for motion capture without a sophisticated tracking setup. Some are even monocular, requiring only one camera. There are some commercial offerings (such as deepmotion.com) but I want to see how far I can get with open source options. I'll be able to modify those as I need and they'll be easier to integrated into a more automated process than commercial options.

The open source projects are usually research projects, so they aren't polished/are somewhat janky and probably don't generalize very well. And their licenses often restrict usage to non-commercial purposes. For example, MocapNET, EasyMocap, and FrankMocap are all non-commercial uses only. I did find MotioNet which does allow commercial usage (under its BSD-2 license) and requires only one camera, so that was promising.

One alternative to the deep learning approach is to just use existing motion capture data and hope that covers all the animations I'd need. A great resource is the CMU Graphics Lab Motion Capture Database, which has generously been converted to .bvh by Bruce Hahne for easy usage in Blender. The collection encompasses 2,500 motions and is "free for all uses". The range of motions is expansive enough (including things like pantomiming a dragon) that it's possible it will have everything I need.

Still, I wanted to try out the deep learning approaches, in part because I was curious.

One note here is that these models typically output motions as .bvh files. These contain motion instructions addressed to a particular skeleton (where, for example, the left leg bone might be named LeftLeg). I used Mixamo's auto-rigger to rig my character and the resulting skeleton has a different naming system. Fortunately there is a Blender addon, "BVH Retargeter", that remaps a .bvh to a differently-named skeleton. It doesn't include a mapping for Mixamo by default, but I set one up myself (available here, it goes into the known_rigs directory).

On this note, there is also this Deep-motion-editing project which has a much more sophisticated retargeter:

Deep-motion-editing retargeter

I don't know yet if I'll have a need for this, but good to know it's there!

On to the tests:

I'm using a Kiros Seagill model (from FF8) for these tests.

Even though the MocapNET license is not what I need, I decided to try it anyways:

MocapNET test

It looks ok, a little janky and all over the place though. And the hands aren't animated.

MotioNet

MotioNet looked promising but unfortunately did not have very good output. The output pose is upside-down for some reason (this is a known issue), which seems like an easy enough fix, but the joint movement is stiff and incorrect.

CMU MoCap

The CMU motion looks great of course, as it's actually captured properly. Again, the only concern here is that it doesn't have a wide enough range of motions.

The last software I tried is FreeMoCap, which is still in very early stages of development, but there's enough to try it out. It was quite a bit more complicated to set up as it works best with multiple cameras (they can still be fairly cheap, e.g. $20-30, webcams), and requires a charuco board for calibration, which I printed off at Kinko's. That cost me about $30 to get it on poster board, but you can probably make something cheaper with an old cardboard box and printing on large-sized computer paper. In total I spent ~$100 on equipment.

The most important thing is to get webcams that work for the size of your recording room, so get your full body in frame for all of them (which may require wide-angle cameras). Then you need to make sure that your charuco board is large enough that its patterns are clear on the webcams—the further you position the webcams, the larger the charuco board you'll need and the lower the resolution you record at, the larger the charuco board you'll need. Note that there's also a resolution/frame-rate trade-off: when running 3 cameras at 1080p I get about 14-15fps, but I needed to run at that size for my charuco board to render clearly. And as another note, the FPS can be bottlenecked if you use a USB hub to run your cameras through (some of the cameras may not even work in that case); I ended up plugging each camera into its own port for best performance.

Getting the program to work was tricky, which wasn't a surprise given the project is in an alpha state. I had to make a few changes to get it to run properly (mainly switching from multithreading to multiprocessing, since the threads were blocked on my system, and manually setting the FPS for my cameras, which for some reason would limit to 5FPS otherwise).

Below is an example of the program's output. My recording environment is less than ideal (my camera setup was super janky, attached to books or shelves in a haphazard way), but the output looks decent. It's jittery, and you'll notice the pose skeleton and camera footage are swapped in the first and last videos. I'm not sure if that's just a bug with this visualization or if it's happening deeper in the program, in which case it may be why there's that jitteriness and the skeleton angle is off.

FreeMoCap output

The program can also output Blender files:

FreeMoCap Blender output

Here the issues are more apparent: the hands are especially all over the place. But even the limbs are too finicky. The demo video (above) has good limb motion, so maybe my setup is lacking (though the hands are also jittery).

FreeMoCap is a really promising project, but unfortunately it's at too early of a stage to be consistent and reliable. For now I'll probably develop the game using the CMU motion data, and then later, when I'm ready and FreeMoCap is likely much more mature I can go through and replace or refine with custom motions. Though at the rate development is going, there's a good chance that FreeMoCap will be much further along by the time I'm ready to start really working on character animations!


Fugue Devlog 15: Divine Demons, In-Game Game, and Seamless Textures

06.21.2022

Modeling demonic divinities

Growing up I was very fond of Buddhist art and I would make an effort to visit various Buddhist temples whenever my family would go to China to see my grandparents. I found the paintings and sculptures of terrifying figures to be strangely calming, and only later did I learn that these fearsome beings are in fact benevolent, wrathful "protectors".

Demonic Divine, Rob Linrothe, Jeff Watt

While visiting Kira's parents last week I had the opportunity to read through the catalogue for one of the Rubin Museum's opening exhibitions, "Demonic Divine" which was exactly on this topic. I've wanted to find some way to draw from this tradition of "divine demons" for Fugue and reading through the catalogue helped me figure out how I could do that.

Without going into too much detail, one fundamental Fugue world-element is the phenomenon of "hauntings", which are mental blocks and fears that characters experience. The character progression system is based around confronting and resolving these hauntings (perhaps in addition to a simpler experience-based system, not really sure yet). The way I've pictured this is that progression happens along a path (the character's "道"/"way" in a sense) and these hauntings are literal roadblocks, represented by these demons, along this path. So a key question is how exactly does this process of resolving hauntings1 work?

One other feature I want to include is an in-game game (tentatively called "Gem"), sort of like Triple Triad in Final Fantasy VIII. Including something like that lets you do a lot of interesting things: explore its cultural significance, show how different places and people interpret the game (perhaps with different house rules and variations), explore all the infrastructure that comes up around a relatively simple game once it becomes a major phenomenon (rules committees, judges, player associations, its role in national conflict, etc), and in general be a means of showing different value systems, thought processes, philosophies, etc, like Azad in Iain M. Banks' The Player of Games. I'm still working out Gem's design, but so far it's like some combination of chess, checkers, and Chinese checkers, with the variability of a card game—i.e. many different pieces with different abilities, and players can build a "deck" of pieces to play with like they might build a deck in Magic: The Gathering.

My current thinking is that to get through a demonic roadblock you challenge the demon to a game of Gem, in a "chess with death" style. The demon itself is represented as a piece on the game board that you need to remove, sort of like the king in chess. However the goal of the game is not necessarily to "defeat" the demon. Each demon represents a mental block/fear that has a complementary virtue; the point is not to excise the demon from your mind but to recognize its value as an ally. In the context of Gem, this means capturing the demon instead of destroying it. If I design Gem right, then capturing the demon will be comparatively difficult to destroying it. If you're able to capture it (maybe "convert" is a better term) then you have access to that piece for future games of Gem and perhaps some kind of bonus "blessing" for the character outside of Gem.

I've been working on modeling the first demon, based on Chitipati. Chitipati is two skeletons depicted with a frame of flames which acts as a memento mori and more generally represent endless change:

Chitipati, from Himalayan Art Resources

The first part I tried to model is the flame halo. Similar features are common in wrathful diety iconography so this model would be useful for other demons too. I made several attempts at modeling these flames using only Blender's geometry nodes with the hope of generating them procedurally, but they never came out well (maybe because of my inexperience with geometry nodes). In the end I hand-modeled five flames and use geometry nodes to distribute them in an arch, which I'm happy with:

Flame arch

When I moved on to modeling Chitipati I started with a fair amount of detail. It did not look very good, so I gave it another shot from scratch using a rougher approach, and it looked a lot better (though the hands look TERRIBLE...I'll give them more attention later maybe, to organize them as particular mudras). Relying on textures for detail rather than capturing it in the mesh makes a lot more sense for this game, especially if I'm using photo-textures. And because I'm rendering out at a lower resolution, finer mesh detail won't really show up anyways.

A side note: I'm modeling this demon as only one skeleton, rather than Chitipati's two.

Test render

This test render's resolution is high enough where the lack of mesh detail is apparent (pretty boxy looking in some parts). When rendered in-game the low detail looks more appropriate:

In-game

Infilling and seamless tiling with GIMP

One idea for the flame halo was to use content-aware fill or a similar infilling algorithm to generate a full image of the flame background and then some how turn that into a 3d model. That route didn't really work—I couldn't make it look good; it just looked flat and janky—but I did find an infilling plugin for GIMP called Resynthesizer that is really promising. It didn't work very well for the flame frame (perhaps because it's an illustration or because there isn't a lot of content to reference for the infilling) but it works much better with texture photography. Here's an example using an image of some overgrowth:

The original image

After applying the Resynthesis plugin (using Filters > Map > Resynthesize) and checking "Make horizontally tileable" and "Make vertically tileable":

The filled-in/resynthesized image

That result looks great, but it wasn't actually seamlessly tileable. Fortunately there is another filter, Filters > Map > Tile Seamless, that handles that:

The seamless tiling image

It looks really good tiled:

The seamless tile in use (2x2)

This texture has a lot going on so it may be an easier example. If you look closely at the seamless tile version you can see some ghost leaves overlapping other leaves, which might be more noticeable in a sparser texture.

It's more apparent in these textures:

You can see some mushiness/blurriness/choppiness in the patterns from the overlapping areas. It's not terrible, especially for textures you won't look closely at and in the context of the game's downsampled resolution. Again, part of the game's aesthetic is about giving a big margin of error for quick-and-dirty assets, whether through low poly modeling or iffy automated tools.

When I have more time I want to see about integrating this directly into the Texture Editor from the last post; it would be nice to not have to open up GIMP every time to process images this way.

As a side note, it's been awhile since I've used GIMP and I'm impressed by this latest version! Feels like it had a big glow up like Blender did.


  1. "Exorcisms", sort of line with how they work in Buddhism, though I don't know that I will call them that in-game. 

>>