Threat Modeling in Digital Communites

It's trolls all the way down

As part of my OpenNews fellowship I've recently started working on the Coral Project, a Knight-funded joint venture between the New York Times, the Washington Post, and Mozilla which, broadly, is focused on improving the experience of digital community online. That mission is an catch-all for lots and lots of subproblems; the set I'm particularly drawn to are those issues around creating inclusive and civil spaces for discussion.

Any attempt at this must contend with a variety of problems which undermine and degrade online communities. To make the problem more explicit, it's helpful to have a taxonomy of these "threats". I'll try to avoid speculating on solutions and save that for another post.


The most visible barrier to discussion spaces are deliberately toxic actors - generally lumped together under the term "trolls"*.

I think most people are familiar with what a troll is, but for the sake of completeness: trolls are the users who go out and deliberately attack, harass, or offend individuals or groups of people.

If you're interested in hearing more about what might motivation a troll, this piece provides some insight.


Any mass conglomeration of spending power or social capital soon becomes a resource to be mined by brands. So many companies (and other organizations) have adopted the practice of astroturfing, which is a simulated grassroots movement.

For instance, a company gets a lot of people to rave about their products until you too, just by sheer exposure (i.e. attrition), adopt a similar attitude as your baseline. This is a much more devious form of spam because it deliberately tries to misshape our perception of reality.

This can increase the amount of noise in the network and reduce the visibility/voice of legitimate members.

Sockpuppeting/Sybil attacks

A common problem in ban-based moderation systems is that barriers-to-entry on the site may be low enough such that malicious actors can create endless new accounts with which to continue their harassment. This type of attack is called a Sybil attack (named after the dissociative identity disorder patient).

Similarly, a user may preemptively create separate accounts to carry out malicious activity, keeping deplorable behavior distinct from their primary account. In this case, the non-primary accounts are sockpuppets.

It seems the problem with Sybil attacks is the ease of account creation, but I don't think a solution to the Sybil attacks is to make barriers-to-entry higher. Rather, you should ask whether banning is the best strategy. Ideally, we should seek to forgive and reform users rather than to exclude them (I'll expand on this in another post). This solution is dependent on whether or not the user is actually trying to participate in good faith.

Witch hunts

This is the madness of crowds that can spawn on social networks. An infraction, whether it exists or not, whether it is big or small, becomes viral to the point that the response is disproportionate by several orders of magnitude. Gamergate, which began last year and now seems to be permanent part of the background radiation of the internet, is an entire movement that blew up from a perceived - that is, non-existent, nor particularly problematic - offense. In these cases, the target often becomes a symbol for some broader issue, and it's too quickly forgotten that this is a person we're talking about.

Eternal September

"Eternal September" refers to September 1993, when AOL expanded access to Usenet caused a large influx of new users, not socialized to the norms of existing Usenet communities. This event is credited with the decline of the quality of those communities, and now generally refers to the anxiety of a similar event. New users who know nothing about what a group values, how they communicate, and so on come in and overwhelm the existing members.

Appeals to "Eternal September"-like problems may themselves be a problem - it may be used to rally existing community members in order to suppress a diversifying membership, in which case it's really no different than any other kind of status quo bias.

To me this is more a question of socialization and plasticity - that is, how should new members be integrated into the community and its norms? How does the community smoothly adapt as its membership changes?


Brigading is the practice where organized groups suss out targets - individuals, articles, etc - which criticize their associated ideas, people, and so on and go en masse to flood the comments in an incendiary way (or otherwise enact harmfully).

This is similar to astroturfing, but I tend to see brigading as being more of a bottom-up movement (i.e. genuinely grassroots and self-organized).


Doxxing - the practice of uncovering and releasing personally identifying information without consent - is by now notorious and is no less terrible than when it first became a thing. Doxxing is made possible by continuity in online identity - the attacker needs to connect one particular account to others, which can be accomplished through linking the same or similar usernames, email addresses, or even personal anecdotes posted across various locations. This is a reason why pseudonyms are so important.


Swatting is a social engineering (i.e. manipulative) "prank" in which police are called in to a investigate possible threat where there is none. It isn't new but seems to have had a resurgence in popularity recently. What was once an activity for revenge (i.e. you might "swat" someone you didn't like) now seems to be purely for the spectacle (i.e. done without consideration of who the target is, just for lulz) - for instance, someone may get swatted while streaming themselves on

The Fluff Principle

The "Fluff Principle" (as it was named by Paul Graham) is where a vote-driven social network eventually comes to be dominated by "low-investment material" (or, in Paul's own words, "the links that are easiest to judge").

The general idea is that if a piece of content takes one second to consume and judge, more people will be able to upvote it in a given amount of time. Thus knee-jerk or image macro-type content come to dominate. Long-form essays and other content which takes time to consume, digest, and judge just can't compete.

Over time, the increased visibility of the low-investment material causes it to be come the norm, and so more of it is submitted, and so the site's demographic comes expecting that, and thus goes the positive feedback loop.

Power-user oligarchies

In order to improve the quality of content or user contributions, many sites rely on voting systems or user reputation systems (or both). Often these systems confer greater influence or control features in accordance with social rank, which can spiral into an oligarchy. A small number of powerful users end up controlling the majority of content and discussion on the site.

Gaming the system

Attempts to solve any of the above typically involve creating some kind of technological system (as opposed to a social or cultural one) to muffle undesirable behavior and/or encourage positive contribution.

Especially clever users often find ways of using these systems in ways contradictory to their purpose. We should never underestimate the creativity of users under constrained conditions (in both bad and good ways!).

Whether or not some of these are problems really depends on the community in question. For instance, maybe a site's purpose is to deliver quick-hit content and not cerebral long-form essays. And the exact nature of these problems - their nuances and idiosyncrasies to a particular community - are critical in determining what an appropriate and effective solution might be. Unfortunately, there are no free lunches :\

(Did I miss any?)

* The term "troll" used to have a much more nuanced meaning. "Troll" used to refer to subtle social manipulators, engaging in a kind of aikido in which they caused people to trip on their own words and fall by the force of their own arguments. They were adept at playing dumb to cull out our own inconsistencies, hypocrisies, failures in thinking, or inappropriate emotional reactions. But you don't see that much anymore...just the real brutish, nasty stuff.


Artwork for The Founder

I've uploaded almost all of the artwork for The Founder. I wasn't sure I would be able to get through it all, but I *think* all the artwork for the game is finished now (whew!).

Here's a taste:

View the rest here.



I started working on the office layouts recently. There will be three office levels, going from Apartment to Office to Campus (there may be another intermediary between Office and Campus). Here's the "apartment-office" so far:

I have been extremely frugal with the textures - the texture atlas for this scene only needs a size of about 64x64. I allocated a 256x256 texture for this so I will probably make some of the textures more detailed.

The area is too cramped at the moment - there's no room for the cone people to move about, so I'll have to make the space bigger. And since the perks you purchase can manifest in the office environment, there will need to be extra space for that too!


New onboarding and UI

This weekend some PubSci friends came over and took a look at the current state of The Founder. There was a lot of really great feedback about improving the onboarding (there wasn't much of one to speak of) and the UI (which was almost entirely in menus, not very "game-like").

So the past couple days I've taken their suggestions and started implementing them. So far I'm really happy with how they're turning out.

The onboarding prior to these changes was really just a screen where you could select your co-founder. And then after that there were a bunch of text boxes introducing all of the game's mechanics and concepts - which there are a lot of.

The onboarding now (below) provides players more flexibility in how they begin the game - they can now select the starting vertical (Information or Hardware) and starting location (Boston/NYC/SF) in addition to their co-founder. So the concepts of vertical and location are more naturally introduced as part of this early game configuration.

For the UI, the general idea was to take it out from these menus and integrate it more directly into the office environment. I went through a few iterations of this today:

Too claustrophobic and disorienting. You lose the sense of the office as a complete space. The perspective limited navigation options on mobile too much.

This is basically the route I ended up taking. It keeps the "god-view" (which is important to the critical aspect of the game) and preserves the player's freedom to pan/zoom around at will. Office objects can be interacted with directly to bring up relevant menus. You can't really see it in this gif, but interact-able objects have pulsing colors.

And this is the most recent build, which is more polished and adds in purchasable expansions to the office. It was important to present these purchasable expansions as noticeable gaps in the space so it feels like your office is "filling up" - i.e. real growth is happening :)


Clipping 3D objects in NGUI

Updated 1/30/2015

This week I've been putting together some of the 3D assets for The Founder and began incorporating them into the game's UI. But I hit a snag:

These objects aren't clipped like NGUI-native elements. I searched around for awhile but only came across half-answers. The two main suggestions were to use a clipping shader or Unity's Render Textures. The latter involves setting up hidden cameras which render to textures and then using those textures as NGUI UITextures. It's kind of a lot of setup and I wasn't sure how it would work with my use case, which involves a grid of 3D objects.

Render Textures are a Unity Pro-only feature anyways so that was out of the question. There is a Unity Free hack which achieves something similar but doesn't have the same optimization that comes with the official implementation.

That left the clipping shader (Note 1/30/2015: This is no longer true, see the update below). I have zero experience with shaders. Fortunately mathiassoeholm was kind enough to post their clipping shader. For the sake of completeness, I've reproduced the code below as well (note: I removed an extra unnecessary subshader from mathiassoeholm's original implementation).

As I understand it, this shader functions the same way NGUI's built-in clipping shaders work. The UIPanel handles the drawing of its children NGUI widgets and passes in the appropriate clipping information to their shaders. Since the UIPanel does not manage the drawing of these 3D objects, the ClippingModel class grabs the necessary clipping information from the nearest UIPanel.

Update: 1/30/2015

I was told of a better approach than the clipping shader I was using before (the original solution is preserved below). The clipping shader itself has issues if you want to render other NGUI UIPanels over your 3D objects. My original solution made it so that 3D objects always rendered on top. I have many semi-transparent overlays and popovers which the objects should render beneath, so this was problematic.

Fortunately, there is a better solution to this clipping problem which does not involve shaders (this solution is completely independent from the clipping shader approach) and works well with multiple UI layers.

All UIPanels in NGUI have a "Render Queue" property which by default is set to "Automatic". This property controls the order in which these panels are drawn. Panels with a higher Render Queue value are drawn later.

Unity's default inspectors for rendering meshes do not provide you access to this value, but this script (from here) gives you that access:

using UnityEngine;

public class SetRenderQueue : MonoBehaviour
    public int renderQueue = 3000;

    Material mMat;

    void Start ()
        Renderer ren = renderer;

        if (ren == null)
            ParticleSystem sys = GetComponent<ParticleSystem>();
            if (sys != null) ren = sys.renderer;

        if (ren != null)
            mMat = new Material(ren.sharedMaterial);
            mMat.renderQueue = renderQueue;
            ren.material = mMat;

    void OnDestroy () { if (mMat != null) Destroy(mMat); }

I attached this script to my 3D objects, set their material to use the Self-Illumin/Diffuse shader (so I wouldn't need to light them), and put them on the "PlayArea" layer (the layer that NGUI uses).

Then it was just a matter of tweaking the related UIPanel's "Render Q" settings. Aside from "Automatic", you have the options of "Explicit" and "Start At". I used "Start At" for my panels. The panels which I wanted to be beneath the 3D objects had a "Render Q: Start At" value lower than the render queue value I set for the objects. The panels which I wanted to be above the 3D objects had a higher "Start At" value. You may have to play around with these values to get exactly what you need.

The old clipping shader solution

There were not any clear instructions on how to actually apply the code provided, so here is how I ended up getting it working.

I attached the ClippedModel.cs to my 3D object and set my 3D object's material's shader to be the UI/ClippedUnlitModel shader.

Then I made sure that the closest UIPanel (in my case, the Scroll View which held the grid of objects) had it's Clipping property set to "Soft Clip".

I thought this would be enough, but then I started having weird transparency issues:

This effect is more like punching a hole through the UI to reveal the game world underneath.

I'm not sure if this is the best or the correct approach, but I managed to solve this by using a separate camera to render the 3D objects.

First I created a new layer and called it "3D".

Then I set my 3D object to be on this layer.

Then I created a new camera, positioned it appropriately (so that your object is in view), and configured like so:

You may need to play with your object's Z position to get it to show up above the UI.

The most important setting here is that the Depth is 2 and the Culling Mask is only the 3D layer.

I also set the NGUI Camera to be at a Depth of 1 so that the 3D camera renders over it. I can't remember if this was the default setting though.



Shader "UI/ClippedUnlitModel"
        _MainTex ("Base (RGB), Alpha (A)", 2D) = "black" {}

        LOD 200

            "Queue" = "Transparent"
            "IgnoreProjector" = "True"
            "RenderType" = "Transparent"

            Cull Off
            Lighting Off
            Offset -1, -1
            Fog { Mode Off }
            ColorMask RGB
            Blend SrcAlpha OneMinusSrcAlpha

            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            sampler2D _MainTex;
            float4 _PanelOffsetAndSharpness;
            float _PanelSizeX, _PanelSizeY;

            struct appdata_t
                float4 vertex : POSITION;
                half4 color : COLOR;
                float2 texcoord : TEXCOORD0;

            struct v2f
                float4 vertex : POSITION;
                half4 color : COLOR;
                float2 texcoord : TEXCOORD0;
                float2 posInPanel : TEXCOORD1;

            v2f vert (appdata_t v)
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                o.color = v.color;
                o.texcoord = v.texcoord;

                float2 clipSpace =  o.vertex.xy / o.vertex.w;

                // Normalize clip space
                o.posInPanel = (clipSpace.xy + 1) * 0.5;

                // Adjust for panel offset
                o.posInPanel.x  -= _PanelOffsetAndSharpness.x;
                o.posInPanel.y  -= _PanelOffsetAndSharpness.y;

                // Adjust for panel size
                o.posInPanel.x  *= (1 / _PanelSizeX);
                o.posInPanel.y  *= (1 / _PanelSizeY);

                // Transform back to clip space
                o.posInPanel *= 2;
                o.posInPanel -= 1;

                return o;

            half4 frag (v2f IN) : COLOR
                // Softness factor
                float2 factor = (float2(1.0, 1.0) - abs(IN.posInPanel)) *;

                // Sample the texture
                half4 col = tex2D(_MainTex, IN.texcoord) * IN.color;
                col.a *= clamp( min(factor.x, factor.y), 0.0, 1.0);

                return col;


using UnityEngine;

public class ClippedModel : MonoBehaviour
    private UIPanel _panel;
    private Material _material;

    private int _panelSizeXProperty;
    private int _panelSizeYProperty;
    private int _panelOffsetAndSharpnessProperty;

    private float _virtualScreenWidth;
    private float _virtualScreenHeight;

    void Start()
        _panel = UIPanel.Find(transform);
        _material = !Application.isPlaying ? renderer.sharedMaterial : renderer.material;

        _virtualScreenWidth = UIRoot.GetPixelSizeAdjustment(gameObject) * Screen.width;
        _virtualScreenHeight = UIRoot.GetPixelSizeAdjustment(gameObject) * Screen.height;

        _panelSizeXProperty = Shader.PropertyToID("_PanelSizeX");
        _panelSizeYProperty = Shader.PropertyToID("_PanelSizeY");
        _panelOffsetAndSharpnessProperty = Shader.PropertyToID("_PanelOffsetAndSharpness");


    void Update()
        if (_panel.hasClipping)
            var soft = _panel.clipSoftness;
            var sharpness = new Vector2(1000.0f, 1000.0f);
            if (soft.x > 0f)
                sharpness.x = _panel.baseClipRegion.z / soft.x;
            if (soft.y > 0f)
                sharpness.y = _panel.baseClipRegion.w / soft.y;

            Vector4 panelOffsetAndSharpness;

            // Get offset
            panelOffsetAndSharpness.x = ((_virtualScreenWidth * 0.5f + _panel.baseClipRegion.x) - (_panel.baseClipRegion.z * 0.5f)) / _virtualScreenWidth;
            panelOffsetAndSharpness.y = ((_virtualScreenHeight * 0.5f + _panel.baseClipRegion.y) - (_panel.baseClipRegion.w * 0.5f)) / _virtualScreenHeight;

            // Get sharpness
            panelOffsetAndSharpness.z = sharpness.x;
            panelOffsetAndSharpness.w = sharpness.y;

            // Set shader properties
            _material.SetFloat(_panelSizeXProperty, _panel.baseClipRegion.z / _virtualScreenWidth);
            _material.SetFloat(_panelSizeYProperty, _panel.baseClipRegion.w / _virtualScreenHeight);
            _material.SetVector(_panelOffsetAndSharpnessProperty, panelOffsetAndSharpness);