Highlights
- FSM remains the simplest entry point for enemy AI with fewer than seven states.
- Behavior trees separate decision-making from task logic, making complex NPC behaviors easier to scale.
- Unity NavMesh and Unreal AI tools handle obstacle-aware movement and form the foundation of modern enemy navigation.
Three systems sit at the core of almost every enemy in every game shipped in the last decade: the finite state machine (FSM), the behavior tree, and NavMesh pathfinding. A gameplay programmer who can implement all three, and choose between them deliberately, holds the practical foundation for professional non-player character (NPC) development.
This tutorial builds each system from scratch in both Unity C# and Unreal Blueprint. It concludes with a direct comparison to show which system fits best for different types of projects.
The Three Systems a Game AI Beginner Must Know
Before writing a single line of code, it helps to understand what problem each system solves. A thorough breakdown of all three systems and their roles in game AI development covers the full architecture in detail.
A finite state machine defines a fixed set of conditions and the rules that move an enemy between them. An enemy occupies exactly one state at any moment: patrolling, idle, chasing, or attacking, and each transition fires when a specific condition is met.
The logic is sequential and easy to read. For a practical look at using state machines for AI behavior in game development, the linked resource walks through the structure in C# with clear examples.
A behavior tree organizes those same decisions into a branching hierarchy of nodes. A Selector node tries each branch in order and moves to the next if the current one returns failure. A Sequence node runs its children one by one and stops the moment any child fails. Tasks at the leaf level execute the actual actions.
A BTService ticks in the background at a set interval, updating shared data on the Blackboard, while a BTTask executes one discrete action, such as moving to a target location, and returns a success or failure result upward through the tree. The complete guide to ‘mastering behavior trees’ in game development on NumberAnalytics covers the node architecture in full.
NavMesh pathfinding handles a separate layer entirely: physical movement through geometry. It precomputes walkable surfaces in the scene, assigns a NavMesh Agent to the enemy character, and recalculates routes dynamically when obstacles shift. Without it, any AI that moves through a three-dimensional (3D) environment will pass straight through walls.
How to Build a Finite State Machine in Unity (C#)
The finite state machine is the right starting point for any game artificial intelligence (AI) beginner.
It is debuggable, small, and appropriate for projects with fewer than six or seven distinct enemy states. The ‘Developers Heaven guide’ on game AI behavior and pathfinding covers the progression from FSM to behavior tree clearly, if additional context is needed alongside this code.
csharp
using UnityEngine;
using UnityEngine.AI;
public enum EnemyState { Idle, Patrol, Chase, Attack }
public class EnemyFSM : MonoBehavior
{
public Transform player;
public Transform[] waypoints;
public float detectionRange = 10f;
public float attackRange = 2f;
private NavMeshAgent agent;
private EnemyState state = EnemyState.Idle;
private int waypointIndex;
private float idleTimer;
void Start() => agent = GetComponent<NavMeshAgent>();
void Update()
{
float dist = Vector3.Distance(transform.position, player.position);
switch (state)
{
case EnemyState.Idle:
idleTimer += Time.deltaTime;
if (dist < detectionRange) SetState(EnemyState.Chase);
else if (idleTimer > 2f) SetState(EnemyState.Patrol);
break;
case EnemyState.Patrol:
if (dist < detectionRange) SetState(EnemyState.Chase);
else if (agent.remainingDistance < 0.5f)
{
waypointIndex = (waypointIndex + 1) % waypoints.Length;
agent.SetDestination(waypoints[waypointIndex].position);
}
break;
case EnemyState.Chase:
agent.SetDestination(player.position);
if (dist < attackRange) SetState(EnemyState.Attack);
else if (dist > detectionRange) SetState(EnemyState.Patrol);
break;
case EnemyState.Attack:
agent.ResetPath();
if (dist > attackRange) SetState(EnemyState.Chase);
break;
}
}
void SetState(EnemyState next) { idleTimer = 0; state = next; }
}
Behavior Trees for Game AI: Unity C# Implementation
Why Behavior Trees Replace State Machines at Scale
When the number of states climbs, the finite state machine generates a tangle of transition conditions that becomes difficult to maintain. Behavior trees solve this by separating the decision structure from the task logic.
A BTService updates shared data each tick, so tasks can read the current world state without doing their own sensing. A BTTask carries out one action and reports success or failure back up the tree.
The 'complete behavior tree implementation tutorial’ for Unity and Unreal on Generalist Programmer, goes deeper into production-grade node architectures and Blackboard systems if more detail is needed beyond the beginner implementation below.
csharp
// Base node
public abstract class BTNode {
public enum Status { Running, Success, Failure }
public abstract Status Tick();
}
// Selector: returns the first non-Failure result
public class Selector : BTNode {
private List<BTNode> children;
public Selector(List<BTNode> children) => this.children = children;
public override Status Tick() {
foreach (var c in children) {
var s = c.Tick();
if (s != Status.Failure) return s;
}
return Status.Failure;
}
}
// BTService: updates shared data each tick
public class BTService_SensePlayer : BTNode {
private Transform enemy, player;
private float range;
public bool Detected { get; private set; }
public BTService_SensePlayer(Transform e, Transform p, float r)
{ enemy = e; player = p; range = r; }
public override Status Tick() {
Detected = Vector3.Distance(enemy.position, player.position) < range;
return Status.Success;
}
}
// BTTask: chase
public class BTTask_Chase : BTNode {
private NavMeshAgent agent;
private Transform player;
private BTService_SensePlayer sense;
public BTTask_Chase(NavMeshAgent a, Transform p, BTService_SensePlayer s)
{ agent = a; player = p; sense = s; }
public override Status Tick() {
if (!sense.Detected) return Status.Failure;
agent.SetDestination(player.position);
return Status.Running;
}
}
// BTTask: patrol
public class BTTask_Patrol : BTNode {
private NavMeshAgent agent;
private Transform[] waypoints;
private int index;
private bool initialized;
public BTTask_Patrol(NavMeshAgent a, Transform[] w) { agent = a; waypoints = w; }
public override Status Tick() {
if (!initialized) {
agent.SetDestination(waypoints[0].position);
initialized = true;
return Status.Running;
}
if (agent.remainingDistance < 0.5f) {
index = (index + 1) % waypoints.Length;
agent.SetDestination(waypoints[index].position);
}
return Status.Running;
}
}
// Wiring: MonoBehavior that owns and drives the tree
public class EnemyAI : MonoBehavior {
public Transform player;
public Transform[] waypoints;
public float detectionRange = 10f;
private BTService_SensePlayer sense;
private BTNode root;
void Start() {
var agent = GetComponent<NavMeshAgent>();
sense = new BTService_SensePlayer(transform, player, detectionRange);
root = new Selector(new List<BTNode> {
new BTTask_Chase(agent, player, sense),
new BTTask_Patrol(agent, waypoints)
});
}
void Update() {
sense.Tick(); // update shared state before the tree evaluates
root.Tick();
}
}
The service is ticked explicitly in ‘Update’ before ‘root.Tick()’ runs. This ensures ‘Detected’ is current before ‘BTTask_Chase’ reads it. The service is not a child of the Selector because a Success return from it would cause the Selector to stop evaluating immediately, preventing the chase and patrol tasks from running.
For a guided walkthrough of building a behavior tree guard AI in Unity from the ground up, Medium’s tutorial on ‘AI behavior trees’ in game development is a useful companion to the code above.
Architecture note: This simplified implementation re-evaluates the Selector from the first child on every tick, which is correct behavior for a stateless tree. Production behavior tree frameworks typically track the currently running child to avoid re-evaluating the tree every tick. For a beginner implementation with only two branches, however, re-evaluating each tick produces the same result and is easier to understand.
NavMesh Unity Setup: Pathfinding for Enemy Movement
Setting Up Unity NavMesh for Enemy AI Navigation
NavMesh pathfinding is the standard approach for obstacle-aware enemy navigation in three-dimensional Unity projects. Without it, enemies move but do not path around obstacles. The setup differs slightly depending on the Unity version in use. Unity's own beginner AI pathfinding course on ‘Unity Learn’ covers baking, agents, and dynamic obstacles across a structured 3.5-hour project if a guided environment is preferred.
Unity 2022.1 and earlier (Legacy Navigation Window):
- Open Window > AI > Navigation.
- Select all floor and walkable geometry and check Navigation Static in the Inspector.
- Select obstacles such as walls and boxes, mark them Navigation Static, and set their Navigation Area to Not Walkable.
- Open the Bake tab and click Bake. A blue overlay confirms the walkable surface.
Unity 2022.2 and Later (AI Navigation Package):
Unity 2022.2 deprecated the legacy Navigation window and moved to the AI Navigation package (com.unity.ai.navigation). Instead of the legacy Navigation window, add a NavMesh Surface component to the scene's ground object via Add Component > AI > NavMesh Surface, then click Bake in its Inspector. The walkable result is identical; only the setup path differs.
A clear breakdown of this newer workflow is available in the ‘Unity AI Navigation guide’ on GeeksforGeeks and in this ‘step-by-step NavMesh Surface setup’ on Medium.
In both versions, add a NavMesh Agent component to the enemy GameObject and apply this script:
csharp
using UnityEngine;
using UnityEngine.AI;
public class EnemyNavMesh : MonoBehavior
{
public Transform player;
private NavMeshAgent agent;
void Start() {
agent = GetComponent<NavMeshAgent>();
if (player == null)
player = GameObject.FindGameObjectWithTag("Player").transform;
}
void Update() => agent.SetDestination(player.position);
}
For a deeper explanation of how the NavMesh system works under the hood, including agent settings, bake parameters, and common configuration mistakes, the official ‘Unity AI Navigation documentation’ and Medium’s ‘walkthrough on core NavMesh setup’ are both worth reading alongside this tutorial.
Increase Radius in the NavMesh Agent if the enemy clips through narrow corridors. If the enemy does not move, confirm the NavMesh has been baked and the NavMesh Agent component is attached.
Unreal Engine Behavior Tree: Blueprint BTService and BTTask
Unreal Engine ships a native behavior tree editor. The setup follows six steps. For a complete reference on the Blackboard, task, and decorator system in Unreal Engine 5, the official ‘Unreal Engine Behavior Tree Quick Start Guide’ and the ‘Epic Developer Community walkthrough’ cover the full setup in detail.
Step 1
- In the Content Browser, right-click and create a Blackboard, a Behavior Tree, and an AI Controller. Assign the controller to the enemy Blueprint under AI Controller Class, and set Auto Possess AI to "Placed in World or Spawned" on the enemy pawn. This ensures the AI logic triggers regardless of how the enemy enters the level.
- In the AI Controller Blueprint, call Run Behavior Tree on Event OnPossess rather than BeginPlay. OnPossess guarantees the controller already holds a valid Pawn reference before the behavior tree starts executing. Using BeginPlay risks starting the tree before possession is complete.
Step 2
- In the Blackboard, add three keys: an Object key named ‘TargetActor,’ a Bool key named ‘IsTargetDetected,’ and a Vector key named ‘TargetLocation.’ Without ‘TargetLocation,’ the patrol task has no destination to pass to a Move To node, and the patrol branch will not function.
Step 3 - BTService
- Create a Blueprint inheriting from ‘BTService_BlueprintBase,’ name it ‘BTS_CheckForTarget.’ In the Receive Tick AI event, use a Multi Sphere Trace by Channel centered on the enemy to check for the player within detection range. On a valid hit, write the player reference to ‘TargetActor’ and set ‘IsTargetDetected’ to true. On no hit, set ‘IsTargetDetected’ to false.
- Attach this service to the root Selector node. In Unreal's behavior tree, a service runs as long as the node it is attached to, or any of its children, is active. Placing it at the root means it ticks continuously for the entire duration of the tree's execution, which is the correct behavior for a global awareness check. Attaching it to a child branch instead would stop it from running whenever that branch is not active.
Note: The Sphere Trace approach works correctly for a basic setup. For larger projects, consider replacing it with the AI Perception Component, which provides built-in support for sight, hearing, and damage senses and is more optimized for production use.
Step 4 - BTTask for Patrol
- Create a Blueprint inheriting from ‘BTTask_BlueprintBase,’ name it ‘BTTask_FindRandomLocation.’ Open the Event Receive Execute AI node and call Get Random Reachable Point in Radius using the enemy's current world position as the origin.
- To write the result to the Blackboard, create a variable of type Blackboard Key Selector, name it ‘TargetLocationKey,’ and enable Instance Editable on it by clicking the eye icon in the Details panel. Then, in the Behavior Tree editor, set this variable to point to the ‘TargetLocation’ key. Use Set Value as Vector with that Key Selector variable to write the result at runtime.
- This Key Selector pattern is strongly recommended over addressing a key by string name directly. Both approaches work at runtime, but the Key Selector is configured in the editor, is refactor-safe, and avoids magic strings that break silently if a key is renamed.
- End the task with Finish Execute set to Success. A missing Finish Execute node locks the behavior tree on that task permanently and is the most common error in beginner Unreal AI setups.
Step 5 - Chase
- Use Unreal's built-in Move To task pointed at ‘TargetActor.’ No custom Blueprint is required.
- Gate this branch with a Blackboard Decorator checking ‘IsTargetDetected’ with the condition Is Set. For Bool keys specifically, Is Set evaluates to true when the value is true, which is the intended behavior here. For Object keys, Is Set means the reference is non-null, so the condition has a different meaning per key type. Do not assume the pattern transfers directly when working with other key types.
Step 6 - Tree Structure
- Place a Selector directly below the root. Attach BTS_CheckForTarget to this root Selector node, not to any child branch, so it runs continuously.
- Left branch: a Sequence containing a Move To task pointed at TargetActor, gated by a Blackboard Decorator checking if IsTargetDetected Is Set. This is the chase branch and runs when a target has been detected.
- Right branch: a Sequence containing BTTask_FindRandomLocation followed by a Move To task pointed at TargetLocation. This is the patrol branch and runs when no target is detected, because the left branch's decorator fails and the Selector falls through to the right.
- The Selector evaluates left to right, so chase takes priority over patrol whenever a target is active.
For attack state logic beyond patrol and chase, Epic's developer community hosts an ‘Unreal Engine 5.4 behavior tree tutorial’ that builds directly on this setup. Follow it once both branches above are confirmed to be working.
Finite State Machine vs Behavior Trees vs NavMesh: When to Choose Each
The three systems are not competitors.
They address separate layers of the same problem, and in most shipped games, all three run simultaneously on the same character. For a broader look at how these tools fit into a full game AI development career, the The Complete Guide to Becoming an AI game developer in 2025 by Kelechi Edeh, covers decision systems, pathfinding, and machine learning tools in a professional context.
Outlook Respawn
The FSM in games remains the most readable entry point for new programmers. Behavior trees take over when complexity demands modularity and the transition graph of an FSM becomes unmanageable. Unity NavMesh and its Unreal equivalent sit underneath both systems, handling the physical act of movement through a scene.
A game AI beginner who builds all three in one project leaves with a toolkit that covers the majority of NPC problems encountered in professional development.
