RL, FEP & AMBP in Alive

The same world, different decisions

Every living creature samples exactly one available action every 50 ms of simulation time. Faster playback executes more of these fixed steps. Turning does not also move. Feeding, drinking, reproduction, and alarm emission require their corresponding actions; physiology continues independently.

The 14 output slots cover four body-relative translations, two turns, feed/bite, reproduce, drink, alarm, rest, inspect, food call, and mating call. The mating-call slot is retained for compatibility but is unavailable to all four asexual founder groups. Availability masks guide the decision. Execution checks energy, live targets, reach, population capacity, and placement again. A target can be consumed by another creature between sensing and execution. Food is finite; bodies collide; terrain and swimming ability constrain movement.

JEPA and RL currently use the same actor-critic and predictive learning architecture. They are inherited founder-group labels, not two independently implemented policy algorithms. Random JEPA, RL, FEP, and AMBP founders share body-trait and learning-rate distributions. All four groups reproduce asexually and can include predators.

RL: a categorical actor with a critic

The actor receives 56 body/environment inputs, 32 eye-ray slots with three channels each, and four directional contact inputs: 156 inputs total. Unused eye slots stay zero. It has 3–15 genetically specified hidden layers, each 16–63 neurons wide, and 14 action outputs. Every layer connects only to its immediate neighbor. Hidden activations use tanh; output logits are linear; hidden and output neurons have trainable biases.

hidden = tanh(W × previous layer + bias)
learned action probability ∝ exp(logit / 0.2)
actual probability = (1 − exploration) × learned probability
                   + exploration / available action count

Founder parameters initialize survival and stereo-sensing features through the hidden layers. These are hand-initialized, trainable priors. The final decision is sampled from the masked probability distribution, rather than taking only its largest entry.

Starting connections guide sideways food approach, brake forward movement at visual hazards, and favor retreat or sideways escape from directional contact and blocked water. Every signal passes through the hidden layers; learning and inherited checkpoints can change these initial preferences. No rule replaces the neural action choice.

Credit arrives from the executed action

The following tick's outcome updates the previous decision's cached activations. Reward combines energy change (excluding allocated birth energy), injury/healing, changes in fear and wellbeing, birth feedback, curiosity, navigation and survival potentials, and failed-movement feedback. The total is bounded to [−1, 1].

advantage = clamp(reward + 0.999 × V(current) − V(previous), −2, 2)

The advantage backpropagates through all actor layers. Exploration correction accounts for the probability that the learned policy, rather than uniform exploration, supplied the action. A small entropy objective (0.002) discourages collapse to a single choice. Bounded eligibility traces on output parameters and critic features retain credit across decisions, with trace decay 0.98. The critic uses a separate value head on the 24-value representation, with normalized updates; values are bounded to ±100 and critic weights to ±10. Actor weights and biases are bounded to ±3. Inherited per-tick learning rates range from 0.002 to 0.012.

Exploration responds to experience and confinement

Fresh founders use 25% uniform exploration for their first 300 decisions (15 seconds), tapering to a 5% baseline by decision 600. Offspring with inherited policies start at the baseline. Boredom and forecast uncertainty can raise it to 35%. After 15 seconds confined within a stable two-unit radius, energy below 15% of individual capacity raises exploration gradually toward 50% near empty. Availability masks still apply.

The predictive heads are auxiliary

A learned action-conditioned predictor forecasts the next 24-value hidden representation. A decoder forecasts sensory inputs; other heads summarize predicted energy, safety, and wellbeing. Predictor/decoder updates use local gradients including the tanh derivative. These forecasts inform inspection and curiosity, but do not replace the MLP's action logits or add actor shortcuts. The implementation has no separate target network, PPO clipping, or general replay buffer; bounded deferred birth lessons are the specific replay mechanism.

FEP: recurrent prediction and active inference

FEP models hunger, health, thirst, fear, wellbeing, food proximity, threat strength, water barrier, cold stress, touch, curiosity, brood pressure, behavioral feedback, offspring investment, food/threat bearings, and left/right food scent. These are 18 fixed state coordinates. Individual eye rays, plant-identity channels, and the remaining raw sensors are outside this model, even though the inspector can display them.

Each of the 14 actions conditions its own dense recurrent network: 324 recurrent weights and 18 biases, or 4,788 parameters across all action banks. The neural state follows:

τ dzᵢ/dt = −zᵢ + biasᵢ + Σⱼ Wᵢⱼ tanh(zⱼ)
τ = 0.1 seconds
one 0.05-second tick = four simultaneous Euler substeps

A Gaussian observation correction to the predicted state becomes the next recurrent state. Signed feedback and net hunger remain signed; predicted states are not clipped. Healthy-state priors calibrate health near 1 and hunger/wellbeing near 0.5, so untried actions do not initially predict large spontaneous injury.

Planning scores predictions, then samples

For each available candidate, the CTRNN integrates one tick of that action followed by one conservative rest tick: a 100 ms horizon. Future feeding, drinking, or reproduction is not assumed available. The planner combines preference cost with information about uncertain model parameters:

G(action) = total preference cost − 0.15 × parameter information gain
q(action) ∝ exp(−32 × G(action))

Only available actions receive probability. FEP samples this posterior directly. MLP logits, critic updates, JEPA learning heads, and uniform exploration do not run in its decision path.

Each preference contributes squared target error plus predicted variance, divided by twice its preference variance; the six terms are averaged. The first and rest-continuation costs are weighted 1 and 0.6, then averaged.

Useful progress must matter within a short tick

The planner adds 0.5 times the predicted change in mean-only physiological preference cost per simulation second. This prevents a useful bite's small normalized hunger change from being diluted into indifference between actions.

While hungry, it also values predicted approach and alignment with food:

food potential = clamp(proximity, 0, 1) × (1 + 0.25 cos(π × bearing))
food approach cost = 2 × clamp((hunger − 0.15) / 0.5, 0, 1)
                   × (current potential − predicted potential) / 0.05

Approach cost is disabled without a food cue, when satiated, or when food-proximity belief is at least 0.99 (the bite-range cue). Consuming food therefore does not incur an approach penalty for losing the source. Both progress costs use the learned CTRNN forecasts; they add no environment reward and force no action. Negative progress cost means predicted improvement.

Learning differentiates the recurrent dynamics

Only the executed action bank learns from its observed endpoint. Gradients pass through all four integration steps, including recurrent paths through other neurons. Each neuron's bias and 18 weights have a full 19 × 19 covariance block. A local extended-Kalman update learns that block from the neuron's own endpoint. Cross-neuron parameter covariance and state cross-covariance are omitted; information gain uses the neuron's own parameter block. This is an approximation to nonlinear Bayesian learning, not an exact posterior over the entire ecology.

Fractional likelihood weight is clamp(learning rate / 0.02, 0.01, 1). Observation variance is 0.0004. Process variance starts at 0.0025 and adapts within [0.0001, 0.25], separating estimated noise from parameter uncertainty. Information is counted once for the candidate, not again for a correlated hypothetical observation during rest. A zero learning rate freezes weights, covariance, and noise; observation filtering and decisions continue.

Variational free energy measures the Gaussian observation fit and posterior/prior divergence. It can be negative because continuous probability densities can exceed one. It is an inference diagnostic, not the RL reward and not the quantity directly sampled for action selection.

AMBP: Adaptive Model-Based Planning

AMBP is a fourth founder group, available in Introduce and Creator. It uses an engineered hybrid controller: resource-management goals, a personal model of observed obstacles, bounded A* route search, remembered food locations, and adaptive action/motion estimates. It has no feed-forward actor, CTRNN, pretrained PPO checkpoint, or deep world model.

The controller receives the same creature senses, including structured visible objects, eye-ray water contacts, and its own position. It cannot query unseen terrain or world indexes. A seen tree is modeled with a conservative body-clearance circle; blocked water and failed movement add local obstacle observations. Unknown terrain is assumed passable until observation or contact contradicts it. This richer geometric interface differs from RL's compressed numerical actor inputs.

Personal memory holds at most 64 obstacles, 128 visited places, and 32 temporarily avoided food targets. Route search uses a 2-unit grid in a 41 × 41 local region, up to 600 expanded nodes, and whole-segment checks against remembered obstacles. Routes refresh while blocked every three seconds, or earlier after new observations or a blocked next waypoint. This is local planning with incomplete knowledge, not a guarantee of finding every route.

Resource goals favor reachable feeding, useful drinking, reproduction with a reserve, recovery and conservation, escaping nearby predators, dispersing from the birthplace, approaching live or remembered food, and exploring less-visited places. Food targets receive a three-second commitment to prevent ranking jitter from redirecting travel; danger, stalled-goal recovery, and arriving at the source can interrupt it. Four translation directions compete by waypoint progress, directional contact, learned utility, and motion efficiency. The body turns toward each route when heading error exceeds 0.35 radians before normal travel; contact escape can still use a side or reverse step. Finite clockwise visual sweeps collect actual eye rays while exploring or conserving, at most once per eight seconds. A sweep finishes after a physical full turn or is interrupted by a useful goal. Exactly one available action is selected each fixed tick from a one-point categorical distribution. Execution still checks current resources, targets, body exclusion, swimming, and placement.

Six seconds without useful displacement inside a two-unit radius triggers a four-second detour and avoids the failed food target for 20 seconds. Turning, jitter, and feeding from a different source do not renew the failed goal. Navigation probes an uncertain blocked route instead of falling back to indefinite Rest. Recovery rest requires a suitable body temperature, hydration, and energy reserve. Routes approach food's bite region rather than insisting on reaching an occupied source center. The inspector shows sweep angle and the navigation progress timer. Visible or smelled predators remain threats even when the prey classifier also considers them food; a sensed safe plant can replace that risky target. Immediate escape can use a reverse or sideways step without waiting to turn.

utility[action] += learning rate × (outcome reward − utility[action])
motion efficiency[action] learns actual displacement / expected displacement

Utility stays within [−1, 1] and motion efficiency within [0.1, 1]. Outcome reward combines net energy change excluding birth investment, health change, and pending feedback. Zero learning rate freezes these estimates. Bite consequences teach food values. Deferred offspring outcomes update reproduction utility and its reserve threshold. This is local adaptive learning alongside engineered planning, not a learned general policy.

Offspring inherit 14 action utilities, four translation motion estimates, and food knowledge, preserving parameter coordinates and encoded mutations. Maps, routes, goals, visits, and birth-credit records start fresh. A dedicated aligned DNA marker identifies AMBP without reassigning any legacy JEPA/RL/FEP locus-18 values. The Brain view displays the actual observed map, route, target, mode, executed action, and learned estimates instead of neural connections.

Birth, inherited experience, and fresh state

Asexual birth requires the reproduction action and 70% of individual maximum energy: 105 of 150 at average size. Birth also requires age over 18 seconds, no active cooldown, room in the population, and dispersal at least eight units from the creature’s own recorded birthplace. A successful asexual birth starts a 60-second cooldown. Failed placement spends no birth energy. Choosing reproduction is an attempt, not proof of a birth. The Body inspector retains the last successful birth’s child ID and physical energy before, spent, and after. Learning feedback excludes the birth investment from reward but never refunds that energy. A newborn receives at most 52 stored energy, bounded by the payment and its capacity. Body genes and neural genes mutate while preserving founder group.

RL/JEPA transfer learned MLP parameters by layer/unit coordinates, preserving correspondence when widths or depth change, plus auxiliary predictive and food knowledge. FEP imprints recurrent parameters by action/neuron/source coordinates and transfers full-precision learned weights plus changes to the child's imprinted parameter genes. It copies process noise and increases parameter covariance diagonals by 0.001 for body mutation. Existing MLP genome records remain compatible but do not drive FEP decisions.

Newborns start with fresh activations or recurrent beliefs, flow, observation errors, current plans, location memories, curiosity records, eligibility traces, and deferred birth lessons. Knowledge is inherited; the parent's live decision state is not.

A birth supplies 0.25 immediate feedback. Up to eight personal birth contexts await offspring assessment: death before age 60 gives −0.75 credit, and survival to age 60 gives up to +0.75 scaled by health. RL replays the recorded decision with a bounded importance correction. FEP updates the recorded reproduction context's feedback model. Only living parents receive this later lesson. Terminal outcomes teach the last executed action and close its episode.

For FEP, the parent's actual energy transfer is normalized by capacity and removed from the learned net-hunger outcome; filtering adds the known transfer back to physical hunger. This separates deliberate offspring investment from wasted energy. Its physiological reward readout remains telemetry, not a reward-trained actor.

Read the Brain view alongside behavior

Start with the selected action, its probability, and the availability rows. An unavailable action cannot indicate a policy preference. The highlighted output is the action actually sampled, not necessarily the most probable one.

• RL / JEPA: inspect real adjacent-layer connections, biases, logits, critic value, temporal-difference advantage, reward, exploration, and sensory/latent forecast errors. Auxiliary forecasts describe predictions rather than a second actor.
• FEP: the graph displays the chosen action's real recurrent bank, including directed self-connections. Hover neurons for belief, flow per second, uncertainty, precision-weighted observation error, and bias. Action rows show expected free energy, total preference cost, information gain, and actual posterior probability; the stats also show the signed food-approach and physiological-progress costs.
• Both: dense connection drawing shows a subset of up to 220 strong weights, not every connection. Unmodeled FEP raw sensors are labeled “not modeled.” Genome parameters describe inherited instructions; current lifetime learning may differ.

What has been checked—and what has not

Regression checks cover action masks and execution, numerical MLP and recurrent gradients, integrated dynamics, finite resource accounting, parameter inheritance/mutation, fresh newborn state, zero-rate controls, deferred credit, and inspection consistency. They establish mechanics; population outcomes require experiments.

In matched 30-second meadow trials with food 15 units away, the progress-preference change increased FEP food arrivals from 0/24 to 21/24, with feeding actions increasing from 0 to 748. JEPA and RL each reached food in 24/24 cases. In three 60-second dense-food learning runs, FEP mean energy increased from 53.5 to 69.8. These trials used fixed seeds 7, 42, and 77 and disabled reproduction. They do not establish long-term survival, successful reproduction, performance across generated terrain, or parity between learning systems.

A separate three-seed obstacle-food trial measured the updated RL starting connections: arrivals rose from 17/24 to 20/24 and mean final energy from 47.0 to 57.4. In open terrain, both versions reached 24/24; the updated policy took 7.2 rather than 6.2 seconds on average and retained slightly less energy. Three additional held-out seeds showed a smaller obstacle gain, 19/24 to 20/24, with higher energy but slower arrival; one seed regressed. These 30-second trials disable reproduction and do not establish general intelligence or population survival.

FEP's 18-state model omits much sensory detail, has a short conservative planning horizon, and does not perform general pathfinding or a future action-tree search. RL's feed-forward actor relies on external memory inputs and approximate temporal credit. Both can make poor decisions, repeat behaviors, deplete food, or die. Computational cost can also prevent the server from meeting a requested playback rate.

In matched three-seed 30-second obstacle trials, AMBP reached food in 24/24 cases versus current RL's 20/24, averaging 4.43 versus 17.61 seconds to arrival including failures. Open arrival averaged 3.76 versus 7.34 seconds, with both reaching 24/24. No founders died. AMBP retained more energy but traveled farther over the full trials. These comparisons use matched body traits, disable reproduction, and do not prove long-term population survival or universal intelligence. AMBP also uses engineered goals and richer geometric observations, so this is not an isolated learning-algorithm comparison.

Direction for smarter creatures

The recommended next step is shared offline RL training across many simulation worlds, followed by individual adaptation, using AMBP as an additional practical baseline. A curriculum should teach food approach, obstacle detours, depleted-food search, threat avoidance, and reproduction, with separate unseen worlds for evaluation. Longer-term memory and learned planning should be added only when measured tasks justify them. This training system is a proposed direction, not part of the current implementation.

This page describes the source bundled into a build. An older running process keeps its embedded code and documentation until restarted. Restarting creates a new world; browser view pause never stops the server.