Minecraft behavioural cloning

What I set out to check

VPT's claim is that a policy can be learned from video alone, given enough labelled play to train an inverse dynamics model that pseudo-labels the rest. The scale of that is out of reach here, so the question I could actually ask was the smaller one underneath it: how much behaviour does a straightforward vision model pick up from a modest amount of directly labelled play, when the labels are exact rather than inferred?

Recording my own gameplay is what makes that tractable. Every frame arrives with the action that produced it, so the inverse dynamics stage disappears and what remains is a supervised learning problem — the part of VPT it is possible to test on one machine.

What I built

A recording pipeline and a training pipeline. The recorder drives Minecraft through MineRL and captures frames at 256×256 alongside the actions taken, writing video and JSON action traces; a surrounding set of scripts converts, cleans and validates them so frames and actions stay aligned, which is where most of the effort actually goes. The recording script is adapted from an existing VPT implementation rather than written from scratch — the training side is mine.

The model is a residual network taking twenty stacked frames as input, sixty channels of RGB, through five stages of residual blocks — eighteen blocks in total, widening 64 to 1024 — with a global average pool and a single linear head emitting eleven binary actions: forward, back, left, right, sprint, jump, attack, and four camera directions. Treating actions as independent binaries rather than one joint category makes the loss binary cross-entropy with logits, and lets the model express combinations — moving forward while turning — that a softmax over a joint action space would have to enumerate. Trained with Adam at 1e-3, batch size 16.

What went well was the data plumbing, which is unglamorous and was most of the work. Frame-action alignment is the failure mode that quietly ruins a behavioural cloning run, and the validation scripts exist because that is worth checking rather than assuming.

Where it diverged

The substitution at the centre of this is not a small one. VPT trains an inverse dynamics model on labelled contractor play and uses it to pseudo-label a very large corpus of unlabelled video; the scale is what makes the method work. Training directly on recorded play swaps that for a dataset that is exact but tiny. So this tests whether the architecture learns from clean labels, not whether VPT's central claim about learning from unlabelled video holds — which is the interesting part of the paper and the part not reproduced here.

Nothing in the repository records how much play was captured, or on what task. For a behavioural cloning result that number matters more than the architecture does, and its absence is the single biggest gap in the writeup.

There is no reported accuracy, held-out or otherwise, and the reason is structural rather than an oversight in reporting: the training script holds out nothing. One loader covers the whole dataset, the loop tracks average training loss, and no evaluation runs at any point. Without a validation split there is no way to tell learning from memorisation, which is the first thing a reader should want to know.

The loss compounds that. Binary cross-entropy over eleven heads, unweighted, against controls that are mostly inactive on any given frame — predicting "no action" everywhere is a strong local optimum and a low loss. A positive-class weight is the standard correction and is not applied, so the training signal itself is biased toward inaction.

Nor is there an RL finetuning stage: the repository is behavioural cloning only, and no reinforcement learning code for this project exists anywhere on my machine. VPT's second stage, the one that turns a policy that imitates into a policy that improves, was never built.

What I would do next

Report a result, with baselines. Eleven binary heads make raw accuracy easy to misread — a policy that predicts "no action" on every head scores well when controls are sparse — so held-out action accuracy only means something beside a random baseline and a majority-class one. That comparison is what would turn this from a description of an architecture into a claim about whether it learned anything.

Record the dataset size and the task alongside it, since neither number currently exists and neither result is interpretable without them.

Then the RL finetune, which is the step that separates imitating from improving, and the natural continuation if the cloned policy turns out to be worth finetuning at all.