DNN Challenge 2: Predicting spectrum occupancy with a small sequence model

The objective is to design and train a small causal sequence model (RNN, transformer, state-space model or a hybrid) that predicts which of 6 frequency sub-bands will be busy in the next time slot, given everything a spectrum sensor has observed so far. The model may have at most 50,000 parameters, it is trained from scratch, and you submit the model code, its weights and your training script.

A simple baseline recurrent network is provided. Doing much better requires a model that understands how the transmitters sharing the band behave.

1. The task

A cognitive-radio receiver monitors a band split into 6 sub-bands (numbered 0–5). In every sensing slot it decides, for each sub-band, whether the sub-band is busy or idle. The 6 decisions of one slot are packed into one token:

\[ x_t = \sum_{b=0}^{5} \mathrm{busy}_b(t)\, 2^b \quad\in\{0,\dots,63\}. \]

For example, token 9 = 0b001001 means that sub-bands 0 and 3 are busy and the others are idle.

Each sequence has 1024 slots: \(x_0, \dots, x_{1023}\). At every slot \(t\), your model sees \(x_0, \dots, x_t\) and outputs 64 scores for the next token \(x_{t+1}\). This is next-token prediction, as in a language model, with a vocabulary of 64.

sequence_examples.png

Figure 1: The first 256 slots of three training sequences. Black means busy. The title of each panel shows the first tokens of that sequence.

2. The data

2.1 The dataset

The sequences are synthetic, made to resemble a shared band such as an unlicensed ISM band.

  • Transmitters. In each sequence, several transmitters of different kinds share the 6 sub-bands. Typical kinds of user in such a band include:
    • scheduled links that repeat a frame structure;
    • pulsed emitters;
    • radios that hop between sub-bands;
    • packet traffic that contends for access and defers to other users.
  • What is not disclosed. The number of transmitters, their kinds, and all of their timing parameters (periods, durations, hop patterns, traffic rates, …) are drawn afresh for every sequence. They are not given to you, and some transmitters start or stop partway through a sequence.
  • Sensing errors. The sensor is imperfect. Every busy/idle decision is wrong with a small probability, which differs between sub-bands and between sequences. The tokens you get, and the tokens you are scored against, are the sensed decisions, errors included.

Some of each sequence is predictable: the structure is there to be found if the model can work out each transmitter's pattern from the slots so far. Some of it is not, for example a packet starting at a random time or a sensing error. No model reaches 100%.

2.2 Files

Necessary files can be found here. The data files consist of two npz files:

File Sequences Tokens Contents
train.npz 8,000 8.2 M x
val.npz 1,000 1.0 M x
Field Shape, dtype Meaning
x (N, 1024) uint8 tokens, values 0–63; row i is one sequence

Load the data with x = np.load("train.npz")["x"]. The sequences are independent of one another: nothing carries over from one row to the next.

2.3 Test data (not distributed)

The organisers hold 3,000 test sequences drawn from the same distribution. They are never released, and the organisers run your model on them:

  • Public test set (1,000 sequences): scores the live leaderboard during the challenge.
  • Private test set (2,000 sequences): decides the final ranking.

3. Model rules

3.1 Interface

Your model is a PyTorch nn.Module that maps a batch of token sequences to scores:

  Shape, dtype Meaning
input (B, L) int64 tokens x[:, 0:L], with L ≤ 1024
output (B, L, 64) float32 out[:, t] are 64 scores (logits) for the token at slot t + 1

The output at the last position, out[:, L-1], is never scored. Return raw scores, without a softmax.

3.2 Causality

out[:, t] may depend only on x[:, 0..t]. Concretely:

  • no bidirectional layers and no non-causal convolutions;
  • no attention to future positions;
  • nothing that pools over the whole sequence.

The checker tests this directly:

  • it changes tokens at several positions and verifies that no earlier output changes;
  • it verifies that the outputs on a prefix x[:, :L] equal the first L outputs on the full sequence;
  • batch elements must not interact either, so it verifies that a sequence gives the same output alone as inside a batch (of 4 and of 16, the batch size the organisers score with).

The organisers repeat the causality test on every batch of test sequences they score. A submission that fails it there is rejected, even if it passed the checker.

Where your model supports it, these tests run in float64 (model.double()), so that float32 rounding does not hide a small leak. Differences above 1e-4 fail. Smaller differences that are still too large for float64 are flagged for review; they are usually a hard-coded .float() cast inside the model.

3.3 Parameter budget

At most 50,000 parameters: every element of every tensor in your submitted weights file counts, whatever its dtype and whatever it is for. The only exception is BatchNorm's num_batches_tracked counter. This includes:

  • embeddings, output layers and biases;
  • normalisation weights;
  • BatchNorm running statistics, if you use BatchNorm.

Fixed, non-learned tensors (positional-encoding tables, causal masks, bit-unpacking tables) are free, provided your model code builds them in __init__ from a formula, registers them as non-persistent buffers (register_buffer(..., persistent=False)), and does not load them from the weights file. "From a formula" means deterministic: random tensors are not free, seeded or not (e.g. a frozen random projection or a random recurrent matrix). If you want one, make it a parameter or a persistent buffer, so that it is in weights.pt and counts. The checker builds your model twice with different seeds and fails it if any tensor outside weights.pt changes. It also flags tensors stored as plain attributes (self.t = torch.zeros(...)) for review.

Weight sharing. A weight may be applied at every time step, which is what a recurrent layer, a convolution or attention does anyway. Across depth, however, each layer's weights may be used by one layer call per time step: no looping a block several times, and no reusing one layer at several depths. The one exception is tying the input embedding to the output layer.

3.4 Allowed building blocks (techniques from the course)

You may combine any of the following, wired however you like:

  • Input and output:
    • token embeddings;
    • fixed re-encodings of the token, such as unpacking it into its 6 busy/idle bits;
    • linear output layers;
    • output scores assembled from per-sub-band scores.
  • Feed-forward: linear layers and MLPs; activations (ReLU, GELU, SiLU, tanh, sigmoid, …); gated linear units (GLU, SwiGLU).
  • Convolutions: causal 1-D convolutions, including dilated, depthwise or grouped ones.
  • Recurrent: RNN, LSTM, GRU, unidirectional and possibly stacked.
  • Attention:
    • causal self-attention, single- or multi-head, including grouped-query and linear attention;
    • positional information: sinusoidal, learned or rotary (RoPE).
  • State-space models: S4/S4D (with HiPPO-style initialisation), selective SSMs (Mamba-style), and Mamba-2/SSD. You must implement them in plain PyTorch.
  • Mixture of experts.
  • Structure and regularisation: residual connections; LayerNorm, RMSNorm or BatchNorm; dropout; weight decay.

3.5 Not allowed

  • Hand-coded prediction algorithms. No component may compute statistics of the history outside learned layers. This includes:

    • n-gram or match counting;
    • lookup tables keyed on past tokens;
    • autocorrelation or period detection;
    • explicit Bayesian filters.

    Fixed re-encodings of the current input (§3.4) are fine; anything that "remembers" the past must be a learned layer from §3.4.

  • External model libraries or custom kernels, e.g. mamba_ssm, flash_attn, xformers or Triton/CUDA extensions. Only torch, numpy and the Python standard library may be imported.
  • Test-time training. No gradient updates or other weight changes at inference.
  • Several networks: ensembles, or several models averaged at inference. Weight averaging within one training run (EMA, SWA) is fine, because the result is one network.

3.6 Training rules

  • Data. Train only on train.npz and val.npz, which you may use in any way, including training on val.npz. No other data: neither real data nor synthetic data you generate yourself.
  • From scratch. No pretrained weights, and no knowledge distillation from any other model.
  • Compute. Your full training run must finish in at most 20 minutes on one Colab T4 GPU, counting every epoch you train for and excluding data loading.
  • Reproducibility. Your train.py (§4) must reproduce your submitted weights from train.npz / val.npz, up to random-seed noise.
  • Free choices: optimiser, schedule, loss, sequence cropping, curriculum and regularisation.

The organisers verify the model rules (§3.1–3.5) with the checker and by reading model.py. After the deadline they retrain the top submissions with their train.py on a T4. A result that is far from the submitted accuracy, or a run over the time limit, is disqualified.

4. Submission format

Submit one zip file, <team_name>.zip, of at most 5 MB, containing these three files and nothing else:

File Contents
model.py defines build_model() -> torch.nn.Module and nothing that runs on import. It may import only torch, numpy, math and the standard library
weights.pt torch.save(model.state_dict(), "weights.pt"): the trained weights, loaded with strict=True
train.py the script that produced weights.pt from train.npz and val.npz (§3.6)

How the organisers run it. They call build_model(), load weights.pt, call model.eval(), and run each test sequence through the model in full (L = 1024) under torch.no_grad().

Check before you submit. Run the provided checker. It is the same script the organisers use: 

python check_submission.py <team_name>.zip --val val.npz

It reports:

  • the parameter count;
  • the causality, prefix and batch-independence tests;
  • the forbidden-import scan (dynamic tricks such as getattr are flagged for review);
  • that fixed tensors outside weights.pt do not depend on the random seed;
  • a count of how often each layer with weights is called per forward pass. Each such layer should be called once, or once per time step; anything else is flagged as possible weight re-use across depth;
  • PASS or FAIL with the reasons;
  • your accuracy and cross-entropy on val.npz.

Submissions that do not PASS are rejected. Warnings do not reject a submission, but the organisers read the code behind each one.

5. Evaluation and ranking

  • Metric: top-1 next-token accuracy, averaged over all test sequences and over the targets \(x_{32},\dots,x_{1023}\) (i.e. outputs out[:, 31:1023]). The first 32 targets are not scored, so that every model has seen a little of each sequence first.
  • Live leaderboard: accuracy on the public test set.
  • Final ranking: accuracy on the private test set. Ties are broken in this order:
    1. lower mean cross-entropy on the same targets (log-softmax of your scores, in nats);
    2. fewer parameters;
    3. earlier submission time.
  • Which submission counts: your last valid submission before the deadline.

For reference, on val.npz:

  • repeating the last token (predicting \(x_{t+1}=x_t\)) scores about 30%;
  • the baseline in the template notebook, a one-layer GRU with 46k parameters, scores about 38%. It trains in a minute or two on a GPU, and more epochs do not help it.

6. Provided files

File What it is
PROBLEM_STATEMENT.md this document
train.npz, val.npz the data (§2.2)
template.ipynb starter notebook: loads the data, trains the baseline, writes the three submission files and runs the checker
check_submission.py the official rule checker and validation scorer (needs torch, numpy)
sequence_examples.png the figure in §1

Software: Python ≥ 3.10, PyTorch ≥ 2.5 and numpy. A free Colab T4 trains the baseline in couple of minutes.

7. FAQ

Can I train on the validation set?
Yes. Holding some data out to choose your final model is still a good idea.

Can I unpack each token into its 6 bits and treat the sub-bands separately?
Yes. That is a fixed re-encoding of the input (§3.4). How you then model the sub-bands, separately, jointly or with shared weights, is up to you.

Can I train on crops shorter than 1024?
Yes. At test time your model receives the full 1024-slot sequence, so make sure it handles that length.

Can my model keep a state that grows with the sequence, such as an attention KV cache?
Yes. Only causality matters, not how much memory the model uses.

Is a fixed positional-encoding table counted as parameters?
No, as long as it is built from a formula in __init__ and is not stored in weights.pt (§3.3).

The checker warns that my outputs change "within float32 rounding". Is my model non-causal?
Probably not. The warning usually means your model casts to float32 somewhere (for example u.float()), so the float64 test cannot rule out a tiny leak. Use .to(x.dtype) or leave dtypes alone and the warning disappears.

Can I use torch.compile, mixed precision or gradient checkpointing?
Yes, for training. The submitted model must also run in plain float32 on CPU.

Do I need a GPU?
Training is much faster on one (e.g. free Colab). Evaluation runs on CPU.

Author: shashank

Created: 2026-11-03 Tue 15:33