DNN Challenge 2: public leaderboard

Round 1 leaderboard

This is the initial leaderboard, and submissions have not been checked carefully for correctness. You can use this information to improve your models. Performance on this leaderboard will not determine your final score.

Evaluated on test_public.npz (1,000 sequences): top-1 next-token accuracy over targets x[32:1024]. Ranked by public accuracy, then public cross-entropy, then fewer parameters, then earlier submission. Each team’s last valid zip counts. 12 teams ranked from 13 zips.

Leaderboard

Rank Team Accuracy CE (nats) Val acc Val CE Params Evaluated Warnings Submitted Zip
1 ML_Mavericks 0.7833 0.7616 0.7806 0.7722 49,576 ONNX (model.onnx)   2026-11-11 23:50:56 ML_Mavericks.zip
2 tensors 0.7780 0.7837 0.7710 0.8076 47,057 ONNX (tensors.onnx)   2026-11-12 00:10:56 tensors.zip
3 Illegal_Activations 0.7746 0.7904 0.7677 0.8139 46,545 model.py + weights.pt   2026-11-11 23:17:30 Illegal_Activations.zip
4 Santa_Claude 0.7708 0.8026 0.7650 0.8230 49,441 notebook + state dict   2026-11-11 22:44:02 Santa_Claude.zip
5 TokenLimitReached 0.7689 0.8129 0.7631 0.8335 48,721 model.py + weights.pt   2026-11-11 23:19:08 TokenLimitReached.zip
6 Team_Transformer 0.7051 1.0366 0.7004 1.0502 47,315 model.py + weights.pt   2026-11-11 22:00:32 Team_Transformer.zip
7 QuadraMinds 0.6971 1.0718 0.6891 1.0998 41,912 model.py + weights.pt   2026-11-11 23:58:46 QuadraMinds.zip
8 Akatsuki 0.6473 1.2341 0.6372 1.2687 49,274 model.py + weights.pt   2026-11-11 23:12:34 Akatsuki.zip
9 Learners 0.6302 1.3160 0.6196 1.3561 48,868 ONNX (Learners.onnx)   2026-11-12 02:42:12 Learners.zip
10 Tensor_Titans 0.6082 1.3752 0.5937 1.4280 44,272 ONNX (Tensor_Titans.onnx)   2026-11-11 23:57:38 Tensor_Titans.zip
11 the_tuning_forks 0.6003 1.4021 0.5899 1.4288 49,600 model.py + weights.pt   2026-11-11 23:45:28 the_tuning_forks.zip
12 dark_knights 0.3385 2.4617 0.3279 2.5049 33,984 model.py + weights.pt   2026-11-11 20:16:26 dark_knights.zip

Reference points on the same sequences (not submissions): repeat last token 0.3078; state genie 0.9528.

Val acc / Val CE: the participants’ val.npz (1,000 sequences, same targets). It is not held out: teams may train on it, so it is reported for comparison and not used for ranking.

Failed submissions (0)

Warnings: allowed, for organizer review (0)

Superseded zips (1)

  • Team_Transformer (2).zip (submitted 2026-11-11 21:56:08, public accuracy 0.7051): team Team_Transformer has a later valid zip

Public test set: accuracy by slot position (all teams)

Also provided are lower and upper bounds on performance: repeat last token does exactly what it says (repeats the last seen token, and any reasonable model should beat this); state genie has access to the entire internal state, so no algorithm with access to only past tokens can beat this.

Rank Team 32-63 64-127 128-255 256-511 512-767 768-1023
1 ML_Mavericks 0.557 0.681 0.748 0.773 0.805 0.843
2 tensors 0.580 0.698 0.749 0.766 0.792 0.836
3 Illegal_Activations 0.581 0.695 0.743 0.761 0.789 0.834
4 Santa_Claude 0.587 0.705 0.750 0.762 0.781 0.819
5 TokenLimitReached 0.593 0.706 0.746 0.754 0.779 0.823
6 Team_Transformer 0.562 0.651 0.682 0.691 0.712 0.756
7 QuadraMinds 0.539 0.636 0.675 0.688 0.704 0.745
8 Akatsuki 0.540 0.616 0.638 0.634 0.645 0.690
9 Learners 0.516 0.583 0.603 0.609 0.634 0.688
10 Tensor_Titans 0.528 0.600 0.606 0.591 0.597 0.649
11 the_tuning_forks 0.531 0.598 0.604 0.584 0.588 0.636
12 dark_knights 0.347 0.355 0.348 0.316 0.319 0.370
- repeat last token 0.347 0.337 0.316 0.286 0.289 0.332
- state genie 0.952 0.952 0.952 0.954 0.953 0.952

Public test set: accuracy by number of hoppers (all teams)

Rank Team 0 (99 seqs) 1 (504 seqs) 2 (397 seqs)
1 ML_Mavericks 0.861 0.817 0.721
2 tensors 0.878 0.830 0.688
3 Illegal_Activations 0.875 0.831 0.677
4 Santa_Claude 0.870 0.824 0.678
5 TokenLimitReached 0.872 0.821 0.677
6 Team_Transformer 0.795 0.743 0.635
7 QuadraMinds 0.803 0.749 0.605
8 Akatsuki 0.767 0.699 0.552
9 Learners 0.762 0.692 0.519
10 Tensor_Titans 0.729 0.680 0.487
11 the_tuning_forks 0.715 0.670 0.483
12 dark_knights 0.651 0.376 0.213
- repeat last token 0.653 0.332 0.191
- state genie 0.943 0.951 0.957

Public test set: accuracy by number of CSMA users (all teams)

Rank Team 0 (104 seqs) 1 (304 seqs) 2 (296 seqs) 3 (193 seqs) 4 (103 seqs)
1 ML_Mavericks 0.843 0.818 0.778 0.741 0.716
2 tensors 0.838 0.809 0.772 0.737 0.720
3 Illegal_Activations 0.829 0.805 0.768 0.735 0.723
4 Santa_Claude 0.833 0.803 0.765 0.726 0.714
5 TokenLimitReached 0.826 0.799 0.763 0.728 0.716
6 Team_Transformer 0.787 0.741 0.700 0.649 0.634
7 QuadraMinds 0.765 0.731 0.692 0.647 0.637
8 Akatsuki 0.717 0.678 0.642 0.599 0.592
9 Learners 0.691 0.665 0.621 0.585 0.576
10 Tensor_Titans 0.679 0.639 0.601 0.560 0.559
11 the_tuning_forks 0.665 0.630 0.594 0.555 0.552
12 dark_knights 0.371 0.346 0.337 0.319 0.325
- repeat last token 0.324 0.308 0.308 0.299 0.306
- state genie 0.982 0.965 0.953 0.936 0.920

Per-team breakdown

1. ML_Mavericks

  • Zip: ML_Mavericks.zip, submitted 2026-11-11 23:50:56
  • Evaluated: ONNX (model.onnx)
  • Parameters: 49,576
  • Note: also in the zip, not used: ML_Mavericks.ipynb, model.py, train.py, weights.pt
  • Validation (val.npz, not held out): accuracy 0.7806, cross-entropy 0.7722 nats; public minus val accuracy +0.0027

Public test set (1,000 sequences)

  • Accuracy: 0.7833
  • Cross-entropy: 0.7616 nats
  • Rank: 1 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.557 0.347 0.952
    64-127 0.681 0.337 0.952
    128-255 0.748 0.316 0.952
    256-511 0.773 0.286 0.954
    512-767 0.805 0.289 0.953
    768-1023 0.843 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.861 0.943
    1 504 0.817 0.951
    2 397 0.721 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.843 0.982
    1 304 0.818 0.965
    2 296 0.778 0.953
    3 193 0.741 0.936
    4 103 0.716 0.920

2. tensors

  • Zip: tensors.zip, submitted 2026-11-12 00:10:56
  • Evaluated: ONNX (tensors.onnx)
  • Parameters: 47,057
  • Note: also in the zip, not used: tensors.ipynb
  • Validation (val.npz, not held out): accuracy 0.7710, cross-entropy 0.8076 nats; public minus val accuracy +0.0070

Public test set (1,000 sequences)

  • Accuracy: 0.7780
  • Cross-entropy: 0.7837 nats
  • Rank: 2 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.580 0.347 0.952
    64-127 0.698 0.337 0.952
    128-255 0.749 0.316 0.952
    256-511 0.766 0.286 0.954
    512-767 0.792 0.289 0.953
    768-1023 0.836 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.878 0.943
    1 504 0.830 0.951
    2 397 0.688 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.838 0.982
    1 304 0.809 0.965
    2 296 0.772 0.953
    3 193 0.737 0.936
    4 103 0.720 0.920

3. Illegal_Activations

  • Zip: Illegal_Activations.zip, submitted 2026-11-11 23:17:30
  • Evaluated: model.py + weights.pt
  • Parameters: 46,545
  • Validation (val.npz, not held out): accuracy 0.7677, cross-entropy 0.8139 nats; public minus val accuracy +0.0069

Public test set (1,000 sequences)

  • Accuracy: 0.7746
  • Cross-entropy: 0.7904 nats
  • Rank: 3 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.581 0.347 0.952
    64-127 0.695 0.337 0.952
    128-255 0.743 0.316 0.952
    256-511 0.761 0.286 0.954
    512-767 0.789 0.289 0.953
    768-1023 0.834 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.875 0.943
    1 504 0.831 0.951
    2 397 0.677 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.829 0.982
    1 304 0.805 0.965
    2 296 0.768 0.953
    3 193 0.735 0.936
    4 103 0.723 0.920

4. Santa_Claude

  • Zip: Santa_Claude.zip, submitted 2026-11-11 22:44:02
  • Evaluated: notebook + state dict
  • Parameters: 49,441
  • Note: model.py and train.py taken from the %%writefile cells of Santa_Claude.ipynb; weights from Santa_Claude.pt
  • Validation (val.npz, not held out): accuracy 0.7650, cross-entropy 0.8230 nats; public minus val accuracy +0.0058

Public test set (1,000 sequences)

  • Accuracy: 0.7708
  • Cross-entropy: 0.8026 nats
  • Rank: 4 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.587 0.347 0.952
    64-127 0.705 0.337 0.952
    128-255 0.750 0.316 0.952
    256-511 0.762 0.286 0.954
    512-767 0.781 0.289 0.953
    768-1023 0.819 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.870 0.943
    1 504 0.824 0.951
    2 397 0.678 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.833 0.982
    1 304 0.803 0.965
    2 296 0.765 0.953
    3 193 0.726 0.936
    4 103 0.714 0.920

5. TokenLimitReached

  • Zip: TokenLimitReached.zip, submitted 2026-11-11 23:19:08
  • Evaluated: model.py + weights.pt
  • Parameters: 48,721
  • Validation (val.npz, not held out): accuracy 0.7631, cross-entropy 0.8335 nats; public minus val accuracy +0.0058

Public test set (1,000 sequences)

  • Accuracy: 0.7689
  • Cross-entropy: 0.8129 nats
  • Rank: 5 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.593 0.347 0.952
    64-127 0.706 0.337 0.952
    128-255 0.746 0.316 0.952
    256-511 0.754 0.286 0.954
    512-767 0.779 0.289 0.953
    768-1023 0.823 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.872 0.943
    1 504 0.821 0.951
    2 397 0.677 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.826 0.982
    1 304 0.799 0.965
    2 296 0.763 0.953
    3 193 0.728 0.936
    4 103 0.716 0.920

6. Team_Transformer

  • Zip: Team_Transformer.zip, submitted 2026-11-11 22:00:32
  • Evaluated: model.py + weights.pt
  • Parameters: 47,315
  • Validation (val.npz, not held out): accuracy 0.7004, cross-entropy 1.0502 nats; public minus val accuracy +0.0046

Public test set (1,000 sequences)

  • Accuracy: 0.7051
  • Cross-entropy: 1.0366 nats
  • Rank: 6 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.562 0.347 0.952
    64-127 0.651 0.337 0.952
    128-255 0.682 0.316 0.952
    256-511 0.691 0.286 0.954
    512-767 0.712 0.289 0.953
    768-1023 0.756 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.795 0.943
    1 504 0.743 0.951
    2 397 0.635 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.787 0.982
    1 304 0.741 0.965
    2 296 0.700 0.953
    3 193 0.649 0.936
    4 103 0.634 0.920

7. QuadraMinds

  • Zip: QuadraMinds.zip, submitted 2026-11-11 23:58:46
  • Evaluated: model.py + weights.pt
  • Parameters: 41,912
  • Validation (val.npz, not held out): accuracy 0.6891, cross-entropy 1.0998 nats; public minus val accuracy +0.0081

Public test set (1,000 sequences)

  • Accuracy: 0.6971
  • Cross-entropy: 1.0718 nats
  • Rank: 7 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.539 0.347 0.952
    64-127 0.636 0.337 0.952
    128-255 0.675 0.316 0.952
    256-511 0.688 0.286 0.954
    512-767 0.704 0.289 0.953
    768-1023 0.745 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.803 0.943
    1 504 0.749 0.951
    2 397 0.605 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.765 0.982
    1 304 0.731 0.965
    2 296 0.692 0.953
    3 193 0.647 0.936
    4 103 0.637 0.920

8. Akatsuki

  • Zip: Akatsuki.zip, submitted 2026-11-11 23:12:34
  • Evaluated: model.py + weights.pt
  • Parameters: 49,274
  • Validation (val.npz, not held out): accuracy 0.6372, cross-entropy 1.2687 nats; public minus val accuracy +0.0101

Public test set (1,000 sequences)

  • Accuracy: 0.6473
  • Cross-entropy: 1.2341 nats
  • Rank: 8 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.540 0.347 0.952
    64-127 0.616 0.337 0.952
    128-255 0.638 0.316 0.952
    256-511 0.634 0.286 0.954
    512-767 0.645 0.289 0.953
    768-1023 0.690 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.767 0.943
    1 504 0.699 0.951
    2 397 0.552 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.717 0.982
    1 304 0.678 0.965
    2 296 0.642 0.953
    3 193 0.599 0.936
    4 103 0.592 0.920

9. Learners

  • Zip: Learners.zip, submitted 2026-11-12 02:42:12
  • Evaluated: ONNX (Learners.onnx)
  • Parameters: 48,868
  • Note: also in the zip, not used: ee6380_challenge2_Learners.ipynb, model.py, train.py, weights.pt
  • Validation (val.npz, not held out): accuracy 0.6196, cross-entropy 1.3561 nats; public minus val accuracy +0.0107

Public test set (1,000 sequences)

  • Accuracy: 0.6302
  • Cross-entropy: 1.3160 nats
  • Rank: 9 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.516 0.347 0.952
    64-127 0.583 0.337 0.952
    128-255 0.603 0.316 0.952
    256-511 0.609 0.286 0.954
    512-767 0.634 0.289 0.953
    768-1023 0.688 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.762 0.943
    1 504 0.692 0.951
    2 397 0.519 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.691 0.982
    1 304 0.665 0.965
    2 296 0.621 0.953
    3 193 0.585 0.936
    4 103 0.576 0.920

10. Tensor_Titans

  • Zip: Tensor_Titans.zip, submitted 2026-11-11 23:57:38
  • Evaluated: ONNX (Tensor_Titans.onnx)
  • Parameters: 44,272
  • Note: also in the zip, not used: Tensor_Titans.ipynb
  • Validation (val.npz, not held out): accuracy 0.5937, cross-entropy 1.4280 nats; public minus val accuracy +0.0145

Public test set (1,000 sequences)

  • Accuracy: 0.6082
  • Cross-entropy: 1.3752 nats
  • Rank: 10 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.528 0.347 0.952
    64-127 0.600 0.337 0.952
    128-255 0.606 0.316 0.952
    256-511 0.591 0.286 0.954
    512-767 0.597 0.289 0.953
    768-1023 0.649 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.729 0.943
    1 504 0.680 0.951
    2 397 0.487 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.679 0.982
    1 304 0.639 0.965
    2 296 0.601 0.953
    3 193 0.560 0.936
    4 103 0.559 0.920

11. the_tuning_forks

  • Zip: the_tuning_forks.zip, submitted 2026-11-11 23:45:28
  • Evaluated: model.py + weights.pt
  • Parameters: 49,600
  • Validation (val.npz, not held out): accuracy 0.5899, cross-entropy 1.4288 nats; public minus val accuracy +0.0104

Public test set (1,000 sequences)

  • Accuracy: 0.6003
  • Cross-entropy: 1.4021 nats
  • Rank: 11 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.531 0.347 0.952
    64-127 0.598 0.337 0.952
    128-255 0.604 0.316 0.952
    256-511 0.584 0.286 0.954
    512-767 0.588 0.289 0.953
    768-1023 0.636 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.715 0.943
    1 504 0.670 0.951
    2 397 0.483 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.665 0.982
    1 304 0.630 0.965
    2 296 0.594 0.953
    3 193 0.555 0.936
    4 103 0.552 0.920

12. dark_knights

  • Zip: dark_knights.zip, submitted 2026-11-11 20:16:26
  • Evaluated: model.py + weights.pt
  • Parameters: 33,984
  • Validation (val.npz, not held out): accuracy 0.3279, cross-entropy 2.5049 nats; public minus val accuracy +0.0106

Public test set (1,000 sequences)

  • Accuracy: 0.3385
  • Cross-entropy: 2.4617 nats
  • Rank: 12 of 12 teams
  • Accuracy by slot position
    Target slots Accuracy Repeat last token State genie
    32-63 0.347 0.347 0.952
    64-127 0.355 0.337 0.952
    128-255 0.348 0.316 0.952
    256-511 0.316 0.286 0.954
    512-767 0.319 0.289 0.953
    768-1023 0.370 0.332 0.952
  • Accuracy by number of hoppers
    Hoppers Sequences Accuracy State genie
    0 99 0.651 0.943
    1 504 0.376 0.951
    2 397 0.213 0.957
  • Accuracy by number of CSMA users
    CSMA users Sequences Accuracy State genie
    0 104 0.371 0.982
    1 304 0.346 0.965
    2 296 0.337 0.953
    3 193 0.319 0.936
    4 103 0.325 0.920

Date: 2026-11-12

Author: shashank

Created: 2026-11-12 Thu 15:26