CleverBirds

Links: 🌐 Website | 📄 Paper | 💾 Data | ⚖️ Model Weights

CleverBirds is a large-scale knowledge tracing benchmark for fine-grained bird species recognition, collected via the eBird citizen-science platform, containing over 17 million multiple-choice questions from 40,000+ participants across 10,000+ bird species, with an average of 400 questions per participant.

Repository Structure

• code/ - Main experiment code
  • configs/ - Training and evaluation configuration files
  • notebooks/ - Analysis and plotting notebooks
  • scripts/ - Training and evaluation scripts
  • src/ - Core implementation code
• mappings/ - Species-to-token mappings and eBird taxonomy files
• pykt_fork/ - Modified pyKT library for knowledge tracing baselines

Installation

Set up a conda environment and install dependencies:

cd code
conda create -n machine_teaching python=3.11
conda activate machine_teaching
conda install pip
pip install uv
uv pip install -r requirements.txt

Data Setup

Downloading and Preprocessing

Download the dataset from HuggingFace and preprocess it for training:

cd code/scripts/preprocessing/processing_cleaned/

# Step 1: Download dataset
python3 step_0_load_from_hf.py --save_path PATH_TO_DATASET

# Step 2: Reorder images
python3 step_1_reorder_imgs.py --local_dir PATH_TO_DATASET --testing False

# Step 3: Add feature columns
python3 step_2_add_feature_columns.py \
    --local_dir PATH_TO_DATASET \
    --logging_dir ./logs \
    --mappings_path ../../mappings \
    --testing False

Replace PATH_TO_DATASET with your desired output directory. The preprocessing creates:

• participant_data/ - User interaction data (train/val/test splits)
• image_features/ - Pre-extracted image features
• reordered_imgs/ - Reordered image files

Dataset Format

Participant Data (participant_data/train.parquet, val.parquet, test.parquet):

• question_id - Question identifier
• quiz_id - Quiz identifier (20 questions per quiz)
• asset_id - Image identifier
• user_id - Anonymized user identifier
• user_interact_count - Chronological question index per user
• species_code - Correct bird species code
• choices - List of candidate choices (last entry is "None of the Above")
• user_answer - User's selected answer
• correct - Binary correctness indicator
• labels - Answer index (4 for NOTA)
• hex3 - Hex3 aggregated location
• week - Calendar week

Image Features (image_features/chunk_IDX.npz):

• NPZ files containing pre-extracted image features
• Match to participant data using asset_id

Training Models

Main Experiments

Training configurations are in code/configs/. To train a model:

cd code/scripts
python3 train.py \
    --base_config ../configs/base/base.yaml \
    --experiment_config ../configs/baselines_dino.yaml

We provide sample configurations, exact model configs can be found in the model weights files. Example configuration files:

• base/base.yaml - Base configuration
• base/base_species_pred.yaml - Species prediction baseline
• incontext_deepseek_readable.yaml - In-context learning with DeepSeek
• incontext_openai_readable_stats.yaml - In-context learning with OpenAI
• baselines_dino.yaml - DINO baseline
• s2s/t5_tiny_eff.yaml - Sequence-to-sequence T5 model
• mc/tinybert.yaml - Multiple choice BERT model

Evaluation

Evaluate trained models:

cd code/scripts
python3 eval.py \
    --base_config ../configs/base/base.yaml \
    --experiment_config ../configs/incontext_deepseek_readable.yaml

Note: Scripts use wandb for experiment tracking. Ensure wandb is configured or disable it in the config files.

Knowledge Tracing Baselines

Baseline knowledge tracing models (DKT, simpleKT, AKT, etc.) are implemented using a modified version of pyKT.

Model Weights

Pre-trained model weights are available on HuggingFace.

Citation

If you use this code or dataset, please cite the CleverBirds paper:

inproceedings{bossemeyercleverbirds,
  title={CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing},
  author={Bossemeyer, Leonie and Heinrich, Samuel and Van Horn, Grant and Mac Aodha, Oisin},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track}
}