Alex Goulielmos agouliel
N8python
/ pretrain.py
Created
January 15, 2025 23:41
Simple character-level pretraining in MLX. Gets a roughly billion tokens/day for an 18M parameter model on one M3 Max.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import json | |
| import random | |
| import mlx.optimizers as optim | |
| import mlx.core as mx | |
| import mlx.nn as nn | |
| import numpy as np | |
| from tqdm import tqdm | |
| import time | |
| from datetime import datetime |