AI Experiment Management: Checkpoints, Logs & Python
Manage a PyTorch experiment with checkpoints, virtual environments, and logs. Use practical commands to inspect runs and avoid common workflow mistakes.
A Practical Workflow for Managing Your First AI Experiment
AI Experiment Management: Checkpoints, Logs & Python
A small neural network can train in seconds. The annoying part is what happens after a few more runs - suddenly it's hard to tell which files belong to which attempt. You might have a model saved in the project root, a training log that gets lost when the session closes, and a nagging question about which Python environment is actually active.
This guide shows a simple, repeatable workflow for keeping your first PyTorch experiment understandable. We'll use an existing WSL 2 Ubuntu project at ~/research/first-neural-network, with a train.py script, a .venv directory, and a saved xor_model.pth file. You'll set up a sensible file layout, capture training output, run a longer job while watching the log, and verify what "activating a virtual environment" changes.
The goal isn't to build a full experiment-tracking platform. It's to put just enough structure in place so your runs don't blur together.
Start from the right project directory
The first failure mode is boring but common: you run a command from the wrong directory. Under WSL, your shell might start in a mounted Windows path like /mnt/c/Users/your-name instead of your Linux home directory. If you assume you're already in the project, you'll get failures - or worse, you'll create files somewhere you didn't intend.
Open Windows Command Prompt and start Ubuntu:
wsl -d UbuntuThen go to the project and check what's there:
cd ~/research/first-neural-network
pwd
ls -la
cd changes your current working directory. pwd prints the absolute path, and ls -la shows files (including hidden ones like .venv). Check this output before you keep going. The current directory affects where relative paths like logs/training.log point, not where Python itself is installed.
If you're working from the earlier tutorial, this assumes your project files already exist. The commands below are for Ubuntu under WSL - so don't mix them with Windows Command Prompt. Mixing shells is a fast way to end up with confusing paths and "where did this file come from?" moments.
Once you've confirmed the project path and files, create a predictable place for model artifacts and logs. Keeping code, run outputs, and environment details separated makes it much easier to inspect (or clean up) later.
Separate model files from training logs
Create folders for checkpoints and logs, then copy the existing model into the checkpoint folder:
mkdir -p checkpoints logs
cp xor_model.pth checkpoints/
ls checkpoints
mkdir -p creates both directories and doesn't error if they already exist. cp makes a copy; it does not move or link the original. After this, you should have xor_model.pth in the project root and also checkpoints/xor_model.pth.
That distinction matters. If train.py later writes a new model to xor_model.pth, the copy you made in checkpoints/ won't change. It's a snapshot from earlier in the workflow. Also, a generic name like xor_model.pth doesn't tell you when it was created or which run produced it - so for repeated experiments, use run-specific names or save into separate run directories rather than quietly replacing the one file you care about.
In this example, xor_model.pth contains the trained model's parameters (weights and biases). Still, don't rely on the .pth extension alone. A PyTorch script might save a state dictionary, a full training checkpoint, or some other object. Check your own save/load code, and for general details see the PyTorch tutorial.
To compare the two files' sizes, run:
ls -lh xor_model.pth checkpoints/xor_model.pth-l gives a detailed listing, and -h makes file sizes human-readable. Matching sizes can be a quick check, but it doesn't prove the contents match. If you want a stronger check on Ubuntu, compute hashes:
sha256sum xor_model.pth checkpoints/xor_model.pthIf the SHA-256 values match, the files are identical. You don't need to hash every tiny experiment, but it's useful when moving artifacts between machines or verifying that a "copy" didn't change unexpectedly.
The practical rule is simple: checkpoints store what your code saves; logs store what your program prints. A log won't preserve model parameters, and a checkpoint won't tell you what happened during training. Keep both if you want to understand a run after the fact.
Capture and monitor training output
With the directories in place, activate the project environment and run training with output redirected to a log:
source .venv/bin/activate
python train.py > logs/training.log 2>&1
The redirection has two pieces. > sends standard output (stdout, file descriptor 1) to logs/training.log. 2>&1 sends standard error (stderr, file descriptor 2) to the same destination as stdout. Order matters here: redirect stdout first, then point stderr at that destination.
Your terminal can look quiet while the command runs because the output is going into the file. When the script finishes, the prompt returns. If you run the same command again with >, you'll overwrite the log. To keep adding to the same file, use >>:
python train.py >> logs/training.log 2>&1Appending can be fine if you truly want a cumulative log, but for separate experiments it's usually clearer to use a distinct log filename.
Also note the limitation: redirected output records what the script prints. Whether the run actually saves new model parameters depends on what train.py does.
After the run, inspect the log using the tool that fits:
cat logs/training.log
head logs/training.log
tail -n 5 logs/training.log
less logs/training.log
cat prints the whole file, head shows the first 10 lines by default, and tail -n 5 shows the last five lines. less lets you scroll; press q to exit. (That extra space before tail in the block above usually won't break anything, but you can just run it cleanly as written.)
For a longer run, open one terminal to start training and another to watch the log. In the first terminal:
cd ~/research/first-neural-network
source .venv/bin/activate
python -u train.py > logs/training.log 2>&1
The -u option makes Python's standard streams unbuffered, so progress messages show up promptly when redirected. Then, in a second Ubuntu session, run:
cd ~/research/first-neural-network
tail -f logs/training.log
tail -f shows new lines as they appear. Press Ctrl+C in the monitoring terminal to stop watching - it won't stop the training process.
This is a lightweight feedback loop: the training process writes, and the monitoring process reads. If the log doesn't update quickly, unbuffered output can help, but it can't create messages the script never prints. Logging every batch can also get noisy and produce huge files, so aim for progress output that helps you spot stalls or crashes without drowning you in text.
Understand what a virtual environment changes
A Python virtual environment isolates installed packages for a project and gives you a Python interpreter via the environment's executables. Here, .venv is a directory inside first-neural-network. Activation is a shell operation - it doesn't magically reorganize your filesystem.
Activate it with:
source .venv/bin/activatesource runs the activation script in your current shell. That script updates PATH so .venv/bin is searched before the system locations. As a result, commands like python and pip usually resolve to the versions inside the virtual environment.
Activation doesn't move you into .venv, doesn't change your current directory, and doesn't stop your shell from reading other files in your project.
To see the difference directly, start from a fresh shell (or run deactivate if you're already inside an environment). Then:
cd ~/research/first-neural-network
pwd
which python3
source .venv/bin/activate
pwd
which python
ls xor_model.pth
pwd should stay the same. which python should point inside .venv/bin. And ls xor_model.pth should still find the file in the project root.
The mental model that matters: activation changes which Python environment your shell selects for commands. It doesn't create a separate Linux filesystem.
Use the selected interpreter to install packages and verify imports:
python -m pip install numpy
python -c "import numpy; print(numpy.__version__)"
Using python -m pip ties the installer to the interpreter you're using. That reduces the chance of installing into a different environment than the one running your script.
Install NumPy only if your project needs it. It's a normal Python package; you don't need it just to read a saved training log. And the log you already wrote doesn't change just because you installed something later - if you want a new log reflecting the updated environment, you run training again.
A virtual environment helps with package control, but it isn't a full record of an experiment. It doesn't automatically capture your exact code revision, data version, random seed, operating system, or hardware. If you care about reproducibility at the "packages" level, record dependencies with something like requirements.txt and keep it updated. The Python documentation covers what venv does and how activation works.
When you're done, exit the environment:
deactivateYour project files, checkpoints, and logs stay where they were.
Use a repeatable run checklist
This workflow stays intentionally small. It gives your first experiment enough structure to answer the practical questions - without pulling in a full tracking system too early. Before you start a run, check:
- Confirm the project path. Run
pwdand verify you're in~/research/first-neural-network. - Activate the intended environment. Run
source .venv/bin/activate, then checkwhich pythonif you're unsure. - Know what the script saves. Look at the model-saving code in
train.py. Redirecting output doesn't automatically create checkpoints. - Choose where this run's output goes. Overwrite, append, or use a unique log filename. A separate filename is usually the least confusing.
- Run and inspect. Capture both output streams, then read the log and confirm the expected model artifacts exist.
For a quick local test, logs/training.log plus a single checkpoint can be enough. As you add runs, a flat directory with generic filenames becomes messy fast. At that point, it helps to include a run identifier in both log and checkpoint filenames, and to record a small set of metadata you'll actually need later: code version, key settings, dependency versions, and where the data came from.
There's a trade-off: more tracking can improve comparison and reproducibility, but it also creates maintenance overhead. Start with the questions you expect to ask later - what changed, what you measured, and which artifact came from that run. For a small XOR project, a clean directory layout and readable logs are usually sufficient.
Conclusion
You don't need a big tool stack to manage your first AI experiment. Start in the right project directory, separate checkpoints from logs, capture stdout and stderr, and verify which interpreter your shell is using. Then make the run identifiable enough that you can connect its log to the model file it produced.
Try this workflow on your next training run: create the artifact folders, save a log, and check what happens when you activate .venv. If you start running experiments often, add run identifiers and a compact record of settings before you add more infrastructure. The "right" system is the smallest one that lets you trust what happened and find the result again.