2026-03-12
In my research I often write scripts that generate data over some large set of
parameters. I have written a python library stet
which makes one particular aspect of this (stopping and starting again) a bit easier.
Often when running a script I might need to stop it to change the parameter range or it might even crash for some other reason. For each research project I will often write a bit of specific code to be able to read the data I have generated and then start from that point. My PhD student was faced with this exact task and I thought I could solve it once and for all.
The python library stet (from the latin "let it stand") provides a simple
decorator which does not affect your running code but writes a store of the
parameters for which it has been run.
import numpy as np
import stet
import time
import pandas as pd@stet.once
def calculation(alpha, beta):
time.sleep(1)%%time
import itertools
parameter_space = list(itertools.product(np.linspace(0, 1, 2), repeat=2))
for alpha, beta in parameter_space:
calculation(alpha, beta)CPU times: user 20.5 ms, sys: 6.95 ms, total: 27.4 ms Wall time: 4.05 s
Running again stet knows to skip the values that have been run once:
%%time
import itertools
for alpha, beta in parameter_space:
calculation(alpha, beta)[stet] Skipping calculation(alpha=np.float64(0.0), beta=np.float64(0.0)) [stet] Skipping calculation(alpha=np.float64(0.0), beta=np.float64(1.0)) [stet] Skipping calculation(alpha=np.float64(1.0), beta=np.float64(0.0)) [stet] Skipping calculation(alpha=np.float64(1.0), beta=np.float64(1.0)) CPU times: user 4.66 ms, sys: 2.51 ms, total: 7.18 ms Wall time: 5.67 ms
If we rest the store it will in fact clear the underlying file.
stet.reset()[stet] Store cleared: _stet_store.csv
%%time
import itertools
for alpha, beta in parameter_space:
calculation(alpha, beta)CPU times: user 11.7 ms, sys: 4.9 ms, total: 16.6 ms Wall time: 4.03 s
stet creates a readable file (it accepts other formats that are probably better suited to large experiments).
df = pd.read_csv("_stet_store.csv")
df| alpha | beta | _stet_timestamp | |
|---|---|---|---|
| 0 | 0.0 | 0.0 | 2026-03-12T11:48:59.428485+00:00 |
| 1 | 0.0 | 1.0 | 2026-03-12T11:49:00.437368+00:00 |
| 2 | 1.0 | 0.0 | 2026-03-12T11:49:01.444970+00:00 |
| 3 | 1.0 | 1.0 | 2026-03-12T11:49:02.453405+00:00 |
For long experiments you can in fact use stet to check the status of an experiment:
stet.status()[stet] Store: _stet_store.csv [stet] 4 completed experiments recorded [stet] Last run: 2026-03-12T11:49:02.453405+00:00 [stet] Key columns: alpha, beta
You can read the docs for the library here and it is pip installable.