Python API

Everything the YAML file does is available from Python, from a file or a pandas DataFrame. The keyword arguments of Benchmark are the YAML sections, and both ways give the same results.

import pandas as pd

from fitrank import Benchmark

df = pd.read_csv("iris.csv")  # the file from the quick start
results = Benchmark(
    data=df,
    target="species",
    task="classification",
    models=["logistic_regression", "svm", {"name": "knn", "params": {"n_neighbors": [3, 5]}}],
    evaluation={"cv": 5, "primary_metric": "f1_macro"},
).run()

print(results.leaderboard()[["rank", "model", "cv_f1_macro", "test_f1_macro"]])
model = results.best_model()  # a fitted scikit-learn Pipeline
print(model.predict(df.drop(columns="species").head()))

A YAML file runs the same way:

results = Benchmark.from_yaml("fitrank.yaml").run()

User errors raise ConfigError or DataError, both FitRankError, with the same messages as the command line.

Reference

Benchmark

Train, compare and rank models on one dataset.

Only data, target, task and models are required. The keyword arguments are the YAML sections with the same name, e.g. evaluation={"cv": 3, "primary_metric": "f1_macro"}. Models are given by name, or as {"name": ..., "params": {...}} to choose the grid.

Raises:

Type	Description
ConfigError	the arguments do not form a valid configuration.

n_candidates property

n_candidates: int

One candidate per declared reduction and model.

from_yaml classmethod

from_yaml(path: str | Path) -> Benchmark

Build a benchmark from a YAML file; the data is read from data.path.

Raises:

Type	Description
ConfigError	the file is missing or invalid.

from_config classmethod

from_config(config: FitRankConfig, data: DataFrame | None = None) -> Benchmark

Build a benchmark from a validated config, optionally with the data in memory.

validate

validate() -> Plan

Check the config and the data without training anything.

Raises:

Type	Description
ConfigError	unknown model, step or reduction, or a bad option.
DataError	the data cannot be read, or cannot be used as declared.

run

run(*, export: bool = True, on_result: Callable[[CandidateResult], None] | None = None) -> Results

Check everything, evaluate every candidate and rank them.

With export (the default), the run folder is written as the project and output sections say; its path is results.run_dir. on_result is called after each candidate.

Raises:

Type	Description
(ConfigError, DataError)	see validate. A failing candidate never raises: it is ranked last with status="failed".

Results

The candidates of a run, best first (ranked on the CV score), failures last.

best property

best: CandidateResult | None

The first-ranked candidate, or None when every candidate failed.

leaderboard

leaderboard() -> pd.DataFrame

One row per candidate, ranked; scores are shown positive (RMSE = 53.8).

Score columns are named after the metrics: cv_f1_weighted, test_f1_weighted, test_accuracy... Secondary metrics are test-set scores.

best_model

best_model() -> Pipeline

The first-ranked pipeline, refitted on the whole training set.

Raises:

Type	Description
FitRankError	every candidate failed.

export

export(output_dir: str | Path | None = None, formats: Sequence[str] | None = None) -> Path

Write the run to a new timestamped folder inside output_dir and return it.

output_dir and formats default to project.output_dir and output.formats. The best model is saved when output.save_best_model is true and at least one candidate succeeded.

load_model

load_model(path: str | Path) -> Pipeline

Load a model saved by a run, ready to predict on new data.

Warns (FitRankWarning) when the scikit-learn version differs from the one recorded in the metadata.json next to the model. Only load files you trust: loading a joblib file can run code.

Raises:

Type	Description
FitRankError	the file does not exist.

register_model

register_model(name: str, task: Task) -> Callable[[Factory], Factory]

Register a model factory under (name, task).

Raises:

Type	Description
ValueError	a model with this name is already registered for this task.

register_reduction

register_reduction(name: str) -> Callable[[ReductionFactory], ReductionFactory]

Register a reduction factory under name.

FitRankError

Bases: Exception

Base class for every error FitRank raises on purpose.

The message says what is wrong, then one line per problem (where it is and how to fix it), then an optional hint, e.g. a YAML snippet. details holds the problem lines without the leading arrow, for programmatic use.

ConfigError

Bases: FitRankError

The configuration file is missing, unreadable or invalid.

DataError

Bases: FitRankError

The data cannot be read, or cannot be used as declared in the configuration.

FitRankWarning

Bases: UserWarning

Something still works but deserves attention, e.g. a scikit-learn version mismatch.