build_token_calibration

Function build_token_calibration 

Source
pub(crate) fn build_token_calibration(
    store: &AtlasStore,
    tokenizer: &str,
) -> Result<TokenCalibrationOverview, CliError>
Expand description

Build an optional local tokenizer calibration over indexed UTF-8 files.