TL;DR: Use the checkpoint index to enumerate required shards before starting a model loader. A local preflight can catch missing or empty files without loading tensors or using a GPU. It cannot prove that present shards are complete, correct, compatible, or affordable to run.

What you’ll have after this guide:

• A CPU-only checker for a conventional local Safetensors shard index.
• A list of required shard files and their observed on-disk bytes.
• A clear boundary between download presence, format validity, and inference readiness.

Honest take: Check the files before allocating inference resources, but never treat a presence check as permission to trust the model.

Why can a model folder look complete and still fail to load?

A sharded checkpoint needs every file referenced by its index. A directory containing a configuration file, tokenizer, and one large weight file can still be missing other required shards. File count alone is a poor guide because unrelated assets and alternative weight formats may live alongside the intended checkpoint.

Hugging Face’s large-model documentation, checked November 9, 2026, explains the index mapping between parameter names and shard files. The matching behavior is visible in Transformers source at a fixed commit: get_checkpoint_shard_files builds its shard list from the unique values in weight_map.

Use that explicit mapping instead of guessing filenames from a pattern or assuming the last numbered shard is present. A manually copied folder, a selective download, and a backup restored from an incomplete manifest can all create this failure. The remedy is to identify the missing files for the intended revision, not immediately upgrade CUDA or rent a larger GPU.

What does the index tell you about memory and cost?

The index tells you where tensors are stored; it does not directly describe the complete runtime memory requirement. Observed file sizes are disk facts. Runtime memory also depends on loading strategy, dtype, quantization, architecture, device placement, and workload.

Check	What it establishes	What remains unknown
Index parses	JSON structure is readable under your policy	Whether the mapping matches shard contents
Referenced paths exist	Local files are available now	Completeness, authenticity, and compatibility
Files are nonempty	They contain at least one byte	Whether all expected bytes arrived
Safetensors format validation passes	That validator accepts the file structure	Model quality, provenance, and runtime fit
Model loads and runs	That exact runtime/workload succeeds	Other revisions, contexts, and hardware

The helper below runs with Python’s standard library and requires no GPU or model-framework installation. It reads a bounded index and inspects file metadata; it does not read all weight payloads. Its 8 MiB index cap is an example policy, not a Hugging Face format limit. It uses existing CPU and disk resources with no service fee, although that does not make a production inference deployment free.

If storage capacity is the question, use the download dry-run workflow before transferring weights. If runtime memory is the question, move to the hardware planning calculator and validate the actual workload afterward. Neither tool replaces the other.

How do you check a local shard index without loading tensors?

Read model.safetensors.index.json, deduplicate its shard names, and check each resolved target. This intentionally narrow helper supports flat .safetensors filenames under the supplied snapshot directory. It rejects nested paths rather than attempting to support every possible checkpoint layout.

Save the following as check_shards.py:

import json
import sys
from pathlib import Path

MAX_INDEX_BYTES = 8 * 1024 * 1024  # Local policy, not a format limit.


def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("duplicate JSON key: " + key)
        result[key] = value
    return result


def check_snapshot(folder):
    root = Path(folder).resolve(strict=True)
    if not root.is_dir():
        raise ValueError("snapshot must be a directory")
    index_path = root / "model.safetensors.index.json"
    resolved_index = index_path.resolve(strict=True)
    if not resolved_index.is_relative_to(root):
        raise ValueError("index resolves outside snapshot")
    with resolved_index.open("rb") as handle:
        raw = handle.read(MAX_INDEX_BYTES + 1)
    if len(raw) > MAX_INDEX_BYTES:
        raise ValueError("index exceeds local size policy")
    index = json.loads(raw, object_pairs_hook=unique_object)
    if not isinstance(index, dict):
        raise ValueError("index must be a JSON object")
    mapping = index.get("weight_map")
    if not isinstance(mapping, dict) or not mapping:
        raise ValueError("weight_map must be a nonempty object")
    shards = set()
    for tensor, name in mapping.items():
        if not isinstance(tensor, str) or not tensor:
            raise ValueError("invalid tensor name")
        if (not isinstance(name, str) or not name
                or "/" in name or "\\" in name or ":" in name
                or not name.endswith(".safetensors")):
            raise ValueError("unsupported shard path: " + repr(name))
        shards.add(name)
    sizes = {}
    for name in sorted(shards):
        target = (root / name).resolve(strict=True)
        if not target.is_relative_to(root):
            raise ValueError("shard resolves outside snapshot: " + name)
        if not target.is_file():
            raise ValueError("shard is not a regular file: " + name)
        size = target.stat().st_size
        if size == 0:
            raise ValueError("empty shard: " + name)
        sizes[name] = size
    return {
        "tensor_assignments": len(mapping),
        "required_shards": len(shards),
        "observed_file_bytes": sum(sizes.values()),
        "files": sizes,
    }


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("usage: check_shards.py SNAPSHOT_DIRECTORY")
    print(json.dumps(check_snapshot(sys.argv[1]), indent=2))

Run it against an existing, quiescent local directory:

python3 check_shards.py ./my-model-snapshot

This command neither downloads files nor invokes from_pretrained. A successful report lists the number of tensor assignments, unique required shards, and actual file sizes. Keep that report with the repository ID and immutable revision used for the download. The helper cannot infer provenance from the directory name.

Why does the checker reject some valid cache layouts?

The checker rejects symlinks that resolve outside the supplied directory as a deliberate containment policy. Hugging Face’s shared cache may use snapshot links pointing into a separate blob store. That arrangement can be valid, while still failing this helper’s stricter self-contained-directory requirement.

Use an intended local deployment directory, or a separately reviewed checker that understands your cache layout. Do not flatten or delete a shared cache merely to obtain a passing result. The Hub download guide describes downloading specific files and snapshots; preserve the revision and selection arguments when creating an export.

The path checks are suitable for a local preflight on files you control. They are not a complete defense against a hostile process changing symlinks between inspection and loading. Do not run the helper concurrently with downloads or edits and claim it validates a stable snapshot. Its report describes the directory at inspection time, not every future read.

What failures did the local fixtures catch?

On November 9, 2026, we exercised the article’s checker with synthetic local shard sets. The valid case contained three tensor assignments spread across two tiny Safetensors files. The test checked that the reported shard count deduplicated filenames and that observed bytes matched the actual files.

Removing a required file raised FileNotFoundError. Replacing a shard with an empty file raised ValueError: empty shard: .... The fixtures also rejected a duplicate JSON key, a traversal path, an absolute path, a nonstring shard name, an empty mapping, an oversized index, and a link escaping the directory.

Those are file-contract tests, not a model benchmark. The tiny files represent simple format fixtures, not a usable language model. No large weights were downloaded, no GPU was allocated, and no generation was performed. The test results do not establish a minimum VRAM capacity or a production throughput figure.

One limitation matters more than the successful checks: replacing a shard with nonempty garbage can still pass. The helper does not open tensor headers or hash payloads. We verified that limitation explicitly rather than letting the success report imply full validation.

How should you repair a missing shard?

Retrieve the exact referenced file from the same trusted repository revision used for the rest of the checkpoint. Keep existing files intact until you understand whether they belong to that revision. A mixture of two revisions can contain every expected filename while still being an invalid model snapshot.

Start by recording the repository ID, revision, missing filename, intended format, and original download command. Compare the index against the authoritative repository file list. If an allow_patterns filter omitted a referenced shard, correct that selection and perform the necessary download. Avoid switching to an unreviewed mirror merely because it has a matching basename.

Do not create an empty placeholder to silence a missing-file error, rename a different shard, or copy the first shard over the missing path. These actions can make a presence check pass while making the actual checkpoint less trustworthy. A failed preflight is useful precisely because it keeps that ambiguity visible before loading.

If the folder is intended for offline recovery, also verify the tokenizer, configuration, custom-code requirements, and license. The offline backup and restore guide addresses the larger bundle. Passing this index check does not establish that an offline application has every dependency it needs.

What should you verify after all shards are present?

Validate file structure and provenance before loading weights into an application. The separate Safetensors header tutorial explains what bounded metadata inspection can and cannot prove. Full format validation and trusted hashes answer different questions from checking that a filename exists.

Compare each shard’s actual tensor names with the index when you need stronger consistency checks. Then use the supported model loader in an appropriate environment and inspect missing or unexpected tensor reports. A generic file-presence tool cannot decide whether a model architecture matches its configuration or whether external custom code is appropriate to execute.

Transformers is distributed under Apache 2.0. That software license does not grant rights to every model loaded through it. Check the specific weights and tokenizer terms independently, especially when moving a snapshot into commercial use or redistributing a backup.

This tutorial references fixed upstream source rather than claiming to have executed the entire current Transformers loader. The checker itself has no Transformers dependency. Its Python behavior relies on the documented pathlib and JSON APIs and was tested with Python 3.13.5.

Honest take: When should you skip this preflight?

Skip this particular helper when your checkpoint is unsharded, uses another naming convention, stores nested shard paths, or depends on shared-cache links outside the selected directory. Its rejection can mean “unsupported by this policy,” not “the model is corrupt.” Choose a checker appropriate to the actual artifact.

Also skip treating the report as an inference-sizing recommendation. A directory can pass every presence check and still exceed your RAM, VRAM, disk capacity during conversion, or maintenance budget. Keep the deployment decision separate from the download diagnosis.

For a conventional flat export, the default workflow is simple: establish provenance, enumerate files from the index, repair missing inputs, validate the format, then test the supported loader. Stop before the expensive step when the cheaper evidence already identifies a missing file.

Sources

• Sharded checkpoints and index mapping — Hugging Face
• Shard enumeration at fixed source revision — Transformers
• Specific files, revisions, and snapshot downloads — Hugging Face Hub
• Apache 2.0 software license — Transformers
• Filesystem resolution and containment — Python
• JSON object parsing hooks — Python

Last updated November 9, 2026. Local fixtures validate file presence and policy rejection only; no model inference, GPU benchmark, or production savings were measured.

What self-hosting actually costs

Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.

Email address Subscribe