Study notes · 2026-10-12 · Beginner

How to choose and read LLMs on Hugging Face: from the model page to GPU memory

This note explains what to look at on a model page on the Hugging Face Hub. It covers how to read the model card, files, name tags, license and benchmarks, and also includes a way to estimate whether a model will run within your GPU memory and a checklist to use when choosing.

Contents

What the Hugging Face Hub is and why it matters
What a model page consists of
How to read model names
Checking the license and commercial use
How to read benchmark tables, popularity metrics and derivative models
Can my GPU run it? Estimating memory
Steps and checklist for choosing your first model
Common mistakes and further study

Key points

A model card (README.md) consists of YAML metadata at the top and descriptive text below. Check the license, base model and training data in the metadata first.
Read model names in the order 'organization/model name-size-type-version-variant'. First determine whether it is a Base or Instruct model, and whether it is a quantized variant such as -GGUF.
Chat models derived from the same base model can still use different chat templates. Using the wrong control tokens can sharply reduce performance.
Estimate weight memory as 'number of parameters × bytes per precision'. Add the KV cache and some headroom to judge whether a model will actually run.
The license is the first gate for deciding whether commercial use is allowed. For models marked license: other, read the original text in the LICENSE file directly.

What the Hugging Face Hub is and why it matters

Most people who want to download and use an LLM (large language model) themselves start at the Hugging Face Hub. The Hugging Face Model Hub documentation source describes it as a place where community members upload, store, discover and share model checkpoints (bundles of files holding trained weights). Models can be downloaded with the huggingface_hub client library or Transformers. According to the same documentation, more than 15 libraries are integrated with the Hub, and for production use you can also use Inference Providers or Inference Endpoints.

The problem is that there are too many models. There are dozens of models with similar names, and quantized, fine-tuned and merged versions of the same model are uploaded separately. If you cannot read names and pages, you may pick a model whose license your company cannot use, or spend hours downloading a model that will not fit on your GPU.

This note sets out the order in which to look at a model when you open it, and what to check to decide whether it is 'a model you can use'.

What a model page consists of

The first screen of a model page is the model card. According to the Hugging Face model card documentation source, the model card is the README.md file in the model repository, a Markdown document with a YAML metadata section at the very top. The metadata includes fields such as language, license, datasets and base_model. The Hub uses these values to filter model listings, display the license and link to the datasets used for training. The same documentation advises that a model card should include a model description, intended uses and limitations (including bias and ethical considerations), training parameters and experiment information, training datasets and evaluation results. This structure comes from the model cards paper paper. That paper proposed releasing, alongside each model, a short document stating its intended context of use, evaluation procedures and performance across various conditions.

The Files tab holds the actual files. config.json is the model architecture configuration file. For Transformers models, the Hub reads this file to automatically infer the task type (pipeline_tag) source. However, repositories created after August 2024 are not automatically treated as Transformers models even if they contain config.json, so it is a good idea to check whether library_name is set in the card source. Tokenizer files contain the rules for splitting text into tokens and the chat template. Weight files are usually in safetensors or GGUF format. The Safetensors documentation source describes safetensors as a format that, unlike pickle, stores tensors safely while still loading them quickly without copying. According to the GGUF documentation source, GGUF is a binary format created by the llama.cpp developer, and unlike safetensors, which holds only tensors, it also holds standardized metadata.

The Community tab is where user questions and discussions gather. Runtime errors, template issues and license questions not covered in the card often surface here first, so it is a good habit to skim it.

Layout of a model page A model repository is divided into the model card, the Files tab and the Community tab. Check the metadata at the top of the card and the three types of files in the Files tab first.

How to read model names

Model IDs on the Hub take the form 'organization/model name'. For example, mistralai/Mistral-7B-Instruct-v0.1, which appears in the Hugging Face chat template documentation source, is a model uploaded by the mistralai organization. The model name reads as size 7B, type Instruct, version v0.1. 7B and 70B mean the model has 7 billion and 70 billion parameters (the numbers the model learned). MoE (mixture of experts) models sometimes add 'active parameters' to the name, as in A3B. This is commonly used to mean that about 3 billion parameters are actually used in computation when processing a single token. Keep in mind that compute follows the active parameters, but memory usually has to hold all of the parameters.

The type tag indicates the intended use. The chat template documentation source explains that a 'Base' model is one made by pretraining on large amounts of text, and a chat model is one fine-tuned from it on message-formatted data. Base models are good at continuing text and have not been refined to follow instructions. It is common to choose a model tagged Instruct or Chat for conversational tasks, and a Base model when you need a starting point for your own fine-tuning.

Variant tags at the end of the name indicate the file format or quantization (reducing numeric precision to shrink size) method. -GGUF means GGUF files for llama.cpp-family runtimes. The GGUF documentation source gives TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF as an example. -AWQ is commonly used as a tag for weight quantization methods such as GPTQ and AWQ, and -FP8 for 8-bit floating-point precision. Repositories with these tags are often uploaded by someone other than the organization that made the original. So use base_model in the card to check what the original is.

Checking the license and commercial use

The first condition to filter on is the license. The Hugging Face licenses documentation source asks you to state the license in the repository card's metadata, and to find and respect a project's license if you use someone else's code or data. The list in the same documentation includes common open-source licenses such as apache-2.0 and mit. It also includes licenses with Non Commercial conditions such as cc-by-nc-4.0, licenses created by model companies such as llama3.1 and gemma, and licenses with use restrictions such as the OpenRAIL family.

A license with 'NC' in its name is a sign that commercial use is not allowed. For community licenses created by companies, do not judge by the name alone; read the original terms. According to the model card documentation source, a license not on the list is marked license: other, with license_name and license_link pointing to the original text, or a LICENSE file placed in the repository. In that case, the license badge on the page tells you nothing.

Derivative models need one more check. The license stated by whoever uploaded a fine-tuned or quantized version can differ from the original model's license. In practice, follow base_model to check the original license as well, and get a legal review when the decision is unclear.

How to read benchmark tables, popularity metrics and derivative models

The benchmark table in a model card is reference material, not a guarantee of performance on your own tasks. The model card paper paper argued that evaluation results should be reported broken down by groups and conditions, along with which metrics were chosen and why, and the evaluation procedure. By this standard, a table that just shows one big number lacks information. When reading the table, check whether the evaluation setup (how many examples were given, which prompt was used) matches that of the models being compared, and whether other models' scores were measured directly or copied from elsewhere. Also keep in mind the possibility that test questions were mixed into the training data, that is, data contamination. In the end, the most reliable check is to test the model directly on a small evaluation set drawn from your own tasks.

Downloads and likes are signals of how widely a model is used. They also mean problems are more likely to be known. But they do not show quality or fit for your tasks. Also keep in mind that older models have had more time to accumulate these numbers.

Derived models are found through the base_model metadata. According to the model card documentation source, the Hub infers the relationship between the current model and its base model as one of adapter (an adapter such as LoRA), merge (merging several models), quantized (quantization) or finetune (fine-tuning). You can also browse derived models by relationship type from the original model's page. If a newer version exists, the new_version field shows a link to the latest model source. Check this link so you do not pick an outdated version.

Derived models linked by base_model The Hub reads the base_model metadata and infers the relationship between a base model and its derived models as one of four types. From the original model, you can browse derived models by relationship type.

Can my GPU run it? Estimating memory

The GPU memory (VRAM) taken by the weights can be estimated as 'number of parameters × bytes per parameter'. By precision, FP32 uses 4 bytes, BF16 and FP16 use 2 bytes, and 8-bit precision uses 1 byte. So a 7B model needs about 28GB in FP32, about 14GB in BF16 and about 7GB in 8-bit. Loading a 70B model in BF16 takes about 140GB, which is hard to fit on a single GPU. The GGUF documentation source gives the 'bits per weight' for each quantization type: Q4_K is 4.5 bits, Q5_K is 5.5 bits and Q6_K is 6.5625 bits. Downloading a 7B model in Q4_K gives 7 billion × 4.5 bits ÷ 8, or about 3.9GB.

On top of this you need to add the KV cache. The KV cache is memory that stores the computed results for tokens the model has already read. It grows with longer inputs and with more requests processed at the same time. For a single short conversation, a small margin on top of the weight memory is enough, but if you feed in long documents or serve many users, the KV cache can grow as large as the weights. The runtime program itself also uses memory, so it is safer to leave plenty of headroom.

The exact precision can be checked in the Files tab. According to the GGUF documentation source, the Hub's GGUF viewer shows the metadata and each tensor's name, shape and precision. GGUF files can be used directly in tools such as llama.cpp, LM Studio, GPT4All and Ollama source. Remember that for MoE models, memory is calculated based on total parameters, even if the active parameters are small.

Estimated weight memory for a 7B model

FP32

28GB

BF16

14GB

8-bit

7GB

GGUF Q4_K

3.9GB

Example values calculated as 'number of parameters × bytes per precision'. KV cache and runtime headroom are not included.

Steps and checklist for choosing your first model

First, decide on the task. For tasks such as conversation or summarization, consider Instruct models; if you plan to fine-tune yourself, consider Base models. Second, filter the Hub list by task type and license. According to the model card documentation source, the list can be filtered by the pipeline_tag and license metadata. Third, use the estimation method above to decide which size and precision fit in your GPU memory. Fourth, read the candidate model cards for intended use, limitations, training data and evaluation methods. Fifth, choose the format you want (original safetensors or a quantized version such as GGUF) and confirm the original model via base_model. Sixth, run the models yourself on examples drawn from your own tasks and compare them.

When running models yourself, make sure to match the chat template. The chat template documentation source shows that even among models based on the same Mistral-7B, Mistral-7B-Instruct uses [INST] tokens while Zephyr uses <|user|> and <|assistant|> tokens. It also warns that using the wrong control tokens sharply degrades performance. It is safest to apply the template bundled with the model using apply_chat_template.

The items to check are as follows. Does the license allow your use (including commercial use)? If it is a derived model, have you also checked the original model's license? Does the model card describe intended use and limitations, training data and evaluation methods? Are the benchmark evaluation conditions the same as for the models being compared? Does the combined memory of the weights and KV cache fit on your GPU? Does the weight format match your runtime tool? Have you applied the model's own chat template? Is a newer version indicated through new_version?

Steps for choosing a model

1
Decide on the taskInstruct for chat, Base as a training starting point
2
Filter by task and licensepipeline_tag and license filters
3
Decide size and precisionEstimate weights + KV cache
4
Read the model cardUse, limitations, data, evaluation
5
Check format and original modelsafetensors·GGUF, base_model
6
Compare directly on your own examplesMatch the chat template

Filtering in order from the top quickly narrows down the candidates.

Model selection checklist

Rights

Does the license allow commercial use?
If it is a derived model, have you checked the original model's license too?
If license: other, have you read the full license text?

Documentation and performance

Does the card cover use, limitations, data and evaluation?
Are the benchmark evaluation conditions the same as for the models being compared?
Does new_version point to a newer version?

Running

Do the weights + KV cache fit on your GPU?
Does the file format match your runtime tool?
Have you applied the model's chat template?

Items to check before downloading and before running the model yourself.

Common mistakes and further study

There are four common mistakes. First, using a Base model for chat and getting off-target answers. Second, adding special tokens twice by re-tokenizing text that already has the template applied. The chat template documentation source recommends using add_special_tokens=False when later tokenizing text created with tokenize=False. Third, judging commercial use from the license badge alone. Fourth, calculating only the weight size and leaving out the KV cache, which causes memory to run out on long inputs. Community quantized versions may differ in quality from the original model, so it is also a good idea to compare them on the same examples as the original.

There are limitations as well. Model cards are written by the uploaders themselves, so their depth and accuracy vary. Few LLM model cards yet properly include the 'evaluation results by condition' proposed in the model card paper paper. Information missing from a card should be treated as absent and checked yourself.

To study further, start with the Hugging Face model card documentation source to learn the metadata fields. Then learn the input format of chat models from the chat template documentation source and the quantization types from the GGUF documentation source. For background on why model documentation is needed, the introduction of the model card paper paper is recommended.

References

Foundational papers

Model Cards for Model Reporting (2018)

Written by AI from the listed references and reviewed. Please check the original sources for details.