Alauda AI

Train on GPU and Run Inference on NPU

Training and inference can use different accelerator types. This page is an index for workflows that fine-tune or train a model on NVIDIA GPUs and serve a compatible exported model on Huawei Ascend NPUs. It does not replace the linked guides or provide deployment steps.

TOC

Supported model artifacts for NPU inferenceFine-tune LLMs on GPU and run LLM inference on NPUHow the fine-tuning method affects NPU inferenceTrain machine-learning models on GPU and run inference on NPUValidated Ascend ModelZoo inference smoke tests

Supported model artifacts for NPU inference

The selected NPU runtime determines which artifacts it can load. The following are the model representations covered by the referenced NPU-serving guides; they are not a universal compatibility matrix for every Ascend device, model architecture, or runtime version.

WorkloadModel artifact or representationNPU serving path and compatibility notes
LLMA Hugging Face model directory with configuration and tokenizer files plus FP16 or BF16 .safetensors weightsServe it with a compatible vLLM-ascend runtime using the transformers model format. The model architecture, selected runtime image, and CANN stack must support the model and precision.
LLMW8A8-quantized model packageServe it only with a compatible vLLM-Ascend/model combination and its required quantization configuration. For a validated W8A8 example, see Qwen3.6-27B (W8A8).
LLMA single-file GGUF modelThe documented vLLM-ascend runtime can detect a single .gguf file. Multi-file GGUF packages are not supported by that loader.
Machine learningTorchScript .pt modelThe YOLOv5 sample loads the exported model with torch_npu. The model must be compatible with the PyTorch and CANN versions in the serving image.
Machine learningONNX .onnx modelThe YOLOv5 native CANN path uses ONNX as the input to ATC conversion; it is not the deployed executable in that path.
Machine learningCANN offline .om modelDeploy it through a CANN acl/pyACL-based service after compiling it for the destination Ascend SKU and compatible CANN/driver stack.

transformers in an InferenceService or ClusterServingRuntime identifies the model-directory format expected by the runtime. It does not by itself state that every precision or quantization method is supported. Before creating an inference service, check the exact model and target NPU SKU against the official Ascend ModelZoo and the selected serving engine's supported-model documentation (for example, the vLLM-Ascend supported-model matrix and model tutorials). Then validate the weight representation, serving image, CANN version, and driver in the target environment.

Fine-tune LLMs on GPU and run LLM inference on NPU

Use one of these guides to fine-tune an LLM on GPU:

How the fine-tuning method affects NPU inference

The training or alignment method does not, by itself, determine whether the result can run on an NPU. What matters at deployment time is the resulting model architecture, checkpoint or adapter format, weight precision or quantization, tokenizer and configuration, and the versions supported by the NPU serving runtime.

Fine-tuning or alignment methodTypical outputNPU deployment tip
Continued pre-training (CPT) or full-parameter SFTA complete Hugging Face checkpointTreat it like the base architecture. It can be served when the architecture, configuration, weight dtype, and target hardware are supported by the selected vLLM-Ascend version.
LoRA or another parameter-efficient SFT methodAn adapter plus the unchanged base modelvLLM-Ascend provides experimental LoRA support. Check the exact base model and feature limitations. If necessary, merge the adapter into the base model and export a supported FP16 or BF16 checkpoint.
QLoRAUsually a LoRA adapter trained while the frozen base model is loaded in 4-bit NF4The bitsandbytes NF4 training representation is not a portable NPU inference artifact. Serve the adapter with a supported full-precision base only when the runtime supports that LoRA combination, or merge it into the full-precision base and then export or quantize it to a format supported by vLLM-Ascend.
Preference optimization or reinforcement-learning post-training, including DPO, PPO, and GRPOA full policy-model checkpoint or a parameter-efficient adapter, depending on the training configurationDeploy the resulting policy checkpoint or adapter according to the corresponding full-model or LoRA guidance above. Optimizer state, reference models, critics, value heads, and reward models used only during training are not part of the generative inference artifact.

Fine-tuning methods can also be combined, such as CPT followed by SFT or DPO with LoRA. Evaluate the final artifact, not the sequence of training algorithms. If the final model is quantized, confirm the exact method in the vLLM-Ascend quantization guide; support for one quantization scheme does not imply support for another.

After the fine-tuned model is published to the model storage location used by your platform, use the NPU-serving guidance in Extend Inference Runtimes. Its vLLM-ascend section covers Ascend NPU runtime and InferenceService configuration. For platform-wide service lifecycle and operations, see Managing Inference Services.

Verify that the exported model, selected NPU serving engine, CANN stack, tokenizer, and model-storage format are compatible before moving a GPU-trained model to an NPU serving environment.

Train machine-learning models on GPU and run inference on NPU

For an end-to-end object-detection example, use Train, Fine-Tune, and Deploy YOLOv5. It covers GPU training with Kubeflow Trainer v2 and two Ascend NPU inference choices:

  • A custom KServe runtime that serves the exported TorchScript model with torch_npu.
  • A native CANN path that converts a YOLOv5 ONNX model to an .om model with ATC and executes it through the CANN acl/pyACL APIs.

The YOLOv5 guide also documents the validated CANN compiler image and NPU environment. Use it as the reference for artifact conversion, model storage, and NPU runtime compatibility for similar machine-learning models.

For a trained machine-learning model that does not have a ready-made NPU serving runtime, create a custom InferenceService: package a FastAPI application in a CANN-compatible image, load the exported TorchScript model with torch_npu, and expose the KServe v2 health and inference endpoints. The YOLOv5 torch_npu sample shows that pattern. Alternatively, first check the official Ascend CANN documentation and Ascend ModelZoo for a CANN acl/pyACL-supported inference method for the exact model. If it is supported, export to the required interchange format, compile the model for the target NPU SKU when required, and deploy it with the matching CANN service implementation. Do not assume that a GPU-trained artifact can run unchanged on an NPU.

Validated Ascend ModelZoo inference smoke tests

The following ModelZoo-PyTorch examples passed inference-only HTTP smoke tests on an Ascend 910B4 NPU on August 14, 2026. The checkout was ModelZoo-PyTorch commit a3c42931e0a70f7366416d4e1fc6222d1ef923a2.

ModelZoo exampleValidated inference pathResult
DLRM_for_PyTorchtorch_npuHTTP inference passed with output shape [2, 1].
ESPnet2_for_PyTorchtorch_npuHTTP inference passed with encoder output shape [1, 16]; the NPU-specific npu_bmmV2 path executed.
MinerUMinerU pipeline on torch_npuThe official /file_parse request passed; layout, OCR detection, and OCR recognition executed on the NPU.
bge-reranker-v2-m3torch_npu with the ModelZoo rewritesThe official checkpoint loaded and returned the expected ranking.
DiTONNX to .om, then pyACLThe compiled .om model returned HTTP 200 with output shape [1, 8, 32, 32].

These results validate model loading, NPU execution, and an HTTP inference boundary, not accuracy, throughput, or production readiness. DLRM, ESPnet2, and DiT used deterministic initialized weights; bge-reranker-v2-m3 used its official checkpoint; and MinerU used a small synthetic PDF. Revalidate each model with production weights, inputs, target NPU SKU, and software stack.