Unable to convert HF model using Olive – “ValueError: Unable to get dummy inputs for the model”

Question

Unable to convert HF model using Olive – “ValueError: Unable to get dummy inputs for the model”

Gowtham CP 6,920 Volunteer Moderator

Hi team,

I am trying to convert a HuggingFace model to ONNX for Foundry Local using Olive (latest version from PyPI). However, every model I try fails with the following error:

ValueError: Unable to get dummy inputs for the model.
Please provide io_config or install an optimum version that supports the model for export.

This happens for models like:

meta-llama/Llama-3.2-1B-Instruct

TinyLlama/TinyLlama-1.1B-Chat-v1.0

Qwen/Qwen2.5-0.5B-Instruct

Here is the full log snippet:

dummy_inputs = _get_dummy_inputs(model, config)
raise ValueError("Unable to get dummy inputs for the model...")
ValueError: Unable to get dummy inputs for the model.
Please provide io_config or install an optimum version that supports the model for export.

I followed the official Microsoft documentation here: https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-local/how-to/how-to-compile-hugging-face-models?view=foundry-classic&tabs=Bash

What I have already tried:

✔ Installed Olive with auto-opt:

pip install olive-ai[auto-opt]

✔ Installed latest optimum version ✔ Installed latest transformers ✔ Tried with and without --trust_remote_code ✔ Tried multiple models (Llama, Qwen, TinyLlama) ✔ Tried CLI commands like:

olive auto-opt --model_name_or_path meta-llama/Llama-3.2-1B-Instruct --trust_remote_code --output_path models/llama --device cpu --provider CPUExecutionProvider --use_ort_genai --precision int4 --log_level 1

Still getting the same error for all models.

Questions:

Is this a known issue with Olive 0.10.x?

Do certain HF architectures still not support dummy input extraction via Optimum?

Do we need to manually define io_config for these newer HF models?

Is there an example of a working JSON config for Olive 0.10.x for HF models?

Any help or updated guidance would be appreciated.

Thank you!

1 answer

Your answer

Answer 1

It seems you are encountering a common issue when trying to convert Hugging Face models to ONNX format using Olive. The error message ValueError: Unable to get dummy inputs for the model typically indicates that the model you are trying to convert does not have the necessary configuration for input extraction, which is often required for the conversion process.

Here are some steps you can take to troubleshoot and potentially resolve this issue:

Check Model Compatibility: Ensure that the models you are trying to convert are compatible with the version of Olive you are using. Some models may require specific configurations or may not yet be supported.
Install Optimum: Make sure you have the latest version of the Optimum library installed, as it provides additional support for model export. You can install it using:
```
   pip install optimum
```
Define io_config: If the models you are using require it, you may need to manually define an io_config for the models. This configuration helps Olive understand how to handle the inputs and outputs of the model.
Consult Documentation: Refer to the official Microsoft documentation for any updates or notes regarding the models you are using. Sometimes, specific models may have known issues or additional requirements.
Community Support: If the issue persists, consider reaching out to the community through forums or GitHub issues, as other users may have encountered similar problems and can offer solutions.

By following these steps, you should be able to address the error and successfully convert your Hugging Face models using Olive.

References:

Compile Hugging Face models to run on Foundry Local

Share via

Unable to convert HF model using Olive – “ValueError: Unable to get dummy inputs for the model”

What I have already tried:

Questions:

1 answer

Your answer