Gently pinging @LysandreJik for his blessing on the latest round of changes :)

Does this ONNX conversion support beam search automatically for BART based summarizers?

Does this ONNX conversion support beam search automatically for BART based summarizers?

Hi @sorenmc, no, you'll have to implement your own .generate() method for the ONNX models. There is a related feature request in the optimum library here. In the meantime, you might be interested in checking out BART summarization example here

hi,
thanks HF team for your great support on this.
trying to export summarization bart
transformers.version == 4.19.0.dev0
onnxruntime.version == 1.11.1

from transformers import pipeline model_name = 'lidiya/bart-base-samsum' summarizer = pipeline("summarization", model=model_name, tokenizer=model_name)

`
from transformers import AutoConfig, AutoModelForSeq2SeqLM
from transformers.models.bart import BartOnnxConfig

config = AutoConfig.from_pretrained(model_name)
onnx_config = BartOnnxConfig(config, task="default")
print(onnx_config.outputs)
`
OrderedDict([('last_hidden_state', {0: 'batch', 1: 'decoder_sequence'})])

I am trying a few export options and non of the gives me the output from the decoder.

option 1:

/Users/aavidan/envs/py39/bin/python3.9 -m transformers.onnx --model=lidiya/bart-base-samsum --feature=seq2seq-lm --atol=5e-5 onnx

output 1:

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Using framework PyTorch: 1.10.2
Overriding 1 configuration item(s)
- use_cache -> False
/Users/aavidan/envs/py39/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:230: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/Users/aavidan/envs/py39/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:236: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/Users/aavidan/envs/py39/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:267: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/Users/aavidan/envs/py39/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:907: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if input_shape[-1] > 1:
Validating ONNX model...
-[✓] ONNX model output names match reference model ({'logits'})
- Validating ONNX Model output "logits":
-[✓] (2, 8, 50265) matches (2, 8, 50265)
-[✓] all values close (atol: 5e-05)
All good, model saved at: onnx/model.onnx

`
from onnxruntime import InferenceSession, SessionOptions, GraphOptimizationLevel

options = SessionOptions()
options.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL

session = InferenceSession(
'onnx/model.onnx',
sess_options=options, providers=["CPUExecutionProvider"]
)

session.disable_fallback()

outputs = [i.name for i in session.get_outputs()]

feed_dict = summarizer.tokenizer(text)
feed_dict['decoder_input_ids'] = feed_dict['input_ids']
feed_dict['decoder_attention_mask'] = feed_dict['attention_mask']
feed_dict = {k: np.array([v]) for k, v in feed_dict.items()}
pred = session.run(None, feed_dict)

for i, p in enumerate(pred):
print(i, outputs[i], p.shape)
`

printout -

0 logits (1, 228, 50265)
1 1209 (1, 228, 768)

summarizer.tokenizer.decode(pred[0][0].argmax(axis=-1), skip_special_tokens=True)

what i get -

gives me back the input text, which basically means logits is simply the input_ids and I am guessing from the shape that 1209 is the encoded vectors for all tokens in the text input. if that is in fact the case, HOW DO I EXPORT THE base_model.decoder ?

option 2:

`
from pathlib import Path
from transformers.convert_graph_to_onnx import convert

convert(framework="pt", model=summarizer.model, output=Path(f"onnx/lidiya_bart1.onnx"),
opset=11, tokenizer=summarizer.tokenizer, pipeline_name="summarization")
`

this results in the following error -

using framework PyTorch: 1.10.2
found input input_ids with shape: {0: 'batch', 1: 'sequence'}
found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
found output output_0 with shape: {0: 'batch', 1: 'sequence'}
found output output_1 with shape: {0: 'batch', 2: 'sequence'}
found output output_1 with shape: {0: 'batch', 2: 'sequence'}
found output output_1 with shape: {0: 'batch', 2: 'sequence'}
found output output_1 with shape: {0: 'batch', 2: 'sequence'}
found output output_2 with shape: {0: 'batch', 2: 'sequence'}
found output output_2 with shape: {0: 'batch', 2: 'sequence'}
found output output_2 with shape: {0: 'batch', 2: 'sequence'}
found output output_2 with shape: {0: 'batch', 2: 'sequence'}
found output output_3 with shape: {0: 'batch', 2: 'sequence'}
found output output_3 with shape: {0: 'batch', 2: 'sequence'}
found output output_3 with shape: {0: 'batch', 2: 'sequence'}
found output output_3 with shape: {0: 'batch', 2: 'sequence'}
found output output_4 with shape: {0: 'batch', 2: 'sequence'}
found output output_4 with shape: {0: 'batch', 2: 'sequence'}
found output output_4 with shape: {0: 'batch', 2: 'sequence'}
found output output_4 with shape: {0: 'batch', 2: 'sequence'}
found output output_5 with shape: {0: 'batch', 2: 'sequence'}
found output output_5 with shape: {0: 'batch', 2: 'sequence'}
found output output_5 with shape: {0: 'batch', 2: 'sequence'}
found output output_5 with shape: {0: 'batch', 2: 'sequence'}
found output output_6 with shape: {0: 'batch', 2: 'sequence'}
found output output_6 with shape: {0: 'batch', 2: 'sequence'}
found output output_6 with shape: {0: 'batch', 2: 'sequence'}
found output output_6 with shape: {0: 'batch', 2: 'sequence'}
found output output_7 with shape: {0: 'batch', 1: 'sequence'}
ensuring inputs are in correct order
decoder_input_ids is not present in the generated input list.
generated inputs order: ['input_ids', 'attention_mask']

ValueError Traceback (most recent call last)
Input In [10], in <cell line: 6>()
3 from transformers.convert_graph_to_onnx import convert
5
----> 6 convert(framework="pt", model=summarizer.model, output=Path(f"onnx/lidiya_bart1.onnx"),
7 opset=11, tokenizer=summarizer.tokenizer, pipeline_name="summarization")

File ~/envs/py39/lib/python3.9/site-packages/transformers/convert_graph_to_onnx.py:395, in convert(framework, model, output, opset, tokenizer, use_external_format, pipeline_name, **model_kwargs)
393 # Export the graph
394 if framework == "pt":
--> 395 convert_pytorch(nlp, opset, output, use_external_format)
396 else:
397 convert_tensorflow(nlp, opset, output)

File ~/envs/py39/lib/python3.9/site-packages/transformers/convert_graph_to_onnx.py:285, in convert_pytorch(nlp, opset, output, use_external_format)
282 # PyTorch deprecated the enable_onnx_checker and use_external_data_format arguments in v1.11,
283 # so we check the torch version for backwards compatibility
284 if parse(torch.version) <= parse("1.10.99"):
--> 285 export(
286 nlp.model,
287 model_args,
288 f=output.as_posix(),
289 input_names=ordered_input_names,
290 output_names=output_names,
291 dynamic_axes=dynamic_axes,
292 do_constant_folding=True,
293 use_external_data_format=use_external_format,
294 enable_onnx_checker=True,
295 opset_version=opset,
296 )
297 else:
298 export(
299 nlp.model,
300 model_args,
(...)
306 opset_version=opset,
307 )

File ~/envs/py39/lib/python3.9/site-packages/torch/onnx/init.py:316, in export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
38 r"""
39 Exports a model into ONNX format. If model is not a
40 :class:torch.jit.ScriptModule nor a :class:torch.jit.ScriptFunction, this runs
(...)
312 model to the file f even if this is raised.
313 """
315 from torch.onnx import utils
--> 316 return utils.export(model, args, f, export_params, verbose, training,
317 input_names, output_names, operator_export_type, opset_version,
318 _retain_param_name, do_constant_folding, example_outputs,
319 strip_doc_string, dynamic_axes, keep_initializers_as_inputs,
320 custom_opsets, enable_onnx_checker, use_external_data_format)

File ~/envs/py39/lib/python3.9/site-packages/torch/onnx/utils.py:109, in export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, opset_version, _retain_param_name, do_constant_folding, example_outputs, strip_doc_string, dynamic_axes, keep_initializers_as_inputs, custom_opsets, enable_onnx_checker, use_external_data_format)
104 if use_external_data_format is not None:
105 warnings.warn("`use_external_data_format' is deprecated and ignored. Will be removed in next "
106 "PyTorch release. The code will work as it is False if models are not larger than 2GB, "
107 "Otherwise set to False because of size limits imposed by Protocol Buffers.")
--> 109 _export(model, args, f, export_params, verbose, training, input_names, output_names,
110 operator_export_type=operator_export_type, opset_version=opset_version,
111 do_constant_folding=do_constant_folding, example_outputs=example_outputs,
112 dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs,
113 custom_opsets=custom_opsets, use_external_data_format=use_external_data_format)

File ~/envs/py39/lib/python3.9/site-packages/torch/onnx/utils.py:728, in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, opset_version, do_constant_folding, dynamic_axes, keep_initializers_as_inputs, fixed_batch_size, custom_opsets, add_node_names, use_external_data_format, onnx_shape_inference)
726 if dynamic_axes is None:
727 dynamic_axes = {}
--> 728 _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
730 graph, params_dict, torch_out =
731 _model_to_graph(model, args, verbose, input_names,
732 output_names, operator_export_type,
(...)
735 training=training,
736 dynamic_axes=dynamic_axes)
738 # TODO: Don't allocate a in-memory string for the protobuf

File ~/envs/py39/lib/python3.9/site-packages/torch/onnx/utils.py:1314, in _validate_dynamic_axes(dynamic_axes, model, input_names, output_names)
1312 for i, x in enumerate(value):
1313 if not isinstance(x, int):
-> 1314 raise ValueError("The type of axis index is expected to be an integer")
1315 if x in value_dict:
1316 warnings.warn("Duplicate dynamic axis index {} was provided for input {}."
1317 .format(x, key))

ValueError: The type of axis index is expected to be an integer

btw, same error when trying to export only the decoder using
convert(framework="pt", model=summarizer.model.base_model.decoder, output=Path(f"onnx/lidiya_dec.onnx"), opset=11, tokenizer=summarizer.tokenizer, pipeline_name="summarization")

option 3:

torch.onnx.export( summarizer.model, (inputs['input_ids'], inputs['attention_mask']), 'onnx/lidiya_torch_onnx_exp.onnx', opset_version=11, )

what i get -

like option 1, successfully exports the encoder (I assume by looking at the exported layers shapes). I still have an issue exporting the decoder.

btw, I saw a bunch of references of how to implement the beam serach, however all links gives are broken/NR, so please can you re-post link to that as well?

thanks a lot!