Skip to content

Extra space in decoded string (CharBPETokenizer) #205

@samsontmr

Description

@samsontmr

The decoder appears to be adding an extra space at the end of the string:

tokenizer.decode(tokenizer.encode('I can feel the magic, can you?').ids) produces 'I can feel the magic, can you? ' instead of 'I can feel the magic, can you?'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions