I think right now if any of the text is too long the entire batch will fail.

It's hard to truncate text automatically because there's no version of a client-side tokenizer that can be used to calculate length, so might have to instead catch when a text is too long and iteratively run it again, reducing it in size by a factor of e.g. 0.8 until it works.

The error message back from the API might help - if it gives a number of tokens I could then truncate based on that ratio.