As suggested by @jcupitt in #19.

You could also consider taking a histogram of L* and setting the range to 1% and 99%, ie. letting the top 1% and the bottom 1% over- and under-expose. This would make it less sensitive to noise and probably produce a more pleasing image in more cases.

vips_percent() does almost this, you'd swap your vips_stats() for this code:
https://github.com/jcupitt/libvips/blob/master/libvips/histogram/percent.c#L82

Python-based example:

low = image.percent(1)
high = image.percent(99)
image = (image - low) * (255 / (high - low))