🐛 Describe the bug

This is most obvious by observing binary sizes, for example on pypi:
torchvision-0.15.1-cp39-cp39-manylinux1_x86_64.whl (6.0 MB view hashes)
torchvision-0.15.1-cp38-cp38-manylinux1_x86_64.whl (33.8 MB view hashes)

Also, from the build log it's clear that 3.8 for some reason build with -g flag:
https://github.com/pytorch/vision/actions/runs/4510717186/jobs/7942008404#step:10:409
https://github.com/pytorch/vision/actions/runs/4510717186/jobs/7942009194#step:10:413

Versions

CI

cc @zou3519