Are you using these CFLAGS for building the kernel?
I haven’t tried building the kernel with anything other than the default CFLAGS. I wouldn’t expect a whole lot of difference, but the maybe letting it use the bit-manipulation extensions (zba, zbb, zbs) would be good?
Do you know off the top of your head what the alignment for vector instructions has to be?
Unfortunately, no idea. I’d expect it to require VLEN aligned. If you try it out, let us know.