The NanoVNA plot of the coaxial stub shows a few ripples that are unphysical (see attached image). It is unlikely the loss of the coax suddenly increases like that with frequency, and as your home made VNA shows that ripple isn't actually there. The most interesting thing here is that the "real" S11 of the coax should cross the "open circuit" point (or very close to it) given that it has low loss, which should also be very close to the open circuit standard. If the actual impedance is close to the impedance of the open circuit standard, how come the measurement is so far from it? I think we can rule out linearity error because there can't be much linearity error between two very close measurements. Calibration error can also be ruled out because the S11 of the open standard would have to be >1 to generate those graphs shown. The only thing left I can think of is that the fundamental signal is bleeding in because the mixer has very low IIP3, which means that in harmonic mode the distortion of the fundamental IF signal will create a 3rd harmonic and not just the 3rd harmonic RF signal.
I think it's pretty clear from those plots that your homemade 3GHz VNA is the most accurate out of the 3.
OwO
