Interesting discussion. I used to work for HP. Later sold Cascade probe stations. Calibration was always "difficult". Especially on wafer: how far do you overdrive (i.e. press the probe down), how many times has that standard been used, etc etc.
Seems to me we're expecting a bit too much from the nanovna. As someone already posted, its not a lab grade instrument. To me its a "how close am I" kind of tool. If I REALLY care I will use a better tool (I am fortunate enough to have an 8753).
Ron
N4XD