Rune,
Yes something is not quite right.
I used the TDR function on the NanoVNA in stand-alone mode and got impedance measurements that made sense to me. I cascaded some 50 ohm (RG58), 75 ohm (RG59) and 93 ohm (RG62) coax and could see the impedance steps.