IB/ipath: Make handling of one subport consistent