Don't try to read/write from an inactive stream. When we, for example,
transfer the second connection in tunneling mode, we are not interested anymore
on read/write activity on the old connection.