Use the unadjusted internal clock times to calculate the internal/external
offset when calibrating the clock.

When going to NULL, unparent and free the ringbuffer, like we do in the source
element.
See #578506