Buffer refcounting is a bit hard, because of the duality between CAPTURE and
OUTPUT mode. In the long term, we should consider having two seperate pool
instead of this mess. At least state should be better kept this way.