matroskademux: Fix possible NULL pointer dereference and assertion that could be caused by invalid files