libdrm is used in multiple places.  Always check for it and set
have_libdrm.  Each user can then check the variable.

This is useful when only EGL and DRI drivers are needed.