Since commit 394053f4 ("kbuild: make single targets work more
correctly"), building single targets is really slow.

Speed it up by not descending into unrelated directories.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>