This was a very difficult place to find. Located in the middle of a dense grid of narrow streets, none of which were named. It didnt help that it was tiny and not really any different from its neighboring houses.
The giant lantern in the yard was the clue that finally helped. There was no-one home and no signboard.
Apparently it is also on the Beppu Saigoku 33 Kannon pilgrimage. Like Beppu itself it was very unmemorable.