You have to remember that in Japan, Arable land is one of the scarcest commodities there is. Therefore, anything you grow will be in a small quantity, driving the prices up. Since Japan is an island country, importing fruits won't help much either, due to high prices. When Japanese people come to California, where I'm from, and get things like an entire melon half for dessert, that's almost unthinkably extravagant to them. To us who live in California, it's no biggie.
Bottom line: it's just really expensive to live in Japan.
(Plus the Japanese are kind of obsessive over first/last things)