Today it gets better situation for mobile data with WIFI in Japan. Actually it is not like your countries. One of the best solution is to get a SIM card for your data communication on your smart phone. It is quite common to change it to local SIM card in Asia and Europe. Surely it is good cost for value.
You can connect by WIFI within only limited areas such as station, cafe, some of sightseeing spots even in Tokyo.
For the ubiquitous use, we recommend to get it with smart way as follows.
Oder it by online before you go, Get it right timing just when you come to your hotel.
As you know you can get all related information in English on the site and order. One thing you need to note is shipping address. Input your hotel address and send it to there. After ordering, it would be perfect to tell your hotel counter that your parcel will come. All you have to do when you reach the hotel, just get it and change new SIM card.
Have a great time in Japan with mobile.
Here are recommended sim package at this moment.
No2.Sakura Mobile