“Polite” and “Japan” are synonymous for many outside of the country, and it is true that by Western standards the customs of how and how not to act are quite rigidly defined. While there is no need to worry that you might inadvertently offend someone by not doing this or that, it might help to remember the following.
It is customary in Japan to remove your shoes before entering someone’s house, and all homes have a porch area where you can change into a pair of slippers. This custom also extends to some of the more traditional restaurants and izakaya, as well as day onsen (where you will need to put your shoes into a small locker near the entrance).
Tattoos are taboo in Japan, and many sport facilities and day onsen have a policy of refusing membership or entry to anyone with a tattoo. This practice of banning those with body markings extends back to the Edo period when criminals were marked with a line on their skin by a hot iron so that others would know that they had committed an offense (later the yakuza—Japanese mafia—took this up as a custom to show that they were deliberately excluding themselves from society, hence the association today).
There is no custom of tipping in Japan, nor will an obligatory service charge be added to your bill (with the exception of some expensive international restaurants). Many restaurants do, however, serve an o-toshi: an obligatory bite-sized starter for which you will be charged a couple of hundred yen. Incidentally, while the international gesture of pretending to scribble on the palm of your hand will work, Japanese make an “X” with their index fingers when they want to pay (“close” the tab).
Before eating in Japan, whether at home or in a restaurant, you will hear Japanese say itadakimasu, which is the honorific form of the verb “to take”. It’s not a religious custom like saying grace—just good manners. Equally, at the end of the meal Japanese say gochiso sama deshita (literally, “I’m a person that has just been treated”).
Equally, food should never be passed between people using chopsticks. At funerals in Japan after the body has been cremated relatives pick the bones out of the ashes and place them in an urn using large chopsticks (a ritual called kotsuage). This often requires two people to hold the bone at either end, and so handling food in the same manner may bring back bad memories for fellow dinners.
Finally, if you want to take food for someone else, it is good manners to flip your chopsticks upside down and pass the food using the thicker (and unused) end.
There is no country-wide ban on smoking in public places. While the situation is much better than it was a decade ago, smoking is still permitted in some restaurants (especially the izakaya) and most bars and nightclubs. On the other hand, you will never see people walking and smoking on the sidewalks—the streets are exceptionally clean and entirely free of cigarette butts. If you do smoke, please use the designated zones found near the stations.