You'll need to check the list of the countries that have the visa exemption agreement with Japan. If your country is included in the list, then you will not be required for a visa when entering Japan under normal circumstances. If your country is not included in the list, you'll need to obtain a short-term stay visa from the embassy of Japan in your country.

Please read the Japanese visa guide provided by the ministry of foreign affairs carefully. And be informed that it is the embassy of Japan in your country that makes a decision on issuing a visa. Whether the attendees need Letter of Invitation fully depends on the relations between countries. The conference organizers hereby reiterate that we are neither able to nor authorized to decide the relations between countries.