Is "gaijin" a racist word?

Many foreigners in Japan complain about "gaijin" (japanese word for "foreigner") being a racist word, and claim that this would be a prove of japanese racism toward foreign people. The reason is that gaijin is formed by gai and jin, where gai means "outside" and jin means "person". So, being the meaning of gaijin that of "person from outside", this is commonly claimed to be a racist word, and Japanese are often vieved as islanders that consider as odd everything that come from outside. But, is there any foundation to this claim? That the Japanese are islanders is surely true, but the word gaijin has probably no relation with their view of foreigners. In fact, can we imagine a word meaning "foreigner" in any language, that doesn't mean "person from outside"? Probably not, since a foreigner "is", actually, a person from outside. In fact, english "foreign" comes from latin "foris", that means "outside", and italian "straniero" comes from latin "extra", that means "outside" too. Even if we look at the german "fremd", the originary meaning was apparently "far". Well, if it means "far", is there really a big difference between something that is "far" and something that is "outside"? Even if there was, the proportion of german speakers between the gaijin in Japan is so small that we cannot think they could ever affect the consideration of the word gaijin. This idea of gaijiin meaning "person from outside" and therefore indicating racism is instead spread between english speakers and speakers of other european languages, that probably don't know the real meaning of the same word in their own language.