This may sound kind of confusing but I'm wondering how the Japanese see Americans who are very interested in their country/culture/language/etc. (Everyone in general of course. You can't know what every person is thinking.)
Whenever someone around me (In the US) finds out that I'm interested in Japan, they assume it's because I like anime. Yes, I like anime. I like it better than most American television, but that's not my reason for liking Japan. I want to learn more about the culture and language. I want to travel there someday and see some of these beautiful places. It's not all about anime. I guess what I'm saying is that (for me) most Americans assume you like the ideal of Japan because you like anime. Is that what Japanese think too? I just think it would be embarrassing to travel there one day and have everyone assume it's because you are crazy for anime.
I know how you feel. I like anime as well (I'm actually part of a podcast for it! aaapodcast.com if you want to be among friends and talk about Japan/anime) but that's not ALL I like about Japan, or all I like in general!
As you said, I can't know what everyone thinks, but I did study abroad in Japan and live with a host family. For the most part, they're curious why you like Japan and they will ask but they don't automatically assume its anime. Unless you're acting like a crazed anime fan they won't think that.
That's been my experience at least.