Junk Mail settings for recipient... need to allow your sending domain. Especially filters like Postini and others...

If your domain records does not configure like it should, the destination email provider may suspect its a spam.

You can use SMTP or check your domain record configuration.

Donamix, your questions is too broad.
Only invitation emails come to spam box?
What kind of emails did you use to test this?

A lot depends on the email where you have tried to send your invitation.
For example if it is gmail, you might want to read http://support.google.com/...8?hl=en&ctx=mail

Too broad? sorry for my question ;/

Well,

Yes, only invitations from the new Invite too, all other emails from site are going to inbox.

I just tried the new invite a test for myself to a hotmail " New Outlook" account it went to junk.

I have tested on my installation and invitation emails come to inbox.
Check  spam filter of your mail client. Seems like your mail client has set of phrases, and if an email containing these phrases comes, it  automatically goes to junk box.