File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes JForum and the fly likes Indexing by Google Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » JForum
Bookmark "Indexing by Google" Watch "Indexing by Google" New topic
Author

Indexing by Google

Migrated From Jforum.net
Ranch Hand

Joined: Apr 22, 2012
Posts: 17424
Hello!
Must I do somyhing special to make the texts in the messages available to indexing (searching) by Google, Yahoo etc.?
Thank you
[originally posted on jforum.net by yanam]
Migrated From Jforum.net
Ranch Hand

Joined: Apr 22, 2012
Posts: 17424
There is no need for special configurations. Just give google's bots enough time to index your board. Of course taht all SEO aspect apply.

Rafael
[originally posted on jforum.net by Rafael Steil]
Migrated From Jforum.net
Ranch Hand

Joined: Apr 22, 2012
Posts: 17424
yanam wrote:Hello!
Must I do somyhing special to make the texts in the messages available to indexing (searching) by Google, Yahoo etc.?
Thank you


We are having the exact opposite problem. One of our users was quite upset that upon Googling their name, the second result was a direct link to their profile page on our forum, even though it is password-protected. ops: It appears the Google robot is able to penetrate these pages without having a login. I was also able to retrieve the entire page with WGET without a login and password. We are using single-sign-on.

Why is this and what can we fix on the backend to stop this from happening?
Google allows you to exclude pages with the use of robots.txt but that is really a band-aid. How can we stop any "dumb client" from accessing these pages?

[originally posted on jforum.net by Skoach]
Migrated From Jforum.net
Ranch Hand

Joined: Apr 22, 2012
Posts: 17424
Which URL? user/edit or user/profile?

Rafael
[originally posted on jforum.net by Rafael Steil]
Migrated From Jforum.net
Ranch Hand

Joined: Apr 22, 2012
Posts: 17424
I can imagine it was /user/profile - as the link is avaiable in the 'header' section. I doubt that bots use brute force techniques to find out about links but rather jump from avaiable link to link.

To avoid it you can edit the header.htm file and make an <#if logged> clause around the member listing entry in the forum header.

Maybe this helps.
[originally posted on jforum.net by Sid]
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Indexing by Google