where do i put robots.txt?

[ro9]

Hi Netweaver guru's

Can someone tell me if it is possible to add a robot.txt file on an external facing netweaver java instance?

We are running an xLSO solution, which is available from the internet.

The solutions i created so that you can access it by goring to: learning.mycompany.com which will then redirect you to learning.mycompany.com/xlso/xxxxx

So, there are no reverse proxies or anything in between, and therefore wee need a robots.txt file  (to keep search engines from cataloging the content).

I can see that it is possible, and sap has some recomendations, when you run an external facing portal for instance.

But i cant seem to find a guide or how-to

Help is therefore much apreciated!

Best regards

Ronni

[ro9 · Accepted solution]

Hi Again!

I followed the guide, and tested it on our development system - with no luck

On the virtual directory page i have only 1 virtual directory, which is the default one.

The path is something like

/usr/sap/xxx/j00/j2ee/docs

When i logon to the server, the docs directory is not there - so i created it

and placed a robots.txt file in that directory - but i am not able to access it from the server when i go to:

http://server:port/robots.txt

should i do more - or is there another why in which i should proceed?

best regards

Ronni

[Former Member · Accepted solution]

I believe the correct directory is \usr\sap\<SID>\<xx>\j2ee\cluster\apps\sap.com\com.sap.engine.docs.examples\servlet_jsp\_default\root. See also SAP KBAs 1906707 and 1908436 for details.

[ro9 · Accepted solution]

That did the trick.
I found the application by looking at the modules - and there i saw the default / (root) pointed to the index site.

Thanks alot for your help!

Best regards

Ronni

[Former Member]

Sure, find out the root directory for your Java virtual host and place the file there, see Virtual Host Configuration - Administration - SAP Library for details. I can't recommend the setup you have but I would recommend you to use ICM filtering to restrict access to only those services that need to be exposed.

[ro9]

Hi Samuli!

Thanks for your answer!

I will try to look at the link you send, and see if i can add the file there.

Regardring ICM filtering - if i get this part right - it the options to "filter" out options not "intended" to be exposed for the whole web - right? 

Best regards Ronni