HANA One root volume running out of space - due to studio/client downloads (?)

[fabian_krger]

Hi everybody,

my HANA One was unaccessible this morning - indexserver not running. Since rebooting HDB instance didn't help, I rebooted the whole AWS instance.

After this I was not able to login via SSH or access the HANA One Portal any more.

AWS System logs had a lot of entries saying there was no more space available on the root device:

scripts/functions: line 307: cannot create temp file for here document: No space left on device


scripts/functions: line 307: cannot create temp file for here document: No space left on device


touch: cannot touch `/tmp//updateInfoInProgress.lck': No space left on device


start executing updateInfo.sh


./updateInfo.sh: line 167: ./../config/info.json.new: No space left on device


chmod: cannot access `./../config/info.json.new': No such file or directory


diff: ./../config/info.json.new: No such file or directory


rm: cannot remove `./../config/info.json.new': No such file or directory

and so on...

I managed to restore the instance by

• Create a snapshot of root volume
• Create a new volume with 20GB size with this snapshot
• Attach the new volume to a new SLES Instance (type c3.8xlarge - smaller instances were forbidden because of marketplace-productcode)
• since resize2fs showed nothing to do, following this guide: https://stackoverflow.com/questions/11014584/ec2-cant-resize-volume-after-increasing-size/14930504#1...
• Attaching the volume to HANA One again as /dev/sda1
• Start HANA One

However, the question was why there was no more space on root volume.

I found out that /srv/www/htdocs allocated 4,7 GB (on root partition, which is only 10 GB!).

To be more specific, there was 1,3GB in /srv/www/htdocs/downloads/client and another 3,5GB in /srv/www/htdocs/downloads/studio.

Turns out that there are Rev. 93 downloads but Rev. 91 as well! Looks like the older downloads are not deleted by the upgrade process.

This means it is just a matter of time and it will happen to other customers as well.

@all other users: I encourage you to delete those older downloads

@HANA One Team: You should delete those old files during upgrade

Kind Regards,

Fabian

[fabian_krger]

I was just running into the same issue again.

Not only space was running out, also inodes were used 100%...

I could identify several reasons for this:

/root/hanaone/logs:

service.err (5 GB), transport.err (1.5 GB), install.err (1.4 GB)

/var/log:

About 2.5 GB each of mail, warn and mail.warn

/var/spool/postfix/maildrop

This directory was responsible for high number of inodes.

I've checked some of the files there and they contain messages like

error: "/root/hanaone/logs" has insecure permissions. It must be owned and be writable by root only to avoid security problems. Set the "su" directive in the config file to tell logrotate which user/group should be used for rotation.

error: "/root/hanaone/jobs/done" has insecure permissions. It must be owned and be writable by root only to avoid security problems. Set the "su" directive in the config file to tell logrotate which user/group should be used for rotation.

error: stat of /root/hanaone/jobs/done/*error failed: No such file or directory

To avoid them as much as possible I just changed permissions on /root/hanaone/logs and /root/hanaone/jobs/done from 0777 to 0700.

Actually this is shouldn't happen on a well configured appliance after a few months of usage...