How does compression works as an advantage when you're writing back?

[Former Member]

As we know BWA uses very high compression ratio and columnar based indexes. This way you can store more data in memory, search faster, transfer more data etc so this is a great advantage when BEx queries are searching, reading data from BWA.

Since HANA 1.0 will use similar approach, I was just curious how this will work as an advantage; everytime you write data back, doesn't it lock the column plus re-compress it?

I would like to read the algorithm/technique if anyone can share it.

Cheers

Tansu

[Vitaliy-R]

Tansu, I cannot tell you for sure, and I would like to get all these deepest levels of details as well But at least based on what I know from corridor discussions with SAP folks during different events:

1/ Changed records are not merged immediately into columnar table, but are stored in "delta" first. Same concept as "delta index" in BWA with the difference that in BWA it is optional and in IMCE delta mechanism is always on.

2/ From time to time delta is asynchronously merged with the main columnar table.

3/ And - this is the most shocking statement for me and I still need to see it to believe - delete and change operations are possible on columnar tables in IMCE, so records are modified during delta merge (if table is not defined as temporal table).

Cheers,

-Vitaliy

Edited by: Vitaliy Rudnytskiy on Feb 9, 2011 11:29 PM

[esjewett]

Hi Tansu,

I don't believe that SAP has shared this level of detailed information. SAP has shared that HANA will support append-only operation, at least in some modes, which means that existing information would not be modified, so there would be no need to uncompress and then compress an entire column.

Even so, depending on the compression algorithm being used (apparently HANA uses several in different situations), it might be necessary to reprocess an entire column at some point to get optimal compression ratios, but I think this could be done without taking the column offline by re-compressing a snapshot.

Please keep in mind, that SAP has not shared this sort of information. What I've written here is extrapolation from what SAP has shared publicly as well as general knowledge about this sort of problem that I've gathered by looking at the open source implementations of related systems like hBase ([http://hbase.apache.org/]) or cStore ([http://db.csail.mit.edu/projects/cstore/]).

Cheers,

Ethan