Can memory faults be a big problem?

[Martin_Lauer]

I'm wondering if memory faults

a) do impact operation of an in-memory DB (and can hard- and software provider prevent it)

b) have a larger impact on in-memory DBs than on disk-based DBs

See related links:

http://en.wikipedia.org/wiki/Memory_tester

http://en.wikipedia.org/wiki/Error_detection_and_correction#Error-correcting_memory

[former_member323511 · Accepted solution]

For more information on how NewDB handles memory errors, please join the KITTY session in Walldorf on 21st February. Intel colleagues who work closely together with the NewDB team will provide details.

"Last but not least, we will demonstrate that in a lot of cases NewDB can continue operation even when uncorrectable memory errors occur."

https://wiki.wdf.sap.corp/wiki/display/TIPcrossPMKT/KittySessionsWalldorf

[esjewett · Accepted solution]

Martin,

That session looks very interesting. Is the wiki page you link to publicly accessible? It looks to me like it is only accessible from SAP's network.

Ethan

[former_member93896]

Hello Martin,

you raise an important point. Especially with large amounts of memory (one or more TB), the likelyhood of memory errors does increase (for example one error per month). With today's RAM chips, software needs to take this into consideration. As mentioned below, HANA does contain a persistency layer that will allow you to recover to a consistent state. Afterall, the memory error could occur not just within data sets but also within code. So we have to make sure that this does not impact a consistent system state.

Regards,

Marc

SAP Techology RIG

[Former Member]

i think there would be raid style backups of the data- keeping memory images ?

[esjewett]

In addition to the possibility Zeeshan mentions of keeping multiple copies in RAM, it is important to note that at least for BWA, RAM is not the primary datastore. All data is written to disk. Data is then loaded into RAM from the disk-based store when it is needed. Hopefully there is room in the RAM for all data to be loaded at all times, but BWA does not require that this is the case. I call this caching, but some disagree with that definition.

I assume, but do not know, that this is the case with HANA and the ICE as well and that all writes are logged to disk so that there is no data-loss in the event of a power or hardware failure.

The way that most in-RAM storage systems achieve transactional persistence is by writing a sequential log file (called a write-ahead-log) and flushing the log file to disk. The log file records each transaction before the client is notified that the transaction is complete. This log file can then be used to reconstruct the most recent state of the database in the event of a failure. At some point the full dataset at a point in time is reconstructed on disk, usually based on the representation in RAM, and then the log file up to that point can be discarded.

Cheers,

Ethan

[former_member93896]

Yes. Ethan, you are right one track. HANA includes a persistency manager that makes sure that committed transactions are also reflected on the (persistent) storage layer (disks or SSD). Like many databases there are undo and redo-logs that allow the system to restore a consistent state after a failure.

Regards,

Marc

SAP Techology RIG