I have left the PI space many years back .
but I remember that a lot of resources are wasted for all the database write /commit operations including writing to the DB for each step in the pipeline . The DB's were disk based DB like oracle or MS SQL , or DB2 where read write is slow as its disk based . Recently In memory DB are making waves , for example SAP HANA . which is a column based DB ( instead of row based DB), but tables can be created and data written as row based operations also.
The primary premise of this In memory DB is that the ENTIRE data resides on the RAM so the access to the read/write to the data is extremely fast. These systems has RAM in order of 512 GB to 2 TB. Since RAM is volatile (i,.e lost when there is power loss) . The entire IN MEMORY data is written periodically to SSD's for backup snapshots .
I believe that PI should move to in memory DB instead of disk based DBs. Then the pipeline steps would superfast amongst other speedups.
and oh, PI should totally drop its ABAP stack , and perhaps also rewrite and refactor its JAVA only stack to make it more lightweight .
rishi
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.