Enhancing the Data Enrichment Process
Chapter 1: Overview
The following paragraphs talk about “data enrichment”, however, we always mean both sales data and stock data enrichment unless specified separately. Performing data enrichment in a different way than the system performs it in standard can mean several different things.
- You might not need a specific key figure for your business and therefore might want to skip the calculation of this key figure.
- You might need a key figure of standard data enrichment, but might want to calculate it slightly or even completely different. This requirement might include to use more specific thresholds than standard calculation does.
- You might need a new key figure for your business other than the key figures standard data enrichment calculates for you.
- You might need to perform parts of the enrichment or even the complete enrichment dependent on the retailer data you upload.
Chapter 2 describes how you add new customer-specific key figures to the data enrichment. This is the most complex way to enhance the data enrichment. Note that in chapter 2, we sometimes mention dependencies between the steps described there. You can ignore these comments if you do not need to run data enrichment regularly in the development system while you enhance it. If you prefer to be able to run enrichment also in the development system, for example for early testing purposes, you should pay attention to the dependencies, though. The easier scenarios 1 and 2 of the list above are described well in the online documentation of DSiM. However, if you need to use more specific thresholds than standard data enrichment does, you should be able to find some useful hints in chapter 4. During standard data enrichment, the thresholds are valid globally.
Chapter 3 describes suggestions how to differentiate the enrichment further, for example by retailer.
Chapter 5 describes possible Testing Functions of Data Enrichment
Chapter 2: Add Customer-Specific Key Figures to Enrichment
If you would like to calculate some new customer-specific key figures during sales or stock data enrichment, you have to
- add corresponding attribute fields to the objects involved
- create code that calculates your new customer-specific key figures
f you use data enrichment and semantic partitioning of the sales / stock data DSOs, you have created a set of involved objects for each semantic partition:
- worklist / result list table
- procedure template instances that access the right worklist / result list table
- Process Chain
If possible, re-use the standard DataSource. This does not result in performance drawbacks even with semantic partitioning of the propagation layer. See also the section about "Retailer-Specific Data Enrichment" for the recommended approach there.
If you follow this approach, you may also re-use InfoSources /DDF/SLS_DATA_ENRICHMENT and /DDF/STCK_DATA_ENRICHMENT. You only need to copy InfoSources /DDF/TD_SALES_02 and /DDF/TD_INV_02 according to your various propagation DSOs.
As a consequence, you have to apply the following to every semantic partition where you need your new customer-specific key figures.
Enhance worklist / result list tables
The standard worklist / result list tables are
We recommend to copy the standard tables into your own tables and add the new attribute fields you need to your copied table(s). You can use the statement
CREATE TABLE “your_new_table” LIKE “corresponding_standard_table”;
Afterwards, you can add the attributes corresponding to your new customer-specific key figures using the statement
ALTER TABLE “your_new_table” ADD ( “your_attribute1” <type>, “your_attribute2” <type>, ..., “your_attributeN” <type> );
- CREATE TABLE “your_SAP<SID>_schema”.”your_sales_worklist_table” LIKE “your_SAP<SID>_schema”.”T_DS11_SALES_WL”;
- CREATE TABLE “your_SAP<SID>_schema”.”your_stock_result_list_table” LIKE “your_SAP<SID>_schema”.”T_DS12_STOCK_RESULT”;
- ALTER TABLE “your_SAP<SID>_schema”.”your_sales_worklist_table” ADD ( “attribute1” NVARCHAR(10), “attribute2” DECIMAL(17,2) );
Create new InfoObjects
This step is optional for helper key figures which you do not need in your InfoProviders. Create an InfoObject for each new customer-specific key figure.
Enhance the BW objects involved
This step is optional for helper key figures which you do not need in your InfoProviders. Add the InfoObjects corresponding to the new customer-specific key figures to the InfoSources and the DSOs involved in data enrichment.
- InfoSource /DDF/SLS_DATA_ENRICHMENT
- InfoSource /DDF/TD_SALES_02 (or your copy of it)
- DataStoreObject /DDF/DS11 (or your own DSO within an SPO)
- InfoSource /DDF/STCK_DATA_ENRICHMENT
- InfoSource /DDF/TD_INV_02 (or your copy of it)
- DataStoreObject /DDF/DS12 (or your own DSO within an SPO)
Adjust the transformations involved
This step is optional for helper key figures which you do not need in your InfoProviders. In the relevant transformations, link the corresponding InfoObjects between source and target object. The transformations involved are the following
- DataSource /DDF/*_DATA_ENRICHMENT -> InfoSource /DDF/*_DATA_ENRICHMENT
- /DDF/*_DATA_ENRICHMENT -> /DDF/TD_*_02
- InfoSource /DDF/TD_*_02 -> DataStoreObject /DDF/DS*
You can continue to use the standard DTPs and the standard process chains (or the ones you used before enhancing the enrichment).
- Process chain /DDF/SALES_ENRICHMT_STAGE
- Process chain /DDF/STOCK_ENRICHMT_STAGE
If the data target is a copy of a standard DSO, create new transformations, DTP’s and process chains with the corresponding standard objects as templates.
Dependencies: You can adjust the transformation(s) from your corresponding DataSource(s) to the “first” InfoSource only after you appended the additional attributes to the corresponding DataSource extract structure(s).
Note that before you append your additional attributes to the DataSource extract structure(s), you have to extend the export table of your corresponding stored procedure(s) for the QUERY / QUERY_RSLT function accordingly.
However, it does not do any harm to an active data enrichment when you adjust the transformations from your DataSource to the corresponding InfoSource at the end of your project to enhance the data enrichment .
New stored procedures (adaption for standard key figures)
You can continue to use the standard stored procedure templates that match your requirements. Create new template instances for them, enter your newly created worklist / result list tables as template parameter values for the corresponding parameters.
Implement BAdI /DDF/CALL_PROCEDURE such that it returns your procedure template instance names for the corresponding functions.
Special case: Assume you have already copied the worklist / result list, for example because you already have enhanced data enrichment before. Now you are enhancing the worklist / result list currently in use (your copied versions), In this case, you don’t need to change anything in the procedure template instances you use. As a consequence, you don't need to adjust your BAdI implementation either. Adding a new attribute to the worklist / result list table does not disturb an active data enrichment.
Dependencies: All stored procedures involved in data enrichment have to access the same worklist / result list tables, see paragraph on implementing BAdI /DDF/CALL_PROCEDURE.
New stored procedures (new key figures)
We recommend that you create the stored procedures that calculate a specific new key figure via procedure templates. Then you can reuse it easier for other worklist / result list tables (for example if you take advantage of semantic partitioning).
Create one or more stored procedure templates that calculate your new customer-specific key figure(s).
If you have several new key figures, you have several options:
- You can create one stored procedure template that calculates more / all key figures.
- Alternatively, you can create several stored procedure templates that calculate one key figure each.
The decision depends on whether you can do the calculations with the same or sufficiently similar selections. Selecting several times the worklist / result list / DSO can have a negative impact on the performance.
Dependencies: If you need to enhance the worklist / result list tables with new customer-specific attributes from time to time, take care of the following:
Access the worklist / result list table (select from / insert into table) only with an explicit field list. Then your procedures continue to run smoothly even if the worklist / result list tables contain more attributes than they contained when you initially created your stored procedures / stored procedure templates.
Coding example how to select from the worklist table. Here, "temporary work table" is a table you use to select data you need for your calculations.
Coding example how to insert data into the worklist table. Here, "temporary work table" is a table which contains the result of your calculation step. It must also contain the records of the worklist that remained unchanged by your calculations.
You can also review the standard stored procedure templates for examples how to do this.
New stored procedure(s) (return the result)
This step is optional for helper key figures which you do not need in your InfoProviders. Otherwise, create your own procedure for the “query worklist” enrichment step (standard function QUERY in sales data enrichment and QUERY_RSLT in stock data enrichment).
- Create your query worklist procedure (template) as a copy of the corresponding existing query worklist procedure template (context menu – copy / paste). Make sure to paste your new stored procedure template into your HANA development package.
- This ensures that your procedure template has the same import / export parameters and the same template parameters as the original procedure template.
- For sales data enrichment, this is procedure template sap.ddf.dsimddf.enr_ext.PT_SALES_ENSTP_QUERY_WORKLIST
- For stock data enrichment, this is procedure template sap.ddf.dsimddf.enr_ext.PT_STOCK_ENSTP_QRY_WORKLIST
- Add your new customer-specific attributes to the export table structure
- Make sure your QUERY procedure also selects your new customer-specific attributes from the enhanced worklist / result list table.
- Make sure your QUERY procedure coding adds the values of your additional attributes to the export table. Example: If you use a copy of the standard code of the query worklist procedure template for sales data enrichment, you have to add the corresponding fields to the function call CE_PROJECTION at the end of the procedure coding. Adding the result of your calculations to the export table makes sure that data is handed over the the ABAP layer.
We reocommend that you use the same import / export parameters as the corresponding standard query worklist procedure for your customer-specific query worklist procedure. Then you can continue to use the standard (enrichment) function to execute the query result step and just implement BAdI /DDF/CALL_PROCEDURE such that it returns your stored procedure name.
Dependencies: You can add your new customer-specific attribute fields to the export table of the corresponding stored procedure (and make your stored procedure coding fill them) before you append the new attributes to the extract structure of the corresponding DataSource currently used in data enrichment processes.
However, the values will only appear in the corresponding PSA tables after you appended your new customer-specific attributes to the corresponding extract structure. But once you did so, the standard logic will also return your new attributes.
Create ABAP class(es) to call your stored procedures
Create one or more ABAP classes that the system can call during data enrichment. This corresponds to the decision to create one or more stored procedures you took in the preceding step.
Your ABAP class(es) must implement interface /DDF/IF_FES_ENRICHMENT to be able to execute an enrichment step by executing the associated function. They can inherit from class /DDF/CL_ENR. Refer to the existing classes involved in data enrichment, /DDF/CL_SLS_ENRMT_* or /DDF/CL_STK_ENRMT_*, as a template.
Some examples are
Your ABAP classes are responsible to call the corresponding stored procedure(s) you created in the preceding step. The stored procedure(s) in turn execute the corresponding calculations.
New function types and functions
In the enrichment customizing, create one or more function types and one or more functions for your new customer-specific attributes. This corresponds to your decision to create one or more stored procedures, see paragraph on the corresponding step. When you create a new function, enter the ABAP class(es) you created in the preceding step into column “Function Class Name”.
You can continue to use the existing functions if they match your requirements.
Implement BAdI /DDF/CALL_PROCEDURE
For each function in sales and stock data enrichment, an implementation of this BAdI can return the name of the stored procedure to call. If you continue to use the standard function with the standard ABAP class, you can have the class call your own stored procedure to execute this enrichment step.
This can be your own procedure template instance if the standard procedure code suits your needs and you just changed the underlying worklist / result list table or the underlying DSO. It can also be your own stored procedure if the stored procedure interface is the same and you needed the stored procedure to execute different code.
As long as you create your own procedure template instances for the sake of semantic partitioning of your propagation layer, check to what extent you can make use of the example implementation of the BAdI.
If you start to enhance the data enrichment for the first time and used the standard enrichment so far, you can safely implement all steps mentioned here as they work on copies of the original worklist / result list tables. Once you are done with your developments, create / adjust your BAdI implementation such that it calls the new procedure template instances / procedures.
Make sure your BAdI implementation returns names of stored procedures that access the same worklist / result list tables for all functions involved in data enrichment. If you use semantic partitioning, you have created different worklist / result list tables per partition – then all stored procedures for one specific partition of data enrichment must access the same worklist / result list tables.
However, the BAdI must not return a mixture of procedures that access the 'old' and the 'new' (enhanced) worklist / result list tables. Also refer to the example in the BAdI documentation.
New enrichment sequence
If you have not already done so, create a new enrichment sequence to orchestrate the sales and stock data enrichment steps respectively. Each step corresponds to a function. Add the standard functions you want to continue to use and your new functions to your new enrichment sequence.
You find the corresponding customizing activities under "Demand Data Foundation" – "Data Enrichment" – "Define Enrichment Sequences".
If you change an enrichment sequence data enrichment currently uses, you could add your new functions once you have successfully tested your new stored procedures. You should make sure that they don’t have unwanted effects on the key figures already calculated and correctly calculate your additional key figures. For testing reasons, you can consider to skip the corresponding enrichment step for all agreements except a specific “test data delivery agreement”, for details refer to the chapter on "Testing Functions of Data Enrichment".
Place the steps that calculate your new customer-specific key figures after the standard enrichment steps and before the “query worklist” enrichment step.
This allows you to continue to use standard logic where it matches your requirements. For all standard key figures that are beneficial for your business, you can continue to use the standard functions / stored procedure template instances you created for your worklist / result list table(s). In case of semantic partitioning, you can continue to use the standard procedure templates and the corresponding template instances you created for your different partitions.
Enhance the DataSource extract structure
This step is optional for helper key figures you do not need in your InfoProviders. Append attribute fields corresponding to your new customer-specific key figures to the extract structure of the corresponding DataSource.
- Structure /DDF/S_SLS_DATA_ENRICHMENT2 of DataSource /DDF/SLS_DATA_ENRICHMENT
- Structure /DDF/S_STCK_DATA_ENRICHMENT2 of DataSource /DDF/STCK_DATA_ENRICHMENT
Then, call transaction RSA6, in order to make the fields you have added to the extract structure visible. For doing so, change the DataSource by navigating from the list of DataSources, and reset the flag “Hide column” on the next window.
Dependencies: Before you append your new customer-specific attribute fields to the extract structure of a DataSource currently used in data enrichment processes, you must add these new attributes to the export table of the corresponding stored procedure for the QUERY / QUERY_RSLT function.
After you appended your new attribute fields to the extract structure, you can adjust the transformation from the DataSource to the corresponding InfoSource (see paragraph on adjusting the transformations involved).
Chapter 3: Retailer-Specific Data Enrichment
In standard data enrichment, sales and stock data enrichment sequences are valid globally.
If you need to calculate a specific key figure differently depending on the retailer (or the agreement), you can achieve this in the ABAP class that executes the function assigned to the corresponding enrichment step.
Depending on the retailer or the agreement, you can make the system call different stored procedures or even skip the calculation of a specific key figure.
An ABAP class that executes an enrichment step can access the process ID currently being processed. Refer to method
for an example how to do so. Import structure IS_PARAMETER of method /DDF/IF_FES_ENRICHMENT~EXECUTE has a component PROCESSID. (IS_PARAMETER is generally typed, but in our context the type is always /DDF/S_DATA_ENRICHM_PARAM).
Knowing the process ID, you can use method
Its export structure ES_INSTANCES contains the data delivery agreement in field ES_INSTANCES-DDEL-DDAGR (it has structure /DDF/S_INSTANCES). With the data delivery agreement, you can access the retailer (data origin), the context and more information you might need.
Your code could for example look like:
CASE <retailer>. (=> use data agreement, context or data origin for differentiation)
WHEN ‘retailer_A’. lv_sp_name = <stored_procedure_for_retailer_A>.
WHEN ‘retailer_B’. lv_sp_name = <stored_procedure_for_retailer_B>.
WHEN ‘retailer_C’. CLEAR: lv_sp_name.
IF lv_sp_name IS INITIAL. RETURN. ENDIF.
lv_statement = <package> && lv_sp_name && <parameters>.
/ddf/cl_enr=>call_stored_procedure EXPORTING . . .
iv_statement = lv_statement
. . .
Retailer-specific key figures
If you need to add new key figures in a more specific way (for example different key figures for different retailers), you would have to create your own objects per set of key figures respectively.
However, we strongly recommend that you continue to use the standard DataSource and the standard extract structure. Just append the full list of corresponding fields for the new key figures to the extract structure.
Note that you can continue to use the standard extractor function modules – unless you have very specific requirements - even if you decide to go for different DataSources with different extract structures.
- (only if absolutely necessary: DataSources with extract structures)
- Transformations (to map the data corresponding to the general extract structure to your more specific InfoSources / DSOs
As for the DataSources, we recommend to evaluate whether it's an option for you to add the full list of all new key figures to the InfoSources / DSOs / Transformations you use. For all the objects involved, we recommend that you use the standard ones as a template if you decide to differentiate nonetheless.
If you use semantic partitioning, you already have created individual objects by semantic partition. Make sure you extend all worklist and result list tables with the corresponding fields. We strongly recommend that you keep the structure of the worklist and result list tables identical, though. Add all fields that you added to your DataSource extract structure also to all worklist / result list tables. This allows you to continue to use one stored procedure template per function and cover the differentiation of different worklist / result list tables with corresponding procedure template instances.
Working with different sets of key figures for different retailers already gives you an increased complexity. Differentiating DataSource, worklist and result list tables by set of key figures appears to introduce significantly more complexity without adding a significant benefit. You might prefer to leave key figures initial when you do not need them for a specific retailer.
Example: Assume you want to calculate the new key figures A and B for Retailer A and the new key figures B and C for retailer B. Then we recommend to add the following attribute fields (corresponding to all three key figures) to your worklist / result list tables:
FIELD_A (corresponding to key figure A)
FIELD_B (corresponding to key figure B)
FIELD_C (corresponding to key figure C).
In your query worklist procedure template for retailer A, make sure you fill FIELD_C with initial value. In the same way, proceed with FIELD_A for retailer B.
Use in CE_PROJECTION:
- CE_CALC('0', DECIMAL) AS "field_c"
- CE_CALC('''''', NVARCHAR(<length>)) AS "field_c"
Use in SELECT commands
- CAST(0 AS DECIMAL(17,3)) AS "field_a"
- CAST('' AS NVARCHAR(<length>)) AS "field_a"
Chapter 4: Working with Different Thresholds
In standard data enrichment, all thresholds involved in sales and stock data enrichment are valid globally. However, you might need to differentiate the threshold values, for example by daily / weekly data.
If you replaced a standard enrichment step by a customer-specific enrichment step including a customer-specific stored procedure, you have the full control how you build your logic anyway. This enhancement scenario of data enrichment is well described in the online help of DSiM.
In case the function(s) of standard data enrichment work well for your business and you just wish to differentiate the thresholds, this chapter should give you some hints how you could do that.
If you have few differentiations of threshold values in your enrichment process, you can create your own threshold table(s) identical to the standard threshold table /DDF/ENR_THRSHLD.
Let’s assume that you need to differentiate by daily or weekly data. Create one more threshold table (use the same key fields and attribute fields like in table /DDF/ENR_THRSHLD). Use one of the tables to read the thresholds for weekly data and the other one to read the thresholds for daily data.
Create different stored procedure template instances with the corresponding value of the threshold table in procedure template parameter THRESHOLD.
Copy the ABAP class the standard enrichment function uses to a customer-specific class and adjust the code such that the system calls the right procedure template instance depending on the fact whether you upload daily or weekly data.
If you have not yet done so, create a new function for the corresponding function type. Enter your copied ABAP class in column "Function Class Name" of your new function (see paragraph "New enrichment sequence" in chapter "Add Customer-Specific Key Figures to Enrichment" for some details on the data enrichment customizing).
You can find some suggestions how to differentiate within the ABAP class coding in the chapter about the "Retailer-Specific Data Enrichment".
If you need to differentiate the thresholds in a more complex way, you have to do customer development. An example might be that you need different thresholds for the different retailers who send their POS data to you. However, as the development you need is more complex than for the approach above, evaluate carefully whether you actually need such a complexity.
If you actually need this complexity, we suggest the following:
- Replace the stored procedures affected by customer-specific stored procedures (create your own stored procedure templates as copies of the standard stored procedure templates).
- Create a customer-specific threshold table with additional key fields. Your additional key fields should reflect your needs to differentiate the thresholds.
- In your copy of the stored procedure template, replace the code that accesses the standard threshold table /DDF/ENR_THRSHLD by code that accesses your customer-specific threshold table instead.
- You might need information to differentiate the threshold values which is not available during runtime of the stored procedure (for example the data delivery agreement, the data origin or the context). In this case, you can consider to add a corresponding import parameter to your copy of the stored procedure template.
- Replace the ABAP class standard enrichment uses and adjust the code such that it passes the right value for this parameter when it calls your stored procedure.
- You can also access your customer-specific threshold table in your ABAP class coding.
- Then you would have to hand over the threshold values to your stored procedures. To be able to do so, add new import parameters corresponding to the threshold values to your stored procedure templates.
- In your new stored procedure (templates), remove the code that selects the threshold values from the standard threshold table /DDF/ENR_THRSHLD.
- Replace the corresponding variables for the threshold values in the code (LV_*) by your import parameters (IV_*).
Chapter 5: Testing Functions of Data Enrichment
Option 1: SQL console of the HANA development studio
You can carry out an initial test in the HANA development studio. Open the SQL console of the corresponding system and execute commands like
CALL “_SYS_BIC”.”your_procedure_name” ( ‘import_parameter1’, . . ., ‘import_parameterN’ );
You can manually call one procedure after the other. After each “enrichment step”, you can evaluate what happend: Simply select the data you need for verification from the corresponding worklist / result list tables.
Option 2: Extractor checker (transaction RSA3)
To test the enrichment in the extractor checker, you need an active enrichment sequence that already contains all relevant functions. To keep the impact as low as possible, consider to implement the corresponding ABAP class(es) such that they skip the new functions to be tested for all data delivery agreements except a specific one for testing. You find a suggestion how you can differentiate data enrichment by data delivery agreement in the chapter on "Retailer-Specific Data Enrichment".
Option 3: Sandbox for data enrichment
You can also create a sandbox for testing:
- sandbox DataSource (you can reuse the standard extractor function module, but should use a sandbox extract structure)
- you can reuse the standard InfoSources
- sandbox transformations (you can reuse the standard transformations between the standard InfoSources)
- sandbox DSO
- sandbox DTP (from your sandbox DataSource to your sandbox DSO)
- sandbox Process Chain
As for option 2, you need an active enrichment sequence that already contains all relevant functions. You might want to consider the corresponding remarks above.