Generating Splunk Data from SMF 110 Records

Introduction

This document describes the process of generating data to be ingested by Splunk from raw SMF data.

Your local SMF data management conventions must be taken into account when adapting this process to your system. However, the following steps outline the basic process.

Isolate the SMF 110 Records

The first step is to isolate the SMF 110 records to be processed. How this is accomplished is customer-dependent, but usually involves running the utility IFASMFDP. Here is the job step to do this.

//**********************************************************************
//* Unload data from the SMF data sets                         
//**********************************************************************
//SMFDUMP  EXEC PGM=IFASMFDP                                           
//STEPLIB  DD DISP=SHR,DSN=SYS1.LINKLIB                                
//INDD1    DD DISP=SHR,DSN=SYS1.S0W1.MAN2                              
//OUTDD1   DD DISP=OLD,DSN=HBR1.HTAC.SMF110.DATA                       
//SYSPRINT DD SYSOUT=*                                                 
//SYSUDUMP DD SYSOUT=*                                                 
//SYSIN    DD *                                                        
  INDD(INDD1,OPTIONS(DUMP))                                            
  OUTDD(OUTDD1,TYPE(110))
  DATE(yyddd,yyddd)
  START(hhmm)     
  END(hhmm)
  SID(xxxx)                                               
/*

Note that DATE, START, END and SID are optional, but probably useful if the SMF dataset you are starting with includes data that spans multiple days.

Isolate or Generate the CICS Dictionary Records

The next step is to extract the CICS dictionary record from the SMF 110 records. Or, if the SMF data does not contain a dictionary record, one must be generated. This is required because the SMF 110’s will be transformed by a subsequent step into their “unloaded” format. The dictionary records are required to do this.

Isolate the CICS Dictionary Records

If the SMF source dataset contains data that spans a long enough time period, or includes a time frame during which the CICS region(s) were started, then it will contain dictionary records. In this case, the following job step can be run to extract just the dictionary records into a separate file.

//**********************************************************************
//* Copy the dictionary records into a separate file           
//**********************************************************************
//DICTCOPY EXEC PGM=SORT,REGION=0M                                     
//SORTIN   DD DISP=OLD,DSN=HBR1.HTAC.SMF110.DATA                       
//SORTOUT  DD DSN=HBR1.HTAC.SMF110.DICT,DISP=(NEW,CATLG),              
//            DCB=(RECFM=VBS,LRECL=32760,BLKSIZE=23476),               
//            SPACE=(CYL,(1,1))                                        
//DFSMSG   DD  SYSOUT=A                                                
//SYSOUT   DD  SYSOUT=A                                                
//SORTDIAG DD  SYSOUT=A                                                
//SYSIN    DD  *                                                       
 OPTION COPY,VLSHRT                                                     
 RECORD TYPE=V                                                         
* POSITIONS OF DATA ARE SPECIFIED AS 1'S BASED NUMBERS                                      
* SELECT RECTYPE=110, SUBTYPE=1(MONITORING), CLASS=1(DICT)             
 INCLUDE COND=(6,1,FI,EQ,110,AND,                                      
              23,2,BI,EQ,X'0001',AND,67,2,BI,EQ,X'0001')               
 END                                                                   
/*

The output of this job step will indicate whether any dictionary records were found. A message like the following will appear in the log (indicating there were 0 output records). If no dictionary records were found, then you will need to generate one or more dictionary records.

ICE054I 0 RECORDS - IN: 158, OUT: 0

In a subsequent step, a HostBridge variant of DFH$MOLS (named HBR$MOLS) will be used to process the SMF 110’s and put them in “unloaded” format. Before that, a SORT must be run to adjust the data and time of the dictionary records.

//**********************************************************************
//* Update Date/Time in Dictionary Records
//**********************************************************************
//DICTUPDT EXEC PGM=SORT,REGION=0M                                   
//SORTIN   DD DISP=OLD,DSN=HBR1.HTAC.SMF110.DICT
//SORTOUT  DD DSN=HBR1.HTAC.SMF110.DICT2,DISP=(NEW,CATLG),              
//            DCB=(RECFM=VBS,LRECL=32760,BLKSIZE=23476),               
//            SPACE=(CYL,(1,1))                                        
//DFSMSG   DD  SYSOUT=A                                                
//SYSOUT   DD  SYSOUT=A                                                 
//SORTDIAG DD  SYSOUT=A                                                
//SYSIN    DD  *                                                       
*                                                             
* IF SMFMNCL (RECORD CLASS) IS 1 (DICTIONARY RECORD),         
* SET THE DATE TO 2020 DAY 1, AND TIME TO 0.                  
* THIS WILL CAUSE THEM TO BE SORTED BEFORE ANY CLASS 3'S      
* (PERFORMANCE CLASS RECORDS). DFH$MOLS WILL THEN BE HAPPY. 
*                                                             
 OPTION COPY,VLSHRT                                                    
 RECORD TYPE=V                                                
 INREC IFTHEN=(WHEN=(67,2,BI,EQ,X'0001'),                     
               OVERLAY=(7:X'00000000',11:X'0120001F'))        
 END  
/*

HBR$MOLS requires that Dictionary records have a date/time BEFORE the data records (otherwise it will issue a 109 ABEND); hence the need for the SORT INREC control card with the OVERLAY clause.

Generate the CICS Dictionary Record(s)

If the SMF dataset does not contain dictionary records, then one or more dictionary records must be generated. Note that this job step must specify the DFHxxx.CICS.SDFHLOAD library for the version of CICS that you are using.

//**********************************************************************
//*   Create new dictionary record for this version of CICS
//**********************************************************************
//MNDUP    EXEC PGM=DFHMNDUP                                         
//STEPLIB DD DISP=SHR,DSN=DFH550.CICS.SDFHLOAD                       
//SYSUT4  DD DISP=(NEW,CATLG),DSN=HBR1.HTAC.SMF110.DICT.DFH550,      
//           UNIT=SYSDA,SPACE=(TRK,(1,1)),                            
//           DCB=BLKSIZE=27998                                       
//SYSPRINT DD SYSOUT=*                                               
//SYSUDUMP DD SYSOUT=*                                               
//*  SPECIFYING NO DATE/TIME WILL CAUSE CURRENT DATE/TIME TO BE USED.
//*  SPECIFY A DATE/TIME BEFORE THE TIME FRAME UNDER ANALYSIS. IF	
//*  THIS IS NOT DONE, THE EXECUTION OF HBO$MOLS THAT FOLLOWS MAY
//*  TERMINATE WITH A 109 ABEND.		              
//SYSIN    DD *                                                      
MCT=NO                                                                
SYSID=S0W1,GAPPLID=gggggggg,SAPPLID=ssssssss                               
DATE=2020001,TIME=000000                                             
/*

gggggggg and ssssssss should specify the Generic and Specific APPLID for the CICS region.
A dictionary record must be generated for each region for which data will be processed.
DFHMNDUP will generate the default dictionary record for a version of CICS.
If your installation uses a custom MCT, then the MCT= parameter should specify the 2 character suffix of the DFHMCTxx module. Also, the library in which that module resides should be added to the STEPLIB DD concatenation list.

Allocate Dataset for Unloaded SMF 110 Records

The IBM-provided program DFH$MOLS (or the HostBridge version HBO$MOLS) is used to transform SMF 110 Performance records into their “unloaded” (fixed record length) format. The program assumes that the output file has been allocated with fixed format records (blocked or unblocked).

When DFH$MOLS/HBO$MOLS is assembled, it includes the data structure specified in DFHxxx.CICS.SDFHSAMP(DFHMNPDA). The LRECL must be the length of that structure. At the end of that structure is the statement:

MNPDRLEN EQU   *-DFHMNPDA          Performance Data Record length

MNPDRLEN is the length of the structure, and the symbol referenced by DFH$MOLS to determine the assumed LRECL of the output file.

The following table lists the values of MNPDRLEN for supported versions of CICS:

CICS Version	MNPDRLEN (LRECL)	BLKSIZE (LRECL*8)
DFH420	X’0BEC’ / 3052	24416
DFH510	X’0D18’ / 3352	26816
DFH520	X’0D18’ / 3352	26816
DFH530	X’0D38’ / 3384	27072
DFH540	X’0D84’ / 3460	27680
DFH550	X’0DD8’ / 3544	28352
DFH560	X'0E14'/ 3604	28832

The following job step allocates the file used for subsequent steps in our example:

//**********************************************************************
//*   Allocate file for unloaded SMF 110 records
//**********************************************************************
//ALLOC  EXEC PGM=IEFBR14                              
//UNLD   DD   DSN=HBR1.HTAC.SMF110.UNLD.DFH550,        
//            DISP=(NEW,CATLG),                        
//            DCB=(RECFM=FB,LRECL=3544,BLKSIZE=28352), 
//            SPACE=(CYL,(100,10)),UNIT=SYSDA

Run HBO$MOLS to Convert SMF 110 to Unloaded Format

The next step is to run HBO$MOLS to perform the conversion to unloaded format. HBO$MOLS is provided by HostBridge (and is functionally identical to DFH$MOLS) but has been enhanced to include fields needed to process data from a customer’s system and create Splunk records from it. Additional details are contained in Appendix A at the end of this article.

The following job step illustrates how to execute HBO$MOLS.

//**********************************************************************
//*   Convert SMF 110 records to unloaded format
//**********************************************************************
//UNLOAD   EXEC PGM=HBO$MOLS,REGION=0M                         
//STEPLIB  DD DISP=SHR,DSN=HBR1.HTAC.DFH550.LIBRARY            
//INPUT    DD DISP=SHR,DSN=HBR1.HTAC.SMF110.DICT.DFH550        
//         DD DISP=SHR,DSN=HBR1.HTAC.SMF110.DATA               
//SYSUT4   DD DISP=OLD,DSN=HBR1.HTAC.SMF110.UNLD.DFH550        
//SORTWK01 DD SPACE=(CYL,(100,10)),UNIT=SYSDA                  
//SORTWK02 DD SPACE=(CYL,(100,10)),UNIT=SYSDA                   
//SORTWK03 DD SPACE=(CYL,(100,10)),UNIT=SYSDA                  
//SORTWK04 DD SPACE=(CYL,(100,10)),UNIT=SYSDA                  
//SORTWK05 DD SPACE=(CYL,(100,10)),UNIT=SYSDA                  
//SORTDIAG DD SYSOUT=*                                          
//SYSOUT   DD SYSOUT=*                                         
//SYSPRINT DD SYSOUT=*                                         
//SYSABEND DD SYSOUT=*                                         
//SYSUDUMP DD SYSOUT=*                                          
//SYSIN    DD *                                                
PRINT DIC                                                      
SELECT APPLID=aaaaaaaa     <= Optional                     
UNLOAD                                                          
/*

HBR$MOLS can issue various user abends when there are problems with the control cards or SMF data. For a list of the possible abends, the messages they generate in the output, and explanations of the abends, click on the following link: https://www.ibm.com/docs/en/cics-ts/5.5?topic=dfhmols-abend-codes-error-messages

The dataset containing the SMF 110 dictionary record(s) is concatenated AHEAD of the dataset containing the SMF 110 data records. This is required.
The SELECT statement is optional. If not specified, all regions will be processed.
A dictionary record must exist for each CICS region processed. If it doesn’t, HBO$MOLS will terminate with a user abend code of 109 or 110. As a result, it’s usually appropriate to specify a SELECT statement to indicate the CICS region(s) you wish to process, and for which you have provided a dictionary record.
DFH$MOLS/HBR$MOLS supports many parameters to control its processing. Consult the CICS Operations and Utilities Guide for additional information.

After this job step runs, the data in the output file (e.g., HBR1.HTAC.SMF110.UNLD.DFH550) can be browsed via ISPF, and will look similar to this:

*********************************************************** Top of Data ************************************************************
CICSA   CICSA   CICSA   S0W1............P=5OÄÏ.Ø...ñ.3,....w...w..............OhsØ.......ï[-HBNL0009CICSA   TO..P=5OÂJ+ P=5OÄÏ.Ø. i.
CICSA   CICSA   CICSA   S0W1............P=+Ê{Í{....ñ.î.+..|J..|J..............OhsØ.......ï[-CSOL....CICSA   U ..P=5¦AuÑ P=+Ê{Í{.....
CICSA   CICSA   CICSA   S0W1............P==5.ll....ñ.ï.p......................OhsØ.......ï[-CSSY....CICSA   U ..P==4Am|ØP==5.ll.. i.
CICSA   CICSA   CICSA   S0W1............P={èa..{...ñ.ï5P...ã...ã..............OhsØ.......ï[-CWBG....CICSA   S ..P={èØ-2ØP={èa..{. i<
CICSA   CICSA   CICSA   S0W1............P=C:{eZ ...ñ..........................OhsØ.......ï[-CSOL....CICSA   U ..P=+Ê{Í{.P=C:{eZ ....
CICSA   CICSA   CICSA   S0W1............P=EO LoØ...ñ..8Z...U...U..............OhsØ.......ï[-CSSY....CICSA   U ..P=EO.^..P=EO LoØ. i%
CICSA   CICSA   CICSA   S0W1............P=F,8+@{...ñ.!.r...3...3..............OhsØ.......ï[-HBNL0009CICSA   TO..P=F,8...P=F,8+@{. i@
CICSA   CICSA   CICSA   S0W1............P=F%.v^Ø...ñ.!.1......................OhsØ.......ï[-HBNL0009CICSA   TO..P=F%.eoØP=F%.v^Ø. i<
CICSA   CICSA   CICSA   S0W1............P=F%Çp6Ø...ñ.!.G......................OhsØ.......ï[-HBNL0009CICSA   TO..P=F%ÅØO P=F%Çp6Ø. io
CICSA   CICSA   CICSA   S0W1............P=F%qGU....ñ.!.^...e...e..............OhsØ.......ï[-HBNL0009CICSA   TO..P=F%qâ3{P=F%qGU.. O.
CICSA   CICSA   CICSA   S0W1............P=F%GÈ9{...ñ.!.....m...m..............OhsØ.......ï[-HBNL0009CICSA   TO..P=F%F\+ P=F%GÈ9{. O.
CICSA   CICSA   CICSA   S0W1............P=F%8F. ...ñ.!.....R...R..............OhsØ.......ï[-HBNL0009CICSA   TO..P=F%7^. P=F%8F. . O.
CICSA   CICSA   CICSA   S0W1............P=F_.$6 ...ñ.!........................OhsØ.......ï[-HBNL0009CICSA   TO..P=F_...{P=F_.$6 . O.
CICSA   CICSA   CICSA   S0W1............P=F_í8Y{...ñ.!.....O...O..............OhsØ.......ï[-HBNL0009CICSA   TO..P=F_í..ØP=F_í8Y{. O<
CICSA   CICSA   CICSA   S0W1............P=F_g}. ...ñ.!........................OhsØ.......ï[-HBNL0009CICSA   TO..P=F_bw.{P=F_g}. . O*
CICSA   CICSA   CICSA   S0W1............P=F_6+w....ñ.!.&...-...-..............OhsØ.......ï[-HBNL0009CICSA   TO..P=F_6...P=F_6+w.. O%
CICSA   CICSA   CICSA   S0W1............P=G1A".Ø...ñ.!k8...î...î..............OhsØ.......ï[-CSSY....CICSA   U ..P=G1A..ØP=G1A".Ø. O@
CICSA   CICSA   CICSA   S0W1............P=H8oo+Ø...ñ.$.6......................OhsØ.......ï[-CATA....CICSA   U ..P=H8+).ØP=H8oo+Ø. O<
CICSA   CICSA   CICSA   S0W1............P=H8~+q ...ñ.$.6......................OhsØ.......ï[-CQRY0011CICSA   S ..P=H8~Î. P=H8~+q . Oo
CICSA   CICSA   CICSA   S0W1............P=H8sd.....ñ.$.7...6..................OhsØ.......ï[-CQRY0011CICSA   S ..P=H8~+q P=H8sd... Oo
CICSA   CICSA   CICSA   S0W1............P=H8toÃØ...ñ.$.7......................OhsØ.......ï[-CESN0011CICSA   S ..P=H8tÄ. P=H8toÃØ. j.
CICSA   CICSA   CICSA   S0W1............P=H=.V. ...ñ.$.7...+...+..............OhsØ.......ï[-CESN0011CICSA   TP..P=H=.|çØP=H=.V. . j.
CICSA   CICSA   CICSA   S0W1............P=H^4). ...ñ.$.....+...+..............OhsØ.......ï[-CEMT0011JOHN    TO..P=H^0~z P=H^4). . j.
CICSA   CICSA   CICSA   S0W1............P=H+<O3....ñ.$.>..Q...................OhsØ.......ï[-CEMT0011JOHN    TO..P=H^4). P=H+<O3.. j.
********************************************************** Bottom of Data **********************************************************

Filter Unloaded Records Based on Intended Splunk Analysis

Analyzing SMF 110 data to for RPA/macro activity is a common use case for using HTAC. This analysis is typically based on 110’s for terminal-oriented transactions only. And, if this is the only objective (at least initially) then it makes send to only send information to Splunk for such transactions.

The following job step will filter the output from HBO$MOLS. The output file will only include unloaded SMF 110 records for transactions run from VTAM terminals.

//COPYTERM EXEC PGM=SORT,REGION=0M                            
//SORTIN   DD DISP=SHR,DSN=input_dataset  
//SORTOUT  DD DISP=OLD,DSN=output_dataset
//DFSMSG   DD  SYSOUT=*                                       
//SYSOUT   DD  SYSOUT=*                                       
//SORTDIAG DD  SYSOUT=*                                       
//SYSIN    DD  *                                              
*  INPUT: FIXED FORMAT RECORDS PRODUCED BY HBR$MOLS.          
*  SELECT RECORDS WITH TERMINFO: NATURE=TERMINAL,ACCMETH=VTAM 
*  REMEMBER, OFFSETS ARE 1'S BASED.                           
 OPTION COPY                                                  
 INCLUDE COND=(233,4,BI,ALL,X'01000100')                      
 END                                                          
/*

This selection can’t be performed before the data has been converted to unloaded format because the TERMINFO data is encoded in the record based on the dictionary.

Allocate Dataset for Splunk Data

The following job step allocates the file that will contain the JSON-formatted documents for Splunk. It will be used in the next topic, Generate Splunk Data from Unloaded SMF Records.

//**********************************************************************
//*   Allocate file for Splunk data
//**********************************************************************
//ALLOC  EXEC PGM=IEFBR14                              
//UNLD   DD   DSN=HBR1.HTAC.SMF110.JSON,
//            DISP=(NEW,CATLG),                        
//            DCB=(RECFM=VB,LRECL=8196,BLKSIZE=27680), 
//            SPACE=(CYL,(50,10)),UNIT=SYSDA

Generate Splunk Data from Unloaded SMF Records

Now that the SMF Performance data records exist in the unloaded format, we will reformat them into the style of records expected by Splunk.

The following job step is used to perform this process:

//FMTJSON  EXEC PGM=HBOSMF55,REGION=0M                          
//STEPLIB  DD  DISP=SHR,DSN=HBR1.HTAC.DFH550.LIBRARY                   
//         DD  DISP=SHR,DSN=HBR1.HTAC.LIBRARY.HBR               
//SYSPRINT DD  SYSOUT=*                                         
//SYSOUT   DD  SYSOUT=*                                         
//SYSPRINT DD  SYSOUT=*                                         
//SYSUDUMP DD  SYSOUT=*                                         
//INPUTFIL DD  DISP=SHR,DSN=HBR1.HTAC.SMF110.UNLD.DFH550        
//OUTJSON  DD  DISP=OLD,DSN=HBR1.HTAC.SMF110.JSON               
//SYSIN DD *                                                    
ILIMIT=99999999                                                 
/*                                                               
//HBOPARAM DD  *                                                
TRACE=0                                                         
STATS=1                                                          
CUSTOMER=INGRAM                                                 
ECOND=Y        ;COND EMIT 0, BLANK, NULL VALUES                 
*                                                               
*  THE DEFAULT IS Y FOR EVERYTHING EXCEPT:                       
*    DFHCBTS                                                    
*    DFHFEPI                                                    
*    DFHTASK_AC                                                 
*                                                                
DFHCBTS=N                                                       
DFHCHNL=N                                                       
DFHCICS_AD=N                                                    
DFHDATA=N                                                        
DFHDEST=N                                                       
DFHDOCH=N                                                       
DFHFEPI=N                                                       
DFHFILE=N                                                        
DFHJOUR=N                                                       
DFHMAPP=N                                                       
DFHPROG=N                                                       
DFHRMI=N                                                        
DFHSOCK=N                                                       
DFHSTOR=Y            
DFHSTOR_PC=N         
DFHSTOR_HWM=N        
DFHSTOR_AMT=N        
DFHSTOR_REQ=N        
DFHSTOR_OCC=N        
DFHSYNC=N            
DFHTASK_AC=N         
DFHTASK_JVM=N        
DFHTEMP=N            
DFHTERM=Y            
DFHWEBB=N            
DFHCPUT_DETAIL=N     
DFHSUSP_DETAIL=N     
DFHDISP_DETAIL=N     
DFHEXEC_DETAIL=N     
DFHMISC_DETAIL=N     
/*

After this job step completes, the output file can be browsed via ISPF, and should look similar to the following (scrolling from left to right):

BROWSE    HBR1.HTAC.SMF110.JSON                                                                        Line 0000000000 Col 001 132
 Command ===>                                                                                                      Scroll ===> CSR 
*********************************************************** Top of Data ************************************************************
{"time":1582038669.433,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582040439.810,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582041113.258,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582041481.835,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582042327.247,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582042964.258,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043117.056,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043117.296,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043117.513,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043117.710,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043117.902,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043118.104,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043118.286,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043118.484,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043118.690,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043118.882,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043358.001,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043734.300,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043734.309,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043734.313,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043734.318,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043740.078,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043753.260,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
{"time":1582043755.195,"CUSTOMER":"INGRAM","CONTEXT":"","TESTCASE":"","MFSOURCETYPE":"SMF110","SUBTYPE":"1","SUBSYSTEM":"CICS","CLAS
********************************************************** Bottom of Data **********************************************************

BROWSE    HBR1.HTAC.SMF110.JSON                                                                        Line 0000000000 Col 133 264
 Command ===>                                                                                                      Scroll ===> CSR 
*********************************************************** Top of Data ************************************************************
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 09:11:09.43 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 09:40:39.81 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 09:51:53.26 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 09:58:01.83 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:12:07.25 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:22:44.26 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:17.06 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:17.30 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:17.51 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:17.71 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:17.90 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:18.10 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:18.29 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:18.48 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:18.69 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:25:18.88 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:29:18.00 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:35:34.30 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:35:34.31 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:35:34.31 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:35:34.32 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:35:40.08 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:35:53.26 -0600","JOBNAME":"CICSA","JOBDAT
S":"3","VERSION":"0720","generated_by":"HBOFSI55:02/19/20 08.44","DATETIME":"2020-02-18 10:35:55.19 -0600","JOBNAME":"CICSA","JOBDAT
********************************************************** Bottom of Data **********************************************************

BROWSE    HBR1.HTAC.SMF110.JSON                                                                        Line 0000000000 Col 265 396
 Command ===>                                                                                                      Scroll ===> CSR 
*********************************************************** Top of Data ************************************************************
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.002648,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":1887.431957,"TRAN":"CSOL",
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.258036,"TRAN":"CSSY","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.001117,"TRAN":"CWBG","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":1887.437058,"TRAN":"CSOL",
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.257613,"TRAN":"CSSY","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.002869,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.000509,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.004461,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.002131,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.002370,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.003470,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.000504,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.002734,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.021162,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.002568,"TRAN":"HBNL","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.001374,"TRAN":"CSSY","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.004425,"TRAN":"CATA","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.000391,"TRAN":"CQRY","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.003930,"TRAN":"CQRY","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.000914,"TRAN":"CESN","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.044717,"TRAN":"CESN","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":0.015286,"TRAN":"CEMT","TE
E":"2020-02-09","JOBTIME":"15:57:40.16","GAPPLID":"CICSA","SAPPLID":"CICSA","TSLEVEL":"050500","DURATION":1.934138,"TRAN":"CEMT","TE
********************************************************** Bottom of Data **********************************************************

BROWSE    HBR1.HTAC.SMF110.JSON                                                                        Line 0000000000 Col 397 528
 Command ===>                                                                                                      Scroll ===> CSR 
*********************************************************** Top of Data ************************************************************
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 09:11:09.430042","STOP":"2020-02-18 09:11:09.432691","TRANNUM":"0040892
"TERM":"","USERID":"CICSA","TTYPE":"U","START":"2020-02-18 09:09:12.377926","STOP":"2020-02-18 09:40:39.809884","TRANNUM":"0000003",
RM":"","USERID":"CICSA","TTYPE":"U","START":"2020-02-18 09:51:53.000260","STOP":"2020-02-18 09:51:53.258297","TRANNUM":"0040893","TR
RM":"","USERID":"CICSA","TTYPE":"S","START":"2020-02-18 09:58:01.833461","STOP":"2020-02-18 09:58:01.834578","TRANNUM":"0040894","TR
"TERM":"","USERID":"CICSA","TTYPE":"U","START":"2020-02-18 09:40:39.809884","STOP":"2020-02-18 10:12:07.246942","TRANNUM":"0000003",
RM":"","USERID":"CICSA","TTYPE":"U","START":"2020-02-18 10:22:44.000498","STOP":"2020-02-18 10:22:44.258111","TRANNUM":"0040896","TR
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:17.052802","STOP":"2020-02-18 10:25:17.055671","TRANNUM":"0040897
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:17.295711","STOP":"2020-02-18 10:25:17.296221","TRANNUM":"0040898
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:17.508621","STOP":"2020-02-18 10:25:17.513083","TRANNUM":"0040899
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:17.708331","STOP":"2020-02-18 10:25:17.710462","TRANNUM":"0040900
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:17.899276","STOP":"2020-02-18 10:25:17.901647","TRANNUM":"0040901
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:18.100179","STOP":"2020-02-18 10:25:18.103650","TRANNUM":"0040902
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:18.285763","STOP":"2020-02-18 10:25:18.286267","TRANNUM":"0040903
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:18.481632","STOP":"2020-02-18 10:25:18.484366","TRANNUM":"0040904
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:18.668385","STOP":"2020-02-18 10:25:18.689547","TRANNUM":"0040905
RM":"0009","USERID":"CICSA","TTYPE":"TO","START":"2020-02-18 10:25:18.879170","STOP":"2020-02-18 10:25:18.881738","TRANNUM":"0040906
RM":"","USERID":"CICSA","TTYPE":"U","START":"2020-02-18 10:29:17.999763","STOP":"2020-02-18 10:29:18.001138","TRANNUM":"0040907","TR
RM":"","USERID":"CICSA","TTYPE":"U","START":"2020-02-18 10:35:34.296017","STOP":"2020-02-18 10:35:34.300442","TRANNUM":"0040908","TR
RM":"0011","USERID":"CICSA","TTYPE":"S","START":"2020-02-18 10:35:34.308706","STOP":"2020-02-18 10:35:34.309097","TRANNUM":"0040909"
RM":"0011","USERID":"CICSA","TTYPE":"S","START":"2020-02-18 10:35:34.309097","STOP":"2020-02-18 10:35:34.313027","TRANNUM":"0040909"
RM":"0011","USERID":"CICSA","TTYPE":"S","START":"2020-02-18 10:35:34.316595","STOP":"2020-02-18 10:35:34.317510","TRANNUM":"0040910"
RM":"0011","USERID":"CICSA","TTYPE":"TP","START":"2020-02-18 10:35:40.032932","STOP":"2020-02-18 10:35:40.077650","TRANNUM":"0040911
RM":"0011","USERID":"JOHN","TTYPE":"TO","START":"2020-02-18 10:35:53.245210","STOP":"2020-02-18 10:35:53.260496","TRANNUM":"0040912"
RM":"0011","USERID":"JOHN","TTYPE":"TO","START":"2020-02-18 10:35:53.260496","STOP":"2020-02-18 10:35:55.194635","TRANNUM":"0040912"
********************************************************** Bottom of Data **********************************************************

Transmit the Data to Splunk

This step is installation dependent.

Appendix A: HBR$MOLS Modifications

The IBM supplied version of DHFMNPDA describes the unloaded format for SMF 110 Performance records. However, DFHMNPDA does not include fields that we need to process data from a customer’s system, and create Splunk records from it. Particularly, it doesn’t include the time zone of the originating system.

To solve this, HBO$MOLS has been modified to reuse the space allotted to field PDRDB2TK in DFHMNPDA, and the 2 bytes following it (a total of 24 bytes). This allows us to add the following fields to the unloaded version of the SMF 110 records:

HBOMNLSO DS    XL8    LEAP SECOND OFFSET TOD FORMAT                    
HBOMNDTO DS    XL8    LOCAL TIME/DATE OFFSET                           
HBOMNRSD DS    XL4    JOB DATE                                         
HBOMNRST DS    XL4    JOB TIME

Here is the DFHMNPDA dsect as provided by IBM:

DFHMNPDA DSECT ,                   Unloaded Performance Data Record       
PDRJOBNM DS    CL8                 Jobname                                
PDRGAPPL DS    CL8                 Generic Applid                         
PDRSAPPL DS    CL8                 Specific Applid                        
PDRSID   DS    CL4                 System identification                  
PDRRVN   DS    XL2                 Record version - x'0vrm'               
PDRMFL   DS    XL2                 Record maintenance indicator           
         DS    XL4                 Reserved - spare                       
PDRCLASS DS    XL2                 Performance record class               
PDRSRTKY DS    0XL10               Cross system report sort key           
PDRSEQNO DS    XL2                 Syncpoint sequence number              
PDRDETT2 DS    XL8                 Transaction stop time                  
         SPACE                                                             
PDRDATE  DS    PL4                 Stop Date (unsigned packed)            
PDRTIME  DS    XL4                 Stop Time (binary)                     
PDRRESP  DS    XL4                 RESPonse Time (stop - start)           
PDRIRESP DS    XL4                 IRESPonse Time (resp - tciowtt)        
         DS    XL4                 Spare - reserved                       
PDRDB2TK DS    XL22                DB2 Accounting Correlation Token      
         DS    XL2                 Spare - reserved

Here is the HBOMNPDA that is used by HBO$MOLS to redefine the 24 bytes starting at PDRDB2TK:

*                                                                      
*  THE IBM SUPPLIED VERSION OF DFHMNPDA DOES NOT INCLUDE SOME          
*  SOME FIELDS THAT WE NEED TO PROCESS DATA FROM A CUSTOMER'S          
*  SYSTEM, AND CREATE SPLUNK RECORDS FROM IT.                          
*  PARTICULARLY, WE DON'T KNOW THE TIME ZONE OF THE ORIGINATING        
*  SYSTEM.                                                              
*                                                                      
*  TO SOLVE THIS, I'M GOING TO MODIFY DFH$MOLS TO REUSE THE SPACE      
*  ALLOTTED TO PDRDB2TK, AND THE 2 BYTES OF SPACE FOLLOWING IT          
*  (TOTAL OF 24 BYTES).                                                
*                                                                      
*  FIELD NAMES WERE CHANGED FROM PDR- TO HBO- TO AVOID NAME            
*  COLLISIONS.                                                           
*                                                                      
*                                                                      
HBOMNPDA DSECT ,                   Unloaded Performance Data Record    
HBOJOBNM DS    CL8                 Jobname                             
HBOGAPPL DS    CL8                 Generic Applid                      
HBOSAPPL DS    CL8                 Specific Applid                      
HBOSID   DS    CL4                 System identification               
HBORVN   DS    XL2                 Record version - x'0vrm'            
HBOMFL   DS    XL2                 Record maintenance indicator        
         DS    XL4                 Reserved - spare                    
HBOCLASS DS    XL2                 Performance record class            
HBOSRTKY DS    0XL10               Cross system report sort key        
HBOSEQNO DS    XL2                 Syncpoint sequence number           
HBODETT2 DS    XL8                 Transaction stop time               
         SPACE                                                         
HBODATE  DS    PL4                 Stop Date (unsigned packed)         
HBOTIME  DS    XL4                 Stop Time (binary)                  
HBORESP  DS    XL4                 RESPonse Time (stop - start)        
HBOIRESP DS    XL4                 IRESPonse Time (resp - tciowtt)     
         DS    XL4                 Spare - reserved                     
HBODB2TK DS    XL22                DB2 Accounting Correlation Token       
*                                                                      
*  BEGIN CHANGE HERE <-------------------------------------------------                                             
         ORG   HBODB2TK                                                
*  ADDED THE FOLLOWING FIELDS FROM THE SMF HEADER:                     
HBOMNLSO DS    XL8    LEAP SECOND OFFSET TOD FORMAT                    
HBOMNDTO DS    XL8    LOCAL TIME/DATE OFFSET                            
HBOMNRSD DS    XL4    JOB DATE                                         
HBOMNRST DS    XL4    JOB TIME                                         
*                                  THE FOLLOWING FIELD SUBSUMED ABOVE 
*        DS    XL2                 Spare - reserved                @P6A
*  END CHANGE HERE <---------------------------------------------------