Extended waits by the CICS® log manager can be caused by problems within the MVS™ logger or other areas of MVS. You can investigate these by looking at the MVS console messages. Look at the following:
Look for:
Log stream data sets are of the form IXGLOGR.stream_name.Annnnnnn. The high level qualifier (IXGLOGR) may be different if the HLQ parameter was specified when the log stream was defined.
Explanations of MVS logger reason codes which are shown in CICS and MVS messages and traces are in the IXGCON macro and in the OS/390 MVS Assembler Services Reference manual.
To check GRS resource contention by displaying GRS enqueues and latch usage on all machines in the sysplex, issue either of the following MVS commands (the RO *ALL phrase means that the command goes to all systems in the sysplex):
RO *ALL,D GRS,C
RO *ALL,D GRS,RES=(SYSZLOGR,*)
A normal response looks like:
D GRS,C
ISG020I 12.06.49 GRS STATUS 647
NO ENQ CONTENTION EXISTS
NO LATCH CONTENTION EXISTS
D GRS,RES=(SYSZLOGR,*)
ISG020I 14.04.28 GRS STATUS 952
NO REQUESTORS FOR RESOURCE SYSZLOGR *
A response showing GRS contention looks like this. You may also see latch set name SYS.IXGLOGER_MISC:
D GRS,C
ISG020I 12.06.31 GRS STATUS 619
LATCH SET NAME: SYS.IXGLOGER_LCBVT
CREATOR JOBNAME: IXGLOGR CREATOR ASID: 0202
LATCH NUMBER: 7
REQUESTOR ASID EXC/SHR OWN/WAIT
IXGLOGR 0202 EXCLUSIVE OWN
IXGLOGR 0202 SHARED WAIT
D GRS,RES=(SYSZLOGR,*)
ISG020I 19.58.33 GRS STATUS 374
S=STEP SYSZLOGR 91
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV26 MSLDELC1 002F 008F6370 EXCLUSIVE OWN
S=STEP SYSZLOGR 93
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV26 MSLWRTC1 002E 008DED90 EXCLUSIVE OWN
MV26 MSLWRTC1 002E 008DB990 EXCLUSIVE WAIT
MV26 MSLWRTC1 002E 008DB700 EXCLUSIVE WAIT
MV26 MSLWRTC1 002E 008F60C8 EXCLUSIVE WAIT
S=SYSTEMS SYSZLOGR LPAYROL.TESTLOG.TLOG1
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV27 IXGLOGR 0011 008F7398 EXCLUSIVE OWN
MV26 IXGLOGR 0011 008F7398 EXCLUSIVE WAIT
This shows which tasks (that is, MVS TCBs) have exclusive enqueues on the log streams, and which tasks are waiting for them. It is quite normal for enqueues and latches to be obtained, occasionally with contention. They are indications of a problem only if they last for more than a minute or so.
Long term enqueuing on the SYSZLOGR resource can be a sign of problems even if there is no contention.
You can choose to display only those log streams exclusively enqueued on by CICS jobs in the sysplex. Issue the following MVS command:
D GRS,RES=(DFHSTRM,*)
A typical response to this command looks like this:
ISG020I 14.51.28 GRS STATUS 541
S=SYSTEMS DFHSTRM PAYROL.CICSVR.DFHLGLOG
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL91 0042 007D9108 SHARE OWN
MV29 PAYROL93 0044 007D9138 SHARE OWN
S=SYSTEMS DFHSTRM PAYROL.FWDRECOV.UTL3
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL91 0042 007D9108 SHARE OWN
MV29 PAYROL93 0044 007D9138 SHARE OWN
S=SYSTEMS DFHSTRM PAYROL.IYK8ZET1.DFHJ02
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL91 0042 007D9108 SHARE OWN
S=SYSTEMS DFHSTRM PAYROL.IYK8ZET1.DFHLOG
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL91 0042 007D9108 EXCLUSIVE OWN
S=SYSTEMS DFHSTRM PAYROL.IYK8ZET1.DFHSHUNT
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL91 0042 007D9108 EXCLUSIVE OWN
S=SYSTEMS DFHSTRM PAYROL.IYK8ZET3.DFHJ02
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL93 0044 007D9138 SHARE OWN
S=SYSTEMS DFHSTRM PAYROL.IYK8ZET3.DFHLOG
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL93 0044 007D9138 EXCLUSIVE OWN
S=SYSTEMS DFHSTRM PAYROL.IYK8ZET3.DFHSHUNT
SYSNAME JOBNAME ASID TCBADDR EXC/SHR OWN/WAIT
MV29 PAYROL93 0044 007D9138 EXCLUSIVE OWN
To display the MVS logger couple data set status, issue the following MVS command:
D XCF,CPL,TYPE=LOGR
A normal response looks like this:
D XCF,CPL,TYPE=LOGR
IXC358I 14.47.51 DISPLAY XCF 391
LOGR COUPLE DATA SETS
PRIMARY DSN: SYS1.SYSPLEX2.SEQ26.PLOGR
VOLSER: P2SS05 DEVN: 230D
FORMAT TOD MAXSYSTEM
12/20/95 09:25:48 8
ALTERNATE DSN: SYS1.SYSPLEX2.SEQ26.ALOGR
VOLSER: P2SS06 DEVN: 2C10
FORMAT TOD MAXSYSTEM
12/20/95 09:27:45 8
LOGR IN USE BY ALL SYSTEMS
If the response shows that LOGR is not in use by all systems, there may be a problem to investigate. Look for IXCxxx messages which might indicate the cause of the problem and issue the following command to attempt reconnection to the couple data set:
SETXCF CPL,TYPE=(LOGR),PCOUPLE=(couple_dataset_name)
To display all structures with Failed_persistent connections, issue the following MVS command:
D XCF,STR,STRNM=*,STATUS=FPCONN
The MVS logger should resolve any failed connections.
To display information about the status of CICS log streams, a batch job should issue the IXCMIAPU command:
LIST LOGSTREAM NAME(streamname) DETAIL(YES)
You can use wildcards to select multiple log streams. For example, the following job produces a report on the system log streams for CICS region IYLX4:
//IYLXLIST JOB NOTIFY=WILLIN,MSGCLASS=A
//LOGLIST EXEC PGM=IXCMIAPU
//SYSPRINT DD SYSOUT=A,DCB=RECFM=FBA
//SYSIN DD *
DATA TYPE(LOGR) REPORT(NO)
LIST LOGSTREAM NAME(WILLIN.IYLX4.DFH*) DETAIL(YES)
Figure 19 shows a typical response to this command, with system logs streams for CICS region IYXL4.
LOGSTREAM NAME(WILLIN.IYLX4.DFHLOG) STRUCTNAME() LS_DATACLAS()
LS_MGMTCLAS() LS_STORCLAS() HLQ(IXGLOGR) MODEL(NO) LS_SIZE(0)
STG_MGMTCLAS() STG_STORCLAS() STG_DATACLAS() STG_SIZE(0)
LOWOFFLOAD(40) HIGHOFFLOAD(85) STG_DUPLEX(YES) DUPLEXMODE(UNCOND)
RMNAME() DESCRIPTION() RETPD(0) AUTODELETE(NO)
DASDONLY(YES)
MAXBUFSIZE(64000)
LOG STREAM ATTRIBUTES:
User Data:
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
LOG STREAM CONNECTION INFO:
SYSTEMS CONNECTED: 1
SYSTEM STRUCTURE CON CONNECTION CONNECTION
NAME VERSION ID VERSION STATE
-------- ---------------- -- ---------- ----------
MV28 0000000000000000 00 00000000 N/A
LOG STREAM DATA SET INFO:
DATA SET NAMES IN USE: IXGLOGR.WILLIN.IYLX4.DFHLOG.<SEQ#>
Ext. <SEQ#> Lowest Blockid Highest GMT Highest Local
----- -------- ---------------- ----------------- ---------------
*00001 A0000007 0000000000496BAB 07/18/97 08:29:13 07/18/97 09:29:
NUMBER OF DATA SETS IN LOG STREAM: 1
POSSIBLE ORPHANED LOG STREAM DATA SETS:
DATA SET NAMES:
--------------------------------------------
IXGLOGR.WILLIN.IYLX4.DFHLOG.A0000037
IXGLOGR.WILLIN.IYLX4.DFHLOG.A0000404
NUMBER OF POSSIBLE ORPHANED LOG STREAM DATA SETS: 2
LOGSTREAM NAME(WILLIN.IYLX4.DFHSHUNT) STRUCTNAME() LS_DATACLAS()
LS_MGMTCLAS() LS_STORCLAS() HLQ(IXGLOGR) MODEL(NO) LS_SIZE(0)
STG_MGMTCLAS() STG_STORCLAS() STG_DATACLAS() STG_SIZE(0)
LOWOFFLOAD(0) HIGHOFFLOAD(80) STG_DUPLEX(YES) DUPLEXMODE(UNCOND)
RMNAME() DESCRIPTION() RETPD(0) AUTODELETE(NO)
DASDONLY(YES)
MAXBUFSIZE(64000)
LOG STREAM ATTRIBUTES:
User Data:
0000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000
LOG STREAM CONNECTION INFO:
SYSTEMS CONNECTED: 1
SYSTEM STRUCTURE CON CONNECTION CONNECTION
NAME VERSION ID VERSION STATE
-------- ---------------- -- ---------- ----------
MV28 0000000000000000 00 00000000 N/A
LOG STREAM DATA SET INFO:
DATA SET NAMES IN USE: IXGLOGR.WILLIN.IYLX4.DFHSHUNT.<SEQ#>
Ext. <SEQ#> Lowest Blockid Highest GMT Highest Local
----- -------- ---------------- ----------------- ---------------
*00001 A0000000 0000000000001F1E 07/16/97 12:52:22 07/16/97 13:52:
NUMBER OF DATA SETS IN LOG STREAM: 1
POSSIBLE ORPHANED LOG STREAM DATA SETS:
NUMBER OF POSSIBLE ORPHANED LOG STREAM DATA SETS: 0
If you are using coupling facility log streams, the IXCMIAPU LIST STRUCTURE NAME(structname) DETAIL(YES) command is useful in finding the status of CICS log stream structures. For further information about these commands, see the OS/390 MVS Setting Up a Sysplex manual.
SMF 88 log stream statistics records and RMF coupling facility usage reports are useful for analyzing problems that are affecting performance. Increasing the amount of coupling facility storage allocated to a structure, or the size of a staging data set, may improve both MVS logger performance and CICS performance.
If you suspect there is a problem within the MVS logger which is not a result of some other resolvable problem, you may need to collect additional diagnostic information. The dumps generated by CICS often don’t contain sufficient information about the MVS logger.
A dump of XCF and MVS logger address spaces from all systems is useful in the diagnosis of such problems. To obtain the dump, issue the following series of MVS commands:
DUMP COMM=(meaningful dump title)
R ww,JOBNAME=(IXGLOGR,XCFAS,cics_jobname),DSPNAME=('IXGLOGR'.*,'XCFAS'.*),CONT
R xx,STRLIST=(STRNAME=structure,(LISTNUM=ALL),ACC=NOLIM),CONT
R yy,REMOTE=(SYSLIST=*('XCFAS','IXGLOGR'),DSPNAME,SDATA),CONT
R zz,SDATA=(COUPLE,ALLNUC,LPA,LSQA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA),END
Use the R xx,STRLIST=(STRNAME=structure,(LISTNUM=ALL),ACC=NOLIM),CONT instruction only where you suspect a problem with the coupling facility structure.
Error records written to the MVS LOGREC data set may also be useful.
The procedure described in the previous section produces "snapshots" of the MVS logger address space and coupling facility structure at the time the commands are issued. However, it is usually more useful to take a dump at the time an error occurs.
If you have applied MVS APAR OW27057, a dump of the MVS logger address space is produced automatically if an MVS IXGBRWSE or IXGDELET request fails because the MVS logger cannot find a specific log stream block identifier. (The MVS logger issues a return code of 8 with a reason code of 804.) To cater for other possible logger errors, or to obtain a dump of the coupling facility structure associated with a failing log stream, you can set an MVS serviceability level indication processing (SLIP) trap. Setting a SLIP trap causes MVS to take a specified set of actions when a specified event occurs. For example, you could specify that MVS is to take a dump of the MVS logger address space if CICS issues a particular message.
Figure 20 shows an example SLIP trap that captures a dump of the CICS address space, the MVS logger address space, and the coupling facility structure associated with the failing logstream.
SLIP SET,IF,LPAMOD=(IGC0003E,0),DATA=(1R?+
4,EQ,C4C6C8D3,+8,EQ,C7F0F7F7,+C,EQ,F2),A=S <change the message
VCD,JOBLIST=(cicsjob,IXGLOGR,XCFAS), <change CICS Job
-->response xx
xx,DSPNAME=('XCFAS'.*,'IXGLOGR'.*),STRLIST
=(STRNAME=structname,LOCKENTRIES,ACC=NOLIM <change STRNAME
,(LISTNUM=ALL,
-->response yy
yy,ENTRYDATA=SERIALIZE,ADJUNCT=CAPTURE)),S
DATA=(RGN,XESDATA,ALLNUC,CSA,LSQA,PSA,SQA,
SWA,TRT,COUPLE,WLM,GRSQ,LPA),
-->response zz
zz,ID=LOGR,REMOTE=(JOBLIST,DSPNAME,SDATA),
END
In this example, the SLIP triggers when a specific CICS log manager message--DFHLG0772--is written to the console. This is specified in the EQ parameter of the SLIP:
+4,EQ,C4C6C8D3,+8,EQ,C7F0F7F7,+C,EQ,F2)
D F H L G 0 7 7 2 <equates to
You can also set a more "generic" trap, that is triggered by the occurence of any one of a range of messages. For example, to cause the SLIP to be triggered by any log manager message in the DFHLG07xx range, alter the value of the EQ parameter to:
+4,EQ,C4C6C8D3,+8,EQ,C7F0F7),
D F H L G 0 7 <equates to
To use the example SLIP, you must:
For system log failures only, you can get the name of the coupling facility structure (or structures) from the two DFHLG0104 messages that were issued when CICS connected to DFHLOG and DFHSHUNT during the run in which the failure occurred.
For all other log streams, to get the name of the coupling facility structure use the LIST LOGSTREAM NAME command already described. For example:
//LOGRRPT EXEC PGM=IXCMIAPU
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DATA
TYPE(LOGR)
REPORT(YES)
LIST LOGSTREAM NAME(logstream_name) DETAIL(YES)
Figure 19 shows example output produced by the LIST LOGSTREAM NAME command. Search for the log stream name; the structure name follows it.
Step 3 assumes two things:
For definitive information about setting SLIP traps, see the OS/390 MVS Diagnostics: Tools and Service Aids manual, SY28-1085-03.
If the MVS logger address space has failed, it can be restarted by the command S IXGLOGRS
Note the 'S' at the end--IXGLOGRS restarts IXGLOGR as a system address space.
After the MVS logger has restarted, you must restart all CICS regions.
If you forcibly cancel the MVS logger address space (by issuing a FORCE IXGLOGR,ARM command) or coupling facility structures used by the MVS logger (by issuing a SETXCF FORCE,CON,STRNAME=structname,CONNAME=ALL command), there is a risk of corruption in the CICS system logs. If the system log is corrupted, CICS issues a message telling you that you need to perform an initial start. Data integrity will be compromised because of the loss of log data required for consistency.