Diagnosing problems in the MVS logger

Extended waits by the CICS® log manager can be caused by problems within the MVS™ logger or other areas of MVS. You can investigate these by looking at the MVS console messages. Look at the following:

Console messages and dumps

Look for:

Log stream data sets are of the form IXGLOGR.stream_name.Annnnnnn. The high level qualifier (IXGLOGR) may be different if the HLQ parameter was specified when the log stream was defined.

Explanations of MVS logger reason codes which are shown in CICS and MVS messages and traces are in the IXGCON macro and in the OS/390 MVS Assembler Services Reference manual.

GRS resource contention

To check GRS resource contention by displaying GRS enqueues and latch usage on all machines in the sysplex, issue either of the following MVS commands (the RO *ALL phrase means that the command goes to all systems in the sysplex):

   RO *ALL,D GRS,C
 
   RO *ALL,D GRS,RES=(SYSZLOGR,*)

A normal response looks like:

   D GRS,C
 
   ISG020I 12.06.49 GRS STATUS 647
   NO ENQ CONTENTION EXISTS
   NO LATCH CONTENTION EXISTS
 
   D GRS,RES=(SYSZLOGR,*)
 
   ISG020I 14.04.28 GRS STATUS 952
   NO REQUESTORS FOR RESOURCE  SYSZLOGR *

A response showing GRS contention looks like this. You may also see latch set name SYS.IXGLOGER_MISC:

   D GRS,C
 
   ISG020I 12.06.31 GRS STATUS 619
   LATCH SET NAME:  SYS.IXGLOGER_LCBVT
   CREATOR JOBNAME: IXGLOGR   CREATOR ASID: 0202
     LATCH NUMBER:  7
       REQUESTOR  ASID  EXC/SHR    OWN/WAIT
       IXGLOGR    0202  EXCLUSIVE  OWN
       IXGLOGR    0202  SHARED     WAIT
 
   D GRS,RES=(SYSZLOGR,*)
 
   ISG020I 19.58.33 GRS STATUS 374
   S=STEP    SYSZLOGR  91
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV26      MSLDELC1           002F       008F6370 EXCLUSIVE   OWN
   S=STEP    SYSZLOGR  93
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV26      MSLWRTC1           002E       008DED90 EXCLUSIVE   OWN
   MV26      MSLWRTC1           002E       008DB990 EXCLUSIVE   WAIT
   MV26      MSLWRTC1           002E       008DB700 EXCLUSIVE   WAIT
   MV26      MSLWRTC1           002E       008F60C8 EXCLUSIVE   WAIT
   S=SYSTEMS SYSZLOGR LPAYROL.TESTLOG.TLOG1
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV27      IXGLOGR            0011       008F7398 EXCLUSIVE   OWN
   MV26      IXGLOGR            0011       008F7398 EXCLUSIVE   WAIT

This shows which tasks (that is, MVS TCBs) have exclusive enqueues on the log streams, and which tasks are waiting for them. It is quite normal for enqueues and latches to be obtained, occasionally with contention. They are indications of a problem only if they last for more than a minute or so.

Long term enqueuing on the SYSZLOGR resource can be a sign of problems even if there is no contention.

You can choose to display only those log streams exclusively enqueued on by CICS jobs in the sysplex. Issue the following MVS command:

   D GRS,RES=(DFHSTRM,*)

A typical response to this command looks like this:

   ISG020I 14.51.28 GRS STATUS 541
   S=SYSTEMS DFHSTRM  PAYROL.CICSVR.DFHLGLOG
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108   SHARE     OWN
   MV29      PAYROL93           0044       007D9138   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.FWDRECOV.UTL3
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108   SHARE     OWN
   MV29      PAYROL93           0044       007D9138   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET1.DFHJ02
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET1.DFHLOG
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108 EXCLUSIVE   OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET1.DFHSHUNT
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL91           0042       007D9108 EXCLUSIVE   OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET3.DFHJ02
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL93           0044       007D9138   SHARE     OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET3.DFHLOG
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL93           0044       007D9138 EXCLUSIVE   OWN
   S=SYSTEMS DFHSTRM  PAYROL.IYK8ZET3.DFHSHUNT
   SYSNAME        JOBNAME         ASID     TCBADDR   EXC/SHR  OWN/WAIT
   MV29      PAYROL93           0044       007D9138 EXCLUSIVE   OWN

Checking coupling facility structure and couple data set status

To display the MVS logger couple data set status, issue the following MVS command:

   D XCF,CPL,TYPE=LOGR

A normal response looks like this:

   D XCF,CPL,TYPE=LOGR
   IXC358I  14.47.51  DISPLAY XCF 391
   LOGR COUPLE DATA SETS
   PRIMARY    DSN: SYS1.SYSPLEX2.SEQ26.PLOGR
              VOLSER: P2SS05     DEVN: 230D
              FORMAT TOD         MAXSYSTEM
              12/20/95 09:25:48          8
   ALTERNATE  DSN: SYS1.SYSPLEX2.SEQ26.ALOGR
              VOLSER: P2SS06     DEVN: 2C10
              FORMAT TOD         MAXSYSTEM
              12/20/95 09:27:45          8
   LOGR IN USE BY ALL SYSTEMS

If the response shows that LOGR is not in use by all systems, there may be a problem to investigate. Look for IXCxxx messages which might indicate the cause of the problem and issue the following command to attempt reconnection to the couple data set:

   SETXCF CPL,TYPE=(LOGR),PCOUPLE=(couple_dataset_name)

To display all structures with Failed_persistent connections, issue the following MVS command:

   D XCF,STR,STRNM=*,STATUS=FPCONN

The MVS logger should resolve any failed connections.

Checking log stream status

To display information about the status of CICS log streams, a batch job should issue the IXCMIAPU command:

   LIST LOGSTREAM NAME(streamname) DETAIL(YES)

You can use wildcards to select multiple log streams. For example, the following job produces a report on the system log streams for CICS region IYLX4:

   //IYLXLIST JOB NOTIFY=WILLIN,MSGCLASS=A
   //LOGLIST  EXEC PGM=IXCMIAPU
   //SYSPRINT DD SYSOUT=A,DCB=RECFM=FBA
   //SYSIN    DD *
     DATA TYPE(LOGR) REPORT(NO)
     LIST LOGSTREAM NAME(WILLIN.IYLX4.DFH*) DETAIL(YES)

Figure 19 shows a typical response to this command, with system logs streams for CICS region IYXL4.

Figure 19. Example output produced by the LIST LOGSTREAM NAME command
   LOGSTREAM NAME(WILLIN.IYLX4.DFHLOG) STRUCTNAME() LS_DATACLAS()
           LS_MGMTCLAS() LS_STORCLAS() HLQ(IXGLOGR) MODEL(NO) LS_SIZE(0)
           STG_MGMTCLAS() STG_STORCLAS() STG_DATACLAS() STG_SIZE(0)
           LOWOFFLOAD(40) HIGHOFFLOAD(85) STG_DUPLEX(YES) DUPLEXMODE(UNCOND)
           RMNAME() DESCRIPTION() RETPD(0) AUTODELETE(NO)
           DASDONLY(YES)
           MAXBUFSIZE(64000)
 
 
      LOG STREAM ATTRIBUTES:
 
        User Data:
          0000000000000000000000000000000000000000000000000000000000000000
          0000000000000000000000000000000000000000000000000000000000000000
 
 
      LOG STREAM CONNECTION INFO:
 
        SYSTEMS CONNECTED: 1
 
        SYSTEM    STRUCTURE         CON CONNECTION  CONNECTION
        NAME      VERSION           ID  VERSION     STATE
        --------  ----------------  --  ----------  ----------
        MV28      0000000000000000  00  00000000    N/A


       LOG STREAM DATA SET INFO:
 
        DATA SET NAMES IN USE: IXGLOGR.WILLIN.IYLX4.DFHLOG.<SEQ#>
 
        Ext.   <SEQ#>    Lowest Blockid    Highest GMT        Highest Local
        -----  --------  ----------------  -----------------  ---------------
       *00001  A0000007  0000000000496BAB  07/18/97 08:29:13  07/18/97 09:29:
 
        NUMBER OF DATA SETS IN LOG STREAM: 1
 
 
      POSSIBLE ORPHANED LOG STREAM DATA SETS:
 
        DATA SET NAMES:
        --------------------------------------------
        IXGLOGR.WILLIN.IYLX4.DFHLOG.A0000037
        IXGLOGR.WILLIN.IYLX4.DFHLOG.A0000404
 
        NUMBER OF POSSIBLE ORPHANED LOG STREAM DATA SETS: 2
 
 LOGSTREAM NAME(WILLIN.IYLX4.DFHSHUNT) STRUCTNAME() LS_DATACLAS()
           LS_MGMTCLAS() LS_STORCLAS() HLQ(IXGLOGR) MODEL(NO) LS_SIZE(0)
           STG_MGMTCLAS() STG_STORCLAS() STG_DATACLAS() STG_SIZE(0)
           LOWOFFLOAD(0) HIGHOFFLOAD(80) STG_DUPLEX(YES) DUPLEXMODE(UNCOND)
           RMNAME() DESCRIPTION() RETPD(0) AUTODELETE(NO)
           DASDONLY(YES)
           MAXBUFSIZE(64000)
 
 
      LOG STREAM ATTRIBUTES:
 
        User Data:
          0000000000000000000000000000000000000000000000000000000000000000
          0000000000000000000000000000000000000000000000000000000000000000
 
 
      LOG STREAM CONNECTION INFO:
 
        SYSTEMS CONNECTED: 1
 
        SYSTEM    STRUCTURE         CON CONNECTION  CONNECTION
        NAME      VERSION           ID  VERSION     STATE
        --------  ----------------  --  ----------  ----------
        MV28      0000000000000000  00  00000000    N/A
 
 
      LOG STREAM DATA SET INFO:
 
        DATA SET NAMES IN USE: IXGLOGR.WILLIN.IYLX4.DFHSHUNT.<SEQ#>
 
        Ext.   <SEQ#>    Lowest Blockid    Highest GMT        Highest Local
        -----  --------  ----------------  -----------------  ---------------
       *00001  A0000000  0000000000001F1E  07/16/97 12:52:22  07/16/97 13:52:
 
        NUMBER OF DATA SETS IN LOG STREAM: 1
 
 
      POSSIBLE ORPHANED LOG STREAM DATA SETS:
 
        NUMBER OF POSSIBLE ORPHANED LOG STREAM DATA SETS: 0

If you are using coupling facility log streams, the IXCMIAPU LIST STRUCTURE NAME(structname) DETAIL(YES) command is useful in finding the status of CICS log stream structures. For further information about these commands, see the OS/390 MVS Setting Up a Sysplex manual.

SMF and RMF statistics

SMF 88 log stream statistics records and RMF coupling facility usage reports are useful for analyzing problems that are affecting performance. Increasing the amount of coupling facility storage allocated to a structure, or the size of a staging data set, may improve both MVS logger performance and CICS performance.

Obtaining MVS logger and coupling facility dumps

If you suspect there is a problem within the MVS logger which is not a result of some other resolvable problem, you may need to collect additional diagnostic information. The dumps generated by CICS often don’t contain sufficient information about the MVS logger.

A dump of XCF and MVS logger address spaces from all systems is useful in the diagnosis of such problems. To obtain the dump, issue the following series of MVS commands:

DUMP COMM=(meaningful dump title)
R ww,JOBNAME=(IXGLOGR,XCFAS,cics_jobname),DSPNAME=('IXGLOGR'.*,'XCFAS'.*),CONT
R xx,STRLIST=(STRNAME=structure,(LISTNUM=ALL),ACC=NOLIM),CONT
R yy,REMOTE=(SYSLIST=*('XCFAS','IXGLOGR'),DSPNAME,SDATA),CONT
R zz,SDATA=(COUPLE,ALLNUC,LPA,LSQA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA),END

Use the R xx,STRLIST=(STRNAME=structure,(LISTNUM=ALL),ACC=NOLIM),CONT instruction only where you suspect a problem with the coupling facility structure.

Error records written to the MVS LOGREC data set may also be useful.

Setting a SLIP trap

The procedure described in the previous section produces "snapshots" of the MVS logger address space and coupling facility structure at the time the commands are issued. However, it is usually more useful to take a dump at the time an error occurs.

If you have applied MVS APAR OW27057, a dump of the MVS logger address space is produced automatically if an MVS IXGBRWSE or IXGDELET request fails because the MVS logger cannot find a specific log stream block identifier. (The MVS logger issues a return code of 8 with a reason code of 804.) To cater for other possible logger errors, or to obtain a dump of the coupling facility structure associated with a failing log stream, you can set an MVS serviceability level indication processing (SLIP) trap. Setting a SLIP trap causes MVS to take a specified set of actions when a specified event occurs. For example, you could specify that MVS is to take a dump of the MVS logger address space if CICS issues a particular message.

Figure 20 shows an example SLIP trap that captures a dump of the CICS address space, the MVS logger address space, and the coupling facility structure associated with the failing logstream.

Figure 20. An example SLIP trap. The trap is triggered when CICS issues a DFHLG0772 message. It captures dumps of the CICS address space, the MVS logger address space, and the coupling facility structure associated with the failing log stream.
SLIP SET,IF,LPAMOD=(IGC0003E,0),DATA=(1R?+
4,EQ,C4C6C8D3,+8,EQ,C7F0F7F7,+C,EQ,F2),A=S    <change the message
VCD,JOBLIST=(cicsjob,IXGLOGR,XCFAS),          <change CICS Job
 
-->response xx
 
xx,DSPNAME=('XCFAS'.*,'IXGLOGR'.*),STRLIST
=(STRNAME=structname,LOCKENTRIES,ACC=NOLIM    <change STRNAME
,(LISTNUM=ALL,
 
-->response yy
 
yy,ENTRYDATA=SERIALIZE,ADJUNCT=CAPTURE)),S
DATA=(RGN,XESDATA,ALLNUC,CSA,LSQA,PSA,SQA,
SWA,TRT,COUPLE,WLM,GRSQ,LPA),
 
-->response zz
 
zz,ID=LOGR,REMOTE=(JOBLIST,DSPNAME,SDATA),
END

In this example, the SLIP triggers when a specific CICS log manager message--DFHLG0772--is written to the console. This is specified in the EQ parameter of the SLIP:

+4,EQ,C4C6C8D3,+8,EQ,C7F0F7F7,+C,EQ,F2)
       D F H L        G 0 7 7        2     <equates to

You can also set a more "generic" trap, that is triggered by the occurence of any one of a range of messages. For example, to cause the SLIP to be triggered by any log manager message in the DFHLG07xx range, alter the value of the EQ parameter to:

+4,EQ,C4C6C8D3,+8,EQ,C7F0F7),
       D F H L        G 0 7      <equates to

To use the example SLIP, you must:

  1. Replace the cicsjob value with the name of the CICS job (or jobs) to be dumped.
  2. Replace the xx, yy, and zz values with the appropriate operator reply numbers, as each segment is entered.
  3. Replace the structname value with the name of the coupling facility structure that contains the failing log stream.

    For system log failures only, you can get the name of the coupling facility structure (or structures) from the two DFHLG0104 messages that were issued when CICS connected to DFHLOG and DFHSHUNT during the run in which the failure occurred.

    For all other log streams, to get the name of the coupling facility structure use the LIST LOGSTREAM NAME command already described. For example:

    //LOGRRPT  EXEC PGM=IXCMIAPU
    //SYSPRINT DD SYSOUT=*
    //SYSIN    DD *
     DATA
       TYPE(LOGR)
       REPORT(YES)
       LIST LOGSTREAM NAME(logstream_name) DETAIL(YES)

    Figure 19 shows example output produced by the LIST LOGSTREAM NAME command. Search for the log stream name; the structure name follows it.

    Step 3 assumes two things:

Notes:
  1. The example SLIP will just fit into the extended operator command area of MVS Version 5 or later.
  2. The example SLIP may result in extra dumps being produced for both CICS and the MVS logger address space.

For definitive information about setting SLIP traps, see the OS/390 MVS Diagnostics: Tools and Service Aids manual, SY28-1085-03.

Restarting the MVS logger address space

If the MVS logger address space has failed, it can be restarted by the command S IXGLOGRS

Note the 'S' at the end--IXGLOGRS restarts IXGLOGR as a system address space.

After the MVS logger has restarted, you must restart all CICS regions.

CAUTION:

If you forcibly cancel the MVS logger address space (by issuing a FORCE IXGLOGR,ARM command) or coupling facility structures used by the MVS logger (by issuing a SETXCF FORCE,CON,STRNAME=structname,CONNAME=ALL command), there is a risk of corruption in the CICS system logs. If the system log is corrupted, CICS issues a message telling you that you need to perform an initial start. Data integrity will be compromised because of the loss of log data required for consistency.

Related concepts
Log manager waits
Some conditions that cause CICS log manager error messages
Related tasks
Dealing with a corrupt system log
Restarting CICS after a system log failure
[[ Contents Previous Page | Next Page Index ]]