HP-UX 11.31 System information & find out part number of a failed disk with sasmgr

18-05-2018 | Remy van Elst


Table of Contents


On one of my regular scheduled datacenter visits one of the older HP-UX Itanium machines had an orange light on the front. These systems are not (yet) monitored, but still in use so the disk had to be replaced. Not knowing anything about this system or which parts were used, I managed to find the exact part number and device type so we could order a spare. This small guide uses sasmgr to get the data on HP-UX 11.31.

If you like this article, consider sponsoring me by trying out a Digital Ocean VPS. With this link you'll get a $5 VPS for 2 months free (as in, you get $10 credit). (referral link)

Hardware Information on HP-UX

Since I normally don't manage these systems and have little experience with HP-UX, I first had to find out some information on the system. Using a few regular Unix tools like netstat, uname and a few HP-UX specific tools like model, machinfo I was able to gather basic hardware information of this machine.

$ TERM=vt100 ssh hpux09 -l root

Last successful login:       Fri May 18 08:18:05 METDST 2018     
Last authentication failure: Fri May 18 08:10:59 METDST 2018     
Last login: Thu Jun 24 09:46:55 2010 from   
(c)Copyright 1983-2006 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993  The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2006 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.

Confidential computer software. Valid license from HP required for
possession, use or copying.  Consistent with FAR 12.211 and 12.212,
Commercial Computer Software, Computer Software Documentation, and
Technical Data for Commercial Items are licensed to the U.S. Government
under vendor's standard commercial license.

You have mail.

Value of TERM has been set to "vt100". 
WARNING:  YOU ARE SUPERUSER !!

hpux09:/root# swlist | grep -i oe
  HPUX11i-BOE                           B.11.31.0809   HP-UX Base Operating Environment 


hpux09:/root# model
ia64 hp server rx3600


hpux09:/root# uname -a
HP-UX hpux09 B.11.31 U ia64 [...] unlimited-user license


hpux09:/root# machinfo
CPU info:
  2 Intel(R) Itanium 2 9100 series processors (1.67 GHz, 18 MB)
          666 MT/s bus, CPU version A1
          4 logical processors (2 per socket)

Firmware info:
   Firmware revision:  04.03
   FP SWA driver revision: 1.18
   IPMI is supported on this system.
   BMC firmware revision: 5.25

Platform info:
   Model:                  "ia64 hp server rx3600"
   Machine ID number:      2e6e7[...]f36
   Machine serial number:  D[...]D

OS info:
   Nodename:  hpux09
   Release:   HP-UX B.11.31
   Version:   U (unlimited-user license)
   Machine:   ia64
   ID Number: [...]
   vmunix _release_version:               
@(#) $Revision: vmunix:    B.11.31_LR FLAVOR=perf 

This system was installed a long while ago:

hpux09:/root# swlist -l bundle -a date | grep HPUX
  HPUX-FTPServer        Wed Dec 10 16:58:16 MET 2008 
  HPUX-MailServer       Wed Dec 10 16:58:16 MET 2008 

print_manifest is a usefull command for information on the system.

hpux09:/root# print_manifest | grep -i created
    The system was created December 10, 2008, 18:02:06 MET.
    It was created with Ignite-UX revision C.7.7.93.

This will go into the hardware documentation, since, sadly, that is very incomplete.

Disk information

Now to the good part, for which we logged in to this machine. Find the serial number and order a new (probably refurbished) disk. The system uses LVM with RAID 1 underneath via a SAS card so when the disk is ordered we can just replace it. If it was just LVM we had to do some configuration

This PDF WhenGoodDisksGoBad.pdf is very helpfull in preparing and researching this issue. When the new disk is here, this page will be helpfull for the replacement.

The first command shows which disk is the failed one plus the raid configuration:

sasmgr get_info -D /dev/sasd0 -q raid

Output:

Wed May 16 14:54:21 2018

---------- LOGICAL DRIVE 2 ----------

Raid Level                                         : RAID 1              
Volume sas address                                 : 0x3c5c0600358833e   
Device Special File                                : /dev/rdsk/c0t3d0    
Raid State                                         : DEGRADED            
Raid Status Flag                                   : ENABLED             
Raid Size                                          : 139136              
Rebuild Rate                                       : 20.00 %             
Rebuild Progress                                   : 100.00 %            

Participating Physical Drive(s) :

SAS Address         Enc   Bay   Size(MB)      Type            State           

0x500000e01d33a9c2   1     8    140014        PRIMARY         ONLINE         
0x5000c50003aeb459   1     2    140014        SECONDARY       FAILED         

---------- LOGICAL DRIVE 7 ----------

Raid Level                                         : RAID 1              
Volume sas address                                 : 0xacb8d6daa52eb4d   
Device Special File                                : /dev/rdsk/c0t2d0    
Raid State                                         : OPTIMAL             
Raid Status Flag                                   : ENABLED             
Raid Size                                          : 69878               
Rebuild Rate                                       : 0.00 %              
Rebuild Progress                                   : 100.00 %            

Participating Physical Drive(s) :

SAS Address         Enc   Bay   Size(MB)      Type            State           

0x5000039668124686   1     1    286102        SECONDARY       ONLINE         
0x5000c5000b9f5965   1     7    70007         PRIMARY         ONLINE         

The second command shows the location (controller-wise) and the part number of the disk:

sasmgr get_info -D /dev/sasd0 -q lun=all -q lun_locate 

Output:

RAID VOL ID is 7 : 
/dev/rdsk/c0t2d0          0/4/1/0.0.0.2.0          

Physical disks in volume are :
        1     1     OFF           HP            EG0300FCSPH        HPD0
        1     7     OFF           HP            DG072BB975         HPDC

RAID VOL ID is 2 : 
/dev/rdsk/c0t3d0          0/4/1/0.0.0.3.0          

Physical disks in volume are :
        1     8     OFF           HP            DG146BABCF         HPD5
        1     2     OFF           HP            DG146ABAB4         HPD5

In my case disk 1,2 failed:

    0x5000c50003aeb459   1     2    140014        SECONDARY       FAILED         
    [...]
        1     2     OFF           HP            DG146ABAB4         HPD5

With this part number we can order a new disk.

dmesg was also spitting out warning about this disk issue:

SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
LVM: VG 64 0x010000: Lost quorum.
This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by current link) must become available:
<3 0x000006> 
LVM: VG 64 0x010000: PVLink 3 0x000006 Failed! The PV is not accessible.
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x0acb8d6daa52eb4d (dev_t 0x03000004) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x03c5c0600358833e (dev_t 0x03000006) failed with CDB status 0x00000400
class : lunpath, instance 3
lun path (class = lunpath, instance = 3) belonging to LUN (default minor = 0x6) has gone offline.  The lunpath hwpath is 0/4/1/0.0x3c5c0600358833e.0x0

SAS driver at 0/4/1/0 (/dev/sasd0) : RAID volume at SAS address 0x03c5c0600358833e is in a degraded state.
SAS driver at 0/4/1/0 (/dev/sasd0) : RAID volume at SAS address 0x03c5c0600358833e is in a degraded state.
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x0acb8d6daa52eb4d (dev_t 0x03000004) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x0acb8d6daa52eb4d (dev_t 0x03000004) failed with CDB status 0x00000400
SAS driver at 0/4/1/0 (/dev/sasd0) : TM (0x00000002) to SAS address 0x0acb8d6daa52eb4d (dev_t 0x03000004) failed with CDB status 0x00000400
LVM: VG 64 0x010000: Reestablished quorum.
LVM: VG 64 0x010000: PVLink 3 0x000006 Recovered.
LVM: NOTICE: VG 64 0x010000: LV 1: All I/O requests to this LV that were
        waiting indefinitely for an unavailable PV have now completed.

Tags: hp  hp-ux  itanium  lvm  raid  sas  sasmgr  tutorials  unix