Raymii.org
Quis custodiet ipsos custodes?Home | About | All pages | Cluster Status | RSS Feed
That time when one of my HP-UX servers lost half of it's RAM (and how to connect to an HP iLO 2 with modern OpenSSH (7.6+))
Published: 06-06-2018 | Author: Remy van Elst | Text only version of this article
❗ This post is over five years old. It may no longer be up to date. Opinions may have changed.
Table of Contents
One of my favorite sayings is: 'Hardware is stupid, move everything to the cloud!'. The cloud is just someone elses computer, but at least I'm not responsible for the hardware anymore, since hardware breaks. When a VM breaks, because you use configuration management and version control, just roll out a new one. We all know that's not true, but still, the thought of it is nice. A man can have hopes and dreams, even if the harsh reality shoots them down every time.
Last week one of the HP-UX machines had a failing disk and this week it's back with a whole new issue. After it was rebooted (due to issues with the services running on it), the Event Monitoring Service (EMS) sent an email regarding RAM issues and after manual checking it seems the machine lost half of it's RAM.
It should have 16 GB and now it only has 8 GB. You might imagine my suprise. This post goes into my troubleshooting, since I was not able to go to the machine, shut it down and check if the RAM was still there or note part numbers. I'll cover the use of cstm (Support Tool Manager), how to connect to the HP iLO (out of band access) with modern OpenSSH (7.2) and the steps I took to gather information on what might have happened.
Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:
I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!
Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.
You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days.
This machine is not under monitoring yet, therefore it is regularly visited and checked manually. You could call that monitoring as well, but I rather have Icinga (or any other tool) doing work for me. An intern doing said things is not considered a tool, sadly (this is a joke). The machine furthermore is part of an older setup (10 years) and is being replaced, so there is not a lot of budget or time allocated. I like to learn new things, so I seize the oppertunity to expand my UNIX knowledge, thus allocating some of my own time to research (and hopefully fix) these issues. Read my other HP-UX articles as well if you like that kind of stuff.
Now that you know some background, let's dive in.
DRAM failure on DIMM XX, deallocate rank
It all started with this nice email from the HP-UX system:
Onderwerp: hpux09: Event Monitor Notification
>------------ Event Monitoring Service Event Notification ------------<
Notification Time: Fri Jun 1 10:59:36 2018
hpux09 sent Event Monitor notification information:
/system/events/ipmi_fpl/ipmi_fpl is >= 3.
Its current value is MAJORWARNING(3).
Event data from monitor:
Event Time..........: Fri Jun 1 10:59:36 2018
Severity............: MAJORWARNING
Monitor.............: fpl_em
Event #.............: 890
System..............: hpux09
Summary:
DRAM failure on DIMM XX, deallocate rank
Description of Error:
SFW has detected that a DRAM is failing on the DIMM specified by the physical location. The rank the failing DIMM is part of will be deallocated.
Probable Cause / Recommended Action:
SFW detected a failing DIMM
Replace the DIMM flagged by SFW
Additional Event Data:
System IP Address...: 198.51.100.30
Event Id............: 0x5b110af800000000
Monitor Version.....: A.01.00
Event Class.........: System
Client Configuration File...........:
/var/stm/config/tools/monitor/default_fpl_em.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: ia64 hp server rx3600
EMS Version.....................: A.04.20.31.02
STM Version.....................: D.04.00
System Serial Number............: DEH48456BK
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/fpl_em.htm#890
v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v
IPMI event hex: 0x7a800fa000e00250 0xffffffff010cff74 Time Stamp: Fri Jun 1 08:44:24 2018 Event keyword: MEM_CHIPSPARE_DEALLOC_RANK Alert level name: Warning Reporting vers: 1 Data field type: Reserved Decoded data field:
Reporting entity ID: 0 ( Cab 0 Cell 0 CPU 0 ) Reporting entity Full Name: System Firmware IPMI Event ID : 4000 (0xfa0)
>---------- End Event Monitoring Service Event Notification ----------<
The HP-UX error message is not very helpful, since no actual DIMM location is given:
DRAM failure on DIMM XX, deallocate rank
SFW has detected that a DRAM is failing on the DIMM specified by the physical location. The rank the failing DIMM is part of will be deallocated.
The machine has no DIMM slot XX. The purchase order stated that the machine came with 16 GB of RAM installed. It now only reported 8 GB of memory.
My first thought was to search the part number and purchase replacement modules (or search our own stash of "old stuff"), then doing my regular procedure with RAM issues, only executing the next step if the previous action did not resolve the issue. I've had my fair share of Dell hardware issues (looking at you, noncritical raid controller error after upgrading OpenManage...) so my procedure has proven itself.
- Apply all available firmware updates for the hardware
- Do a DIMM swap (slot XX to slot YY) to see if the error is in the slot or in the DIMM
- Replace the DIMM
- Replace the motherboard and CPU
- Contact the manufacturer for warranty
Since this machine is not under a support contract anymore, step 1 and 5 are off the table. HP requires an active support contract to get firmware updates. I do not have access to replacement DIMM's and I was not able to go to the machine to do a DIMM swap (or note part numbers).
The only thing left to do was to dig into the issue and gather as much logging and information as possible, to prepare for a visit to the machine.
However, first I ended up researching a term unfamiliar to me, Memory Ranks.
Memory Ranking
I had never heard of the term Rank in RAM context, but Wikipedia came to the rescue:
Sometimes memory modules are designed with two or more independent sets of DRAM chips connected to the same address and data buses; each such set is called a rank.
There is even an article on Memory Ranking:
A memory rank is a set of DRAM chips connected to the same chip select, which are therefore accessed simultaneously.
That still didn't tell me much, how correct it might be. This site had a very helpfull and practical explanation:
The term "rank" simply refers to a 64-bit chunk of data. In its simplest form, a DIMM with DRAM chips on just one side would contain a single 64-bit chunk of data and would be called a single-rank (1R) module. DIMMs with chips on both sides often contain at least two 64-bit chunks of data and are referred to as dual-rank (2R) modules. Some DIMMs can have DRAM chips on both sides but are configured so that they contain two 64-bit data chunks on each sidefour in totaland are referred to as quad-rank (4R) modules. Quad-rank DIMMs run at a maximum PC3-8500 (DDR3-1066) speed in current architecture.
Now that I had a better understanding of what a memory rank is, my suspicion is that there is one failed memory module and the rest of the modules (after that failed module) are not loaded anymore.
Support Tool Manager
Using cstm I hope to find the part number of the DIMM's so I can at least
order a few replacement modules. According to multiple HPe forum posts
ctsm should report the part number with this command:
echo "selclass qualifier memory;info;wait;infolog" | /usr/sbin/cstm
In my case, the output did not contain the part number:
Running Command File (/usr/sbin/stm/ui/config/.stmrc).
-- Information --
Support Tools Manager
Version D.04.00
Product Number B4708AA
(C) Copyright Hewlett Packard Co. 1995-2008
All Rights Reserved
Use of this program is subject to the licensing restrictions described
in "Help-->On Version". HP shall not be liable for any damages resulting
from misuse or unauthorized use of this program.
cstm>selclass qualifier memory;info;wait;infolog
-- Updating Map --
Updating Map...
-- Converting a (4036) byte raw log file to text. --
Preparing the Information Tool Log for IPF_MEMORY on path memory File ...
.... hpux09 : 198.51.100.30 ....
-- Information Tool Log for IPF_MEMORY on path memory --
Log creation time: Fri Jun 1 12:13:01 2018
Hardware path: memory
Basic Memory Description
Module Type: MEMORY
Page Size: 4096 Bytes
Total Physical Memory: N/A
Total Configured Memory: 8192 MB
Total Deconfigured Memory: N/A
Memory Board Inventory
DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
Ext 0 DIMM 0A 2048 Ext 0 DIMM 0B 2048
Ext 0 DIMM 0C 2048 Ext 0 DIMM 0D 2048
Ext 0 DIMM 1A ---- Ext 0 DIMM 1B ----
Ext 0 DIMM 1C ---- Ext 0 DIMM 1D ----
Ext 0 DIMM 2A ---- Ext 0 DIMM 2B ----
Ext 0 DIMM 2C ---- Ext 0 DIMM 2D ----
Ext 0 Total: 8192 (MB)
===========================================================================
DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
Ext 1 DIMM 0A ---- Ext 1 DIMM 0B ----
Ext 1 DIMM 0C ---- Ext 1 DIMM 0D ----
Ext 1 DIMM 1A ---- Ext 1 DIMM 1B ----
Ext 1 DIMM 1C ---- Ext 1 DIMM 1D ----
Ext 1 DIMM 2A ---- Ext 1 DIMM 2B ----
Ext 1 DIMM 2C ---- Ext 1 DIMM 2D ----
Ext 1 Total: 0 (MB)
===========================================================================
Memory Error Log Summary
The memory error log is empty.
Page Deallocation Table (PDT)
The Page Deallocation Table is empty.
PDT Entries Used: 0
PDT Entries Free: 100
PDT Total Size: 100
-- Information Tool Log for IPF_MEMORY on path memory --
View - To View the file.
Print - To Print the file.
SaveAs - To Save the file.
Enter Done, Help, Print, SaveAs, or View: [Done] #.
It did gave me a better idea of the physical memory layout.
According to this post, the output with part number should look like this:
DIMM Location Size(MB) State Serial Num Part Num
-------------------- -------- ------- ---------------- ------------------
Cab 0 Cell 0 DIMM 0A 2048 Config PRY07064US A9846-60301
Cab 0 Cell 0 DIMM 0B 2048 Config PRY07063JF A9846-60301
Cab 0 Cell 0 DIMM 1A 2048 Config PRY06372A2 A9846-60301
-------------------- -------- ------- ---------------- ------------------
But, as can be seen, no part numbers in my output. I guess it's a version difference. I found another command to get ALL the hardware in the machine:
echo "selall;infolog;wait"|cstm
but except for a ton of output, it did not contain any part number. The post did reference the following:
You can get the memory dimm part number from cstm and also from GSP/MP
Run the following command..
GSP>cm
GSP:CM>df ==> Select A for selecting all and D to dump.
From this out put you would get the exact part number for the dimm and also other HW connected to this box.
The MP referred here is the "HP Integrated Lights Out Management Processor",
shortly known as the iLO. Dell calls them iDrac (integrated Dell remote access
controller) and on a SuperMicro server it's just called IPMI or OOB (out of band
access). It provides a way to power on/off and troubleshoot the server when you
cannot access it, often also a remote console.
Since I was not able to go to the machine and reboot into some kind of BIOS or iLO console, I had to resort to connecting via the web or SSH. The web interface was useless on gathering RAM information, so SSH was my last resort.
SSH with modern OpenSSH (7.6) to an HP iLO2
With good hope I connected to the iLO IP from my Ubuntu 18.04 box. Only to be greeted by a happy little error message:
$ ssh Admin@192.0.2.30
Unable to negotiate with 192.0.2.30 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1
Time to configure some old settings. In my ~/.ssh/config file I started with
the following:
Host hpux09-ilo
HostName 192.0.2.30
KexAlgorithms diffie-hellman-group1-sha1
But of course, just a KeyAlgorithm is not enough:
$ ssh Admin@hpux09-ilo
Unable to negotiate with 192.0.2.30 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc
Let's add that ciphersuite to my ~/.ssh/config:
Host hpux09-ilo
HostName 192.0.2.30
KexAlgorithms diffie-hellman-group1-sha1
Ciphers aes128-cbc,3des-cbc
We know that the setting did something, because now it just fails with no helpful error:
$ ssh Admin@hpux09-ilo
Received disconnect from 192.0.2.30 port 22:11: Client Disconnect
Disconnected from 192.0.2.30 port 22
Lucky for me, the iLO 2 was horribly old and insecure in 2013 already, as this post shows. With OpenSSH 6.2 there were problems connecting to the iLO, back then. I'm on OpenSSH 7.6 so let's hope that their fix works for me as well.
In a firmware update for the iLO2 some of these bugs are fixed, but not all of them, and quoting Oscar A. Perez (who lists "Senior Embedded System Engineer, 100% committed to make Embedded Systems reliable, safe and secure" on it's LinkedIn for 15 years, so I guess probably is legit), it will be hard to fix in the future due to the limited iLO 2 memory:
I had to make lots of changes to the mpSSH server code to get it to work with the new OpenSSH 6.2p1. I hope this is the last time we have to make changes like this one. iLO2 memory is very limited and already full so, we won't be able to spin new firmware releases, every time the OpenSSH folks decide to increase the size of the payload during Key Exchange.
Lower on in the post I do find the correct OpenSSH options to connect. I missed
the HostKeyAlgorithms and the MACs. The complete, working configuration in
my ~/.ssh/config file looks like this:
Host hpux09-ilo
HostName 192.0.2.30
HostKeyAlgorithms ssh-rsa,ssh-dss
KexAlgorithms diffie-hellman-group1-sha1
Ciphers aes128-cbc,3des-cbc
MACs hmac-md5,hmac-sha1
A one-liner with these options:
ssh -o HostKeyAlgorithms=ssh-rsa,ssh-dss -o KexAlgorithms=diffie-hellman-group1-sha1 -o Ciphers=aes128-cbc,3des-cbc -o MACs=hmac-md5,hmac-sha1 Admin@hpux09-ilo
*Do note that a better solution here is to upgrade the hardware and get it under a support contract.
iLO Management Processor hardware information
Logging in gives me a few options to work with:
Admin@192.0.2.30's password:
Hewlett-Packard Integrity Integrated Lights-Out 2
(c) Copyright Hewlett-Packard Company 1999-2008. All Rights Reserved.
MP Host Name: mphpux09
Revision F.02.23
MP MAIN MENU:
CO: Console
VFP: Virtual Front Panel
CM: Command Menu
SMCLP: Server Management Command Line Protocol
CL: Console Log
SL: Show Event Logs
HE: Main Help Menu
X: Exit Connection
[mphpux09] MP>
The forum post stated to go into the Command Menu:
[mphpux09] MP> cm
(Use Ctrl-B to return to MP main menu.)
Then to enter the following command:
[mphpux09] MP:CM> df -nc -a
The output contains a long, long list of hardware. You can find it at the bottom of this article. We are interested in the RAM parts. To my pleasent suprise it did list the actual RAM in the machine, the 16 GB, including the part numbers. The Operating system does not see the Ext1 DIMM's, the iLO does:
Ext0
The cstm output showed me that Ext0 is filled with 4 DIMM's of 2 GB each:
Memory Board Inventory
DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
Ext 0 DIMM 0A 2048 Ext 0 DIMM 0B 2048
Ext 0 DIMM 0C 2048 Ext 0 DIMM 0D 2048
Ext 0 DIMM 1A ---- Ext 0 DIMM 1B ----
Ext 0 DIMM 1C ---- Ext 0 DIMM 1D ----
Ext 0 DIMM 2A ---- Ext 0 DIMM 2B ----
Ext 0 DIMM 2C ---- Ext 0 DIMM 2D ----
Ext 0 Total: 8192 (MB)
The ILO confirms that:
PRODUCT INFO:
FRU Entry # 16 :
FRU NAME : MemExt0 DIMM0A
FRU ID : 0128
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D209E
Mfg Unique Serial # : 0x00CE010640711D209E
FRU Entry # 17 :
FRU NAME : MemExt0 DIMM0B
FRU ID : 0136
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D20AA
Mfg Unique Serial # : 0x00CE010640711D20AA
FRU Entry # 18 :
FRU NAME : MemExt0 DIMM0C
FRU ID : 0144
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D20A2
Mfg Unique Serial # : 0x00CE010640711D20A2
FRU Entry # 19 :
FRU NAME : MemExt0 DIMM0D
FRU ID : 0152
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D20A8
Mfg Unique Serial # : 0x00CE010640711D20A8
I cannot easily find a replacement for this part number, but it comes up in a HP forum post as a Samsung 2GB module.
Ext1
Ext1 according to the operating system is empty:
DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
Ext 1 DIMM 0A ---- Ext 1 DIMM 0B ----
Ext 1 DIMM 0C ---- Ext 1 DIMM 0D ----
Ext 1 DIMM 1A ---- Ext 1 DIMM 1B ----
Ext 1 DIMM 1C ---- Ext 1 DIMM 1D ----
Ext 1 DIMM 2A ---- Ext 1 DIMM 2B ----
Ext 1 DIMM 2C ---- Ext 1 DIMM 2D ----
Ext 1 Total: 0 (MB)
The iLO thinks differently:
FRU Entry # 20 :
FRU NAME : MemExt1 DIMM0A
FRU ID : 0160
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD925C4D3
Mfg Unique Serial # : 0x002C0C0836D925C4D3
FRU Entry # 21 :
FRU NAME : MemExt1 DIMM0B
FRU ID : 0168
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD72FB723
Mfg Unique Serial # : 0x002C0C0836D72FB723
FRU Entry # 22 :
FRU NAME : MemExt1 DIMM0C
FRU ID : 0176
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD925C4D9
Mfg Unique Serial # : 0x002C0C0836D925C4D9
FRU Entry # 23 :
FRU NAME : MemExt1 DIMM0D
FRU ID : 0184
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD925C4CB
Mfg Unique Serial # : 0x002C0C0836D925C4CB
So the DIMM's are still in the system, and of all places, Amazon sells
these DIMM's. The type is 2GB DDR2 PC2-5300 667MHz 240pin ECC, which is
exactly what I need to order a replacement or look into our hardware stash. It
took me a good hour, but the part number and some more type information has been
found.
I still need to figure out which specific DIMM broke, but that is not in the above output.
iLO Event Log
In the iLO menu, I also saw SL: Show Event Logs. Maybe that will tell me
specifically which DIMM could be the culprit.
[mphpux09] MP> sl
Event Log Viewer Menu:
Log Name Entries % Full Latest Timestamped Entry
---------------------------------------------------------------------------
E - System Event 32 4 % 01 Jun 2018 08:47:06
F - Forward Progress 4000 100 %
B - Current Boot 130 43 %
P - Previous Boot 130 43 %
I - iLO Event 316 63 % 01 Jun 2018 11:14:55
C - Clear All Logs
L - Live Events
Let's view the System Event log, using option E:
Enter menu item or [Ctrl-B] to Quit: E
Log Name Entries % Full Latest Timestamped Entry
---------------------------------------------------------------------------
E - System Event 32 4 % 01 Jun 2018 08:47:06
Event Log Navigation Help:
+ View next block (forward in time, e.g. from 3 to 4)
- View previous block (backward in time, e.g. from 3 to 2)
<CR> Continue to the next or previous block
D Dump the entire log
F First entry
L Last entry
J Jump to entry number
H View mode configuration - Hex
K View mode configuration - Keyword
T View mode configuration - Text
A Alert Level Filter options
U Alert Level Unfiltered
? Display this Help menu
Q Quit and return to the Event Log Viewer Menu
Ctrl-B Exit command, and return to the MP Main Menu
Just give me everything, D it is:
MP:SL (+,-,<CR>,D, F, L, J, H, K, T, A, U, ? for Help, Q or Ctrl-B to Quit) >d
Confirm? (Y/[N]): y
# Location|Alert| Encoded Field | Data Field | Keyword / Timestamp
-------------------------------------------------------------------------------
0 BMC 2 0x205A9918E5020010 FFFF0103FCC00300 TIME_SET
02 Mar 2018 09:27:01
1 BMC 2 0x205A991B79020020 FFFF0103FCC00300 TIME_SET
02 Mar 2018 09:38:01
2 BMC 2 0x205ABBA1FC020030 FFFF0103FCC00300 TIME_SET
28 Mar 2018 14:09:00
3 BMC 2 0x205ABBA490020040 FFFF0103FCC00300 TIME_SET
28 Mar 2018 14:20:00
4 BMC 2 0x205B0FEF26020050 FFFF006F04140300 POWER_BUTTON_PRESSED
31 May 2018 12:48:38
5 BMC 2 0x205B0FEF28020060 040EA37004120300 CHASSIS_CONTROL_REQUEST
31 May 2018 12:48:40
6 HPUX 2 2 0x54801C3002E00070 00000000001A100C HP-UX_OS_NORMAL_SHUTDOWN
31 May 2018 12:48:54
7 BMC 2 0x205B0FEF38020090 FFFF006F04140300 POWER_BUTTON_PRESSED
31 May 2018 12:48:56
8 BMC 2 0x205B0FEF3B0200A0 040EA37004120300 CHASSIS_CONTROL_REQUEST
31 May 2018 12:48:59
9 BMC 2 0x205B0FEF420200B0 FFFF056FFA220300 ACPI_SOFT_OFF
31 May 2018 12:49:06
10 BMC 2 0x205B0FEF430200C0 FA00A370FA120300 CHASSIS_CONTROL_REQUEST
31 May 2018 12:49:07
11 BMC 2 0x205B0FEF440200D0 FFFF000943090300 POWER_UNIT_DISABLED
31 May 2018 12:49:08
12 BMC 2 0x205B0FEF550200E0 FFFF006F04140300 POWER_BUTTON_PRESSED
31 May 2018 12:49:25
13 BMC 2 0x205B0FEF560200F0 FFFF027000120300 SOFT_RESET
31 May 2018 12:49:26
14 BMC 2 0x205B0FEF56020100 FFFF010943090300 POWER_UNIT_ENABLED
31 May 2018 12:49:26
15 BMC 2 0x205B0FEF57020110 FFFF006FFA220300 ACPI_ON
31 May 2018 12:49:27
16 BMC 2 0x205B0FEF57020120 0401A37004120300 CHASSIS_CONTROL_REQUEST
31 May 2018 12:49:27
17 BMC 2 0x205B0FEF64020130 FFFF027000120300 SOFT_RESET
31 May 2018 12:49:40
18 SFW 2 0xC15B0FEF71020140 FFFF000A001D0300 CPU_START_BOOT
31 May 2018 12:49:53
19 SFW 0 2 0x5480006300E00150 0000000000000000 BOOT_START
31 May 2018 12:49:53
20 SFW 0 *3 0x7A800FA000E00170 FFFFFFFF010CFF74 MEM_CHIPSPARE_DEALLOC_RANK
31 May 2018 12:50:01
21 SFW 0 *3 0x7A800FA000E00190 FFFFFFFF010BFF74 MEM_CHIPSPARE_DEALLOC_RANK
31 May 2018 12:50:01
22 SFW 0 2 0x40801CBB00E001B0 0000000000000000 BOOT_SWITCH_INSECURE_MODE
31 May 2018 12:50:24
23 HPUX 0 2 0x54801C2F00E001D0 0000000000001001 HP-UX_BOOT_COMPLETE
31 May 2018 12:52:39
24 HPUX 0 2 0x54801C3000E001F0 00000000001A100C HP-UX_OS_NORMAL_SHUTDOWN
01 Jun 2018 08:44:01
25 BMC 2 0x205B110757020210 FFFF027000120300 SOFT_RESET
01 Jun 2018 08:44:07
26 SFW 2 0xC15B110760020220 FFFF000A001D0300 CPU_START_BOOT
01 Jun 2018 08:44:16
27 SFW 0 2 0x5480006300E00230 0000000000000000 BOOT_START
01 Jun 2018 08:44:16
28 SFW 0 *3 0x7A800FA000E00250 FFFFFFFF010CFF74 MEM_CHIPSPARE_DEALLOC_RANK
01 Jun 2018 08:44:24
29 SFW 0 *3 0x7A800FA000E00270 FFFFFFFF010BFF74 MEM_CHIPSPARE_DEALLOC_RANK
01 Jun 2018 08:44:24
30 SFW 0 2 0x40801CBB00E00290 0000000000000000 BOOT_SWITCH_INSECURE_MODE
01 Jun 2018 08:44:48
31 HPUX 0 2 0x54801C2F00E002B0 0000000000001001 HP-UX_BOOT_COMPLETE
01 Jun 2018 08:47:06
-> This is the last entry in the selected log.
MP:SL (+,-,<CR>,D, F, L, J, H, K, T, A, U, ? for Help, Q or Ctrl-B to Quit) >
Filtering out all the logs (reboots were expected), focussing on the DIMM parts:
28 SFW 0 *3 0x7A800FA000E00250 FFFFFFFF010CFF74 MEM_CHIPSPARE_DEALLOC_RANK
29 SFW 0 *3 0x7A800FA000E00270 FFFFFFFF010BFF74 MEM_CHIPSPARE_DEALLOC_RANK
Not much help, no clear DIMM location yet. But, I did search around and found this HP support page titled "HP Integrity rx3600 Servers - BOOT DECONFIG CPU Can Be Caused by Memory Dimm Failure". In the log output there are these lines, looking a lot like the above output:
SFW 0 0 0x040000E500E00000 FFFFFFFF000AFF74 MEM_SPD_2G_DIMM_FOUND
SFW 0 0 0x040000E500E00000 FFFFFFFF000BFF74 MEM_SPD_2G_DIMM_FOUND
SFW 0 0 0x040000E500E00000 FFFFFFFF001AFF74 MEM_SPD_2G_DIMM_FOUND
SFW 0 0 0x040000E500E00000 FFFFFFFF001BFF74 MEM_SPD_2G_DIMM_FOUND
SFW 0 0 0x040000E500E00000 FFFFFFFF010AFF74 MEM_SPD_2G_DIMM_FOUND
SFW 0 0 0x040000E500E00000 FFFFFFFF010BFF74 MEM_SPD_2G_DIMM_FOUND
SFW 0 0 0x040000E500E00000 FFFFFFFF011AFF74 MEM_SPD_2G_DIMM_FOUND
SFW 0 0 0x040000E500E00000 FFFFFFFF011BFF74 MEM_SPD_2G_DIMM_FOUND
In this part I do see a pattern in the Data Field column, the only thing changing looks an awfull lot like a memory location:
- 00 0A
- 00 0B
- 00 1A
- 00 1B
- 01 0A
- 01 0B
- 01 1A
- 01 AB
Comparing that to the DIMM layout output from earlier:
DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
Ext 0 DIMM 0A 2048 Ext 0 DIMM 0B 2048
Ext 0 DIMM 0C 2048 Ext 0 DIMM 0D 2048
DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
Ext 1 DIMM 0A ---- Ext 1 DIMM 0B ----
Ext 1 DIMM 0C ---- Ext 1 DIMM 0D ----
Not exactly the same but good enough, since it's a different server (the above output seems to be for the 8 port memory carrier board). This post confirms my suspicion on the HEX values corresponding to the DIMM Slots.
HP Integrity rx3600 Server User Service Guide
The HP Integrity rx3600 Server User Service Guide, chapter 5 Troubleshooting, subsection CPU, Memory and SBA, subsection Troubleshooting rx3600 memory has a picture of the 24 slot memory carrier board:
I know this server has 2 24 slot memory carrier boards due to the full output of
the hardware list, it states 4 12 DIMM Memory Extender components.
The service guide also lists the error message:
- IPMI events: Type E0h, 4000d:26d MEM CHIPSPARE DEALLOC_RANK
- Cause: An SDRAM is failing on the DIMM.
- Notes: The failing DIMM quad will be deallocated.
Furthermore, the service manual states this:
* Troubleshooting rx3600 Memory
The memory controller logic in the zx2 chip supports three versions of memory expanders. An eight
DIMM memory carrier provides two memory boards that hold two or four memory DIMMs in both
memory cells 0 and 1. A 24 DIMM memory carrier provides two 12-DIMM memory boards that
hold four, eight, or twelve DIMMs in both memory cells 0 and 1.
All three versions of memory expanders must have their memory DIMMs installed in groups of four,
known as a quad. DIMM quads of different sizes can be installed in any physical rank on all
versions of memory expanders, but they must be grouped by their size.
* Memory Subsystem Behaviors
You must replace DIMMs or memory carriers when a threshold is reached for multiple double-byte
errors from one or more DIMMs on the same board. When any uncorrectable memory error (more
than 2 bytes) or when no quad of like memory DIMMs is loaded in rank 0 of side 0, you must
replace the DIMMs. All other DIMM errors are corrected by zx2 and reported to the Page
Deallocation Table (PDT) and the diagnostic LED panel.
* Memory DIMM Load Order
For a minimally loaded server, four equal-size memory DIMMs must be installed in slots 0A, 0B,
0C, and 0D on the same side of the 24/48 slot memory expander; and in the 0A and 0B slots
on both 0 and 1 sides of the 8 slot memory expander.
The first quad of DIMMs are always loaded into rank 0s slots for side 0 then in the rank 0s slots
for side 1. The next quad of DIMMs are loaded into rank 1s slots for side 0, then for side 1, and
so on, until all ranks slots for both sides are full.
Best memory subsystem performance result when both memory sides 0 and 1 have the same number
of DIMM quads in them.
In chapter 6. "Removing and replacing server components", there are 11 pages (185-196), with picutes and example configurations, on replacing the DIMM's. It also has Memory loading guidelines:
Use the following rules and guidelines when installing memory:
- Install DIMMs in pairs in the 8-DIMM memory carrier and in quads in the 24-DIMM memory carrier.
- Ensure all DIMMs within a pair or quad are identical.
- Install quads in order of capacity from largest to smallest. For example, install all 2 GB quads before 1 GB or smaller quads, and install all 1 GB quads before 512 MB quads.
- Side 0 must have equal or greater memory capacity than side 1.
- Install DIMM pairs or quads based on the following rules:
- 1. Load pairs or quads into the memory carrier in order, starting with slot 0 and ending with slot 2.
- 2. Install the first pair or quad in side 0.
- 3. Install the second pair or quad in side 1.
- 4. For the remaining pairs or quads:
- a. If both sides of the memory carrier contain the same capacity of memory, install the next pair or quad in side 0.
- b. If side 0 contains more memory capacity than side 1, install the next quad in side 1.
- c. If side 1 is full, install the remaining quads in side 0.
(Nested lists in Markdown are fun)
The guide even has a list of Customer Replacable parts including HP and
replacement part numbers. For my 2GB memory module, it would be AD328A, for
sale on lots of places.
Conclusion
Combining all the knowledge and logging, my best guess is that the following slots have issues:
- Side 1 Slot 0B
- Side 1 Slot 0C
Replacement DIMM's are ordered and on their way, soon to be replaced in the correct order. Let's hope that the machine get's the other half of it's RAM back and the problem is fixed.
Reference, complete output of MP:CM> df -nc -a
FRU Entry # 0 :
FRU NAME: Core I/O Board ID:0000
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 7462080
Manufacturer : CELESTICA
Product Name : Core IO Board with VGA
S/N : MYL011Y014
Part Number : AB463-60003
Fru File ID : 10
Custom Info : A
Custom Info : 4848
Custom Info : A5
Custom Info : 0
PRODUCT INFO:
FRU Entry # 1 :
FRU NAME: Mem Extender 0 ID:0001
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6729120
Manufacturer : CELESTICA
Product Name : 12 DIMM Memory Extender
S/N : TH9842X51H
Part Number : AB463-60112
Fru File ID : 10
Custom Info : A
Custom Info : 4721
Custom Info : A2
Custom Info : 0
PRODUCT INFO:
FRU Entry # 2 :
FRU NAME: Mem Extender 1 ID:0002
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6729120
Manufacturer : CELESTICA
Product Name : 12 DIMM Memory Extender
S/N : TH9842X51J
Part Number : AB463-60112
Fru File ID : 10
Custom Info : A
Custom Info : 4721
Custom Info : A2
Custom Info : 0
PRODUCT INFO:
FRU Entry # 3 :
FRU NAME: Power Supply 0 ID:0003
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6662537
Manufacturer : C&D
Product Name : BULK POWER SUPPLY
S/N : A804070ET3
Part Number : 0957-2198
Fru File ID : 10
Custom Info : 00000000
Custom Info : 0804
Custom Info : 07
Custom Info : 0
PRODUCT INFO:
FRU Entry # 4 :
FRU NAME: Power Supply 1 ID:0004
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6675416
Manufacturer : muRata-ps
Product Name : BULK POWER SUPPLY
S/N : A8371000F7
Part Number : 0957-2198
Fru File ID : 10
Custom Info : 00000000
Custom Info : 0837
Custom Info : 10
Custom Info : 0
PRODUCT INFO:
FRU Entry # 5 :
FRU NAME: I/O Assembly ID:0005
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6655680
Manufacturer : CELESTICA
Product Name : 10 Slot PCI-E 1.1 IOBP
S/N : TH9834955H
Part Number : AB463-60028
Fru File ID : 10
Custom Info : A
Custom Info : 4832
Custom Info : A2
Custom Info : 0
PRODUCT INFO:
FRU Entry # 6 :
FRU NAME: Display Board ID:0006
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6703200
Manufacturer : CELESTICA
Product Name : DVD/Display Board
S/N : TH983910DA
Part Number : AB463-60020
Fru File ID : 10
Custom Info : A
Custom Info : 4814
Custom Info : A3
Custom Info : 0
PRODUCT INFO:
FRU Entry # 7 :
FRU NAME: Disk Backplane ID:0007
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6668640
Manufacturer : CELESTICA
Product Name : 8 Disk Drive SAS Backplane
S/N : TH9835V650
Part Number : AB463-60006
Fru File ID : 10
Custom Info : A
Custom Info : 4814
Custom Info : A3
Custom Info : 0
PRODUCT INFO:
FRU Entry # 8 :
FRU NAME: ProcessorCarrier ID:0009
CHASSIS INFO:
Type:Rack Mount Chassis
Part Number :
Serial Number :
BOARD INFO:
Mfg Date/Time : 6732745
Manufacturer : JABIL
Product Name : 2 Socket CPU Carrier
S/N : MYJ84203PH
Part Number : AB463-60113
Fru File ID : 10
Custom Info : C
Custom Info : 4818
Custom Info : A3
Custom Info : 0
PRODUCT INFO:
Manufacturer : hp
Product Name : server rx3600
Part/Model :
Version :
S/N :
Asset Tag :
FRU File ID : 11
Custom Info : 411
FRU Entry # 9 :
FRU NAME: Interconnect Bd ID:0010
CHASSIS INFO:
Type:Rack Mount Chassis
Part Number :
Serial Number :
BOARD INFO:
Mfg Date/Time : 6694560
Manufacturer : CELESTICA
Product Name : SAS Interconnect Board
S/N : TH9838Z061
Part Number : AB463-60025
Fru File ID : 10
Custom Info : A
Custom Info : 4814
Custom Info : A2
Custom Info : 0
PRODUCT INFO:
Manufacturer : hp
Product Name : server rx3600
Part/Model : AB596A
Version :
S/N : DEH48456BK
Asset Tag :
FRU File ID : 11
Custom Info : 411
FRU Entry # 10 :
FRU NAME: Hot-Plug Board ID:0011
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 6658560
Manufacturer : CELESTICA
Product Name : PCI Hot Plug Control Board
S/N : TH983515H7
Part Number : AB463-60002
Fru File ID : 10
Custom Info : A
Custom Info : 4711
Custom Info : A2
Custom Info : 0
PRODUCT INFO:
FRU Entry # 11 :
FRU NAME: I/O Power Module ID:0015
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 2105376
Manufacturer : C&D
Product Name : PCI POWER BOARD
S/N : 9080708250F8
Part Number : 0950-4677
Fru File ID : 10
Custom Info : X1
Custom Info : 0825
Custom Info : A1
Custom Info : 0
PRODUCT INFO:
FRU Entry # 12 :
FRU NAME: Processor 0 ID:0032
PROCESSOR DATA
S-spec/QDF: LAB5
Sample/Prod: 01
CORE DATA
Arch Revision : 00
Core Family : 20
Core Model : 01
Core Stepping : 01
Max Core Frequency : 1666 MHZ
Max SysBus Frequency : 333 MHZ
Core Voltage : 1150 mV
Core Voltage Tolerance,High : 64 mV
Core Voltage Tolerance,Low : 3F mV
CACHE DATA
Cache Size : 18 MB
PACKAGE DATA
Package Revision : NE Substrate Revision: 01
PROC PART NUMBER DATA
Part Number : 80567KF
Electronic Signature : 0003AB71B8C89784
THERMAL REF DATA
Upper Temp Ref : 92 C
Calibr Offset : 18 C
FEATURES DATA
IA-32 Proc Core Feature Flags: FFFB8743
IA-64 Proc Core Feature Flags: 1B81806300000000
Package Feature Flags : 3F010000
Devices on TAP Chain : 2
FRU Entry # 13 :
FRU NAME: Processor 1 ID:0033
PROCESSOR DATA
S-spec/QDF: LAB5
Sample/Prod: 01
CORE DATA
Arch Revision : 00
Core Family : 20
Core Model : 01
Core Stepping : 01
Max Core Frequency : 1666 MHZ
Max SysBus Frequency : 333 MHZ
Core Voltage : 1150 mV
Core Voltage Tolerance,High : 64 mV
Core Voltage Tolerance,Low : 3F mV
CACHE DATA
Cache Size : 18 MB
PACKAGE DATA
Package Revision : NE Substrate Revision: 01
PROC PART NUMBER DATA
Part Number : 80567KF
Electronic Signature : 0001E4BA8B93CDD5
THERMAL REF DATA
Upper Temp Ref : 92 C
Calibr Offset : 17 C
FEATURES DATA
IA-32 Proc Core Feature Flags: FFFB8743
IA-64 Proc Core Feature Flags: 1B81806300000000
Package Feature Flags : 3F010000
Devices on TAP Chain : 2
FRU Entry # 14 :
FRU NAME: Processor 0 RAM ID:0036
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 0
Manufacturer :
Product Name : MTV_A1_1618
S/N : PR1084441T
Part Number : AD391-2100C
Fru File ID : B
Custom Info :
Custom Info : 4733
Custom Info :
Custom Info : 2H64014D2b82
Custom Info : 1
PRODUCT INFO:
FRU Entry # 15 :
FRU NAME: Processor 1 RAM ID:0037
CHASSIS INFO:
BOARD INFO:
Mfg Date/Time : 0
Manufacturer :
Product Name : MTV_A1_1618
S/N : PR108443XV
Part Number : AD391-2100C
Fru File ID : B
Custom Info :
Custom Info : 4733
Custom Info :
Custom Info : 2H64014D2b82
Custom Info : 1
PRODUCT INFO:
FRU Entry # 16 :
FRU NAME : MemExt0 DIMM0A
FRU ID : 0128
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D209E
Mfg Unique Serial # : 0x00CE010640711D209E
FRU Entry # 17 :
FRU NAME : MemExt0 DIMM0B
FRU ID : 0136
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D20AA
Mfg Unique Serial # : 0x00CE010640711D20AA
FRU Entry # 18 :
FRU NAME : MemExt0 DIMM0C
FRU ID : 0144
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D20A2
Mfg Unique Serial # : 0x00CE010640711D20A2
FRU Entry # 19 :
FRU NAME : MemExt0 DIMM0D
FRU ID : 0152
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0xCE00000000000000
JEDEC Mfg Location : 0x01
JEDEC Mfg Part # : M3 93T5750CZ3-CD5
JEDEC Mfg Revision Code : 0x3343
JEDEC Mfg Year : 0x06
JEDEC Mfg Week : 0x40
JEDEC Mfg Serial # : 0x711D20A8
Mfg Unique Serial # : 0x00CE010640711D20A8
FRU Entry # 20 :
FRU NAME : MemExt1 DIMM0A
FRU ID : 0160
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD925C4D3
Mfg Unique Serial # : 0x002C0C0836D925C4D3
FRU Entry # 21 :
FRU NAME : MemExt1 DIMM0B
FRU ID : 0168
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD72FB723
Mfg Unique Serial # : 0x002C0C0836D72FB723
FRU Entry # 22 :
FRU NAME : MemExt1 DIMM0C
FRU ID : 0176
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD925C4D9
Mfg Unique Serial # : 0x002C0C0836D925C4D9
FRU Entry # 23 :
FRU NAME : MemExt1 DIMM0D
FRU ID : 0184
JEDEC SPD Rev : 0x12
JEDEC Mfg ID : 0x2C00000000000000
JEDEC Mfg Location : 0x0C
JEDEC Mfg Part # : 36HTF25672PY-667D1
JEDEC Mfg Revision Code : 0x0100
JEDEC Mfg Year : 0x08
JEDEC Mfg Week : 0x36
JEDEC Mfg Serial # : 0xD925C4CB
Mfg Unique Serial # : 0x002C0C0836D925C4CB
-> This is the last entry in the selected list.
-> Command successful.
[mphpux09] MP:CM> sysrev
SYSREV
Current firmware revisions
MP FW : F.02.23
BMC FW : 05.25
EFI FW : ROM A 07.12, ROM B 07.12
System FW : ROM A 04.03, ROM B 04.03, Boot ROM A
PDH FW : 50.07
DHPC FW : 01.23
UCIO FW : 03.0b
PRS FW : 00.08 UpSeqRev: 0c, DownSeqRev: 08
HFC FW : 00.04 SetRev: 00