RPM DB Corrupted Completely unable to run yum & rpm commands

sakthv10 · April 26, 2016, 6:05am

Hello All

We are facing issue of rpm db corrupted completely on RHEL Server 6.5 , unable to run rpm and yum commands

[root@INMYZCG0035 ~]# rpm -qa
[root@INMYZCG0035 ~]# rpm -qa | grep -i sshd
[root@INMYZCG0035 ~]# rpm -qa | grep -i python
[root@INMYZCG0035 ~]# rpm -qa | grep -i glibc
[root@INMYZCG0035 ~]# rpm -qa | wc -l
0

though we have tried as
#cd /var/lib/rpm/

rm -rf __db*

#rpm --rebulddb
and even we tried copying file ‘/var/lib/rpm/’ from rpm working server to this server and tried above commands after that we were facing “error : rpmdbNextIterator : Skipping h# header v3 RSA/SHA256 Signature, Key Id fd431d51, BAD”

Still we are unable to resolve this issue!!!

It would really appreciated, if someone gave idea of resolving this issue

Thanks in Advance!!!

raghuu · April 26, 2016, 6:20am

Hi

execute the below command and try once

#ln -sv libfreebl3.so /usr/lib64/libfreeblpriv3.so            for 64bit

#ln -sv libfreebl3.so /usr/lib/libfreeblpriv3.so              for 32bit

let us know the outpout of below command after executing the above command

#rpm -qa

or

#rpm -qa

sakthv10 · April 26, 2016, 6:34am

Hi raghu

Thanks for the reply

here what i tried as you said

[root@INMYZCG0035 ~]# find / -name libfreebl3.so
/lib64/libfreebl3.so
/lib/libfreebl3.so
/usr/lib64/libfreebl3.so
/usr/lib/libfreebl3.so
[root@INMYZCG0035 ~]# ln -sv libfreebl3.so /usr/lib/libfreeblpriv3.so
ln: creating symbolic link `/usr/lib/libfreeblpriv3.so’: File exists
[root@INMYZCG0035 ~]# rpm -qa
[root@INMYZCG0035 ~]# rpm -qa | wc -l
0
[root@INMYZCG0035 ~]#

Thanks!!!

raghuu · April 26, 2016, 7:04am

Hmm it was’t showing any installed packages you have to rebuild the rpm database once follow below steps.

Check the size of the /var/lib/rpm/Packages using ls -lh

if its size can be measured in K instead of M, it has been wiped by someone running an rpm --rebuilddb command.

actaully you said that you ran rpm --rebuilddb, but before running rpm --rebuilddb you have to make sure taht there are no processes with the RPM database files open.

#ps -aux | grep -e rpm -e yum -e up2date

(or)

# lsof | grep /var/lib/rpm

The only way to recover from this is by replacing the Packages file from a backup or an identical system
After replacing /var/lib/rpm/Packages, run below commands

Verify integrity of the Packages file, which is taken from backup or an identical system.

# cd /var/lib/rpm

# rm -f __db*     

# /usr/lib/rpm/rpmdb_verify Packages

if you dont find any errors then you can go for below step if you face any issues then a dump and load of the database is required

#rpm -vv --rebuilddb

 and then 

#rpm -qa

once again verify RPM database

#cd /var/lib/rpm

# /usr/lib/rpm/rpmdb_verify Packages

4)Finally, check rpm -Va output to look for inconsistentencies ---- it will take long time.

anilkumarmullapudi · April 26, 2016, 7:36am

Login as root

cd /root
wget http://mirror.centos.org/centos-6/6.6/updates/x86_64/Packages/nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm
rpm2cpio nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm | cpio -idmv
cp ./lib64/libfreeblpriv3.* /lib64
yum update

sakthv10 · April 26, 2016, 7:39am

Hi Raghu

This is the error i am getting

i copied ‘/var/lib/rpm/Packages’ from identical server to the offline server and replaced it with new Packages in ‘/var/lib/rpm/’

2.[root@INMYZCG0035 tmp]# cd rpm_Apr26_2016/
[root@INMYZCG0035 rpm_Apr26_2016]# ls
Basenames __db.002 Dirnames Installtid Packages Pubkeys
Conflictname __db.003 Filedigests Name Providename Requirename
__db.001 __db.004 Group Obsoletename Provideversion Requireversio
[root@INMYZCG0035 rpm_Apr26_2016]# cp -P Packages /var/lib/rpm/
cp: overwrite `/var/lib/rpm/Packages’? y
[root@INMYZCG0035 rpm_Apr26_2016]# cd /var/lib/rpm
[root@INMYZCG0035 rpm]# rm -rf __db*
[root@INMYZCG0035 rpm]# ls -lh Packages
-rwxrwxrwx 1 root root 110M Apr 26 18:36 Packages
[root@INMYZCG0035 rpm]# /usr/lib/rpm/rpmdb_verify Packages
[root@INMYZCG0035 rpm]# rpm -vv --rebuilddb
y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 479 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 1507 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 228 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 2025 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 746 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 2284 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 1774 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 495 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 1523 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 244 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 2041 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 2300 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 1790 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
error: rpmdbNextIterator: skipping h# 511 Header V3 RSA/SHA256 Signature, ke y ID fd431d51: BAD
D: closed db index /var/lib/rpm/Packages
D: closed db environment /var/lib/rpm
D: closed db index /var/lib/rpmrebuilddb.8405/Packages
D: closed db environment /var/lib/rpmrebuilddb.8405
D: removing directory /var/lib/rpmrebuilddb.8405
D: closed db index /var/lib/rpm/Name
D: closed db index /var/lib/rpm/Packages
D: closed db environment /var/lib/rpm
[root@INMYZCG0035 rpm]# rpm -vv --rebuilddb
D: rebuilding database /var/lib/rpm into /var/lib/rpmrebuilddb.8406
D: creating directory /var/lib/rpmrebuilddb.8406
D: opening old database with dbapi 3
D: opening db environment /var/lib/rpm create:cdb:mpool:joinenv
D: opening db index /var/lib/rpm/Packages rdonly mode=0x0
D: locked db index /var/lib/rpm/Packages
D: opening new database with dbapi 3
D: opening db environment /var/lib/rpmrebuilddb.8406 create:mpool:joinenv
D: opening db index /var/lib/rpmrebuilddb.8406/Packages create mode=0x42
D: closed db index /var/lib/rpm/Packages
D: closed db environment /var/lib/rpm
D: closed db index /var/lib/rpmrebuilddb.8406/Packages
D: closed db environment /var/lib/rpmrebuilddb.8406
D: removing directory /var/lib/rpmrebuilddb.8406

[root@INMYZCG0035 ~]# cd /var/lib/rpm
[root@INMYZCG0035 rpm]# /usr/lib/rpm/rpmdb_verify Packages
[root@INMYZCG0035 rpm]# pwd
/var/lib/rpm
[root@INMYZCG0035 rpm]# ls -lh Packages
-rwxrwxrwx 1 root root 12K Apr 26 18:36 Packages
[root@INMYZCG0035 rpm]#

Thanks!!!

raghuu · April 26, 2016, 7:47am

It’s strange it should work do one thing delete all files in /var/spool/up2date

# cd /var/spool/up2date

# rm *

# rm .*

and repeat the processes what you did before, do one thing once reboot the server before going to perform above task again.

and also i want to know the arch of your server

#uname -p

share the output

sakthv10 · April 26, 2016, 7:59am

Hi Anil

[root@INMYZCG0035 server]# rpm2cpio nss-softokn-freebl-3.14.3-9.el6.x86_64.rpm | cpio -idmv
./lib64/libfreebl3.chk
./lib64/libfreebl3.so
./usr/lib64/libfreebl3.chk
./usr/lib64/libfreebl3.so
921 blocks
[root@INMYZCG0035 server]# find / -name libfreeblpriv3.so
[root@INMYZCG0035 server]#

‘libfreeblpriv3’ is not installed though i have installed ‘nss-softokn-freebl’

Thanks!!!

sakthv10 · April 26, 2016, 9:08am

Hi Raghu

There is no files under ‘/var/spool/up2date/’

Thanks!!!

anilkumarmullapudi · April 26, 2016, 9:17am

Try to check these two packages(SHOULD HAVE set a dependency from nss-softokn to nss-softokn-freebl ) has installed with dependencies. if not install and try to run yum update once.
And try to run above rpm commands to re-build rpm DB.