记一次郁闷的故障处理
首先来看一下故障发生的来龙去脉：
数据库版本：9.2.0.4
操作系统版本：5200-04
1、日志推进至95761，准备归档95760，未释放redo02.log
Fri Feb 25 04:22:00 2011
Thread 1 advanced to log sequence 95760
  Current log# 2 seq# 95760 mem# 0: /zzrkdata/zzrk/redo02.log
。。。
Thread 1 advanced to log sequence 95761
  Current log# 1 seq# 95761 mem# 0: /zzrkdata/zzrk/redo01.log
Fri Feb 25 04:23:11 2011
ARC0: Evaluating archive   log 2 thread 1 sequence 95760
ARC0: Beginning to archive log 2 thread 1 sequence 95760
Creating archive destination LOG_ARCHIVE_DEST_1: '/zzrkarch/zzrk/arch_1_95760.arc'

2、ARC1不能归档95760，和ARC0进程发生争用。
Fri Feb 25 04:24:15 2011
ARC1: Completed archiving  log 1 thread 1 sequence 95761
ARC1: Evaluating archive   log 2 thread 1 sequence 95760
ARC1: Unable to archive log 2 thread 1 sequence 95760
      Log actively being archived by another process
     
3、由于不能归档，导致rman备份失败，进而导致归档目录慢，业务挂起
Mon Mar 14 04:20:09 2011
Errors in file /oracle/app/admin/zzrk/bdump/zzrk_arc1_3846274.trc:
ORA-19502: write error on file "/zzrkarch/zzrk/arch_1_97795.arc", blockno 114689 (blocksize=512)
ORA-27063: skgfospo: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 28: No space left on device

4、日志准备推进至98186，准备归档98184，未释放redo04.log
Thu Mar 17 04:26:54 2011
ARCH: Connecting to console port...
Thread 1 cannot allocate new log, sequence 98185
All online logs needed archiving
  Current log# 4 seq# 98184 mem# 0: /zzrkdata/zzrk/redo04.log
。。。
Thu Mar 17 04:27:09 2011
Thread 1 advanced to log sequence 98185
  Current log# 1 seq# 98185 mem# 0: /zzrkdata/zzrk/redo01.log
Thu Mar 17 04:27:09 2011
ARC1: Evaluating archive   log 2 thread 1 sequence 95760
ARC1: Unable to archive log 2 thread 1 sequence 95760
      Log actively being archived by another process
ARC1: Evaluating archive   log 4 thread 1 sequence 98184
ARC1: Beginning to archive log 4 thread 1 sequence 98184
Creating archive destination LOG_ARCHIVE_DEST_1: '/zzrkarch/zzrk/arch_1_98184.arc'

5、由于归档失败，98185占用redo01.log，未能推进至98186
Thu Mar 17 04:27:56 2011
ORACLE Instance zzrk - Can not allocate log, archival required
Thu Mar 17 04:27:56 2011
ARCH: Connecting to console port...
Thread 1 cannot allocate new log, sequence 98186
All online logs needed archiving
  Current log# 1 seq# 98185 mem# 0: /zzrkdata/zzrk/redo01.log
Thu Mar 17 09:52:12 2011
ARCH: Evaluating archive   log 2 thread 1 sequence 95760
ARCH: Unable to archive log 2 thread 1 sequence 95760
      Log actively being archived by another process
ARCH: Evaluating archive   log 4 thread 1 sequence 98184
ARCH: Unable to archive log 4 thread 1 sequence 98184
      Log actively being archived by another process
Thu Mar 17 09:53:41 2011
Shutting down instance: further logons disabled
Shutting down instance (immediate)
License high water mark = 360

6、手动添加日志文件,推进至98186，并占用redo03.log
Thu Mar 17 09:57:12 2011
/* OracleOEM */ ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 3 ('/zzrkdata/zzrk/redo03.log') SIZE 102400K REUSE;
Thu Mar 17 09:57:17 2011
Thread 1 advanced to log sequence 98186
  Current log# 3 seq# 98186 mem# 0: /zzrkdata/zzrk/redo03.log
 
7、手动添加数据文件，推进至98187，并占用redo05.log，未能推进至98188
Thread 1 cannot allocate new log, sequence 98187
All online logs needed archiving
  Current log# 3 seq# 98186 mem# 0: /zzrkdata/zzrk/redo03.log
Thu Mar 17 09:57:45 2011
/* OracleOEM */ ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 5 ('/zzrkdata/zzrk/redo05.lo