Tuesday, April 26, 2011

Script for finding more than 7 days old files and remove permanantly

Script for finding more than 7 days old files and remove permanantly
find "Source Directory" -type f -mtime +7 -exec rm {} \;

Script for Finding more than 7 days old files and perform compression and move it to destination folder

Script for Finding more than 7 days old files and perform compression and move it to destination folder

find "Source Directory" -type f -mtime +7 | \
> while read file
> do
> compress $file
> mv $file.Z "Destination Directory"
> done


Saturday, April 23, 2011

How to restore non bootable mksysb image

Never use restbyname command in NIM server.
If you want to restore in client use below
tctl -f /dev/rmt0.1 rewind
tctl -f /dev/rmt0.1 fsf 3
restbyname -xqf /dev/rmt0.1

Removing duplicate default route



lsattr -El inet0 -> sees bad route
chdev -l inet0 -a delroute=
lsattr -El inet0 -> bad routes removed

Wednesday, April 20, 2011

NIM installation procedure

TABLE OF CONTENTS

1 Introduction. 2

2 Contacts. Error! Bookmark not defined.

3 High Level overview: 3

4 Helpful NIM commands: 4

5. NIM Install and Configuration. 5

5.1. Introduction to Network Installation Management 5

5.2. Create the NIM master 7

5.2.1 Build NIM Filesystem.. 7

5.2.2 Install NIM Master 7

5.2.3 Build the LPP SOURCE.. 7

5.2.4 Update the SPOT. Error! Bookmark not defined.

5.3 Configure NIM master 8

5.3.1 Configure Master Network Resource. 9

5.3.2 Configure LPP Resources. 10

5.3.3 Configure SPOT Resources. 13

5.3.4 Create mksysb NIM Resources. 14

5.4 Install NIM Clients. 15

5.4.1 Adding the Client to the NIM Master 15

5.4.1.1 Add the Client Server to NIM master 15

5.4.1.2 Preparing the NIM Client for a bos_install 16

5.4.2 Configure the Client Servers. 19

5.4.2.1 Booting into SMS Mode. 19

5.4.2.2 Configuring Remote IPL.. 20

5.4.2.3 Adapter Configuration and Test 20

5.4.2.4 Booting the Server from the NIM Master 22

6 Server Backups and Restores. 24

6.1 Structure of the NIM Master 24

6.2 Preparation for NIM Backups and Restores. 24

6.3 Backup a NIM Client 24

6.4 Restore a NIM Client 25

6.5 Reconfiguration on the cleint after mksysb backup is installed. 25


1 Introduction

The intent of this document is to provide detail steps for AIX install, backup, restore, migration using Network Installation Management tool (NIM) for LPARs in NHSS environment. It includes guidance in setting up a NIM environment.

Assumptions used during this document,

à There is already a backup strategy in place and mksysb’s are being saved on the NIM Master in the /export/nim/mksysb NFS directory.

à P51A Server has been designated at the NIM Master.


3. High Level overview:

The following is a fundamental list of activities to be performed in order to create the NIM environment and perform a NIM install on the client partition.

q Planning the NIM Configuration

o Plan the NIM Master network config (what network is being used)

o Plan the NIM Master and Client NIM names

o Plan the NIM Master directory structure

o Plan the NIM Client network config.

o Plan the NIM Master Resource names (lppsource, spot…)

o Plan the NIM Master Mksysb resource.

q Implement the NIM Master

o Install the bos.sysmgt file sets

o Create the required filesystems

o Use smitty nim to configure nim environment

o Use smitty nim_mkres to build the lpp resource, spot,bosinst and mksysb resources

q Create the nim clients

o “smitty nim_mkmac”

q Allocate the NIM resources to the client in preparation for mksysb restore.

o “smitty nim_bosinst”

q Boot the client into SMS and select boot from Ethernet (add the IP’s as required).

q Client boots and performs the mksysb restore.

o Once finished, the nim resources are deallocated.

o Additional AIX configuration such as etherchannel may be required.


4 Helpful NIM commands:

Þ smitty nim

§ smitty nim_mkmac

§ smitty nim_mknet

§ smitty nim_bosinst

Þ lsnim

Þ lsnim –l

Þ lsnim –l

Þ lsnim –a spot (who has spot allocated)

Þ nim –Fo reset

Þ nim –o deallocate –a subclass=all

Þ nim –Fo reset

Þ nim –o check

Þ nim –o check spot52

Þ nim –o lslpp

Þ nim –o showres

Þ nim –o fix_query

Þ /usr/lib/instl/lppmgr –d / -u –b –r

§ –d = lppsource directory

§ –u = remove duplicate updates

§ –b = remove duplicate base levels

§ –k = remove languages

§ – x = remove supercedes

§ –r = remove files

§ –m = move files

§ –l = list files

§ –V = verbose


5. NIM Install and Configuration

There are different ways which can be used for AIX installation process. Due to the physical configuration of a managed system, use of the Network Installation Management (NIM) environment to install AIX is recommended.

5.1. Introduction to Network Installation Management

This section provides an introduction to the NIM environment and the operations you can perform to manage the installation of the AIX Base Operating System (BOS) and optional software on one or more machines. NIM gives you the ability to install and maintain not only the AIX operating system, but also any additional software and fixes that may be applied over time. NIM also allows you to customize the configuration of machines both during and after installation. NIM eliminates the need for access to physical media, such as tapes and CD-ROMs, because the media is a NIM resource on the NIM master server. System backups can be created with NIM, and stored on any server in the NIM environment, including the NIM master. Use NIM to restore a system backup to the same server or to another server. Before you begin configuring the NIM environment, you should already have the following:

· NFS and TCP/IP installed

  • TCP/IP configured correctly

· Name resolution configured

For any installation procedure, you need a software source to install from, such as the AIX 5.2 product CDs (in NHSS environment). The AIX 5.2 product CDs contains boot images used to boot the system from the CD-ROM, installation images, and the installation commands used to install the installation images.

In the NIM environment, the software source is separated into two NIM resources, the LPP_Source and the SPOT. The LPP_Source is a directory on your NIM server. When the LPP_Source is created, installation images are copied from the product CDs to the LPP_Source directory. The product CDs also contains boot images that enable the system to boot from the CD-ROM and installation commands that are used to install the installation images. This equivalent NIM resource is called the SPOT (Shared Product Object Tree). The SPOT is a directory that contains the installation commands that are used to install the installation images from the LPP_Source onto a system. The SPOT is also used to build the necessary boot images to boot a client system. Separate boot images exist for each type of adapter (Ethernet, token ring, and so on). This illustration above shows that when an LPP_Source resource is created, installation images are copied from the product CDs to the LPP_Source directory and a SPOT resource contains the installation commands used to install the installation images from the LPP_Source resource onto a system.

When using the nim_master_setup script to install a NIM master on a system, it creates an LPP_Source and SPOT resource for you and defines the resources in the NIM environment. The nim_master_setup script will also copy the AIX update images from your update CD to the LPP_Source and then install the update images into the SPOT resource. In addition to the LPP_Source and SPOT resources, several NIM resources can help customize the BOS installation process and basic network configuration. The following table shows all the NIM resources that are created with the nim_master_setup script:

Table 5. NIM resources created by nim_master_setup script

NIM Resource

Name Given

Description

spot*

Spot_52

Commands used during installation. The network boot images are built from the SPOT.

lpp_source

Lppsource_52

Directory containing installation images.

mksysb

Hostname.mksysb

System backup image

bosinst_data

bosinst_ow

Answers questions asked during the BOS installation, which allows for a non-prompted new and complete overwrite installation.

resolv_conf

resolv_res

Provides the domain name and name server information.

res_group

basic_res_grp

Used by the nim_clients_setup script to allocate the bosinst_data, mksysb, lpp_source, spot, and resolv_conf to install the client partitions.

* Required resource for installation

Beside lpp_source and spot resource which represent BOS, a system backup image called mksysb is usually created. mksysb is a generic system backup which includes BOS plus customer software. As a standard AIX BOS image, a mksysb can be used to clone customized AIX to the servers who has no AIX installed. Also as a dedicated server backup on the NIM master, in the event a client server has to be recovered, such as a catastrophic hardware or software problem, the mksysb can be “pushed” down to the server to restore the server operation. Scripts are available that create the resources necessary to perform a mksysb installation.

Each LPAR on P5 server then will be defined in the NIM environment as a standalone system resource, also referred to as a NIM client. Use smitty nim_mkmac to add the servers as NIM clients. A nim_clients_setup script is also available you to define NIM clients and initiates a AIX installation on the NIM clients.

5.2. Create the NIM master

This section describes how to install the NIM master from scratch. It starts to build a separate NIM logic volume group called nimvg volume group if additional hard disk is available. Then it runs nim_master_setup script to install NIM from CDs. In case the NIM has been installed though smitty, the NIM master can also be configured through smitty nim, which will be described in next section.

5.2.1 Build NIM Filesystem

It is optional but recommended to build the NIM logic volume group in a separate nimvg volume group if additional hard disk is available. For the limit of hard disks, the NIM file system can be established on rootvg.

5.2.2 Install NIM Master

To install the NIM Master, do the following:

  • Ensure that the AIX 5.3 CD 1 of 8 to install AIX
  • NIM Server is on AIX 5.3 ML05 and all LPAR’s are on AIX 5.2 ML09
  • Issue the following command to configure the NIM environment:

§ smitty nim

§ nim easy create option

§ select the network name, lppsource name , spot name

§ select the filesystem size and volume group you want to create to (like in our case its nimvg)

§ Note: The estimated run time is 40 minutes.

  • After the above command has completed, verify that the following directories are created:

§ /export/nim;

§ /export/nim/spot/spot_52;

§ /export/nim/lpp_source/lppsource_52

§ /tftpboot

o Manually create the mksysb and backup directory using the following command:

§ mkdir /export/nim/mksysb

§ mklv

§ crfs

§ mount

Build the LPP SOURCE

Note: lppsource and SPOT was created during the nim easy install process. However we can manually create the lppsource as below

Copying fileset from AIX 5.2 CDs to /media/AIX , this is used as a media server

o Insert the AIX 5.2 CD 1 of 8 CD into the CD-ROM drive.

o Enter the following command at the command line:

§ smitty bffcreate

o The Copy Software to Hard Disk for Future Installation screen appears. Do the following:

o Enter /dev/cd0 in the INPUT device / directory for software field, then press the Enter key.

§ In the DIRECTORY for storing software package field, enter /media/AIX/AIX_52.

o Update the TOC

§ Inutoc

5.3 Configure NIM master

This section describes how to configure the NIM master when the NIM fileset (and lpp source) has been installed onto the server. There are two lpp sources – AIX 5.3 and AIX 5.2 – installed on the NIM master.

5.3.1 Configure Master Network Resource

To create a lpp resource, do the following:

o smitty nim_mknet

o The Network Type screen appears. Select ent, then press the Enter key.


  • The Define a Network screen appears. enter the appropriate information as the following.

Define a Network

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Network Name [master]

Network Type ent

Ethernet Type Standard

Network IP Address []

Subnetmask []

Other Network Type

Comments [master network]

F1=Help F2=Refresh F3=Cancel F4=List

Esc+5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

5.3.2 Configure LPP Resources

To create a lpp source, do the following:

o smitty nim_mkres

o The Resource Type screen appears. Select the lpp_source type, and then press the Enter key.

o The Define a Resource screen appears. Do the following:

o In the Resource Name field, enter lppsource_52.

o In the Server of Resource field, press the F4 key, then select master from the list.

o In the Location of Resource field, enter /media/AIX/AIX52.

o In the Comment field, enter your comment such as “This is the latest AIX 5.2 OS filesets”

o Press the Enter key to add the resource.


Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [lppsource_52]

* Resource Type lpp_source

* Server of Resource [master]

* Location of Resource [/export/software+]

Source of Install Images []

Names of Option Packages []

Comments [This is the lates+]

F1=Help F2=Refresh F3=Cancel F4=List

Esc+5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

o Once the command has completed, press the F10 key to exit to the command prompt.


5.3.3 Configure SPOT Resources

To create a spot resource, do the following:

o smitty nim_mkres

o The Resource Type screen appears. Select the spot type, then press the Enter key.

o The Define a Resource screen appears. Do the following:

o In the Resource Name field, enter spot52.

o In the Server of Resource field, press the F4 key, then select master from the list.

o In the Source of Install Images, press the F4 key, then select lppsource_52 from the list.

o In the Location of Resource field, enter /export/nim/spot.

o In the Comment field, enter your comment such as “This is the spot for AIX 5.2 ML7”

o Press the Enter key to add the resource.

Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* Resource Name [spot52]

* Resource Type spot

* Server of Resource [master]

* Source of Install Images []

* Location of Resource [/export/NIM/spot]

Expand file systems if space needed? yes

Comments [This is the spot for+]

installp Flags

COMMIT software updates? no

SAVE replaced files? yes

AUTOMATICALLY install requisite software? yes

OVERWRITE same or newer versions? no

VERIFY install and check file sizes? no

F1=Help F2=Refresh F3=Cancel F4=List

Esc+5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

o It may take 10 -15 minutes to create the spot resource from the lpp_source. Once the command has completed without error, press the F10 key to exit to the command prompt.

5.3.4 Create mksysb NIM Resources

You must create at least one mksysb resource as your system backup image. The actual mksysb image can be created from each LPAR weekly as a backup AIX 5.x image for each LPAR, saved in the /export/nim/mksysb/hostname directory. It can also be copied to a CD as a separate backup, especially for the NIM master. To create a mksysb resource, do the following:

o Assume the mksysb is a file in /export/nim/mksysb/hostname/hostname.mksysb

o Define the new mksysb resource in NIM by entering the following command from the command line:

o smitty nim_mkres

o The Resource Type screen appears. Select the mksysb resource type, then press the Enter key.

o The Define a Resource screen appears. Do the following:

o In the Resource Name field, enter hostname.mksysb.

o In the Server of Resource field, press the F4 key, then select master from the list.

o In the Location of Resource field, enter /export/nim/mksysb/hostname.

o In the Comment field, enter your comment as required

o Press the Enter key to add the resource.


Define a Resource

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[TOP] [Entry Fields]

* Resource Name [52mksysb]

* Resource Type mksysb

* Server of Resource [master]

* Location of Resource [/export/nim/mksysb+]

Comments []

Source for Replication []

-OR-

System Backup Image Creation Options:

CREATE system backup image? no

NIM CLIENT to backup []

PREVIEW only? no

IGNORE space requirements? no

[MORE...9]

F1=Help F2=Refresh F3=Cancel F4=List

Esc+5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

o Once the command has completed, press the F10 key to exit to the command prompt.

5.4 Install NIM Clients

Now you can start to configure and install AIX to each LPAR (NIM Client). The following sections describe the procedure of installation on a NIM client and you can repeat it until all servers are installed. There are two stages to configuring and installing a new client server from the NIM master. The first step involves adding the client on the NIM master. The second step involves configuring the client server for a NIM action, and then initiating a NIM boot over the network. This will automatically install the new server using the client template definition in the NIM master.

5.4.1 Adding the Client to the NIM Master

5.4.1.1 Add the Client Server to NIM master

The next step adds the client server to the NIM server. Run the following command:

smitty nim_mkmac

Enter the host name of NIM client press Enter, then select ent as the attached network. Fill in the fields of the next screen using the following values and press Enter when complete:


Define a Machine

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]

* NIM Machine Name [catom-mddbpca01]

* Machine Type [standalone]

* Hardware Platform Type [chrp]

Kernel to use for Network Boot [mp]

Primary Network Install Interface

* Cable Type tp

* Network Speed Setting [auto]

* Network Duplex Setting [auto]

* NIM Network master_net

* Host Name catos-nimpa00

Network Adapter Hardware Address [0]

Network Adapter Logical Device Name []

IPL ROM Emulation Device []

CPU Id []

Machine Group []

Comments [] F1=Help F2=Refresh F3=Cancel F4=List

Esc+5=Reset F6=Command F7=Edit F8=Image

F9=Shell F10=Exit Enter=Do

To verify the new client was added correctly, run the following command:

§ lsnim –l

5.4.1.2 Preparing the NIM Client for a bos_install

At this point, the newly created NIM client must have a bosinst_data resources allocated to it in order to make it serviceable.

To allocate the bosinst.data, run the following command:

§ smitty nim_bosinst

§ Select the previously defined client name, and press Enter.

§ Choose “mksysb” for the installation type, and press Enter.

§ Select the desired mksysb resource (52mksysb for instance), and press Enter.

Select the MKSYSB to use for the installation

Move cursor to desired item and press Enter.

52mksysb resources mksysb

F1=Help F2=Refresh F3=Cancel F4=List

Esc+5=Reset F6=Command F7=Edit F8=Image F9=Shell

§ Select the previously defined SPOT resource, and press Enter.

Select the MKSYSB to use for the installation

Move cursor to desired item and press Enter.

spot52 resources spot

F1=Help F2=Refresh F3=Cancel F4=List

Esc+5=Reset F6=Command F7=Edit F8=Image F9=Shell

§ Complete the next screen by filling in the following fields, and pressing Enter twice to confirm when complete.

Install the Base Operating System on Standalone Clients

[TOP] [Entry Fields]

* Installation Target catos-mddbbca02

* Installation TYPE mksysb

* SPOT spot52

* LPP_SOURCE 52_lppres

MKSYSB 52mksysb

BOSINST_DATA to use during installation [52bid_64bit_2dr]

IMAGE_DATA to use during installation []

RESOLV_CONF to use for network configuration [resolv_conf]

Customization SCRIPT to run after installation []

Remain NIM client after install? [yes]

PRESERVE NIM definitions for resources on target [yes]

FORCE PUSH the installation [no]

Initiate reboot and reboot now? [no]

-OR-

Set bootlist for installation at the [no]

next reboot?

Additional BUNDLES to install []

-OR-

Additional FILESETS to install []

(bundles will be ignored)

installp Flags

COMMIT software updates? [yes]

SAVE replaced files? [no]

AUTOMATICALLY install requisite software? [yes]

EXTEND filesystems if space needed? [yes]

OVERWRITE same or newer versions? [no]

VERIFY install and check file sizes? [no]

Group controls (only valid for group targets):

Number of concurrent operations []

Time limit (hours) []

Schedule a Job [no]

YEAR []

MONTH []

DAY (1-31) []

HOUR (0-23) []

MINUTES (0-59) []

[BOTTOM]

F1=Help F2=Refresh F3=Cancel F4=List

§ Press the F10 key to exit this screen and return to the command line.

To verify the completed NIM client profile, run the following command:

lsnim –l

5.4.2 Configure the Client Servers

Now that the NIM master has been updated with the client information, the client server needs to be configured for network boot through the HMC console.

5.4.2.1 Booting into SMS Mode.

This procedure assumes that the target server is currently “down”

o Open the HMC console.

§ Use Websm to do

In the Navigation Area (left pane), and right click it to open the terminal window.

1. Click Activate. The Activate Partition menu opens with a selection of partition profiles.

2. Select Open a terminal window to open a virtual terminal (vterm) window.

3. Click (Advanced...) to open the advanced options menu.

4. For the boot mode, select SMS.

5. Click OK to close the advanced options menu.

6. Click OK. You will see the following message:

After a few moments, the terminal screen will open, and the system will boot to the SMS prompt.


5.4.2.2 Configuring Remote IPL

In SMS menu, choose “Setup Remote IPL (Initial Program Load)” from the main menu.

o Choose the Port 1 press Enter, then select “1. IP Parameter”, press enter.

o Fill in each field using the IP addresses of both NIM server and NIM client from the Server Install Worksheets.

Note: the “Gateway IP Address” field is always set to the IP Address of the NIM Master

5.4.2.3 Adapter Configuration and Test

o Back to the previous screen and choose option “2. Adapter Configuration”.

o Select network speed auto and Duplex auto

o Press ESC twice and select option 3. Ping Test.

o Select “1. Execute Ping Test” on next screen to attempt the ping operation. The NIM client will now attempt to ping the NIM server.


After a 60 second wait, a success message should appear. If not, review and change the network adapter configuration until the ping is successful.


5.4.2.4 Booting the Server from the NIM Master

After the successful ping, press any key and then select “M” to return to the SMS main menu screen. From this screen, select “1. Select Install/Boot Devices” and press Enter


From next screen, choose “7. List all Devices”, and press Enter.

o After the buses are scanned, a list of boot devices will be presented. Select “1. Ethernet” and press Enter.

o Choose “2. Normal Mode Boot”. and press Enter

o Finally, choose “1. Yes” to exit the SMS menu and initiate the boot process.

After the “STARTING SOFTWARE PLEASE WAIT…” message, the new client will start booting and install the new image automatically. Once the “Welcome to AIX” message appears, the client has successfully started the boot process. Approximately 30 minutes later, the new client will be installed and ready for login.

Now you can follow the procedure in 5.4.2 to install the NIM client.


6 Server Backups and Restores

It is high recommended to make a completed system backups after implementing AIX on all p570 LPAR’s. This section is intended to highlight a few key points about backups and restores on the AIX platform.

6.1 Structure of the NIM Master

Usually OS Backups (mksysb) will occur through regularly scheduled jobs on each of the servers. These servers will nfs mount the /export/nim/mksysb directory from the NIM Master, over the network, and perform a mksysb to a file in the NFS directory. Once the mksysb completes, the nfs mount will be unmounted, leaving the completed mksysb file on the nim master.

In the event a server has to be recovered, such as a catastrophic hardware or software problem, this mksysb can be “pushed” down to the server to restore the server operation.

The /export/NIM directory holds the required NIM master resources such as the mksysb and SPOT and LPPSOURCE.

6.2 Preparation for NIM Backups and Restores

Some key components required for Backups and Restores are listed below.

  • The /etc/hosts file on all servers must contain all server hostnames and IP Addresses. It must be accurate.
  • The network must be fully functional.
  • The /export/nim/mksysb directory is nfs exported as Read/Write from the NIM master to all Clients. As part of this, the /etc/hosts file must be accurate.
  • The client must have an mksysb mount point directory created

· You will need to create a scripts for the same

6.3 Backup a NIM Client

The backup process begins on the NIM Client (for instance – catos-mddbbca01).

o Step 1 – the client nfs mounts the /export/nim/mksysb directory from the NIM master.

o Step 2 – the client issues the mksysb command and places the resulting file in the nfs mounted mksysb directory. This mksysb file contains all info in regards to the Operating System as well as customer data.

o Step 3 – the client unmounts the nfs directory. The mksysb backup now resides on the NIM master.

o Step 4 – Backup software such as TSM can then be used to manage the backup of the mksysbs now residing on the NIM Master. TSM is also used to backup all application “data”. This is beyond the scope of this document.

6.4 Restore a NIM Client

The restore process involves the use of NIM to restore a previously saved mksysb image of a server in the event of a catastrophic failure.

The restore process begins on the NIM master but then shifts to the NIM Client Server at step 3 (perform a netboot).

o Step 1 – The desired mksysb image mksysb.hostname to restore is selected in the /export/nim/mksysb/hostname directory.

o Step 2 – Remove and recreate the mksysb resource on the NIM master. Use smitty nim_mkres

o Step 3 – the NIM master is configured to allocate the desired mksysb image to the desired client and set the client to netboot. Use “smitty nim_bosinst”.

o Step 4 – the client is netbooted. During this process, it follows the instructions setup on the NIM master and performs a non-prompted OS install.

o Step 5 – at the end of the netboot, the NIM resources are automatically deallocated on the NIM Master and the server is rebooted to the login screen.

6.5 Reconfiguration on the client after mksysb backup is installed

To restore original environment which mksysb was built, do the following steps:

o Step 1 – Check hostname.

o Step 2 – Check all Ethernet adapater and lables.

o Step 3 – Check all etherchannels

o Step 4 – Check HACMP filesets and start HACMP .

o Step 5 – Check all SAN disk are attached and configured

o Step 6 – Start GPFS.

o Step 7 - Mount filesystems.

o Step 8 – run smitty aio and select change/show Asynchronous I/O.

select “STATE to be configured at system restart” as “available”. This is necessary for oracle application.

o Step 9 – Application People to check Oracle RAC and its working

o