LDOM

From pressy's brainbackup
Jump to: navigation, search

LDOM

My first T5-2 impressions with the new Logical Domain Manager 3. These are only quick&dirty notes for myself... Based on this blogs, thanks: https://blogs.oracle.com/cmt/entry/what_s_up_with_ldoms

Simple Setup

Activation

I installed Solaris 11.1 on my T5, so the LDOM software was already installed... First I had to "activate" the LDOM partitioning and some services like the virtual network terminal server daemon. To create new LDOMs, you need to remove HW from the primary domain.

# ldm add-vconscon port-range=5000-5100 primary-console primary:
# svcadm enable vntsd
# svcs vntsd
STATE          STIME    FMRI
online         17:38:35 svc:/ldoms/vntsd:default
# ldm set-vcpu 128 primary
# ldm start-reconf primary      <-- reconfigure after reboot
# ldm set-memory 127G primary
# ldm add-config initial
# ldm add-vds primary-vds primary
# ldm add-vswitch net-dev=net0 switch-primary primary
# init 6

After the reboot we have resources for other guest domains...

Create Guest LDOM

You can always choose the names by yourself, I think that could become complicated in a huge environment... this is a small configuration with on a ZVOL as hard drive based on my data ZPOOL:

# ldm create sol10-test01
# ldm list-devices -S         <-- list free resources 
# ldm set-vcpu 128 sol10-test01
# ldm set-memory 128G sol10-test01
# zfs create -o mountpoint=none data/ldoms
# zfs create -V 128G data/ldoms/sol10-test01.hdd0
# ldm add-vdsdev /dev/zvol/dsk/data/ldoms/sol10-test01.hdd0 sol10-test01.hdd0@primary-vds
# ldm add-vdisk hdd0 sol10-test01.hdd0@primary-vds sol10-test01
# ldm add-vnet net0 switch-primary sol10-test01
# telnet localhost 5000 

Install LDOM

I've installed Solaris 10 from a DVD image:

# ldm stop  sol10-test01
# ldm unbind  sol10-test01
# ldm add-vdsdev /downloads/sol-10-u11-ga-sparc-dvd.iso sol10u11.iso@primary-vds
# ldm add-vdisk sol10u11.iso sol10u11.iso@primary-vds sol10-test01
# ldm bind  sol10-test01
# ldm start  sol10-test01
# telnet localhost 5000
{0} ok boot sol10u11.iso:f

And yes, it worked ;)

# telnet localhost 5000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to <name>.
Escape character is '^]'.
{0} ok banner

SPARC T5-2, No Keyboard
Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.35.0.a, 128.0000 GB memory available, Serial #833XXX86.
Ethernet address 0:14:4f:f8:XX:XX, Host ID: 84fXXX9a.



{0} ok
{0} ok boot 
Boot device: disk:a  File and args:
SunOS Release 5.10 Version Generic_148888-05 64-bit
Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.
Hostname: sol10-test01

sol10-test01 console login: root
Password:
Last login: Tue Jul  2 17:49:28 from 192.168.50.101
Jul  3 15:12:30 sol10-test01 login: ROOT LOGIN /dev/console
Oracle Corporation      SunOS 5.10      Generic Patch   January 2005
Sourcing //.profile-EIS.....
# uname -a
SunOS sol10-test01 5.10 Generic_148888-05 sun4v sparc sun4v
# psrinfo -pv
The physical processor has 128 virtual processors (0-127)
  SPARC-T5 (chipid 0, clock 3600 MHz)
# prtconf | grep Memory
Memory size: 131072 Megabytes

Split I/O Domain

This Time, reseted to default, I will setup an own I/O domain.

Primary Setup

First of all, we need free resources, so let's shrink the service domain

root@t5pd:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    256   255G     0.0%  0.0%  3h 19m
root@t5pd:~# ldm list-services
root@t5pd:~#
root@t5pd:~# ldm add-vconscon port-range=5000-5100 primary-console primary
root@t5pd:~# ldm list-services
VCC
    NAME             LDOM             PORT-RANGE
    primary-console  primary          5000-5100

root@t5pd:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    256   255G     0.0%  0.0%  3h 19m
root@t5pd:~#
root@t5pd:~# svcadm enable vntsd
root@t5pd:~# svcs -l vntsd
fmri         svc:/ldoms/vntsd:default
name         virtual network terminal server
enabled      true
state        online
next_state   none
state_time   Thu Sep 19 13:20:31 2013
logfile      /var/svc/log/ldoms-vntsd:default.log
restarter    svc:/system/svc/restarter:default
contract_id  125
manifest     /lib/svc/manifest/platform/sun4v/vntsd.xml
dependency   optional_all/error svc:/milestone/network (online)
dependency   optional_all/none svc:/system/system-log (online)
root@t5pd:~#

I/O domain

OK... we will split "the backplane", so what devices are needed for the current installation...

root@t5pd:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        rpool                      ONLINE       0     0     0
          c0t5000CCA0160BCAB8d0s0  ONLINE       0     0     0

errors: No known data errors
root@t5pd:~# mpathadm show lu /dev/rdsk/c0t5000CCA0160BCAB8d0s0
Logical Unit:  /dev/rdsk/c0t5000CCA0160BCAB8d0s2
        mpath-support:  libmpscsi_vhci.so
        Vendor:  HITACHI
        Product:  H109030SESUN300G
        Revision:  A31A
        Name Type:  unknown type
        Name:  5000cca0160bcab8
        Asymmetric:  no
        Current Load Balance:  round-robin
        Logical Unit Group ID:  NA
        Auto Failback:  on
        Auto Probing:  NA

        Paths:
                Initiator Port Name:  w508002000159c458
                Target Port Name:  w5000cca0160bcab9
                Override Path:  NA
                Path State:  OK
                Disabled:  no

        Target Ports:
                Name:  w5000cca0160bcab9
                Relative ID:  0
root@t5pd:~# mpathadm show initiator-port w508002000159c458
Initiator Port:  w508002000159c458
        Transport Type:  unknown
        OS Device File:  /devices/pci@300/pci@1/pci@0/pci@2/scsi@0/iport@1
Initiator Port:  w508002000159c458
        Transport Type:  unknown
        OS Device File:  /devices/pci@300/pci@1/pci@0/pci@2/scsi@0/iport@2
root@t5pd:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
net0              ip         ok           --         --
   net0/v4        static     ok           --         192.168.56.200/24
net5              ip         ok           --         --
   net5/v4        static     ok           --         169.254.182.77/24
root@t5pd:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net5              Ethernet             up         10     full      usbecm2
net0              Ethernet             up         1000   full      ixgbe0
net2              Ethernet             unknown    0      unknown   ixgbe2
net1              Ethernet             unknown    0      unknown   ixgbe1
net3              Ethernet             unknown    0      unknown   ixgbe3
root@t5pd:~# ls -l /dev/ixgbe0
lrwxrwxrwx   1 root     root          53 Sep 17 18:24 /dev/ixgbe0 -> ../devices/pci@300/pci@1/pci@0/pci@1/network@0:ixgbe0
soo... we need pci@300 for my devices... 
root@t5pd:~# ldm list -l -o physio primary
NAME
primary

FLAGS
normal,delayed(),control,vio-service

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@340                          pci_1
    pci@300                          pci_0
    pci@3c0                          pci_3
    pci@380                          pci_2
    pci@340/pci@1/pci@0/pci@4        /SYS/MB/PCIE5
    pci@340/pci@1/pci@0/pci@5        /SYS/MB/PCIE6
    pci@340/pci@1/pci@0/pci@6        /SYS/MB/PCIE7
    pci@300/pci@1/pci@0/pci@4        /SYS/MB/PCIE1
    pci@300/pci@1/pci@0/pci@2        /SYS/MB/SASHBA0
    pci@300/pci@1/pci@0/pci@1        /SYS/MB/NET0
    pci@3c0/pci@1/pci@0/pci@7        /SYS/MB/PCIE8
    pci@3c0/pci@1/pci@0/pci@2        /SYS/MB/SASHBA1
    pci@3c0/pci@1/pci@0/pci@1        /SYS/MB/NET2
    pci@380/pci@1/pci@0/pci@5        /SYS/MB/PCIE2
    pci@380/pci@1/pci@0/pci@6        /SYS/MB/PCIE3
    pci@380/pci@1/pci@0/pci@7        /SYS/MB/PCIE4
-------------> I will remove pci@3c0 for my I/O Domain (alias pci_3). 
root@t5pd:~# ldm remove-io pci_3 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

root@t5pd:~#
root@t5pd:~# ldm set-vcpu -c 1 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
root@t5pd:~# ldm set-memory 8G primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
root@t5pd:~#
root@t5pd:~# ldm list-config
factory-default [next poweron]
root@t5pd:~# ldm add-config mpr-split-io
root@t5pd:~# ldm list-config
factory-default
mpr-split-io [current]
root@t5pd:~#
root@t5pd:~# init 6
[...]
root@t5pd:~# psrinfo
0       on-line   since 09/20/2013 14:50:18
1       on-line   since 09/20/2013 14:50:22
2       on-line   since 09/20/2013 14:50:22
3       on-line   since 09/20/2013 14:50:22
4       on-line   since 09/20/2013 14:50:22
5       on-line   since 09/20/2013 14:50:22
6       on-line   since 09/20/2013 14:50:22
7       on-line   since 09/20/2013 14:50:22
root@t5pd:~# psrinfo -p
1
root@t5pd:~# prtconf | grep Memory
Memory size: 8192 Megabytes
root@t5pd:~#
root@t5pd:~# ldm list -l -o physio primary
NAME
primary

IO
    DEVICE                           PSEUDONYM        OPTIONS
    pci@340                          pci_1
    pci@300                          pci_0
    pci@380                          pci_2
    pci@340/pci@1/pci@0/pci@4        /SYS/MB/PCIE5
    pci@340/pci@1/pci@0/pci@5        /SYS/MB/PCIE6
    pci@340/pci@1/pci@0/pci@6        /SYS/MB/PCIE7
    pci@300/pci@1/pci@0/pci@4        /SYS/MB/PCIE1
    pci@300/pci@1/pci@0/pci@2        /SYS/MB/SASHBA0
    pci@300/pci@1/pci@0/pci@1        /SYS/MB/NET0
    pci@380/pci@1/pci@0/pci@5        /SYS/MB/PCIE2
    pci@380/pci@1/pci@0/pci@6        /SYS/MB/PCIE3
    pci@380/pci@1/pci@0/pci@7        /SYS/MB/PCIE4
root@t5pd:~#
root@t5pd:~# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
pci_1                                     BUS    pci_1    primary
pci_0                                     BUS    pci_0    primary
pci_3                                     BUS    pci_3
pci_2                                     BUS    pci_2    primary
/SYS/MB/PCIE5                             PCIE   pci_1    primary  EMP
/SYS/MB/PCIE6                             PCIE   pci_1    primary  EMP
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP
/SYS/MB/PCIE1                             PCIE   pci_0    primary  EMP
/SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC
/SYS/MB/PCIE8                             PCIE   pci_3             UNK
/SYS/MB/SASHBA1                           PCIE   pci_3             UNK
/SYS/MB/NET2                              PCIE   pci_3             UNK
/SYS/MB/PCIE2                             PCIE   pci_2    primary  EMP
/SYS/MB/PCIE3                             PCIE   pci_2    primary  EMP
/SYS/MB/PCIE4                             PCIE   pci_2    primary  EMP
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
root@t5pd:~#

OK, now let's create the second domain with the "2nd half" of the box

root@t5pd:~# ldm create io-dom
root@t5pd:~# ldm set-vcpu -c 1 io-dom
root@t5pd:~# ldm set-memory 8G io-dom
root@t5pd:~# ldm add-io pci_3 io-dom
root@t5pd:~#
root@t5pd:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    8     8G       0.0%  0.0%  19m
io-dom           inactive   ------          8     8G
root@t5pd:~#
root@t5pd:~# ldm bind io-dom
root@t5pd:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    8     8G       1.0%  0.9%  23m
io-dom           bound      ------  5000    8     8G
root@t5pd:~#
root@t5pd:~# ldm start io-dom
LDom io-dom started
root@t5pd:~#
root@t5pd:~# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "io-dom" in group "io-dom" ....
Press ~? for control options ..

{0} ok
{0} ok
{0} ok banner

SPARC T5-2, No Keyboard
Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.35.3, 8.0000 GB memory available, Serial #83xxxxxx.
Ethernet address 0:14:xx:xx:xx:xx, Host ID: 84xxxxxx.



{0} ok

Now we need to install that I/O domain. Best way would be an AI. Had a small issue, because the "net" alias was not set. But after setting the OBP-devalias it worked to boot/install:

{0} ok banner

SPARC T5-2, No Keyboard
Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.35.3, 8.0000 GB memory available, Serial #83xxxxxx.
Ethernet address 0:14:xx:xx:xx:xx, Host ID: 84xxxxxx.



{0} ok
{0} ok boot net2:dhcp - install
Boot device: /pci@3c0/pci@1/pci@0/pci@1/network@0:dhcp  File and args: - install
1000 Mbps full duplex Link up
Timed out waiting for BOOTP/DHCP reply
<time unavailable> wanboot info: WAN boot messages->console
<time unavailable> wanboot crit: No network device available for wanboot!
<time unavailable> wanboot crit: (Ensure that the 'net' alias is set correctly)
Type  'go' to resume
{0} ok
{0} ok
{0} ok nvalias net /pci@3c0/pci@1/pci@0/pci@1/network@0
{0} ok boot net:dhcp - install
NOTICE: Entering OpenBoot.
NOTICE: Fetching Guest MD from HV.
NOTICE: Starting additional cpus.
NOTICE: Initializing LDC services.
NOTICE: Probing PCI devices.
NOTICE: Finished PCI probing.


SPARC T5-2, No Keyboard
Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.35.3, 8.0000 GB memory available, Serial #83378927.
Ethernet address 0:14:4f:f8:42:ef, Host ID: 84f842ef.



Boot device: /pci@3c0/pci@1/pci@0/pci@1/network@0:dhcp  File and args: - install
1000 Mbps full duplex Link up
<time unavailable> wanboot info: WAN boot messages->console
<time unavailable> wanboot info: configuring /pci@3c0/pci@1/pci@0/pci@1/network@0:dhcp

1000 Mbps full duplex Link up
<time unavailable> wanboot info: Starting DHCP configuration
<time unavailable> wanboot info: DHCP configuration succeeded
<time unavailable> wanboot progress: wanbootfs: Read 368 of 368 kB (100%)
<time unavailable> wanboot info: wanbootfs: Download complete
Fri Sep 20 13:21:43 wanboot progress: miniroot: Read 251058 of 251058 kB (100%)
Fri Sep 20 13:21:43 wanboot info: miniroot: Download complete
SunOS Release 5.11 Version 11.1 64-bit
Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.
Remounting root read/write
Probing for device nodes ...
Preparing network image for use
Downloading solaris.zlib
--2013-09-20 13:23:04--  http://192.168.56.113:5555/export/install/sparc-ai//solaris.zlib
Connecting to 192.168.56.113:5555... connected.
HTTP request sent, awaiting response... 200 OK
Length: 158720512 (151M) [text/plain]
Saving to: `/tmp/solaris.zlib'

100%[======================================>] 158,720,512  112M/s   in 1.4s

2013-09-20 13:23:06 (112 MB/s) - `/tmp/solaris.zlib' saved [158720512/158720512]

Downloading solarismisc.zlib
--2013-09-20 13:23:06--  http://192.168.56.113:5555/export/install/sparc-ai//solarismisc.zlib
Connecting to 192.168.56.113:5555... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12876800 (12M) [text/plain]
Saving to: `/tmp/solarismisc.zlib'

100%[======================================>] 12,876,800  --.-K/s   in 0.1s

2013-09-20 13:23:06 (111 MB/s) - `/tmp/solarismisc.zlib' saved [12876800/12876800]

Downloading .image_info
--2013-09-20 13:23:06--  http://192.168.56.113:5555/export/install/sparc-ai//.image_info
Connecting to 192.168.56.113:5555... connected.
HTTP request sent, awaiting response... 200 OK
Length: 88 [text/plain]
Saving to: `/tmp/.image_info'

100%[======================================>] 88          --.-K/s   in 0s

2013-09-20 13:23:06 (7.44 MB/s) - `/tmp/.image_info' saved [88/88]

Done mounting image
Configuring devices.
Hostname: solaris
Service discovery phase initiated
Service name to look up: default-sparc

############
### a.s.o ####
############

Super, now I have two LDOMs which can provide I/O... let's configure a LDOM using both pci complexes

root@t5pd:~# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS
----                                      ----   ---      ------   ------
pci_1                                     BUS    pci_1    primary
pci_0                                     BUS    pci_0    primary
pci_3                                     BUS    pci_3    io-dom
pci_2                                     BUS    pci_2    primary
/SYS/MB/PCIE5                             PCIE   pci_1    primary  EMP
/SYS/MB/PCIE6                             PCIE   pci_1    primary  EMP
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP
/SYS/MB/PCIE1                             PCIE   pci_0    primary  EMP
/SYS/MB/SASHBA0                           PCIE   pci_0    primary  OCC
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC
/SYS/MB/PCIE8                             PCIE   pci_3    io-dom   EMP
/SYS/MB/SASHBA1                           PCIE   pci_3    io-dom   OCC
/SYS/MB/NET2                              PCIE   pci_3    io-dom   OCC
/SYS/MB/PCIE2                             PCIE   pci_2    primary  EMP
/SYS/MB/PCIE3                             PCIE   pci_2    primary  EMP
/SYS/MB/PCIE4                             PCIE   pci_2    primary  EMP
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_3    io-dom
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_3    io-dom
root@t5pd:~#
root@t5pd:~# ldm add-vds iodom-vds primary
root@t5pd:~# ldm add-vds iodom-vds io-dom
root@t5pd:~# ldm add-vswitch net-dev=net0 switch-primary primary
root@t5pd:~# ldm add-vswitch net-dev=net0 switch-io io-dom
root@t5pd:~#
root@t5pd:~# ldm create ldom01
root@t5pd:~# ldm set-vcpu -c 1 ldom01
root@t5pd:~# ldm set-memory 8g ldom01
root@t5pd:~#
root@t5pd:~# ldm add-vdsdev /dev/zvol/dsk/rpool/ldom/ldom01/ldom01-prim01vol ldom01-prim01vol@primary-vds
root@t5pd:~# ldm add-vdsdev /dev/zvol/dsk/rpool/ldom/ldom01/ldom01-iodom01vol ldom01-iodom01vol@iodom-vds
root@t5pd:~#
root@t5pd:~# ldm add-vdisk hdd0 ldom01-prim01vol@primary-vds ldom01
root@t5pd:~# ldm add-vdisk hdd1 ldom01-iodom01vol@iodom-vds ldom01
root@t5pd:~#
root@t5pd:~# ldm add-vnet net0 switch-primary ldom01
root@t5pd:~# ldm add-vnet net1 switch-io ldom01
root@t5pd:~# ldm set-vsw linkprop=phys-state switch-primary
root@t5pd:~# ldm set-vsw linkprop=phys-state switch-io
root@t5pd:~# ldm set-vnet linkprop=phys-state net0 ldom01
root@t5pd:~# ldm set-vnet linkprop=phys-state net1 ldom01
root@t5pd:~#
root@t5pd:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    8     8G       0.1%  0.1%  3h 22m
io-dom           active     -n--v-  5000    8     8G       0.0%  0.0%  16m
ldom01           inactive   ------          8     8G
root@t5pd:~# ldm bind ldom01
root@t5pd:~#
root@t5pd:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    8     8G       0.8%  0.8%  3h 23m
io-dom           active     -n--v-  5000    8     8G       0.1%  0.1%  16m
ldom01           bound      ------  5001    8     8G
root@t5pd:~# ldm start ldom01
LDom ldom01 started
root@t5pd:~# telnet localhost 5001
SPARC T5-2, No Keyboard
Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.35.3, 8.0000 GB memory available, Serial #83490061.
Ethernet address 0:14:4f:f9:f5:d, Host ID: 84f9f50d.



{0} ok
{0} ok
{0} ok show-disks
a) /virtual-devices@100/channel-devices@200/disk@1
b) /virtual-devices@100/channel-devices@200/disk@0
c) /iscsi-hba/disk
q) NO SELECTION
Enter Selection, q to quit: q
{0} ok
{0} ok devalias
hdd1                     /virtual-devices@100/channel-devices@200/disk@1
hdd0                     /virtual-devices@100/channel-devices@200/disk@0
net1                     /virtual-devices@100/channel-devices@200/network@1
net0                     /virtual-devices@100/channel-devices@200/network@0
net                      /virtual-devices@100/channel-devices@200/network@0
disk                     /virtual-devices@100/channel-devices@200/disk@0
virtual-console          /virtual-devices/console@1
name                     aliases
{0} ok
[...install...]

huhu... let's use a HA setup

root@ldom01:~# echo | format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c3d0 <Unknown-Unknown-0001-128.00GB>
          /virtual-devices@100/channel-devices@200/disk@0
       1. c3d1 <Unknown-Unknown-0001-128.00GB>
          /virtual-devices@100/channel-devices@200/disk@1
Specify disk (enter its number): Specify disk (enter its number):
root@ldom01:~# zpool attach rpool c3d0 c3d1
root@ldom01:~# zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 7.91G in 0h3m with 0 errors on Fri Sep 20 18:42:02 2013
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c3d0    ONLINE       0     0     0
            c3d1    ONLINE       0     0     0

errors: No known data errors
root@ldom01:~#
root@ldom01:~#
root@ldom01:~# ipadm show-if
IFNAME     CLASS    STATE    ACTIVE OVER
lo0        loopback ok       yes    --
net0       ip       ok       yes    --
root@ldom01:~#
root@ldom01:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net1              Ethernet             unknown    0      unknown   vnet1
net0              Ethernet             up         0      unknown   vnet0
root@ldom01:~#
root@ldom01:~# route -p delete default 192.168.56.1 -ifp net0
delete net default: gateway 192.168.56.1: not in table
delete persistent net default: gateway 192.168.56.1
root@ldom01:~# route -p add default 192.168.56.1
add net default: gateway 192.168.56.1: entry exists
add persistent net default: gateway 192.168.56.1
root@ldom01:~# svcadm restart routing-setup
root@ldom01:~#
root@ldom01:~# ipadm create-ip net1
root@ldom01:~# ipadm create-ipmp ipmp0
root@ldom01:~# ipadm add-ipmp -i net0 -i net1 ipmp0
root@ldom01:~# ipadm create-addr -T static -a 192.168.56.202/24 ipmp0/v4
root@ldom01:~# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
ipmp0/v4          static   ok           192.168.56.202/24
lo0/v6            static   ok           ::1/128
root@ldom01:~#
root@ldom01:~# ipmpstat -a
ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND
::                        down   ipmp0       --          --
ldom01                    up     ipmp0       net0        net1 net0
root@ldom01:~# ipmpstat -i
INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
net1        yes     ipmp0       -------   up        disabled  ok
net0        yes     ipmp0       --mbM--   up        disabled  ok
root@ldom01:~#