Veritas Cluster Server
Veritas Cluster Server = VCS.  High Aavailability Clusters (HAS) NOT High-Perf.
http://datadisk.co.uk/html_docs/veritas/veritas_cluster.htm

Fencing is a FS locking strategy to thwart split-brain condition.

License management
    vxlicinst
    vxlicrep
    vxlicense -p

Created by Veritas Software, now part of Symantec.
Packaged in SFHA product:  Storage Foundation High Availability
Similar to HP ServiceGuard.
Supports all major OSes and 40+ major app/replication technologies.
Only the ethernet heartbeat protocol requies kernel space kernel modules.
 (And the group membership protocol on top of that).
S Server product includes VCS Managemetn Console, a multi-cluster management...
Veritas Volume Manager is just like MetaDisk, but somewhat more capable.
Installation.  Old but simple:  http://unixway.com/vm/veritasvm/install.html
Bundled with SFHA (Storage Foundation HA), but also available alone for Unixes.
http://unixadvice.formyjob.net/t5-veritas-cluster-server-a-short-document-about-vcs
CHEAT SHEET:  http://www.datadisk.co.uk/html_docs/veritas/veritas_cluster_cs.htm
Proprietary Layer 2 ethernet-based protocol and group membership for heartbeat
 run in kernel space with modules.  Rest is user space.

Secure mode requires installation of a root Broker (Win or Sol) and all nodes
operate as authentication brokers.

VCS Management Console manages multiple clusters.  Win or Sol.

Shared disk prep
Disk Groups are the migratable partitionst that are exported and imported.
SCSI with unique controller IDs from the hosts
OR FC.
Some DB agents required shared disk usage.
Supports Veritias Volume Manager and Solstice DiskSuite as LV Managers.

Heartbeat networks
Use cross over cable, hub, switch.  Sometimes need an eeprom command to
enforce unique Mac addrs.

Architectural Components {
    LLT  kernel-to-kernel over LAN heartbeatlinks.  Config by file mostly.
        /etc/llttab  Ids integer node id, int. cluster id, links, 'start'.
        /etc/llthosts  Maps integer node ids to host names
    GAB  Shared disk comm and messaging.  Config by file mostly.
        /etc/gabtab   (node count at least.
            It contains just command '/sbin/gabconfig -c -n2'
            '/sbin/gabdisk...'
             to config optional comm disks... so why is this config-by file???
    VCS  Service management.  Config by file or CLI.
        /etc/VRTSvcs/conf/config/*
            main.cf   C-like Configurations for Clusters, Groups, Resources
                // Case Sensitive and no name-space independence.
                // comments
                includes
                Must include types.cf
                Sets maxes.
                cluster iMS()
                system aSysMemberName()  // Must be in llthosts
                system bSysMemberName()
                snmp vcs (...)
                group iMS5a (  // Service Group definition
                    SystemList = { aSysMemberName,... }
                    AutoStartList = {aSysMemberName }
                    NIC networka (  // A resource def
                        Device = hme0
                        NetworkType = ether
                    )
                    IP logical_IPa (  // A resource def.  TYPE name ( vars
                        Device = hme0
                        Address = "10.10.30.156"
                    )
                    logical_IPa requires networka  // Dependency tree of res.s
                                                   // Start sequence.
                )
            types.cf    C-like prototypes for each Resource
            FailOverPolicy = Priority (node-id-value), Load, RoundRobin // 1of
            ManualOps = true // CLI permitted?  Guessing about syntax
            Parallel = true  // I guess load sharing instead of failover?
}

Cluster {
    Systems  (hosts)
    Service Groups.   Subset of systems; start-group-systems; Resources
        I think res.s of a SG is what gets atomically migrated.
    Resources.  Monitor target: NIC, logical IP, mount point, disk, procs, file
}
Agents monitor Resources.

Heartbeats.  Requires at least 2.  1 may comm over a shared disk.
Standard 'link' heartbeat is high-bandwidth.  'link-lowpri' better suited for
 Internet.

COMMANDS.  All in /opt/VRTSvcs/bin unless noted otherwise.
    hacf -verify    # verifies configuration
    hastart
    hastop [-force] [-local] [-sys otherNodeName]
    hastatus [-summary]  # runs continuously without the switch
    lltstat [-nvv]  # network stats
    /sbin/gabconfig -a  # reports on membership with "jeopardy" meaning bad.
    hasys [-display]  # can add/delete mod systems
    hagrp
    hares  // Finest granularity tool
    hares -offline ResourceA -sys sysa
    hares -online ResourceA -sys sysa  # For failover groups ONLY ON active sys

    SWING:  hagrp -switch GroupA -to sysb

    CLEARING FAULTS:
    hagrp -clear GroupA
    hares -clear ResourceA
    ?/vxdg list

TODO:  What in hell is rootdg?  root Disk Group

GOTCHAS
VCS likes to have sole control of resources, so YOU don't bring up IPs, mount
disk volumes, etc.
Don't put VCS and non-VCS volumes into the same VG.
Don't use non-virtual interfaces.

All cluster members share the newest working VCS config setup.

DETAILED TROUBLESHOOTING
http://www.blacksheepnetworks.com/security/resources/veritas-cluster-server-debug.html

Recommended book: http://www.amazon.com/exec/obidos/ASIN/047118070X/jewlerymall

GUI
hagui  (On Solaris VRTScscm package must be installed).  Obsolete, but can
 download VCS Single Cluster Manager (is this hagui???)
VCS Management Console is recommended GUI.


CUSTOM AGENTS
Server Process Agent.
Veritas will only support yours if strict numerical + binary config.

INSTALLATION
Set up passwordless sftp access (definitely used by 'installvcs').
Can run "installer" (which also installs packages) or "installvcs" which you
can run anytime to install a cluster.
("installvcs" takes care of licenses somewhat)

Other files
/var/VRTSvcs/log
/etc/vx/elm  licenses

Daemons:  had + hashadow