View source for What is an Anvil! and why do I care?

{{howto_header}}

The most basic answer to this question is;

* An ''Anvil!'' is one of the most resilient platforms for running servers on. If you have a server that, if it failed, would really hurt, you probably want to put it on an ''Anvil!'' system.

Now, the longer answer.

= Overview =

The Anvil! platform is an [[Intelligent Availability]] platform. This means it has the following core features;

* It proactively protects hosted servers. It predicts failures where possible and accounts for changing threats.
* Full stack redundancy. All components can be completely removed and replaced without a maintenance window.
* Availability is prioritized before all else. Thin provisioning is not supported and compartmentalization is prioritized.
* Performance is kept consistent in a degraded state. It's not enough to be up if your performance falls below what is needed.
* Focus on automation. Even the best people make mistakes, so automate as much as possible.

It's natural to compare the ''Anvil!'' platform to a cloud solution. 

* Both host virtual servers.
* Both simplify the creation and management of servers.

This is not valid, though.

* Cloud solutions generally focus on resource utilization efficiency.
* Cloud solutions generally integrate with web-based services.
* Cloud solutions often run "agents" or custom kernels on the guest servers.

On the other hand;

* The ''Anvil!'' puts '''nothing''' before stability and availability. Period.
* The ''Anvil!'' is designed to be self-contained and is fully functional without any external network connection.
* The ''Anvil!'' emulates traditional hardware as completely as possible, requiring no special software inside the guest servers.

Before we proceed, lets cover some terminology;

* '''Fault Tolerant'''; This is the gold-standard of availability. A fault-tolerant system is one that can suffer an unexpected failure without causing any interruption. 
* '''High Availability'''; HA system are ones that can detect a fault and recover automatically and rapidly, but causes an interruption.
* '''Intelligent Availability'''&trade;; IA systems use intelligence software to detect changing threats and autonomously takes action to mitigate threats.

In short; Intelligent Availability tries to bridge to take action so that traditionally HA systems behave in a more FT manner.

{{note|1=Caveat: "[[Intelligent Availability]]" is a trademarked term created by Alteeve in order to define the next generation of availability. Any system can be made IA and describe itself as such. Alteeve's purpose in trademarking the term was to avoid the diluting of the definition.}}

Let's get into detail now!

= How Does The Anvil! Work? =

Many projects and products promise you the world. It's made us cynical, too.

== Important Note! ==

'''''Nothing is perfect'''''.

'''Period'''.

The ''Anvil!'' is no different. We know we're not perfect and that is a strength. 

Once you think you are perfect, that you have it all figured out, you will stop looking for ways to improve. You will start to make assumptions. You will eat humble pie and it will taste bitter.

== Full Stack Redundancy - An Overview ==

In traditional "High Availability", full-stack redundancy is not required. Many "HA" solutions are simply a [[SAN]] and a few nodes, which leaves several single points of failure. 

* A bad SAN firmware update, pulling the wrong drive from a degraded array and so on could take the storage down.
* A tripped [[UPS]] or [[PDU]] breaker can sever all power.
* A failed ethernet switch could sever network communication.

The ''Anvil!'' has been designed to provide no single points of failure. In fact, a guiding ethos of "Intelligent Availability" says that it is not enough to survive a failure, but also to survive recovery.

"So does this rule out blade chassis? SANs, too?"

Yup.

Consider this;

* Your blade/SAN chassis has redundant power backplanes. A fault causes a short and physically damages one of the power rails. You survive this fault successfully. 

How do you repair it?

You would have to cut all power to safely open that chassis to remove and replace the shorted circuit board. This means you need to schedule a maintenance window and hope that, once you open the chassis, you in fact have all the parts you need to effect repair.

The ''Anvil!'' solves this problem by sticking to old fashioned, mechanically and electrically separated components. 

* Standard rack mount machines for the nodes.
* Standard rack mount ethernet switches.

It's not sexy, but if there was an electrical short like above, we simply pull the dead part out and replace it. There is no need to take the backup component offline, so you can repair or replace without a maintenance window.

* "What about DC-sized UPSes? Can I use them instead of a couple smaller UPSes?"

No. 

Or rather, yes, sure, but you will still need the two small UPSes.

We've seen DC-sized UPSes fail numerous times. Causes include human error, like forgetting to put the power into bypass before maintenance, firmware faults and other failures. In all cases, all power was lost and all systems went offline. 

A pair of UPSes will insulate you against a failure in the UPS as the second UPS is fully independent and sufficient to power the full ''Anvil!''. Second, the ''Anvil!''uses the data from the UPSes in the [[ScanCore]] decision engine, which we will talk about later.

* "What's wrong with a SAN? <span class="code">$Vendor</span> tells me they are totally redundant!"

Well, maybe. Consider the issues discussed above; Can the vendor's SAN solve those problems? We've not seen it. Further, SANs generally use fiber-channel or iSCSI network connections to access the storage. This is avoidable complexity, and complexity is the enemy of availability.

Now, if you really want to use a SAN for capacity or performance reasons that local (direct-attached) storage can't provide, then fine. In this case, to be compatible with the ''Anvil!'', use two SANs and attach each SAN's LUN to either node and treat them as DAS after that.

== Storage Replication ==

This part of the ''Anvil!'' doesn't have hardware, per-se, but it needs to be discussed right off the top.

We discussed in the previous section how most SANs have a fundamental flaw, from a pure availability perspective. So then, how to we solve this problem? Replicated Storage!

In the ''Anvil!'', we have a special partitions on each node, and both nodes in the node pair have the same sized partition. We use [[DRBD]], an open-source software package, to "mirror" the data on these partitions. If you have ever build a RAID level 1 array before, then you already familiar with how data can be written to two locations at the same time. Later, if a drive failed, you would be protected because the data is still available on the remaining good drive.

DRBD works the same way, just over the network.

So imagine this traditional software <span class="code">[https://en.wikipedia.org/wiki/Mdadm mdadm]</span> array;

* <span class="code">/dev/sda5</span> + <span class="code">/dev/sdb5</span> -> <span class="code">/dev/md0</span>

In this case, once the <span class="code">md0</span> device is created, that is what you use from then on. The <span class="code">mdadm</span> software deals behind the scenes to copy the data to both locations, deal with failed drives and rebuild on a replacement drive.

In the DRBD world, it works like this:

* <span class="code">node1:/dev/sda5</span> + <span class="code">node2:/dev/sda5</span> -> <span class="code">/dev/drbd0</span>

Both nodes would be able to see the new <span class="code">/dev/drbd0</span> device, and writes to either node will be synchronously replicated to the peer node.

In our world, if we had a server running on a node's 'subnode 1', then each write it makes will be committed to the backing device on both subnodes before the write is considered complete. This way, if 'subnode 1' were destroyed, the server would boot up on 'subnode 2' right where it left off. File systems would replay their journals, databases would replay their write-ahead logs, etc. The recovery would be the same as a traditional bare iron server having lost power and then turned back on.

{{note|1=Unlike RAID, if a node disconnects, it '''won't''' need a full rebuild. DRBD keeps a bitmap of "dirty blocks" so that when the peer reconnects, only those dirty blocks need to be copied over to bring the peer back up to speed. However, the ''Anvil!'' is considered to be in a degraded state until this rebuild completes.}}

= The Anvil! Dissected =

To understand how the ''Anvil!'' achieves full-stack redundancy, we need to need a few parts.

* [[Selecting Hardware For Your Anvil!]]

== Physical Components ==

The ''Anvil!'' is based on three main component groups;

#. Foundation Pack
#. Node Pairs
#. Striker Dashboards

Briefly;

* A foundation pack consists on the foundation equipment; Two [[UPS]]es, two switched [[PDU]]s and two ethernet switches.
* Node Pairs are where the hosted servers live. Servers can migrate back and forth between nodes without interruption. Servers can be cold-migrated between pairs.
* Striker Dashboards host the web interface and store the [[ScanCore]] database data. 

Now lets look at these parts in more detail.

=== Foundation Pack ===

Lets look closer at the foundation on top of which the ''Anvil!'' node pairs and Striker dashboards will live.

==== Networked UPSes ====

[[image:APC_SMT1500RM2U_Front-Right.jpg|thumb|right|400px|APC [http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=SMT1500RM2U SmartUPS 1500 RM2U] 120vAC UPS. Photo by [http://www.apcmedia.com/prod_image_library/index.cfm?search_item=SMT1500RM2U APC].]]

Each UPS is connected to a dedicated mains circuit providing enough power (voltage and amperage) suitable to power the entire ''Anvil!''. 

When a building is serviced by two power companies, one circuit from either provider will be used to power either UPS. When only one provider is available, then two separate circuits are required. When possible, take two circuits from different panels.

The goal is to minimize the chance that both UPSes will lose power at the same time.

{{warning|1=It is critical that the power feeding the UPS, and the UPS itself, is of sufficient capacity that it can provide the required minimum hold-up time when the peer UPS has failed!}}

The network connection is used by [[ScanCore]] scan agents to monitor the quality of the input power, the health of the UPSes, the estimated hold-up time and the charge rate. We will cover this in more detail later.

==== Switched PDUs ====

[[image:APC_AP7900_Front-Right.jpg|thumb|right|400px|APC [http://www.apc.com/products/resource/include/techspec_index.cfm?base_sku=AP7900 AP7900 8-Outlet 1U] 120vAC PDU. Photo by [http://www.apcmedia.com/prod_image_library/index.cfm?search_item=AP7900# APC].]]

The switched PDUs each plug into their respective UPSes. These are, at their heart, network-connected power strips that can turn each outlet on and off, independent of one another.

This capability provides a critical safety capability as a backup fence device. We will go into more detail on this later.

All other devices with redundant power supplies will plug one supply into the first PDU and the backup supply to the second PDU. When devices without redundant power supplies are used, such as most Striker dashboard machines, the first device will draw power from the first UPS and the second, backup system will draw power from the second PDU.

Given that each PDU connects to a different UPS, and each UPS comes from a different circuit, power feeding the ''Anvil!'' is made as redundant as possible.

==== Ethernet Switches ====

[[Image:brocade_icx6610-24_iso_01.png|thumb|right|400px|The Brocade [http://www.brocade.com/en/products-services/switches/campus-network-switches/icx-6610-switch.html ICX 6610-24]. Picture by Brocade.]]

The ''Anvil!'' uses three separate, [[VLAN]]-isolated networks. Each network is designed to provide isolation from the others, for both security and performance stability reasons. 

They are:

* [[Back-Channel Network]] - Used for all internal ''Anvil!'' communication.
* [[Storage Network]] - Used for synchronous storage replication.
* [[Internet-Facing Network]] - Linkage between hosted servers and the outside network. 

The ''Anvil!'' node pairs use all three networks. Connections to each network are over two active-passive bonded network interfaces. Each link in a bonded pair must span two different network interfaces (we'll explore this more later).

<syntaxhighlight lang="text">
 _________________________                                                     
| [ an-a01n01 ]           |                                                    
|         ________________|                        ______________              
|        |     ___________|                       |  bcn1_bond1  |             
|        | P  | bcn1_link1 =----------------------=-----+--------=----{ Switch 
|        | C  |__________||                       |     |        |             
|        | I              |    /------------------=-----/        |             
|        | e   ___________|    |                  |______________|             
|        |    | ifn1_link1 =----------------\      ______________              
|        | 1  |__________||    |            |     |  ifn1_bond1  |             
|        |                |    |            \-----=-----+--------=----{ Switch 
|        |     ___________|    |                  |     |        |             
|        |    |  sn1_link1 =---------\  /---------=-----/        |             
|        |    |__________||    |     |  |         |______________|             
|        |                |    |     |  |          ______________              
|        |     ___________|    |     |  |         |  sn1_bond1   |             
|        |    |  mn1_link1 =------\  \------------=-----+--------=----{ Switch 
|        |    |__________||    |  |     |         |     |        |      or Peer
|        |________________|    |  |     |  /------=-----/        |             
|         ________________|    |  |     |  |      |______________|             
|        |     ___________|    |  |     |  |       ______________              
|        | P  | bcn1_link2 =---/  |     |  |      |  mn1_bond1   |             
|        | C  |__________||       \---------------=-----+--------=----{ Peer   
|        | I              |             |  |      |     |        |             
|        | e   ___________|             |  |  /---=-----/        |             
|        |    | ifn1_link2 =------------/  |  |   |______________|             
|        | 2  |__________||                |  |                                
|        |                |                |  |                                
|        |     ___________|                |  |                                
|        |    |  sn1_link2 =---------------/  |                                
|        |    |__________||                   |                                
|        |                |                   |                                
|        |     ___________|                   |                                
|        |    |  mn1_link2 =------------------/                                
|        |    |__________||                                                    
|        |________________|                                                    
|_________________________|                                                    
</syntaxhighlight>

With this configuration, the network on the node pairs are made fully redundant. Lets look at a few common failure scenarios;

* If the primary switch or the first PCIe controller is lost, all bonded interfaces will fail over to the backup links.
* If the backup switch or the second PCIe controller is lost, the bonds will degrade but no cut-over will be required.

As we will see later, if a node needs to be ejected from the ''Anvil!'' system, the [[IPMI]] interfaces will be connected to the primary switch and the switched PDUs will be connected to the backup switch. These two devices provide alternate mechanism for ejecting a failed node, so the loss of either switch will not remove the ability to eject a failed node.

Even the ''Anvil!'' platform's safety systems have backups!

=== Anvil! Nodes ===

[[image:Fujitsu_Primergy_RX2540-M2_Front-Left.jpg|thumb|right|700px|The capable Fujitsu Primergy [http://www.fujitsu.com/global/products/computing/servers/primergy/rack/rx2540m2/ RX2540 M2]; Photo by [http://mediaportal.ts.fujitsu.com/pages/view.php?ref=39130&search=rx2540&order_by=field12&sort=DESC&offset=0&archive=0&k= Fujitsu].]]

An ''Anvil!'' node consists of two physical subnodes, each capable alone of powering all hosted servers with sufficient speed to deliver operation performance requirements.

The nodes themselves can vary in capacity and performance a great deal from one pair to the next. Despite this, there are architectural requirements that must always be met. They are:

* CPUs that support hardware virtualization ([https://en.wikipedia.org/wiki/X86_virtualization#Intel_virtualization_.28VT-x.29 VT-d]). Almost all modern CPUs provide this, though it is often disabled in the BIOS by default.
* Sufficient RAM for all hosted servers, plus at least 4 [[GiB]] for the host.
* Six network interfaces.
* [[IPMI]] (out-of-band) support, known as 'iRMC' on Fujitsu/Siemens, 'RSA' on IBM/Lenove, 'DRAC' on Dell 'iLO' on HP, and so on.
* Redundant Power Supplies

This configuration is the minimum configuration needed to provide no single point of failure. Additional features, like RAID for storage, mirrored RAM, 3x dual-port network cards and similar are highly recommended as they increase the fault tolerance of each node.

==== Wait, Why Pairs? ====

'''Buckle up, we're going on a tangent'''.

A frequently asked question is "why does the ''Anvil!'' only support pairs of nodes? How do you scale?".

The answer is, quite simply, "simplicity".

In availability, there is nothing more important that continuous operation. Every single proposed feature must be looked at through this lens.

A 2-node setup, much like [[TLUG_Talk:_Storage_Technologies_and_Theory#Level_1|RAID level 1]], trades efficiency for simplicity. In a two-node configuration, you are able to gain access to all the protection you require to protect the hosted servers. 

Adding a third or fourth node would increase complexity in exchange for better resource utilization. The idea, and it is a sensible one, is that one spare node could protect two or more active server hosts. This is true, but then you need to introduce external, fully redundant storage which will be accessed over less reliable network or fiber-channel links. You also have much more complex cluster resource management configurations. You also increase the network traffic needed to keep all the nodes in sync, and so on.

Availability is not an efficiency game. Cloud technologies are.

==== But I Was Told That 2-Nodes Can't Be Highly Available? ====

Here's the longer answer;

* [[The 2-Node Myth]]

This is a common myth in the availability industry, and it comes from people and vendors misunderstanding the role of "[[quorum]]" and "[[fencing]]" (also called "[[stonith]]").

In short; Fencing is mandatory, quorum is optional.

People who say that 2-node clusters can't work say so because they get this backwards.

To answer this, we first must now explain fencing.

===== Problem; Split-Brains =====

In a fully redundant system like the ''Anvil!'', it is theoretically possible that both nodes could partition. That is to say, they can each think that their peer is dead and thus try to operate alone. This condition is called a "split-brain" and it is one of the most dangerous conditions for an availability platform.

* At the very least, you have a very troublesome and drawn out recovery process.
* At worst, you corrupt your data.

Consider this scenario;

A fault in the network stack is such that inter-node communication is lost. This could be something as simple as an errant firewall rule, or it could be a weird fault in the switches. The cause itself doesn't matter. The result, though, is that both nodes lose contact with their peer, but are otherwise functional.

If an assumption was made that "Hey, I can't talk to my peer, so it must be dead", then this is what would happen.

#. The node that was hosting the servers goes back to operating normally, as though nothing happened.
#. The peer node, thinking the old host is dead, now boots all of the servers in an attempt to restore service.

'''How bad is this?'''

{|
|[[Image:family-guy_oh-no.jpg|thumb|left|236px|Consuela image copyright 20th Century Fox Television]]
|}

''That depends, how lucky were you''?

If the replicated storage split at the same time, then it means that each copy of the ''identical'' server is now operating independently and your data is diverging. Setting aside concerns of two identical servers claiming the same hostname, IP and [[MAC]] address, you now have a problem of deciding which copy's data is "more" better.

To recover, you have to tell one node to shut down it's copy and then discard all of the changes that happened after the break. If you are super lucky, this might be an easy decision to make. In most cases though, you are going to have to lose data.

You could take one node's servers, backup the data to an external source, discard the changes and then manually restore just the data you need to merge back. How hard is this depends on your application and data structure. It certainly won't be a fast recovery. 

You can't automate this. Do you discard the side with the smallest changes? Perhaps the side with the oldest changes? What if you had some users connect to one server and they recorded financial information, patient data or similarly small but important data. Later, other users who accessed the other node uploaded a DVD disc image. If you discard oldest or smallest changes, you wipe the obviously more important data.

In the worst case scenario, you might still have storage replicating normally. The replicated storage subsystem looks at byte streams, not data structures. If data comes in from both nodes, as happens normally with cluster-aware filesystems, the replicated storage simply ensures that the changes from either node make it to the peer. 

If you boot the same server on the second node, it will start to write data to the same blocks. Congratulations, you just corrupted your server's storage.

Q. So how do you prevent this?

A. Fencing.

===== Fencing - Ejecting Nodes For Great Success! =====

In the above horror scenario, the root of the problem was that the nodes were allowed to make assumptions. Sure, in a demo to a client this is safe because you actually kill a node. Great. 

That's not how it goes in the real world, though.

When a node stops responding, the only thing we know is that we don't know what it is doing...

* Maybe it's running fine but we can't talk to it.
* Maybe it's hung and might recover later.
* Maybe it's a smouldering pile for iron and sadness.

Fencing is, simply, a tool for blocking the cluster until the peer can be put into a known state. The most common way of doing this is to turn the node off.

In the ''Anvil!'', this is done by first connecting to the lost node's [[IPMI]] baseboard management controller. The BMC is basically an independent computer that lives on the host's mainboard. It has its own CPU, RAM, MAC and IP address and it can function no matter whether it's host is powered on or not. It can read the host's sensors and press its power and reset buttons. 

We saw above that each node's IPMI interface is plugged into ethernet switch 1 on the [[BCN]]. The good node will log into the peer's IPMI and ask the BMC "hey, please power off your host". The BMC does this by pressing and holding the power switch until the host shuts down. Once off, the good node is informed that the peer is dead and recovery (if needed) can begin safely.

There's a flaw with this, however.

The BMC draws its power from the same source as the host itself. So in the case of a catastrophic failure, the BMC will die along with its host. So in this case, the good node will say ''try'' to talk to the BMC, but it will get no response. 

So what then?

If this was the only fence device, then the good node would stay blocked. The logic is that, as bad as it is to hang the cluster, it is better than risking data loss.

Thankfully, we have a backup!

In the ''Anvil!'', the good node would time-out waiting for a reply from the BMC and give up. It would then move on to the backup fence method; The switched PDUs. Note that the PDUs are plugged into the second switch.

Node 1 knows which outlets node 2 draws its power from. Likewise, Node 2 knows which outlets node 1 is plugged into. So if the lost peer can't be reached over IPMI, the good node logs into both PDUs and cuts the power to both power supplies feeding the peer. 

Now, we can be sure that the peer is dead! Recovery will proceed, safe in the knowledge that the peer is not running the servers.

It's brutal, but it works.

===== Quorum - When All Is Well =====

The basic principle of quorum is that you have a "tie-breaker" node. 

The idea is that a majority of nodes need to be communicating with each other in order to be allowed to host and run any cluster resources, like virtual servers.

The mistake people make is that they think that this tie-breaker vote can prevent a split-brain.

'''''It can not'''''.

The problem is that quorum only works when all nodes are behaving in a predictable manner.

In an ideal world, if a node lost access to the peers, it would shut down any hosted servers because it would become ''inquorate'' it had a minority of votes. If it really did this, then yes, quorum would prevent a split-brain. Unfortunately, the real world of failures is too messy to ever make this assumption. 

Consider this ideal 3-node, 3-vote scenario;

Your server is running on Node 1. Someone throws up an errant firewall rule and blocks communication with nodes 2 and 3. Node 1 says "well, I am by myself, so I have 1 vote. I'm inquorate, so I am going to kill my hosted servers". Meanwhile, nodes 2 and 3 declare node 1 lost, see that they have 2 votes and are therefore quorate. They assume that node 1 must be shutting down it's servers, so they decide to go ahead and boot the servers on node 2.

Excellent. Wonderful. Too bad that doesn't always happen.

Consider this realistic 3-node, 3-vote scenario;

Node 1 is again hosting a server. That server is in the middle of a write to disk, and for some reason the host node hangs hard. Nodes 2 and 3 re-vote, declare they have quorum and recover the server by booting it on node 2. 

Now here is where it all goes sideways;

At some point later, Node 1 recovers. It has no concept of time passing, so it has no reason in that moment to check if it is still quorate or not. The server completes its write and voila, you've corrupted your storage. Sure, node 1 will realize it has lost quorum and try to shut down, but by then the damage has been done.

Quorum can be useful, but it is not required. It is certainly '''NOT''' a substitute for fencing.

===== Can Both Nodes Fence Each Other? =====

Short answer; Yes.

Long answer; Almost never.

Consider again the scenario where both nodes in a 2-node cluster are alive, but communication is severed. Both nodes will try to shoot the other. 

{|
|[[Image:ourobengr_stonith.png|thumb|center|626px|Image from [https://ourobengr.com/ha/ Ourobengr].]]
|}

This is often portrayed as a "wild-west" style shoot-out. The fastest node to pull the trigger lives and the slower node dies.

This is, basically, what happens in real life. However, we can bias fencing to ensure that one node will live in a case like this.

In the ''Anvil!'', the node hosting the servers is given a 15 second head start of fencing the peer. In a case where both nodes are alive, the node without servers is told to wait 15 seconds, where the node with the server pulls the trigger immediately. So in this case, the node with the servers will always win, avoiding the need to recover the servers at all.

If, however, node 1 was truly failed, then node 2 would exit the wait and shoot node 1. Once shot, it would reboot the servers, no different than if a bare-iron server lost power and was restarted.

===== Fencing vs. Quorum - In Summary =====

This provides, I hope, a complete answer to why the myth that you can't do availability on 2-node platforms is not just wrong, but backwards.

There is a saying in the availability world;

* "A cluster is not beautiful when there is nothing left to add. It is beautiful when there is nothing left to take away."

A 2-node platform is significantly simpler and totally safe, thanks to fencing.

==== Scaling The Anvil! - Compartmentalization ====

The ''Anvil!'' scales by adding node pairs. 

Each pair is an autonomous unit that operates independent of other nodes. The servers on one pair can live-migrate between the nodes without interruption. Servers can, when needed, be cold-migrated to a new node pair if needed.

A single foundation pack can host two or three node pairs. Once a foundation pack is filled, a new foundation pack is deployed and then additional node pairs can be deployed. In this way, you can run as many pairs as you like.

''Wait, isn't that really wasteful''?

Yes, but recall that availability is the goal, not resource utilization efficiency.

The benefit of keeping the node pair to foundation pack ratio high is that is provides healthy compartmentalization. A multi-component disaster on one foundation pack will effect only its hosted node pairs. In a large deployement, this could mean that a subset of systems are lost, which is certainly a very bad thing, but the damage will be limited.

Think of it like floodable compartments on a ship (an imperfect analogy, but work with me on this). 

{|
|[[Image:double-hauled-ship.jpg|thumb|center|596px|Image copyright [http://www.marine-knowledge.com/single-vs-double-hull-ships/single-vs-double-hull-ships/ Marine-Knowledge].]]
|}

In a normal double-haul ship, you can think of the outer and inner hauls as you traditional redundancy. If you hit an iceberg, you will tear the outer haul, but not the inner haul. Excellent.

{|
|[[Image:double-hauled-with-transverse-watertight-compartments.jpg|thumb|center|582px|The USS South Dakota's interior amidships, including six major watertight transverse bulkheads and anti-torpedo longitudinal bulkheads along the sides. Image source [https://en.wikipedia.org/wiki/File:USS_South_Dakota_(BB-57)_under_construction,_1_April_1940.jpg wikipedia].]]
|}

What if your ship is a military vessel and is hit by a torpedo? Both hauls will be breached and water will start to ingress, and now you need to focus on [https://en.wikipedia.org/wiki/Ship_floodability damage control]. Whatever was on the other side of the haul is lost, but it doesn't mean your ship is going to sink. If your ship has water tight compartments, you can seal off the destroyed portion and limit the amount of water taken on. Your ship is crippled, but not sunk.

That is the idea behind compartmentalization in the ''Anvil!'' system.

=== Striker Dashboards ===

Striker dashboard systems play three roles;

#. They provide a web-interface designed to provide a simple user interface that covers 90% of day-to-day use of the ''Anvil!'' systems.
#. They host the [[ScanCore]] database where all sensor data from all connected ''Anvil!'' node pairs is stored.
#. They determine when it is safe to recover an 'Anvil!'' node or node pair after an emergency shut down or unexpected power loss.

A single pair of Striker dashboards can, theoretically, manage any number of ''Anvil!'' node pairs. 

In practice, the number of functionally supported node pairs is limited to the dashboard's performance. The incoming ScanCore sensor and state data is auto-archived, but there is still a functional limit to what the dashboard hardware can support. The more powerful the dashboard hardware, the more ''Anvil!'' pairs they can manage.

Striker dashboards are rarely inherently redundant. Redundancy is achieved by there being two of them.

* The first dashboard draws its power from the first power rail and connects into the first ethernet switch.
* The second dashboard draws its power from the second power rail and connects into the second ethernet switch.

In this way, no matter what fails, one of the dashboards will be up and running, providing full redundancy.

The dashboards themselves do not have a concept of "primary" or "backup". Each one is fully independent, but they are able to merge in data that was collected by another dashboard while it was offline. The resyncronization can happen in both directions at the same time, and the synchronization is not limited to two dashboards. There is no hard limit on how many dashboards can work together and maintain data sync. Of course, the same limitations on load versus hardware capacity remain.

We will explore ScanCore later, as it is the intelligence in "Intelligent Availability" and requires a section of its own.

== Rack Configuration ==

Below is a very typical ''Anvil!'' load-out. Note that this is a half-height rack and a single foundation pack hosting two node pairs. The Striker dashboards are typically Intel NUCs and sit on the shelf.

{|class="wikitable"
|style="background-color: #cfcfcf; text-align: center;" colspan="3"|<span class="plainlinks">[http://www.apc.com/shop/ca/en/products/NetShelter-SX-24U-600mm-Wide-x-1070mm-Deep-Enclosure-with-Sides-Black/P-AR3104 <span style="color: #2f2f2f; font-weight: bold;">NetShelter SX 24U</span>]</span>
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;" colspan="3"|
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|24
|style="background-color: #7f7f7f; color: #2f2f2f; text-align: center; width: 200px;"|--
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|24
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|23
|style="background-color: #7f7f7f; color: #2f2f2f; text-align: center;"|--
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|23
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|22
|style="background-color: #7f7f7f; color: #2f2f2f; text-align: center;"|--
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|22
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|21
|style="background-color: #7f7f7f; color: #2f2f2f; text-align: center;"|--
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|21
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|20
|style="background-color: #cfcfcf; color: #2f2f2f; text-align: center;" class="code"|<span class="plainlinks">[http://www.apc.com/shop/ca/en/products/Sliding-Shelf-100lbs-45-5kg-Black/P-AR8123BLK <span style="color: #2f2f2f;">Shelf</span>]</span>
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|20
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|19
|style="background-color: #dfdfdf; color: #2f2f2f; text-align: center;" class="code" rowspan="2"|an-a01n01
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|19
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|18
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|18
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|17
|style="background-color: #cfcfcf; color: #2f2f2f; text-align: center;" class="code" rowspan="2"|an-a01n02
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|17
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|16
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|16
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|15
|style="background-color: #dfdfdf; color: #2f2f2f; text-align: center;" class="code" rowspan="2"|an-a02n01
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|15
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|14
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|14
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|13
|style="background-color: #cfcfcf; color: #2f2f2f; text-align: center;" class="code" rowspan="2"|an-a02n02
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|13
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|12
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|12
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|11
|style="background-color: #dfdfdf; color: #2f2f2f; text-align: center;" class="code"|an-switch01
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|11
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|10
|style="background-color: #cfcfcf; color: #2f2f2f; text-align: center;" class="code"|an-switch02
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|10
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|9
|style="background-color: #afafaf; color: #676767; text-align: center;"|<network cables>
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|9
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|8
|style="background-color: #dfdfdf; color: #2f2f2f; text-align: center;" class="code"|an-pdu01
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|8
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|7
|style="background-color: #afafaf; color: #676767; text-align: center;"|<power cables>
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|7
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|6
|style="background-color: #cfcfcf; color: #2f2f2f; text-align: center;" class="code"|an-pdu02
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|6
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|5
|style="background-color: #afafaf; color: #676767; text-align: center;"|<power cables>
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|5
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|4
|style="background-color: #dfdfdf; color: #2f2f2f; text-align: center;" class="code" rowspan="2"|an-ups01
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|4
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|3
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|3
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|2
|style="background-color: #cfcfcf; color: #2f2f2f; text-align: center;" class="code" rowspan="2"|an-ups02
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|2
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|1
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;"|1
|-
|style="background-color: #2f2f2f; color: #cfcfcf; text-align: center;" colspan="3"|
|}

=== Physical Configuration - Consistency ===

Recall how we said that nothing is perfect? We said this is a strength, and now we will explain why.

When you accept that everything can fail, you start to ask yourself, "If there is a failure, what can I do now to minimize the mean time to recover?".

One of the strongest tools you have is consistency. 

If you make sure that every installation is as identical as possible, then you won't have to reference documentation to find out what ports were used, how devices were racks, etc. Building on this, you will want to label ''everything''. You want to be able to walk up to any system and immediately confirm that a given cable is, in fact, the cable you expect it to be. 

A few hours of work pre-deployment can significantly reduce your recovery time in a disaster. This can not be emphasized enough.

In the ''Anvil!'' system, this means that the ports used on the ethernet switches, switched PDUs and position of devices in the rack are always the same. You can walk up to an ''Anvil!'' that was installed years ago that you have never seen before and how it is racked and assembled will be familiar.

This extends into the system as well. 

Network interface names, node names, resource names... It is all consistent install after install. 

A good ''Anvil!'' system is one that you will pretty much forget that you have. In the early days, we would provide training to clients so that they could learn the guts of the system. In some cases, we would be hired to walk the client's staff thought the entire manual build process so that they could fully understand the system. These training sessions had value, but one thing became very clear; People forget.

The problem is that the time span between training and problem could be 6, 12, 18 months or more. No matter how smart the user, it was just too little exposure.

Providing consistency is one of the best tools you have to help mitigate this. 

=== Beneficial Assumptions ===

One of the biggest benefits of consistency in design is that it allows you to make a very large number of assumptions in your software.

Humans make mistakes, and that is unavoidable. 

Accepting this means that you will design the software to minimize the surface area where users can break things. If you ask the user to tell you if this given system has this or that setup, they will eventually get it wrong. By building on top of an extremely consistent system, you can remove most of the guess work and dramatically simplify your user interface.

Striker has taken advantage of this fact. It is far from the most powerful UI, but that's by design. Recall; ''Nothing is more important than availability''.

'''The purpose of the Striker UI is ''not'' to help the user. It is to protect the system.'''

When you start looking at the user interface this way, your design decisions change fairly dramatically. You are not competing against the feature set of cloud platforms anymore. 

= A Logical Breakdown =

Thus far, we've talked about the ''Anvil!'' at a fairly high level. 

Lets break it down.

== Logical Block Diagram ==

Below is a full block-diagram map of the ''Anvil!'' cluster, with two nodes. The first node also shows the striker dashboards. 

Consider what we covered above, and this should start to make sense.

{|
|style="white-space:nowrap; text-align:center;"|
<syntaxhighlight lang="text">
-=] Strikers and an-anvil-01                                                                                                                                                                              
                                                                                               \_  IFN 1  _/                                                                                              
  Striker Dashboards                                                                             \_ LAN _/                                                                                                
                _______________________________                                                    \___/                                                      _______________________________             
               | an-striker01.alteeve.com      |                                                     |                                                       |      an-striker02.alteeve.com |            
               |             __________________|                                                     |                                                       |___________________            |            
               |           | ifn1_link1        =---------------------------------\ /-------------------------------------------------------------------------=        ifn1_likn1 |           |            
               |           |       10.255.4.1 ||                                 | |                 |                                                       || 10.255.4.2       |           |            
               |           |__________________||                                 | |                 |                                                       ||__________________|           |            
               |            ___________________|                                 | |                 |                                                       |___________________            |            
               |           | bcn1_link1        =---------------------------------|-|-----------------------------------\ /-----------------------------------=        bcn1_link1 |           |            
               |           |       10.201.4.1 ||                                 | |                 |                 | |                                   || 10.201.4.2       |           |            
               |           |__________________||                                 | |                 |                 | |                                   ||__________________|           |            
               |                               |                                 | |                 |                 | |                                   |                               |            
               |                        _______|                                 | |                 |                 | |                                   |_______                        |            
               |                       | PSU 1 |    ____________________         | |                 |                 | |           ____________________    | PSU 1 |                       |            
               |_______________________|_______|==={_to_an-pdu01_port-8_}        | |                 |                 | |          {_to_an-pdu02_port-8_}===|_______|_______________________|            
                                                                                 | |                 |                 | |                                                                                
-=] an-anvil-02 (Node 2)                                                         | |                 |                 | |                                                                                
  ____________________________________________________________________________   | |            _____|____             | |   ____________________________________________________________________________ 
 | an-a01n01.alteeve.com                                                      |  | | /---------{_intranet_}----------\ | |  |                                                      an-a01n02.alteeve.com |
 |                                    Network:               _________________|  | | |                               | | |  |_________________               Network:                                    |
 |                                    _________________     | ifn1_bond1      |  | | |   _________________________   | | |  |      ifn1_bond1 |     _________________                                    |
 |      Servers:                     |  ifn1_bridge1   |----| <ip on bridge>  |  | | |  | an-switch01             |  | | |  |  <ip on bridge> |----|  ifn1_bridge1   |                     Servers:      |
 |      __________________________   |   10.255.10.1   |    |    _____________|  | | |  |____ Internet-Facing ____|  | | |  |____________     |    |   10.255.10.2   |  ............................     |
 |     | [ srv01-sql ]            |  |_________________|    |   | ifn1_link1  =---------=_01_]    Network    [_02_=---------=  ifn_link1 |    |    |_________________|  :            [ srv01-sql ] :     |
 |     |        __________________|    | | | | |            |   |____________||  \------=_09_]_______________[_24_=--/ | |  ||___________|    |            : : : : :    :__________________        :     |
 |     |       | NIC 1            =----/ | | | |            |                 |    | |  | an-switch02             |    | |  |                 |            : : : : -----=            NIC 1 |       :     |
 |     |       |      10.255.1.1 ||      | | | |            |    _____________|    | |  |____                 ____|    | |  |____________     |            : : : :      :| 10.255.1.1      |       :     |
 |     |       |_________________||      | | | |            |   | ifn1_link2  =---------=_01_]  VLAN ID 300  [_02_=---------=  ifn_link2 |    |            : : : :      :|_________________|       :     |
 |     |                          |      | | | |            |   |____________||    \-|--=_09_]_______________[_24_=--\ | |  ||___________|    |            : : : :      :                          :     |
 |     |   ____                   |      | | | |            |_________________|      |                               | | |  |_________________|            : : : :      :                   ____   :     |
 |  /--=--[_c:_]                  |      | | | |                              |      \-------------------------------/ | |  |                              : : : :      :                  [_c:_]--=--\  |
 |  |  |__________________________|      | | | |             _________________|                                        | |  |_________________             : : : :      :..........................:  |  |
 |  |                                    | | | |            | sn1_bond1       |                                        | |  |       sn1_bond1 |            : : : :                                    |  |
 |  |    __________________________      | | | |            |     10.101.10.1 |                                        | |  | 10.101.10.2     |            : : : :     ............................   |  |
 |  |   | [ srv02-app1 ]           |     | | | |            |     ____________|                                        | |  |____________     |            : : : :     :           [ srv02-app1 ] :   |  |
 |  |   |        __________________|     | | | |            |    | sn1_link1  =---------------------------------------------=  sn1_link1 |    |            : : : :     :__________________        :   |  |
 |  |   |       | NIC 1            =-----/ | | |            |    |___________||                                        | |  ||___________|    |            : : : ------=            NIC 1 |       :   |  |
 |  |   |       |      10.255.1.2 ||       | | |            |                 |                                        | |  |                 |            : : :       :| 10.255.1.2      |       :   |  |
 |  |   |       |_________________||       | | |            |     ____________|                                        | |  |____________     |            : : :       :|_________________|       :   |  |
 |  |   |                          |       | | |            |    | sn1_link2  =---------------------------------------------=  sn1_link2 |    |            : : :       :                          :   |  |
 |  |   |   ____                   |       | | |  /---------|    |___________||                                        | |  ||___________|    |---------\  : : :       :                   ____   :   |  |
 |  +---=--[_c:_]                  |       | | |  |         |_________________|                                        | |  |_________________|         |  : : :       :                  [_c:_]--=---+  |
 |  |   |__________________________|       | | |  |                           |                                        | |  |                           |  : : :       :..........................:   |  |
 |  |                                      | | |  |          _________________|                                        | |  |_________________          |  : : :                                      |  |
 |  |    __________________________        | | |  |         | mn1_bond1       |                                        | |  |       mn1_bond1 |         |  : : :       ............................   |  |
 |  |   | [ srv03-app2 ]           |       | | |  |         |     10.199.10.1 |                                        | |  | 10.199.10.2     |         |  : : :       :           [ srv03-app2 ] :   |  |
 |  |   |        __________________|       | | |  |         |     ____________|                                        | |  |____________     |         |  : : :       :__________________        :   |  |
 |  |   |       | NIC 1            =-------/ | |  |         |    | mn1_link1  =---------------------------------------------=  mn1_link1 |    |         |  : : --------=            NIC 1 |       :   |  |
 |  |   |       |      10.255.1.3 ||         | |  |         |    |___________||                                        | |  ||___________|    |         |  : :         :| 10.255.1.3      |       :   |  |
 |  |   |       |_________________||         | |  |         |                 |                                        | |  |                 |         |  : :         :|_________________|       :   |  |
 |  |   |                          |         | |  |         |     ____________|                                        | |  |____________     |         |  : :         :                          :   |  |
 |  |   |  ____                    |         | |  |         |    | mn1 link2  =---------------------------------------------=  mn1_link2 |    |         |  : :         :                   ____   :   |  |
 |  +--=--[_c:_]                   |         | |  |         |    |___________||                                        | |  ||___________|    |         |  : :         :                  [_c:_]--=---+  |
 |  |   |__________________________|         | |  |         |_________________|                                        | |  |_________________|         |  : :         :..........................:   |  |
 |  |                                        | |  |                           |                                        | |  |                           |  : :                                        |  |
 |  |    __________________________          | |  |          _________________|                                        | |  |_________________          |  : :         ............................   |  |
 |  |   | [ srv04-app3 ]           |         | |  |         | bcn1_bond1      |          _________________________     | |  |      bcn1_bond1 |         |  : :         :           [ srv04-app3 ] :   |  |
 |  |   |        __________________|         | |  |         | <ip on bridge>  |         | an-switch01             |    | |  |  <ip on bridge> |         |  : :         :__________________        :   |  |
 |  |   |       | NIC 1            =---------/ |  |         |    _____________|         |____  Back-Channel   ____|    | |  |_____________    |         |  : ----------=            NIC 1 |       :   |  |
 |  |   |       |      10.255.1.4 ||           |  |         |   | bcn1_link1  =---------=_13_]    Network    [_14_=---------=  bcn1_link1 |   |         |  :           :| 10.255.1.4      |       :   |  |
 |  |   |       |_________________||           |  |         |   |____________||         |____________________[_19_=----/ |  ||____________|   |         |  :           :|_________________|       :   |  |
 |  |   |                          |           |  |         |                 |         | an-switch02             |      |  |                 |         |  :           :                          :   |  |
 |  |   |  ____                    |           |  |         |    _____________|         |____                 ____|      |  |_____ _______    |         |  :           :                   ____   :   |  |
 |  +--=--[_c:_]                   |           |  |         |   | bcn1_link2  =---------=_13_]  VLAN ID 100  [_14_=---------=  bcn1_link2 |   |         |  :           :                  [_c:_]--=---+  |
 |  |   |__________________________|           |  |         |   |____________||         |____________________[_19_=------/  ||____________|   |         |  :           :..........................:   |  |
 |  |                                          |  |         |_________________|                                             |_________________|         |  :                                          |  |
 |  |    __________________________            |  |            |              |                                             |              |            |  :           ............................   |  |
 |  |   | [ srv05-admin ]          |           |  |     _______|_________     |                                             |     _________|_______     |  :           :          [ srv05-admin ] :   |  |
 |  |   |        __________________|           |  |    |  bcn1_bridge1   |    |                                             |    |  bcn1_bridge1   |    |  :           :__________________        :   |  |
 |  |   |       | NIC 1            =-----------/  |    |   10.201.10.1   |    |                                             |    |   10.201.10.2   |    |  ------------=            NIC 1 |       :   |  |
 |  |   |       |    10.255.1.250 ||              |    |_________________|    |                                             |    |_________________|    |              :| 10.255.1.250    |       :   |  |
 |  |   |       |_________________||              |       |                   |                                             |                   |       |              :|_________________|       :   |  |
 |  |   |                          |              |       |                   |                                             |                   |       |              :                          :   |  |
 |  |   |        __________________|              |       |                   |                                             |                   |       |              :__________________        :   |  |
 |  |   |       | NIC 2            =----------------------/                   |                                             |                   \----------------------=            NIC 2 |       :   |  |
 |  |   |       |    10.201.1.250 ||              |                           |                                             |                           |              :| 10.201.1.250    |       :   |  |
 |  |   |       |_________________||              |                           |                                             |                           |              :|_________________|       :   |  |
 |  |   |   _____                  |              |                           |                                             |                           |              :                  _____   :   |  |
 |  +---=--[_vda_]                 |              |                           |                                             |                           |              :                 [_vda_]--=---+  |
 |  |   |__________________________|              |                           |                                             |                           |              :..........................:   |  |
 |  |                                             |                           |                                             |                           |                                             |  |
 |  |                                             |                           |                                             |                           |                                             |  |
 |  |                                             |                           |                                             |                           |                                             |  |
 |  |                                             |                           |                                             |                           |                                             |  |
 |  |     Storage:                                |                           |                                             |                           |                                Storage:     |  |
 |  |     __________                              |                           |                                             |                           |                              __________     |  |
 |  |    [_/dev/sda_]                             |                           |                                             |                           |                             [_/dev/sda_]    |  |
 |  |      |   ___________         ___________    |                           |                                             |                           |    ___________         ___________   |      |  |
 |  |      +--[_/dev/sda1_]-------[_/boot/efi_]   |                           |                                             |                           |   [_/boot/efi_]-------[_/dev/sda1_]--+      |  |
 |  |      |   ___________         _______        |                           |                                             |                           |        _______         ___________   |      |  |
 |  |      +--[_/dev/sda2_]-------[_/boot_]       |                           |                                             |                           |       [_/boot_]-------[_/dev/sda1_]--+      |  |
 |  |      |   ___________         ___________    |                           |                                             |                           |    ___________         ___________   |      |  |
 |  |      +--[_/dev/sda3_]-------[_<lvm:_pv>_]   |                           |                                             |                           |   [_<lvm:_pv>_]-------[_/dev/sda1_]--+      |  |
 |  |            ___________________|             \------------------\        |                                             |        /------------------/             |___________________            |  |
 |  |          _|___________________                                 |        |                                             |        |                                 ___________________|_          |  |
 |  |         [_<lvm:_vg_an-a01n01>_]                                |        |                                             |        |                                [_<lvm:_vg_an-a01n01>_]         |  |
 |  |          _|___________________    ________                     |        |                                             |        |                     ________    ___________________|_          |  |
 |  |         [_/dev/an-a01n01/swap_]--[_<swap>_]                    |        |                                             |        |                    [_<swap>_]--[_/dev/an-a01n02/swap_]         |  |
 |  |          _|___________________    ___                          |        |                                             |        |                          ___    ___________________|_          |  |
 |  |         [_/dev/an-a01n01/root_]--[_/_]                         |        |                                             |        |                         [_/_]--[_/dev/an-a01n02/root_]         |  |
 |  |           |___________________________                         |        |                                             |        |                         ___________________________|_          |  |
 |  |         [_/dev/an-a01n01/srv01-psql_0_]                        |        |                                             |        |                        [_/dev/an-a01n01/srv01-psql_0_]         |  |
 |  |           |  |   _______________________________               |        |                                             |        |               _______________________________   |  |           |  |
 |  +-----------|--+--[_/dev/drbd/by-res/srv01-psql/0_]--------------+        |                                             |        +--------------[_/dev/drbd/by-res/srv01-psql/0_]--+--|-----------+  |
 |  |          _|_______________________________                     |        |                                             |        |                     _______________________________|_          |  |
 |  |         [_/dev/an-a01n01_vg0/srv02-app1_0_]                    |        |                                             |        |                    [_/dev/an-a01n01_vg0/srv02-app1_0_]         |  |
 |  |           |  |   _______________________________               |        |                                             |        |               _______________________________   |  |           |  |
 |  +-----------|--+--[_/dev/drbd/by-res/srv02-app1/0_]--------------+        |                                             |        +--------------[_/dev/drbd/by-res/srv02-app1/0_]--+--|-----------+  |
 |  |          _|_____________________________                       |        |                                             |        |                     _______________________________|_          |  |
 |  |         [_/dev/an-a01n01_vg0/srv03-app2_]                      |        |                                             |        |                    [_/dev/an-a01n01_vg0/srv03-app2_0_]         |  |
 |  |           |  |   _______________________________               |        |                                             |        |               _______________________________   |  |           |  |
 |  +-----------|--+--[_/dev/drbd/by-res/srv03-app2/0_]--------------+        |                                             |        +--------------[_/dev/drbd/by-res/srv03-app2/0_]--+--|-----------+  |
 |  |          _|_____________________________                       |        |                                             |        |                     _______________________________|_          |  |
 |  |         [_/dev/an-a01n01_vg0/srv04-app3_]                      |        |                                             |        |                    [_/dev/an-a01n01_vg0/srv04-app3_0_]         |  |
 |  |           |  |   _______________________________               |        |                                             |        |               _______________________________   |  |           |  |
 |  +-----------|--+--[_/dev/drbd/by-res/srv04-app3/0_]--------------+        |                                             |        +--------------[_/dev/drbd/by-res/srv04-app3/0_]--+--|-----------+  |
 |  |          _|________________________________                    |        |                                             |        |                    ________________________________|_          |  |
 |  |         [_/dev/an-a01n01_vg0/srv05-admin_0_]                   |        |                                             |        |                   [_/dev/an-a01n01_vg0/srv05-admin_0_]         |  |
 |  |              |   ________________________________              |        |                                             |        |              ________________________________   |              |  |
 |  \--------------+--[_/dev/drbd/by-res/srv05-admin/0_]-------------/        |                                             |        \-------------[_/dev/drbd/by-res/srv05-admin/0_]--+--------------/  |
 |                                                                            |          _________________________          |                                                                            |
 |                                                                            |         | an-switch01             |         |                                                                            |
 |                                                          __________________|         |___        BCN       ____|         |__________________                                                          |
 |                                                         | IPMI            =----------=_03_]    VID 100    [_04_=---------=             IPMI |                                                         |
 |                                    _________    _____   |     10.201.11.1 ||         |_________________________|         || 10.201.11.2     |   _____    _________                                    |
 |                                   {_sensors_}--[_BMC_]--|_________________||         | an-switch02             |         ||_________________|--[_BMC_]--{_sensors_}                                   |
 |                                                                            |         |           BCN           |         |                                                                            |
 |                                                             ______ ______  |         |         VID 100         |         |  ______ ______                                                             |
 |                                                            | PSU1 | PSU2 | |         |____   ____   ____   ____|         | | PSU1 | PSU2 |                                                            |
 |____________________________________________________________|______|______|_|         |_03_]_[_07_]_[_08_]_[_04_|         |_|______|______|____________________________________________________________|
                                                                    || ||                  |      |      |       |                  || ||                                                                 
                                        /---------------------------||-||------------------|------/      \-------|------------------||-||---------------------------\                                     
                                        |                           || ||                  |                     |                  || ||                           |                                     
                         _______________|___                        || ||        __________|________     ________|__________        || ||                        ___|_______________                      
              _______   | an-ups01          |                       || ||       | an-pdu01          |   |          an-pdu02 |       || ||                       |          an-ups02 |   _______           
             {_Mains_}==|        10.201.3.1 |=======================||=||=======|        10.201.2.1 |   | 10.201.2.2        |=======||=||=======================| 10.201.3.2        |=={_Mains_}          
                        |___________________|                       || ||       |___________________|   |___________________|       || ||                       |___________________|                     
                                                                    || ||                      || ||     || ||                      || ||                                                                 
                                                                    || \\========[ Port 1 ]====// ||     || \\====[ Port 2 ]========// ||                                                                 
                                                                    \\===========[ Port 1 ]=======||=====//                            ||                                                                 
                                                                                                  \\==============[ Port 2 ]===========//                                                                 
                                                                                                                                                                                                           
===========================================================================================================================================================================================================
                                                                                                                                                                                                           
-=] an-anvil-02 (Node 2)                                                                                                                                                                                   
  ____________________________________________________________________________                                               ____________________________________________________________________________ 
 | an-a02n01.alteeve.com                                                      |                                             |                                                      an-a02n02.alteeve.com |
 |                                    Network:               _________________|                                             |_________________               Network:                                    |
 |                                    _________________     | ifn1_bond1      |          _________________________          |       ifn_bond1 |     _________________                                    |
 |      Servers:                     |   ifn_bridge1   |----| <ip on bridge>  |         | an-switch01             |         |  <ip on bridge> |----|   ifn_bridge1   |                     Servers:      |
 |      __________________________   |   10.255.12.1   |    |    _____________|         |____ Internet-Facing ____|         |____________     |    |   10.255.12.2   |  ............................     |
 |     | [ srv06-sql-qa ]         |  |_________________|    |   | ifn1_link1  =---------=_01_]    Network    [_02_=---------=  ifn_link1 |    |    |_________________|  :         [ srv06-sql-qa ] :     |
 |     |        __________________|    | |                  |   |____________||         |_________________________|         ||___________|    |                  : :    :__________________        :     |
 |     |       | NIC 1            =----/ |                  |                 |         | an-switch02             |         |                 |                  : -----=            NIC 1 |       :     |
 |     |       |      10.255.1.6 ||      |                  |    _____________|         |____                 ____|         |____________     |                  :      :| 10.255.1.6      |       :     |
 |     |       |_________________||      |                  |   | ifn1_link2  =---------=_01_]  VLAN ID 300  [_02_=---------=  ifn_link2 |    |                  :      :|_________________|       :     |
 |     |                          |      |                  |   |____________||         |_________________________|         ||___________|    |                  :      :                          :     |
 |     |   ____                   |      |                  |_________________|                                             |_________________|                  :      :                   ____   :     |
 |  /--=--[_c:_]                  |      |                                    |                                             |                                    :      :                  [_c:_]--=--\  |
 |  |  |__________________________|      |                   _________________|                                             |_________________                   :      :..........................:  |  |
 |  |                                    |                  | sn1_bond1       |                                             |       sn1_bond1 |                  :                                    |  |
 |  |    __________________________      |                  |     10.121.12.1 |                                             | 10.121.12.2     |                  :     ............................   |  |
 |  |   | [ srv07-app-qa ]         |     |                  |     ____________|                                             |____________     |                  :     :         [ srv07-app-qa ] :   |  |
 |  |   |        __________________|     |                  |    | sn1_link1  =---------------------------------------------=  sn1_link1 |    |                  :     :__________________        :   |  |
 |  |   |       | NIC 1            =-----/                  |    |___________||                                             ||___________|    |                  ------=            NIC 1 |       :   |  |
 |  |   |       |      10.255.1.7 ||                        |                 |                                             |                 |                        :| 10.255.1.7      |       :   |  |
 |  |   |       |_________________||                        |     ____________|                                             |____________     |                        :|_________________|       :   |  |
 |  |   |                          |                        |    | sn1_link2  =---------------------------------------------=  sn1_link2 |    |                        :                          :   |  |
 |  |   |   ____                   |              /---------|    |___________||                                             ||___________|    |---------\              :                   ____   :   |  |
 |  +---=--[_c:_]                  |              |         |_________________|                                             |_________________|         |              :                  [_c:_]--=---+  |
 |  |   |__________________________|              |                           |                                             |                           |              :..........................:   |  |
 |  |                                             |          _________________|                                             |_________________          |                                             |  |
 |  |                                             |         | mn1_bond1       |                                             |       mn1_bond1 |         |                                             |  |
 |  |                                             |         |     10.199.12.1 |                                             | 10.199.12.2     |         |                                             |  |
 |  |                                             |         |     ____________|                                             |____________     |         |                                             |  |
 |  |                                             |         |    | mn1_link1  =---------------------------------------------=  mn1_link1 |    |         |                                             |  |
 |  |                                             |         |    |___________||                                             ||___________|    |         |                                             |  |
 |  |                                             |         |                 |                                             |                 |         |                                             |  |
 |  |                                             |         |     ____________|                                             |____________     |         |                                             |  |
 |  |                                             |         |    | mn1_link2  =---------------------------------------------=  mn1_link2 |    |         |                                             |  |
 |  |                                             |         |    |___________||                                             ||___________|    |         |                                             |  |
 |  |                                             |         |_________________|                                             |_________________|         |                                             |  |
 |  |                                             |                           |                                             |                           |                                             |  |
 |  |                                             |          _________________|                                             |_________________          |                                             |  |
 |  |                                             |         | bcn1_bond1      |          _________________________          |      bcn1_bond1 |         |                                             |  |
 |  |                                             |         |     10.201.12.1 |         | an-switch01             |         | 10.201.12.2     |         |                                             |  |
 |  |                                             |         |    _____________|         |____  Back-Channel   ____|         |_____________    |         |                                             |  |
 |  |                                             |         |   | bcn1_link1  =---------=_13_]    Network    [_14_=---------=  bcn1_link1 |   |         |                                             |  |
 |  |                                             |         |   |____________||         |_________________________|         ||____________|   |         |                                             |  |
 |  |                                             |         |                 |         | an-switch02             |         |                 |         |                                             |  |
 |  |                                             |         |    _____________|         |____                 ____|         |_____________    |         |                                             |  |
 |  |                                             |         |   | bcn1 link2  =---------=_13_]  VLAN ID 100  [_14_=---------=  bcn1_link2 |   |         |                                             |  |
 |  |                                             |         |   |____________||         |_________________________|         ||____________|   |         |                                             |  |
 |  |                                             |         |_________________|                                             |_________________|         |                                             |  |
 |  |                                             |            |              |                                             |              |            |                                             |  |
 |  |                                             |     _______|_________     |                                             |     _________|_______     |                                             |  |
 |  |                                             |    |  bcn1_bridge1   |    |                                             |    |  bcn1_bridge1   |    |                                             |  |
 |  |                                             |    |   10.201.10.1   |    |                                             |    |   10.201.10.2   |    |                                             |  |
 |  |                                             |    |_________________|    |                                             |    |_________________|    |                                             |  |
 |  |                                             |                           |                                             |                           |                                             |  |
 |  |                                             |                           |                                             |                           |                                             |  |
 |  |     Storage:                                |                           |                                             |                           |                                Storage:     |  |
 |  |     __________                              |                           |                                             |                           |                              __________     |  |
 |  |    [_/dev/sda_]                             |                           |                                             |                           |                             [_/dev/sda_]    |  |
 |  |      |   ___________         ___________    |                           |                                             |                           |    ___________         ___________   |      |  |
 |  |      +--[_/dev/sda1_]-------[_/boot/efi_]   |                           |                                             |                           |   [_/boot/efi_]-------[_/dev/sda1_]--+      |  |
 |  |      |   ___________         _______        |                           |                                             |                           |        _______         ___________   |      |  |
 |  |      +--[_/dev/sda2_]-------[_/boot_]       |                           |                                             |                           |       [_/boot_]-------[_/dev/sda1_]--+      |  |
 |  |      |   ___________         ___________    |                           |                                             |                           |    ___________         ___________   |      |  |
 |  |      +--[_/dev/sda3_]-------[_<lvm:_pv>_]   |                           |                                             |                           |   [_<lvm:_pv>_]-------[_/dev/sda1_]--+      |  |
 |  |            ___________________|             \------------------\        |                                             |        /------------------/             |___________________            |  |
 |  |          _|___________________                                 |        |                                             |        |                                 ___________________|_          |  |
 |  |         [_<lvm:_vg_an-a02n01>_]                                |        |                                             |        |                                [_<lvm:_vg_an-a02n01>_]         |  |
 |  |          _|___________________    ________                     |        |                                             |        |                     ________    ___________________|_          |  |
 |  |         [_/dev/an-a02n01/swap_]--[_<swap>_]                    |        |                                             |        |                    [_<swap>_]--[_/dev/an-a02n02/swap_]         |  |
 |  |          _|___________________    ___                          |        |                                             |        |                          ___    ___________________|_          |  |
 |  |         [_/dev/an-a02n01/root_]--[_/_]                         |        |                                             |        |                         [_/_]--[_/dev/an-a02n02/root_]         |  |
 |  |           |_____________________________                       |        |                                             |        |                       _____________________________|_          |  |
 |  |         [_/dev/an-a02n01/srv06-sql-qa_0_]                      |        |                                             |        |                      [_/dev/an-a02n01/srv06-sql-qa_0_]         |  |
 |  |           |  |   _________________________________             |        |                                             |        |             _________________________________   |  |           |  |
 |  +-----------|--+--[_/dev/drbd/by-res/srv06-sql-qa/0_]------------+        |                                             |        +------------[_/dev/drbd/by-res/srv06-sql-qa/0_]--+--|-----------+  |
 |  |          _|_________________________________                   |        |                                             |        |                   _________________________________|_          |  |
 |  |         [_/dev/an-a02n01_vg0/srv07-app-qa_0_]                  |        |                                             |        |                  [_/dev/an-a02n01_vg0/srv07-app-qa_0_]         |  |
 |  |              |   _________________________________             |        |                                             |        |             _________________________________   |              |  |
 |  \--------------+--[_/dev/drbd/by-res/srv07-app-qa/0_]------------/        |                                             |        \------------[_/dev/drbd/by-res/srv07-app-qa/0_]--+--------------/  |
 |                                                                            |          _________________________          |                                                                            |
 |                                                                            |         | an-switch01             |         |                                                                            |
 |                                                          __________________|         |___        BCN       ____|         |__________________                                                          |
 |                                                         | IPMI            =----------=_03_]    VID 100    [_04_=---------=             IPMI |                                                         |
 |                                    _________    _____   |     10.201.13.1 ||         |_________________________|         || 10.201.13.2     |   _____    _________                                    |
 |                                   {_sensors_}--[_BMC_]--|_________________||         | an-switch02             |         ||_________________|--[_BMC_]--{_sensors_}                                   |
 |                                                                            |         |           BCN           |         |                                                                            |
 |                                                             ______ ______  |         |         VID 100         |         |  ______ ______                                                             |
 |                                                            | PSU1 | PSU2 | |         |____   ____   ____   ____|         | | PSU1 | PSU2 |                                                            |
 |____________________________________________________________|______|______|_|         |_03_]_[_07_]_[_08_]_[_04_|         |_|______|______|____________________________________________________________|
                                                                    || ||                  |      |      |       |                  || ||                                                                 
                                        /---------------------------||-||------------------|------/      \-------|------------------||-||---------------------------\                                     
                                        |                           || ||                  |                     |                  || ||                           |                                     
                         _______________|___                        || ||        __________|________     ________|__________        || ||                        ___|_______________                      
              _______   | an-ups01          |                       || ||       | an-pdu01          |   |          an-pdu02 |       || ||                       |          an-ups02 |   _______           
             {_Mains_}==|        10.201.3.1 |=======================||=||=======|        10.201.2.1 |   | 10.201.2.2        |=======||=||=======================| 10.201.3.2        |=={_Mains_}          
                        |___________________|                       || ||       |___________________|   |___________________|       || ||                       |___________________|                     
                                                                    || ||                      || ||     || ||                      || ||                                                                 
                                                                    || \\========[ Port 3 ]====// ||     || \\====[ Port 4 ]========// ||                                                                 
                                                                    \\===========[ Port 3 ]=======||=====//                            ||                                                                 
                                                                                                  \\==============[ Port 4 ]===========//                                                                 
                                                                                                                                                                                                          
</syntaxhighlight>
|}

[[Image:oh_my.png|thumb|center|350px|Screenshot from [http://time.com/3067261/george-takei-oh-my/ CNN interview] with George Takei. Oh my!]]

In the diagram, we see the two dashboards, two nodes, the hosted servers within the subnodes and the foundation pack equipment. The servers with dotted lines shows that the server is ready to run there, but isn't at the time the diagram was made.

The dashboards and the foundation pack equipment has been discussed already, as has the networking. Below is the switch port assignment for a very typical foundation pack hosting two node pairs. Above, the two switches are split into four groups, just to try and make the diagram marginally sane. In truth, there are just the two switches. You can correlate the ports in the diagram above to the map below.

== Ethernet Switch Port Assignment ==

The ethernet switch port assignment plays a central role in the consistency and security of the ''Anvil!'' platform. The port assignment below is the same across all installed ''Anvil!'' systems. The make, model and performance may change dramatically from install to install, but not the layout.

{{note|1=Here we show the [[Storage Network]] using the switch. If you plan to run the SN back to back between the subnodes, the SN VLAN can be removed. The [[Migration Network]] is never run through a switch, and is always back to back.}}

{|class="wikitable"
!an-switch01
!colspan="15"|&nbsp;
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #f7f7f7; text-align: center;" colspan="2"|Stack
|class="null"|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #FFFFFF; text-align: center;" colspan="5"|Back-Channel Network]]
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #99FF66; text-align: center;" colspan="3"|Storage Network
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #2f2f2f; color: #cfcfcf; text-align: center;" colspan="5"|Internet-Facing Network
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X1 (1/2/1)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X3 (1/2/3)
|rowspan="4"|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|1 (1/1/1)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|3 (1/1/3)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|5 (1/1/5)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|7 (1/1/7)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|9 (1/1/9)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|11 (1/1/11)
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|13 (1/1/13)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|15 (1/1/15)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|17 (1/1/17)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|19 (1/1/19)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|21 (1/1/21)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|23 (1/1/23)
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|To an-switch02<br />X3
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|To an-switch02<br />X1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a01n01<br />BCN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a01n01<br />IPMI / Fencing
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a02n01<br />BCN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a02n01<br />IPMI / Fencing
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-striker01<br />BCN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a01n01<br />SN - Link 1
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a02n01<br />SN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a01n01<br />IFN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a02n01<br />IFN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-striker01<br />IFN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a01n02<br />BCN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a01n02<br />IPMI / Fencing
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a02n02<br />BCN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a02n02<br />BCN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a01n02<br />SN - Link 1
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a02n02<br />SN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a01n02<br />IFN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a02n02<br />IFN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|Uplink 1
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X2 (1/2/2)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X4 (1/2/4)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|2 (1/1/2)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|4 (1/1/4)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|6 (1/1/6)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|8 (1/1/8)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|10 (1/1/10)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|12 (1/1/12)
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|14 (1/1/14)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|16 (1/1/16)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|18 (1/1/18)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|20 (1/1/20)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|22 (1/1/22)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|24 (1/1/24)
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #f7f7f7; text-align: center;" colspan="2"|Unused
|class="null"|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #FFFFFF; text-align: center;" colspan="5"|VID 100
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #99FF66; text-align: center;" colspan="3"|VID 200
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #2f2f2f; color: #cfcfcf; text-align: center;" colspan="5"|VID 300
|-
|colspan="16"|&nbsp;
|-
!an-switch02
!colspan="15"|&nbsp;
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #f7f7f7; text-align: center;" colspan="2"|Stack
|class="null"|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #FFFFFF; text-align: center;" colspan="5"|Back-Channel Network]]
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #99FF66; text-align: center;" colspan="3"|Storage Network
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #2f2f2f; color: #cfcfcf; text-align: center;" colspan="5"|Internet-Facing Network
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X1 (2/2/1)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X3 (2/2/3)
|rowspan="4"|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|1 (2/1/1)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|3 (2/1/3)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|5 (2/1/5)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|7 (2/1/7)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|9 (2/1/9)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|11 (2/1/11)
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|13 (2/1/13)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|15 (2/1/15)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|17 (2/1/17)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|19 (2/1/19)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|21 (2/1/21)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|23 (2/1/23)
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|To an-switch01<br />X3
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|To an-switch01<br />X1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a01n01<br />BCN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-pdu01
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a02n01<br />BCN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-ups01
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-striker02<br />BCN - Link 1
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a01n01<br />SN - Link 2
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a02n01<br />SN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a01n01<br />IFN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a02n01<br />IFN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-striker01<br />IFN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #1f1f1f; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a01n02<br />BCN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-pdu02
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-a02n02<br />BCN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|an-ups02
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #2f2f2f; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a01n02<br />SN - Link 2
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #005C00; text-align: center;"|an-a02n02<br />SN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a01n02<br />IFN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|an-a02n02<br />IFN - Link 2
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|--
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #000000; text-align: center;"|Uplink 2
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X2 (2/2/2)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|X4 (2/2/4)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|2 (2/1/2)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|4 (2/1/4)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|6 (2/1/6)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|8 (2/1/8)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|10 (2/1/10)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|12 (2/1/12)
|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|14 (2/1/14)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|16 (2/1/16)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|18 (2/1/18)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|20 (2/1/20)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|22 (2/1/22)
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; white-space: nowrap; color: #5f5f5f; text-align: center;"|24 (2/1/24)
|-
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #f7f7f7; text-align: center;" colspan="2"|Unused
|class="null"|&nbsp;
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #FFFFFF; text-align: center;" colspan="5"|VID 100
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #99FF66; text-align: center;" colspan="3"|VID 200
|style="font-family: 'Dejavu Sans Mono', 'Courier 10 pitch', monospace, Courier; font-size: 12px; background-color: #2f2f2f; color: #cfcfcf; text-align: center;" colspan="5"|VID 300
|}

== Subnets - Giving IP Addresses Meaning ==

{{note|1=This example below is specific to this sample install. For a more complete review of Networking in an Anvil! Cluster, please see: [[Anvil! Networking]].}}

To further build on consistency, the ''Anvil!'' platform uses, everywhere possible, the same subnets for the [[BCN]] and [[SN]] across all installs. The [[IFN]] is often outside of our control, so there is little we can do to enforce consistency there. That doesn't matter much though, because as stated earlier, all inter-''Anvil!'' communication occurs on the [[BCN]], save for the storage replication.

We use the subnets to both identify role and IP.

* The first two octets define the subnet.
* The third octet defines the device role.
* The fourth octet is identifies the device.

Lets look at this closer.

{|class="wikitable"
!Purpose
!Subnet
!Notes
|-
|[[Back-Channel Network]] 1
|class="code"|10.201.x.y/16
|
* The third octet describes the role of the device;
** <span class="code">1</span>: Ethernet switches.
** <span class="code">2</span>: Switched PDUs.
** <span class="code">3</span>: UPSes.
** <span class="code">4</span>: Striker dashboards.
** ''Anvil!'' pairs are identified by multiplying their sequence number by ten. So, the first ''Anvil!'' would use '<span class="code">10</span>', the second would use '<span class="code">20</span>' and so on.
** The IPMI interfaces use the ''Anvil!'' pair's third octet, plus 1. So the IPMI interfaces for the first ''Anvil!'' would use '<span class="code">11</span>', the second would use '<span class="code">21</span>' and so on.
Each node will use <span class="code">10.201.50.x</span> where <span class="code">x</span> matches the node ID.
* The fourth octet is used to identify the sequence number of the device. 
** The first ethernet switch, PDU, UPS and striker dashboard will end in '<span class="code">1</span>'. The second of each would end in '<span class="code">2</span>' and so on.
|-
|style="white-space: nowrap;"|[[Internet-Facing Network]] 1
|class="code"|''variable''
|
In most all cases, the IFN exists before you arrive, and you have no influence over the available range of IP addresses. The existing network administrator will need to provide a static IP address for each subnode and dashboard on the network.

{{note|1=The ''Anvil!'' itself avoids using the IFN as much as possible. Administrators will use it to gain access to the ''Anvil!'' subnodes and dashboards, and the admin might use the IFN to do work like download updates. Generally, we ignore the [[IFN]] because it is meant to link the client's servers to their network, and not something we concern ourselves with.}}
|-
|[[Storage Network]] 1
|class="code"|10.101.x.y/16
|
* Only nodes use this network. The third and fourth octets matches the third and fourth octet from the BCN.
|-
|[[Migration Network]] 1
|class="code"|10.199.x.y/16
|
* Only nodes use this network. The third and fourth octets matches the third and fourth octet from the BCN.
|}

== Hostnames - Cattle ==

Naming machines in an ''Anvil!'' definitely falls into the 'cattle' side of "[http://www.slideshare.net/randybias/the-cloud-revolution-cyber-press-forum-philippines Pets vs. Cattle]" (ref: slide #29). The host names used in ''Anvil!'' systems serve a purpose.

All host names in ''Anvil!'' systems look like this;

* <span class="code"><prefix>-<purpose><sequence>.<domain_or_suffix></span>

#. The prefix is used to identify a group on ''Anvil!'' systems. Generally it is a short string used to identify a client, location or division. 
#. The purpose indicates the role the device plays.
##. Nodes use the format '<span class="code">aXXnYY</span>', where '<span class="code">XX</span>' is the ''Anvil!'' sequence number and '<span class="code">YY</span>' is the node number.
##. All other devices have a descriptive name; '<span class="code">switch</span>' for ethernet switches, '<span class="code">pdu</span>' for switched PDUs, '<span class="code">ups</span>' for UPSes and '<span class="code">striker</span>' for dashboards.
#. The sequence number is a simple integer, zero-padded. 

As an example, Alteeve's general use prefix is simply '<span class="code">an</span>' (Alteeve's Niche! Inc. is our full name) and our domain is '<span class="code">alteeve.ca</span>'. So with this, the first UPS on the first foundation pack would be '<span class="code">an-ups01.alteeve.ca</span>'. The first UPS in the second foundation pack would be '<span class="code">an-ups03.alteeve.ca</span>' and so on. The second node of the first node pair would be '<span class="code">an-a01n02.alteeve.ca</span>' (anvil 01, node 02). The first node of the fifth pair would be '<span class="code">an-a05n01.alteeve.ca</span>'.

To simplify things further, the ''Anvil!'' uses the local '<span class="code">/etc/hosts</span>' file and a set of suffixes to denote specific networks on multi-homed devices, like nodes and dashboards. They are;
* '<span class="code"><short_hostname>.bcn</span>' - Names that resolve to the IP of the target on the Back-Channel Network.
* '<span class="code"><short_hostname>.sn</span>' - Names that resolve to the IP of the target on the Storage Network.
* '<span class="code"><short_hostname>.ifn</span>' - Names that resolve to the IP of the target on the Internet-Facing Network.
* '<span class="code"><short_hostname>.ipmi</span>' - Names that resolve to a node's IPMI [[BMC]], used when [[fencing]] a target node.

Pretty simple, but again, consistency is a key part of availability.

== Putting it together ==

This will be more clear with a diagram of a typical ''Anvil!'' IP load-out. In this example, we will use the IFN subnet '<span class="code">10.255.0.0/16</span>' and use the third and fourth octets as we do in the BCN and SN. Of course, this is an ideal situation and rarely possible.

{|class="wikitable"
!Device
!Back-Channel Network 1 IP
!Internet-Facing Network 1 IP
!Storage Network 1 IP
!Migration Network 1 IP
|-
|class="code"|an-switch01.alteeve.ca
|class="code"|10.201.1.1
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-switch02.alteeve.ca
|class="code"|10.201.1.2 &diams;
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-pdu01.alteeve.ca
|class="code"|10.201.2.1
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-pdu02.alteeve.ca
|class="code"|10.201.2.2
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-ups01.alteeve.ca
|class="code"|10.201.3.1
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-ups02.alteeve.ca
|class="code"|10.201.3.2
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-striker01.alteeve.ca
|class="code"|10.201.4.1
|class="code"|10.255.4.1
|class="code"|--
|class="code"|--
|-
|class="code"|an-striker02.alteeve.ca
|class="code"|10.201.4.2
|class="code"|10.255.4.2
|class="code"|--
|class="code"|--
|-
|class="code"|an-a01n01.alteeve.ca
|class="code"|10.201.10.1
|class="code"|10.255.10.1
|class="code"|10.101.10.1
|class="code"|10.199.10.1
|-
|class="code"|an-a01n01.ipmi
|class="code"|10.201.11.1
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-a01n02.alteeve.ca
|class="code"|10.201.10.2
|class="code"|10.255.10.2
|class="code"|10.101.10.2
|class="code"|10.199.10.2
|-
|class="code"|an-a01n02.ipmi
|class="code"|10.201.11.2
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-a02n01.alteeve.ca
|class="code"|10.201.12.1
|class="code"|10.255.12.1
|class="code"|10.101.12.1
|class="code"|10.199.12.1
|-
|class="code"|an-a02n01.ipmi
|class="code"|10.201.13.1
|class="code"|--
|class="code"|--
|class="code"|--
|-
|class="code"|an-a02n02.alteeve.ca
|class="code"|10.201.12.2
|class="code"|10.255.12.2
|class="code"|10.101.12.2
|class="code"|10.199.12.2
|-
|class="code"|an-a02n02.ipmi
|class="code"|10.201.13.2
|class="code"|--
|class="code"|--
|class="code"|--
|}
<span style="font-size: 75%;">&diams; Rarely used as most installs use stacked switches.</span>

That's it. It might take a bit to explain at first, but once you see the pattern, it is pretty easy to follow. With this, you can walk up to any ''Anvil!'' and immediately know what everything is, how to access it and what the parts do.

= ScanCore - The "Intelligence" In 'Intelligent Availability' =

Thus far, what we have talked about describes a really good High Availability setup. We've covered all the ways that the system will be protected from a failure, but not all of these backup systems are fault tolerant.

More specifically, the failure of a node will cause any servers hosted on it to lose power, requiring a fresh boot to recover them. This is generally completed in 30~90 seconds, but the interruption can be significant.

ScanCore is Alteeve's answer to try and solve this problem. It runs on all nodes and dashboards, and collectively works together to try to predict failures before they happen, and migrate servers to the healthier peer before the node can fail. We'll explore this more in a moment.

''But wait, there's more!''

ScanCore also determines when threat models change. Consider this;

Under normal operation, the most likely cause of interruption is component failure. So during normal operation, the best thing to do is to maintain full redundancy. 

* Want more detail? Please see the [[ScanCore]] page.

== Load Shedding ==

What about when the power fails?

What about when HVAC fails and the room starts heating up?

In these scenarios, the greatest threat is no longer component failure. In both cases, though using different metrics, ScanCore will decide that the best thing to do is to shed load. This will reduce the draw on the batteries in the first scenario, or reduce the thermal output in the second case.

The goal in these cases is to buy time. Shedding load doesn't fix the underlying problem, but it gives you more time to hopefully fix the underlying problem in time to avoid a full shut down. If, however, the problem lasts for too long, then ScanCore will decide when it can't wait any longer and will gracefully shutdown hosted servers and then power off the remaining node.

What then?

The Striker dashboards are considered sacrificial lambs. That is to say, we're willing to risk them in order to protect the node pairs. So they are allowed to keep running until the hardware forcibly shuts down (either because the UPSes completely drain or the temperature goes so high that hardware safety system kick in). If this happens, the nodes are configured to boot on power restore, after which they will again start monitoring the general environmental health.

When power is restored and the UPSes have charged to a safe level, or when the temperature returns to safe levels, ScanCore on the dashboards will boot the nodes back up. If power was restored, or temperatures drop to safe levels, while in a load shed state, ScanCore will wait until minimum safe levels (charge or temperature) return before restoring full redundancy.

== Failure Prediction ==

Sometimes, there is no warning. A server will fail dramatically and even in the post-mortem, no signs of the impending failure could be found.

An example of this was a rare, faulty voltage regulators on some node mainboards a tier-1 vendor was providing us. Up to a year or two into production, the voltage regulator would fail catastrophically. There were no voltage irregularities before hand that we could detect, and so, the failure took down any hosted servers.

In a scenario like this, Intelligent Availability provided no comfort.

Thankfully, this is the exception, not the rule!

There are many ways to predict a pending failure. Lets look at storage as an example, as this is something most people have experience with.

[[SAS]] hard drives and proper hardware [[RAID]] controllers will report a lot of useful information about each drive. Temperature, media error counters and so on. Drives almost always start showing signs of failure long before they actually die. Here's how ScanCore deal with this.

#. A drive starts to throw error counters. It's not unexpected for drives to throw periodic errors and so, this alone isn't enough for ScanCore to take action on its own. It will notify the admin, however.
#. If the error counters start to climb, the admin will contact their hardware vendor and request a pre-failure replacement. 
#. If the drive is not replaced in time, the drive will fail and the RAID array will enter a degraded state. Now ScanCore acts!

In this scenario, the peer node is assumed to have no faults. So ScanCore says "ok, sure, the current host is alive, but it is degraded. So lets be safe and live migrate the servers over to the peer. Now, if a second drive were to fail and the node go offline, there would be no interruption because the servers have already vacated the old host!

== All Failures Are Not Created Equal ==

In the example above, we looked at how ScanCore would deal with a faulty drive. Lets step back now, and start a new scenario where both nodes are healthy.

In this scenario, we'll say that node 1 is hosting the servers. So with the stage set, lets look at a case where both nodes develop problems.

#. Node 1 loses a cooling fan. The system can handle this without overheating, which is confirmed by no thermal sensors registering problems.
#. ScanCore says "well, node 2 is perfectly fine, so lets move the servers over" and then proceeds to do just that.
#. Within a few minutes, all servers have moved over to Node 2 without any user noticing anything and without any input from an admin.
#. Later, Node 2 loses a hard drive and its RAID array degrades.
#. ScanCore looks at the problems on both nodes and decides, "ya, you know what? A failed fan doesn't look so bad anymore" and live-migrates the servers back to Node 1.
#. A replacement hard drive is found and Node 2's array starts rebuilding. Nothing yet happens because the array remains degraded until the rebuild completes.
#. The rebuild completes, and once again, Node 2 is perfectly fine. For some reason though, Node 1 is still without a fan. So ScanCore says "Welp, Node 2 is healthier" and live-migrates the servers back to Node 2.
#. Finally, someone finds an unobtanium fan and fixes Node 1. ScanCore now sees that both nodes are healthy, but it decides to leave the servers on Node 2.

Why?

It is important to understand that in an ''Anvil!'' system, nodes don't have a "primary" or "backup" role. These roles are determined purely by where the servers happen to be at any given time. So in the scenario above, when Node 1 was fixed, there was no reason to do anything. Node 2 is just as good a host as Node 1, so ScanCore decides that Node 1 is now the backup and leaves things as they are.

Live-migrations of servers are considered very safe, but not perfectly safe. 

Remember; '''Everything can fail'''.

This is another example of how accepting that everything can fail is a strength. It's natural to want to move the servers back to node 1. Why not? That's where they normally are and normal is good. This way of thinking saves us from that temptation.

= Bonus! Self-Destruct Sequence, Engage! =

We were challenged by a client to produce an ''Anvil!'' system that could render data forensically unrecoverable in a maximum of 300 seconds.

{|
|[[Image:t2_thunbs-up.png|thumb|center|528px|"Terminator 2: Judgement Day", TriStar Pictures.]]
|}

''I won't lie, this was the most fun I've had in my 20+ year professional career.''

For the record, no, [https://en.wikipedia.org/wiki/Thermite thermite] on the hard drives was not good enough. It make for great movies, but anything that is physical couldn't be sure to destroy the data thoroughly enough.

''Wait, backup... Why would you want to destroy a platform designed to protect your systems?''

For privacy reasons, we can't speak to details. Generally though, the threat model was one where an attacker with effectively limitless resources would gain physical control of the ''Anvil!'' system. This attacker could spend considerable time and money, with access to top level experts, to try and recover the data. 

The ''Anvil!'' system is ideal for use in air-gapped environments. It can be extremely rugged and take a significant beating. It can be deployed in places where the user may not care if a server goes down for a few hours, but are located in a place where replacement parts can't be ordered. In these cases, the system must be able to live, even if it is limping, for weeks or months until repairs can be made.

This capability means that ''Anvil!'' systems can be found in potentially hostile locations. It is feasible that people will have to abandon their equipment and flee for safety. In such a scenario, the value of the hardware is hardly a concern. Securing the data, however, is still critical.

== Lock-Out ==

{{note|1=This feature is available in M2, but pending addition to M3. If you have a use-case for this feature, please [[contact us]] and we'll try push this feature up the ToDo list.}}

In this particular case, the client's equipment would be deployed in a secure location with extremely limited human access. If a node rebooted, it was not possible for a user to enter a boot-time passphrase. 

With all this in mind, here is what we did.

* We used [https://en.wikipedia.org/wiki/Hardware-based_full_disk_encryption Self-Encrypting Drives] that used hardware [https://en.wikipedia.org/wiki/Advanced_Encryption_Standard AES-256] encryption.
* We used Broadcom (formerly LSI) controllers with their [https://www.broadcom.com/products/storage/raid-controllers/megaraid-safestore-software SafeStore] option.

Details of how we configure this can be [[Encrypted Arrays with LSI SafeStore|found here]].

Given the requirement for unattended booting, we had to store the passphrase in the RAID controller. The up-side to this is that we were able to develop an intermediary "lock-out" option, to sit between normal operation and full-on self-destruct.

If the client enters a situation where they think they might lose control over the system, they have the option to (remotely) [https://github.com/ClusterLabs/striker/blob/master/tools/disable_boot.sh set and require] a boot-time passphrase. Once set, the nodes can be rebooted them. This will leave them recoverable by someone who knows the passphrase.

Of course, this leaves the system vulnerable to [https://en.wikipedia.org/wiki/Rubber-hose_cryptanalysis rubber-hose decryption]. In the client's case, this was considered acceptable because they had a few secure remote access links available that would allow out-of-band access to both fire the script and restore normal access when the threat was passed. Thanks to this ability to remotely lock the system, it was deemed fairly safe as anyone who might be captured would not know the lock-out passphrase.

Later, when the threat passed, the client would use this access to unlock the storage and boot the nodes. Once booted, they could then [https://github.com/ClusterLabs/striker/blob/master/tools/reenable_boot.sh re-enable normal boot].

Obviously, this approach is not without risk.

== Killing Me Softly ==

When an evacuation was certain, local staff or remote users could access the nodes and fire the [https://github.com/ClusterLabs/striker/blob/master/tools/anvil-self-destruct self-destruct] script. Once complete, the system would have no data at all to recover. Even if the old passphrase was somehow known, the [https://en.wikipedia.org/wiki/Salt_(cryptography) salt] on the drives would be gone, so the data would be forever lost.

This is the core of the self-destruct. It doesn't make for great movies, but it is exceptionally effective.

Oh, and we can use this to destroy nearly 40 TB per node in under 30 seconds, 1/10th the maximum time.

So these are the steps that happen when the self-destruct sequence is initiated;

#. The script is invoked with a passphrase. The validity of this passphrase is checked. Note that this is weak protection, but is enough to prevent accidental invocation.
#. A [https://en.wikipedia.org/wiki/RAM_drive RAM disk] is created and certain needed tools are copied to this disk.
#. A list of virtual drives (arrays) and their disks are created.
#. The actual self-destruct sequence begins;
##. The cache policy is switched to write-through.
##. The cache is flushed.
##. The virtual drive(s) are deleted. At this point, the OS starts to throw significant errors.
##. The drives are sent the "instant secure erase" command, deleting their internal passphrase and salt. The data is now unrecoverable.
##. The old key is deleted from the controller, just to be safe.
##. The [https://en.wikipedia.org/wiki/Magic_SysRq_key#Commands magic SysRq] '<span class="code">o</span> is sent to the kernel, triggering a power off.

This leaves approximately 240 seconds for the RAM to drain, reducing the risk of any latent data being recoverable via an [https://en.wikipedia.org/wiki/Cold_boot_attack ice attack] on the RAM itself.

Once complete, the hardware is left 100% intact and can be re-used.

{{footer}}