AN!Wiki - Changes [en]

DRBD on Fedora 13

Digimer — Tue, 07 Sep 2010 20:06:55 GMT

AN! Provided DRBD RPMs:

← Older revision		Revision as of 20:06, 7 September 2010
(3 intermediate revisions not shown)
Line 46:		Line 46:

	<source lang="bash">		<source lang="bash">
-	yum install drbd.x86_64 drbd-xen.x86_64	+	yum install drbd.x86_64 drbd-xen.x86_64 drbd-utils.x86_64
	</source>		</source>

Line 53:		Line 53:
	If you are running the [[Xen_on_Fedora_13#dom0\|custom Xen dom0]], you will probably need to build the DRBD module from the source RPM. You can try the following RPMs, but be aware that if the [[dom0]] kernel gets updated, you will need to rebuild these RPMs using the following steps.		If you are running the [[Xen_on_Fedora_13#dom0\|custom Xen dom0]], you will probably need to build the DRBD module from the source RPM. You can try the following RPMs, but be aware that if the [[dom0]] kernel gets updated, you will need to rebuild these RPMs using the following steps.

-	* [http://wiki.alteeve.com/files/an-cluster/drbd-km-2.6.32.~~17_157~~.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm drbd-km-2.6.32.~~17_157~~.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm] - DRBD kernel module for myoung's 2.6.32.~~17_157~~ dom0 kernel (897 KiB)	+	== AN! Provided DRBD RPMs ==
-	* [http://wiki.alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm] - Debug info for DRBD kernel module for myoung's 2.6.32.~~17_157~~ dom0 kernel (3.2 KiB)	+
		+	* [http://wiki.alteeve.com/files/an-cluster/drbd-km-2.6.32.21_167.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm drbd-km-2.6.32.21_167.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm] - DRBD kernel module for myoung's 2.6.32.21_167 dom0 kernel (897 KiB)
		+	* [http://wiki.alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm] - Debug info for DRBD kernel module for myoung's 2.6.32.21_167 dom0 kernel (3.2 KiB)

	You can install the two above RPMs with this command:		You can install the two above RPMs with this command:

	<source lang="bash">		<source lang="bash">
-	rpm -ivh http://wiki.alteeve.com/files/an-cluster/drbd-km-2.6.32.~~17_157~~.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm http://wiki.alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm	+	rpm -ivh http://wiki.alteeve.com/files/an-cluster/drbd-km-2.6.32.21_167.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm http://wiki.alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm
	</source>		</source>
		+
		+	== Building RPMs From Source ==

	If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please continue.		If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please continue.
Line 76:		Line 80:

	<source lang="bash">		<source lang="bash">
-	rpm -ivh --force http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.17-~~157~~.xendom0.fc12.x86_64.rpm http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.17-~~157~~.xendom0.fc12.x86_64.rpm	+	rpm -ivh --force http://fedorapeople.org/~myoung/dom0/x86_64/kernel-headers-2.6.32.21-167.xendom0.fc12.x86_64.rpm http://fedorapeople.org/~myoung/dom0/x86_64/kernel-devel-2.6.32.21-167.xendom0.fc12.x86_64.rpm
	</source>		</source>

Xen on Fedora 13

Digimer — Tue, 07 Sep 2010 19:47:20 GMT

dom0:

← Older revision		Revision as of 19:47, 7 September 2010
(3 intermediate revisions not shown)
Line 50:		Line 50:

	<source lang="bash">		<source lang="bash">
-	yum install kernel-2.6.32.17-~~157~~.xendom0.fc12.x86_64	+	yum install kernel-2.6.32.21-167.xendom0.fc12.x86_64
	</source>		</source>

	The entry in grub's <span class="code">/boot/grub/menu.lst</span> won't work. You will need to edit it to look like this:		The entry in grub's <span class="code">/boot/grub/menu.lst</span> won't work. You will need to edit it to look like this:

-	'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your <span class="code">root=</span> is likely different and your <span class="code">rd_MD_UUID=</span> will definitely be different, even on the same machine across installs. Generally speaking, what follows the <span class="code">kernel /vmlinuz-2.6.32.17-~~157~~.xendom0.fc12.x86_64 ...</span> entry made by the dom0 kernel can be copied after the <span class="code">module /vmlinuz-2.6.32.17-~~157~~.xendom0.fc12.x86_64 ...</span> entry in the example below.	+	'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your <span class="code">root=</span> is likely different and your <span class="code">rd_MD_UUID=</span> will definitely be different, even on the same machine across installs. Generally speaking, what follows the <span class="code">kernel /vmlinuz-2.6.32.21-167.xendom0.fc12.x86_64 ...</span> entry made by the dom0 kernel can be copied after the <span class="code">module /vmlinuz-2.6.32.21-167.xendom0.fc12.x86_64 ...</span> entry in the example below.

	<source lang="bash">		<source lang="bash">
Line 61:		Line 61:
	</source>		</source>
	<source lang="bash">		<source lang="bash">
-	title Xen 3.4.x, Linux kernel 2.6.32.17-~~157~~.xendom0.fc12.x86_64	+	title Xen 3.4.x, Linux kernel 2.6.32.21-167.xendom0.fc12.x86_64
-	root (hd0,0)	+	root (hd0,2)
	kernel /xen.gz dom0_mem=1024M		kernel /xen.gz dom0_mem=1024M
-	module /vmlinuz-2.6.32.17-~~157~~.xendom0.fc12.x86_64 ~~ro root=/dev/mapper/vg_01-lv_root rd_MD_UUID=4f48439e:39ec22e8:a6154857:c340036f rd_LVM_LV=vg_01/lv_root rd_LVM_LV=vg_01/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US~~.~~UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us~~	+	module /vmlinuz-2.6.32.21-167.xendom0.fc12.x86_64 ...
-	module /initramfs-2.6.32.17-~~157~~.xendom0.fc12.x86_64.img	+	module /initramfs-2.6.32.21-167.xendom0.fc12.x86_64.img
	</source>		</source>

Two Node Fedora 13 Cluster

Digimer — Tue, 07 Sep 2010 18:54:27 GMT

← Older revision		Revision as of 18:54, 7 September 2010
(One intermediate revision not shown)
Line 408:		Line 408:

	==== cpg; Closed Process Group ====		==== cpg; Closed Process Group ====
		+
		+	'''Note''': Only install the following if you wish to review the <span class="code">man</span> page.

	<source lang="bash">		<source lang="bash">
Line 425:		Line 427:

	Please note that the <span class="code">openais_overview</span> man page is considered out of date at the time of this writing.		Please note that the <span class="code">openais_overview</span> man page is considered out of date at the time of this writing.
		+
		+	'''Note''': Only install the following if you wish to review the <span class="code">man</span> page.

	<source lang="bash">		<source lang="bash">
Line 432:		Line 436:
	=== Pacemaker ===		=== Pacemaker ===

-	[http://www.clusterlabs.org Pacemaker] is the cluster resource manager. It can be used to trigger scripts to control non cluster-aware applications. In this way, these non cluster-aware applications can be made more highly available.	+	[http://www.clusterlabs.org Pacemaker] is the cluster resource manager. It can be used to trigger scripts to control non cluster-aware applications. In this way, these non cluster-aware applications can be made more highly available. It is considered a replacement for <span class="code">cman</span>'s <span class="code">[[rgmanager]]</span>.
-		+
-	<~~source lang~~="~~bash~~">	+
-	~~yum install pacemaker~~	+
-	</~~source~~>	+

	== Setup Core Programs ==		== Setup Core Programs ==
Line 623:		Line 623:

	'''Full Disclosure''': Node Assassin was created by me, with much help from others, for this paper.		'''Full Disclosure''': Node Assassin was created by me, with much help from others, for this paper.
-
-	~~= DRBD =~~
-
-	~~This is the rough-in for setting up [[DRBD on Fedora 13]].~~

	= Xen =		= Xen =

	This is the rough-in for getting [[Xen on Fedora 13]].		This is the rough-in for getting [[Xen on Fedora 13]].
		+
		+	= DRBD =
		+
		+	If you plan to use [[Xen]], please make sure it is setup and running before proceeding.
		+
		+	This is the rough-in for setting up [[DRBD on Fedora 13]].

	= Thanks =		= Thanks =

DRBD on Fedora 13

Digimer — Tue, 07 Sep 2010 17:53:32 GMT

A Map of the Cluster's Storage:

← Older revision		Revision as of 17:53, 7 September 2010
Line 12:		Line 12:

	<source lang="text">		<source lang="text">
-	Node1 Node2	+	Node1 Node2
-	_____ _____ _____ _____	+	_____ _____ _____ _____
-	\| sda \| \| sdb \| \| sda \| \| sdb \|	+	\| sda \| \| sdb \| \| sda \| \| sdb \|
-	\|_____\| \|_____\| \|_____\| \|_____\|	+	\|_____\| \|_____\| \|_____\| \|_____\|
-	\|_______\| \|_______\|	+	\|_______\| \|_______\|
-	__\|__ __\|__	+	_______ ____\|___ _______ _______ ____\|___ _______
-	__\|__ __\|__ __\|__ __\|__	+	__\|__ __\|__ __\|__ __\|__ __\|__ __\|__ __\|__ __\|__
-	\| md0 \| ~~md1 \| \| md0~~ \| md1 \|	+	\| md0 \| \| md1 \| \| md2 \| \| md3 \| \| md3 \| \| md2 \| \| md1 \| \| md0 \|
-	\|~~_____~~\|~~_____~~\| \|~~_____\|_____\|~~	+	\|_____\| \|_____\| \|_____\| \|_____\| \|_____\| \|_____\| \|_____\| \|_____\|
-	\| \| \| \|	+	\| \| \| \| \| \| \| \|
-	~~___~~\|~~___~~ \| ~~___~~\|~~___~~ \|	+	___\|___ _\|_ ____\|____ \|___________\| ____\|____ _\|_ ___\|___
-	\| ~~/boot~~ \| \| \| ~~/boot~~ \| \|	+	\| /boot \| \| / \| \| <swap> \| \| \| <swap> \| \| / \| \| /boot \|
-	\|~~_______~~\| \| \|~~_______~~\| \|	+	\|_______\| \|___\| \|_________\| ______\|______ \|_________\| \|___\| \|_______\|
-	~~____~~\|~~____ ____~~\|~~____~~	+	\| /dev/drbd0 \|
-	\| ~~LVM PV \| \| LVM PV~~ \|	+	\|_____________\|
-	\|~~_________~~\| \|~~_________\|~~	+	\|
-	~~____~~\|~~____ ____~~\|~~____~~	+	____\|______
-	\| ~~vg_01~~ \| \| ~~vg_01 \|~~	+	\| clvm PV \|
-	~~\|_________\| \|_________\|~~	+	\|___________\|
-	~~\| \|~~	+	\|
-	___~~\|_______ ___________ ___________ _______~~\|___	+	_____\|_____
-	~~____~~\|~~____~~ ____\|____ ~~____~~\|~~____ ____~~\|~~____~~ ____\|____ ~~____~~\|~~____~~	+	\| drbd_vg0 \|
-	~~\| lv_swap \| \| lv_root \| \| lv_drbd \| \| lv_drbd \| \| lv_root \| \| lv_swap~~ \|	+	\|___________\|
-	\|~~_________~~\| \|~~_________~~\| ~~\|_________\| \|_________\| \|_________\| \|_________\|~~	+	\|
-	~~\| \| \| \| \| \|~~	+	_____\|_____ ___...____
-	~~____\|____ ____\|____ \| \| ____\|____ ____\|____~~	+	\| \| \|
-	\| <swap> \| \| ~~/ \| \| \|~~ \| <swap> \| \| / \|	+	___\|___ ___\|___ ___\|___
-	\|~~_________~~\| \|~~_________~~\| \| \| \|_________\| \|_________\|	+	\| lv_X \| \| lv_Y \| \| lv_N \|
-	\|~~________________~~\|	+	\|_______\| \|_______\| \|_______\|
-	\|	+
-	~~______~~\|~~______~~	+
-	\| /dev/drbd0 \|	+
-	\|_____________\|	+
-	\|	+
-	____\|______	+
-	\| clvm PV \|	+
-	\|___________\|	+
-	\|	+
-	_____\|_____	+
-	\| drbd_vg0 \|	+
-	\|___________\|	+
-	\|	+
-	_____\|_____ ___...____	+
-	\| \| \|	+
-	___\|___ ___\|___ ___\|___	+
-	\| lv_X \| \| lv_Y \| \| lv_N \|	+
-	\|_______\| \|_______\| \|_______\|	+
	</source>		</source>

Cluster.conf

Digimer — Tue, 07 Sep 2010 16:51:53 GMT

rrp_mode:

← Older revision		Revision as of 16:51, 7 September 2010
(2 intermediate revisions not shown)
Line 298:		Line 298:

	'''NOTE''': Be sure to set this if you are using redundant rings! If you wish to use a redundant ring, it must be configured in each node's <span class="code">[[#clusternode\|clusternode]]</span> entry. See below for an example.		'''NOTE''': Be sure to set this if you are using redundant rings! If you wish to use a redundant ring, it must be configured in each node's <span class="code">[[#clusternode\|clusternode]]</span> entry. See below for an example.
		+
		+	* Default is <span class="code">none</span>
		+	* Valid options are <span class="code">active</span>, <span class="code">passive</span>, or <span class="code">none</span>
		+
		+	==== Manual Ring Recovery ====

	If a ring fails and then is restored, you must manually run the following to re-enable the ring.		If a ring fails and then is restored, you must manually run the following to re-enable the ring.
Line 310:		Line 315:
	corosync-cfgtool -s		corosync-cfgtool -s
	</source>		</source>
-
-	* Default is <span class="code">none</span>
-	* Valid options are <span class="code">active</span>, <span class="code">passive</span>, or <span class="code">none</span>

	=== secauth ===		=== secauth ===
Line 945:		Line 947:
	This provides a means of setting up a second, [[#rrp_mode\|redundant]] <span class="code">[[#totem; Totem Ring and the Redundant Ring Protocol\|totem]]</span> ring. This provides a means of communication between the nodes in the event of a failure of the primary ring, which could happen if a main switch failed. Without this, should the primary network fail, your nodes would all separate and there would be no way to gain quorum, stopping all hosted services. In corosync or openais, this would be known as ring number <span class="code">1</span>.		This provides a means of setting up a second, [[#rrp_mode\|redundant]] <span class="code">[[#totem; Totem Ring and the Redundant Ring Protocol\|totem]]</span> ring. This provides a means of communication between the nodes in the event of a failure of the primary ring, which could happen if a main switch failed. Without this, should the primary network fail, your nodes would all separate and there would be no way to gain quorum, stopping all hosted services. In corosync or openais, this would be known as ring number <span class="code">1</span>.

-	At the time of this writing, totem rings will not automatically recover after a failure. Please see the ~~node~~ in [[#rrp_mode\|rrp_mode]] for information on how to manually recover a ring.	+	At the time of this writing, totem rings will not automatically recover after a failure. Please see the note in [[#rrp_mode\|rrp_mode]] for information on how to manually recover a ring.

	===== name =====		===== name =====

Fedora13 KS an-node02.ks

Digimer — Tue, 07 Sep 2010 16:42:28 GMT

← Older revision		Revision as of 16:42, 7 September 2010
Line 1:		Line 1:
	{{2node_header}}		{{2node_header}}

-	This is the current kickstart script for the Fedora 13 <span class="code">an-~~node02~~</span> general purpose cluster node.	+	This is the current kickstart script for the Fedora 13 <span class="code">an-node01</span> general purpose cluster node.

-	'''Updated''': ~~Jul~~. 03, 2010	+	'''Updated''': Sep. 07, 2010

	<source lang="bash">		<source lang="bash">
	# Kickstart file created by Digimer		# Kickstart file created by Digimer
	# Email: digimer@alteeve.com		# Email: digimer@alteeve.com
-	# Updated: ~~Jul~~. 03, 2010	+	# Updated: Sep. 07, 2010
	# Based on: Fedora 13		# Based on: Fedora 13
	# Spin Name: AN!Cluster		# Spin Name: AN!Cluster
-	# Spin Ver.: 0.2.~~001~~	+	# Spin Ver.: 0.2.002
-	# Node: AN!Node Base-Type 01	+	# Node: AN!Node Base-Type 02
		+

	### Setup values.		### Setup values.
	install		install
		+	# If you've built a PXE server, you can change 'cdrom' to
		+	#url --url=http://192.168.1.10/f13/x86_64/img/
		+	# For more info on setting up a PXE server, see:
		+	# - http://wiki.alteeve.com/index.php/Setting_Up_a_PXE_Server_in_Fedora)
	cdrom		cdrom
	lang en_US.UTF-8		lang en_US.UTF-8
	keyboard us		keyboard us
	timezone --utc America/Toronto		timezone --utc America/Toronto
		+

	### NICs; Ideal Setup: eth0 = Internet-facing, eth1 = DRBD, eth2 = back_channel		### NICs; Ideal Setup: eth0 = Internet-facing, eth1 = DRBD, eth2 = back_channel
	network --device eth0 --bootproto dhcp --onboot yes --hostname an-node02.alteeve.com		network --device eth0 --bootproto dhcp --onboot yes --hostname an-node02.alteeve.com
-	# ~~This is~~ the ~~desired setup~~, ~~but is currently not that reliable as it's not~~	+	# If you know what your MAC addresses are on the node, you can pre-configure your
-	# ~~possible to link an ethX device to a particular physical NIC~~.	+	# network in advance. Replace the MAC addresses below with yours.
-	#network --device eth0 --bootproto static --onboot yes --ip 192.168.1.72 --netmask 255.255.255.0 --gateway 192.168.1.1 --nameserver 192.139.81.117,192.139.81.1 --hostname an-node02.alteeve.com	+	#network --device 90:E6:BA:71:82:D8 --device eth0 --bootproto static --onboot yes --ip 192.168.1.72 --netmask 255.255.255.0 --gateway 192.168.1.1 --nameserver 192.139.81.117,192.139.81.1 --hostname an-node02.alteeve.com
-	#network --device eth1 --bootproto static --onboot yes --ip 10.0.0.72 --netmask 255.255.255.0	+	#network --device 00:21:91:19:96:5A --device eth1 --bootproto static --onboot yes --ip 10.0.0.72 --netmask 255.255.255.0
-	#network --device eth2 --bootproto static --onboot yes --ip 10.0.1.72 --netmask 255.255.255.0	+	#network --device 00:0E:0C:59:45:78 --device eth2 --bootproto static --onboot yes --ip 10.0.1.72 --netmask 255.255.255.0
		+

	### Security		### Security
Line 38:		Line 45:
	# timezone, root password and initial user account.		# timezone, root password and initial user account.
	firstboot --reconfig		firstboot --reconfig
		+

	### Hard drive info.		### Hard drive info.
		+	# A note here: LVM is desirable on each node's system, but isn't the
		+	# most reliable when also using CLVM. For this reason, the local nodes
		+	# will use simple RAID. All LVM stuff will be handled post-install.
	zerombr		zerombr
-	clearpart --~~linux~~ --drives=sda	+	clearpart --all --initlabel --drives=sda,sdb
-	ignoredisk --only-use=sda	+	ignoredisk --only-use=sda,sdb
-	~~bootloader~~ --~~location~~=~~mbr~~ --~~driveorder~~=sda --~~append~~=~~"rhgb quiet"~~	+
		+	### Setup the two RAID arrays
		+	# /boot
		+	part raid.01 --ondisk=sda --asprimary --size=250
		+	part raid.02 --ondisk=sdb --asprimary --size=250
		+	# /
		+	part raid.11 --ondisk=sda --asprimary --size=40000
		+	part raid.12 --ondisk=sdb --asprimary --size=40000
		+	# <swap>
		+	part raid.21 --ondisk=sda --asprimary --size=4096
		+	part raid.22 --ondisk=sdb --asprimary --size=4096
		+	# DRBD
		+	#part raid.5 --ondisk=sda --asprimary --size=50000 --grow
		+	#part raid.6 --ondisk=sdb --asprimary --size=50000 --grow
		+
		+	# Format /boot, / 'root' and <swap>.
		+	raid /boot --fstype=ext3 --level=1 --device=md0 raid.01 raid.02
		+	raid / --fstype=ext3 --level=1 --device=md1 raid.11 raid.12
		+	raid swap --fstype=swap --level=1 --device=md2 raid.21 raid.22

-	~~part /boot --fstype ext3 --size=250 --asprimary~~
-	~~part pv.11 --size=100 --grow~~
-	~~volgroup vg01 --pesize=32768 pv.11~~
-	~~logvol / --fstype ext4 --name=lv_root --vgname=vg01 --size=40000~~
-	~~logvol swap --fstype swap --name=lv_swap --vgname=vg01 --size=2048~~

	### Repositories. Need to change these once the AN!Repo is up.		### Repositories. Need to change these once the AN!Repo is up.
	#repo --name="Fedora 13 - x86_64" --baseurl=http://fedora.mirror.iweb.ca/releases/13/Everything/x86_64/os/		#repo --name="Fedora 13 - x86_64" --baseurl=http://fedora.mirror.iweb.ca/releases/13/Everything/x86_64/os/
	#repo --name="Fedora 13 - x86_64 - Updates" --baseurl=http://fedora.mirror.iweb.ca/updates/13/x86_64/		#repo --name="Fedora 13 - x86_64 - Updates" --baseurl=http://fedora.mirror.iweb.ca/updates/13/x86_64/
		+

	### Services. Multiple entries are comma-seperated with no spaces.		### Services. Multiple entries are comma-seperated with no spaces.

Fedora13 KS an-node01.ks

Digimer — Tue, 07 Sep 2010 16:40:42 GMT

← Older revision		Revision as of 16:40, 7 September 2010
Line 3:		Line 3:
	This is the current kickstart script for the Fedora 13 <span class="code">an-node01</span> general purpose cluster node.		This is the current kickstart script for the Fedora 13 <span class="code">an-node01</span> general purpose cluster node.

-	'''Updated''': ~~Jul~~. 03, 2010	+	'''Updated''': Sep. 07, 2010

	<source lang="bash">		<source lang="bash">
	# Kickstart file created by Digimer		# Kickstart file created by Digimer
	# Email: digimer@alteeve.com		# Email: digimer@alteeve.com
-	# Updated: ~~Jul~~. 03, 2010	+	# Updated: Sep. 07, 2010
	# Based on: Fedora 13		# Based on: Fedora 13
	# Spin Name: AN!Cluster		# Spin Name: AN!Cluster
-	# Spin Ver.: 0.2.~~001~~	+	# Spin Ver.: 0.2.002
	# Node: AN!Node Base-Type 01		# Node: AN!Node Base-Type 01
		+

	### Setup values.		### Setup values.
	install		install
		+	# If you've built a PXE server, you can change 'cdrom' to
		+	#url --url=http://192.168.1.10/f13/x86_64/img/
		+	# For more info on setting up a PXE server, see:
		+	# - http://wiki.alteeve.com/index.php/Setting_Up_a_PXE_Server_in_Fedora)
	cdrom		cdrom
	lang en_US.UTF-8		lang en_US.UTF-8
	keyboard us		keyboard us
	timezone --utc America/Toronto		timezone --utc America/Toronto
		+

	### NICs; Ideal Setup: eth0 = Internet-facing, eth1 = DRBD, eth2 = back_channel		### NICs; Ideal Setup: eth0 = Internet-facing, eth1 = DRBD, eth2 = back_channel
	network --device eth0 --bootproto dhcp --onboot yes --hostname an-node01.alteeve.com		network --device eth0 --bootproto dhcp --onboot yes --hostname an-node01.alteeve.com
-	# ~~This is~~ the ~~desired setup~~, ~~but is currently not that reliable as it's not~~	+	# If you know what your MAC addresses are on the node, you can pre-configure your
-	# ~~possible to link an ethX device to a particular physical NIC~~.	+	# network in advance. Replace the MAC addresses below with yours.
-	#network --device eth0 --bootproto static --onboot yes --ip 192.168.1.72 --netmask 255.255.255.0 --gateway 192.168.1.1 --nameserver 192.139.81.117,192.139.81.1 --hostname an-node01.alteeve.com	+	#network --device 90:E6:BA:71:82:EA --device eth0 --bootproto static --onboot yes --ip 192.168.1.72 --netmask 255.255.255.0 --gateway 192.168.1.1 --nameserver 192.139.81.117,192.139.81.1 --hostname an-node01.alteeve.com
-	#network --device eth1 --bootproto static --onboot yes --ip 10.0.0.72 --netmask 255.255.255.0	+	#network --device 00:21:91:19:96:53 --device eth1 --bootproto static --onboot yes --ip 10.0.0.72 --netmask 255.255.255.0
-	#network --device eth2 --bootproto static --onboot yes --ip 10.0.1.72 --netmask 255.255.255.0	+	#network --device 00:0E:0C:59:46:E4 --device eth2 --bootproto static --onboot yes --ip 10.0.1.72 --netmask 255.255.255.0
		+

	### Security		### Security
Line 38:		Line 45:
	# timezone, root password and initial user account.		# timezone, root password and initial user account.
	firstboot --reconfig		firstboot --reconfig
		+

	### Hard drive info.		### Hard drive info.
		+	# A note here: LVM is desirable on each node's system, but isn't the
		+	# most reliable when also using CLVM. For this reason, the local nodes
		+	# will use simple RAID. All LVM stuff will be handled post-install.
	zerombr		zerombr
-	clearpart --~~linux~~ --drives=sda	+	clearpart --all --initlabel --drives=sda,sdb
-	ignoredisk --only-use=sda	+	ignoredisk --only-use=sda,sdb
-	~~bootloader~~ --~~location~~=~~mbr~~ --~~driveorder~~=sda --~~append~~=~~"rhgb quiet"~~	+
		+	### Setup the two RAID arrays
		+	# /boot
		+	part raid.01 --ondisk=sda --asprimary --size=250
		+	part raid.02 --ondisk=sdb --asprimary --size=250
		+	# /
		+	part raid.11 --ondisk=sda --asprimary --size=40000
		+	part raid.12 --ondisk=sdb --asprimary --size=40000
		+	# <swap>
		+	part raid.21 --ondisk=sda --asprimary --size=4096
		+	part raid.22 --ondisk=sdb --asprimary --size=4096
		+	# DRBD
		+	#part raid.5 --ondisk=sda --asprimary --size=50000 --grow
		+	#part raid.6 --ondisk=sdb --asprimary --size=50000 --grow
		+
		+	# Format /boot, / 'root' and <swap>.
		+	raid /boot --fstype=ext3 --level=1 --device=md0 raid.01 raid.02
		+	raid / --fstype=ext3 --level=1 --device=md1 raid.11 raid.12
		+	raid swap --fstype=swap --level=1 --device=md2 raid.21 raid.22

-	~~part /boot --fstype ext3 --size=250 --asprimary~~
-	~~part pv.11 --size=100 --grow~~
-	~~volgroup vg01 --pesize=32768 pv.11~~
-	~~logvol / --fstype ext4 --name=lv_root --vgname=vg01 --size=40000~~
-	~~logvol swap --fstype swap --name=lv_swap --vgname=vg01 --size=2048~~

	### Repositories. Need to change these once the AN!Repo is up.		### Repositories. Need to change these once the AN!Repo is up.
	#repo --name="Fedora 13 - x86_64" --baseurl=http://fedora.mirror.iweb.ca/releases/13/Everything/x86_64/os/		#repo --name="Fedora 13 - x86_64" --baseurl=http://fedora.mirror.iweb.ca/releases/13/Everything/x86_64/os/
	#repo --name="Fedora 13 - x86_64 - Updates" --baseurl=http://fedora.mirror.iweb.ca/updates/13/x86_64/		#repo --name="Fedora 13 - x86_64 - Updates" --baseurl=http://fedora.mirror.iweb.ca/updates/13/x86_64/
		+

	### Services. Multiple entries are comma-seperated with no spaces.		### Services. Multiple entries are comma-seperated with no spaces.

DRBD on Fedora 13

Digimer — Thu, 02 Sep 2010 15:17:40 GMT

Why DRBD?:

← Older revision		Revision as of 15:17, 2 September 2010
(One intermediate revision not shown)
Line 8:		Line 8:

	[[DRBD]] is useful in small clusters as it provides real-time mirroring of data across two (or more) nodes. In [[Two Node Fedora 13 Cluster\|two-node clusters]], this can be used to host clustered [[LVM]] physical volumes. On these volumes you can create logical volumes to host [[GFS2]] partitions, virtual machines, [[iSCSI]] and so forth.		[[DRBD]] is useful in small clusters as it provides real-time mirroring of data across two (or more) nodes. In [[Two Node Fedora 13 Cluster\|two-node clusters]], this can be used to host clustered [[LVM]] physical volumes. On these volumes you can create logical volumes to host [[GFS2]] partitions, virtual machines, [[iSCSI]] and so forth.
		+
		+	== A Map of the Cluster's Storage ==
		+
		+	<source lang="text">
		+	Node1 Node2
		+	_____ _____ _____ _____
		+	\| sda \| \| sdb \| \| sda \| \| sdb \|
		+	\|_____\| \|_____\| \|_____\| \|_____\|
		+	\|_______\| \|_______\|
		+	__\|__ __\|__
		+	__\|__ __\|__ __\|__ __\|__
		+	\| md0 \| md1 \| \| md0 \| md1 \|
		+	\|_____\|_____\| \|_____\|_____\|
		+	\| \| \| \|
		+	___\|___ \| ___\|___ \|
		+	\| /boot \| \| \| /boot \| \|
		+	\|_______\| \| \|_______\| \|
		+	____\|____ ____\|____
		+	\| LVM PV \| \| LVM PV \|
		+	\|_________\| \|_________\|
		+	____\|____ ____\|____
		+	\| vg_01 \| \| vg_01 \|
		+	\|_________\| \|_________\|
		+	\| \|
		+	___\|_______ ___________ ___________ _______\|___
		+	____\|____ ____\|____ ____\|____ ____\|____ ____\|____ ____\|____
		+	\| lv_swap \| \| lv_root \| \| lv_drbd \| \| lv_drbd \| \| lv_root \| \| lv_swap \|
		+	\|_________\| \|_________\| \|_________\| \|_________\| \|_________\| \|_________\|
		+	\| \| \| \| \| \|
		+	____\|____ ____\|____ \| \| ____\|____ ____\|____
		+	\| <swap> \| \| / \| \| \| \| <swap> \| \| / \|
		+	\|_________\| \|_________\| \| \| \|_________\| \|_________\|
		+	\|________________\|
		+	\|
		+	______\|______
		+	\| /dev/drbd0 \|
		+	\|_____________\|
		+	\|
		+	____\|______
		+	\| clvm PV \|
		+	\|___________\|
		+	\|
		+	_____\|_____
		+	\| drbd_vg0 \|
		+	\|___________\|
		+	\|
		+	_____\|_____ ___...____
		+	\| \| \|
		+	___\|___ ___\|___ ___\|___
		+	\| lv_X \| \| lv_Y \| \| lv_N \|
		+	\|_______\| \|_______\| \|_______\|
		+	</source>

	= Install =		= Install =
Line 398:		Line 450:
	### END INIT INFO		### END INIT INFO
	</source>		</source>
-

	Change the start order by removing and re-adding all cluster-related daemons:		Change the start order by removing and re-adding all cluster-related daemons:
-
-	~~'''Note'''': This has extra entries to make sure all relevant entries are sorted when re-added. If you do not have some of the daemons listed below installed yet, simply skip them.~~

	<source lang="bash">		<source lang="bash">
-	chkconfig xend off; chkconfig cman ~~off; chkconfig drbd~~ off; chkconfig clvmd off; ~~chkconfig xendomains off~~	+	chkconfig xend off; chkconfig cman off; chkconfig clvmd off; chkconfig xend on; chkconfig cman on; chkconfig clvmd on
-	chkconfig xend on; chkconfig cman ~~on; chkconfig drbd~~ on; chkconfig clvmd ~~on; chkconfig xendomains~~ on	+
	</source>		</source>

Xen on Fedora 13

Digimer — Wed, 01 Sep 2010 23:52:43 GMT

Make xend play nice with clustering:

← Older revision		Revision as of 23:52, 1 September 2010
Line 161:		Line 161:
	By default under Fedora 13, <span class="code">cman</span> will start before <span class="code">xend</span>. This is a problem because <span class="code">xend</span> takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.		By default under Fedora 13, <span class="code">cman</span> will start before <span class="code">xend</span>. This is a problem because <span class="code">xend</span> takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.

-	To avoid this, edit <span class="code">/etc/init.d/xend</span> and tell it to start earlier than position <span class="code">98</span>. This is done by changing the line <span class="code">chkconfig: 2345 98 01</span> to <span class="code">chkconfig: 2345 11 98</span>.	+	To avoid this, edit <span class="code">/etc/init.d/xend</span> and tell it to start earlier than position <span class="code">98</span>. This is done by changing the line <span class="code">chkconfig: 2345 98 01</span> to <span class="code">chkconfig: 2345 11 98</span>. We also don't want it to stop until <span class="code">cman</span> has stopped. We accomplish this by adding <span class="code">cman</span> to the <span class="code">Required-Stop</span> line.

	<source lang="bash">		<source lang="bash">
Line 179:		Line 179:
	# Required-Start: $syslog $remote_fs		# Required-Start: $syslog $remote_fs
	# Should-Start:		# Should-Start:
-	# Required-Stop: $syslog $remote_fs	+	# Required-Stop: $syslog $remote_fs cman
	# Should-Stop:		# Should-Stop:
	# Default-Start: 3 4 5		# Default-Start: 3 4 5
Line 189:		Line 189:
	</source>		</source>

-	~~Lastly~~, ~~remove~~ and ~~re-~~add ~~the~~ <span class="code">xend</span> ~~daemon in~~ the ~~start list:~~	+	With <span class="code">xend</span> set to start at a position lower than <span class="code">98</span>, we now have room for <span class="code">chkconfig</span> to put other daemons after it in the start order. First and foremost, we now need to tell <span class="code">cman</span> to not start until after <span class="code">xend</span> is up.
		+
		+	As above, we will now edit <span class="code">cman</span>'s <span class="code">/etc/init.d/cman</span> script. This time though, we will not edit it's <span class="code">chkconfig</span> line. Instead, we will add <span class="code">xend</span> to the <span class="code">Required-Start</span> entry.

	<source lang="bash">		<source lang="bash">
-	chkconfig xend ~~off~~	+	vim /etc/init.d/cman
-	chkconfig xend on	+	</source>
		+	<source lang="bash">
		+	#!/bin/bash
		+	#
		+	# cman - Cluster Manager init script
		+	#
		+	# chkconfig: - 21 79
		+	# description: Starts and stops cman
		+	#
		+	#
		+	### BEGIN INIT INFO
		+	# Provides: cman
		+	# Required-Start: $network $time xend
		+	# Required-Stop: $network $time
		+	# Default-Start:
		+	# Default-Stop:
		+	# Short-Description: Starts and stops cman
		+	# Description: Starts and stops the Cluster Manager set of daemons
		+	### END INIT INFO
		+	</source>
		+
		+	Lastly, remove and re-add the <span class="code">xend</span> and <span class="code">cman</span> daemons to re-order them in the start list:
		+
		+	<source lang="bash">
		+	chkconfig xend off; chkconfig cman off; chkconfig xend on; chkconfig cman on
		+	</source>
		+
		+	Confirm that the order has changed so that <span class="code">xend</span> is earlier in the boot sequence than <span class="code">cman</span>. Assuming you've switched to run-level 3, run:
		+
		+	<source lang="bash">
		+	ls -lah /etc/rc3.d/
		+	</source>
		+
		+	Your start sequence should now look like:
		+
		+	<source lang="text">
		+	lrwxrwxrwx. 1 root root 14 Sep 1 19:26 S26xend -> ../init.d/xend
		+	lrwxrwxrwx. 1 root root 14 Sep 1 19:26 S27cman -> ../init.d/cman
	</source>		</source>

DRBD on Fedora 13

Digimer — Wed, 01 Sep 2010 23:47:33 GMT

Making LVM Cluster-Aware:

Show changes

Xen on Fedora 13

Digimer — Wed, 01 Sep 2010 21:10:46 GMT

← Older revision		Revision as of 21:10, 1 September 2010
(7 intermediate revisions not shown)
Line 54:		Line 54:

	The entry in grub's <span class="code">/boot/grub/menu.lst</span> won't work. You will need to edit it to look like this:		The entry in grub's <span class="code">/boot/grub/menu.lst</span> won't work. You will need to edit it to look like this:
		+
		+	'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your <span class="code">root=</span> is likely different and your <span class="code">rd_MD_UUID=</span> will definitely be different, even on the same machine across installs. Generally speaking, what follows the <span class="code">kernel /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ...</span> entry made by the dom0 kernel can be copied after the <span class="code">module /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ...</span> entry in the example below.

		+	<source lang="bash">
		+	vim /boot/grub/menu.lst
		+	</source>
	<source lang="bash">		<source lang="bash">
	title Xen 3.4.x, Linux kernel 2.6.32.17-157.xendom0.fc12.x86_64		title Xen 3.4.x, Linux kernel 2.6.32.17-157.xendom0.fc12.x86_64
	root (hd0,0)		root (hd0,0)
	kernel /xen.gz dom0_mem=1024M		kernel /xen.gz dom0_mem=1024M
-	module /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ro root=/dev/mapper/vg_01-lv_root rd_MD_UUID=4f48439e:39ec22e8:a6154857:c340036f rd_LVM_LV=vg_01/lv_root rd_LVM_LV=vg_01/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us ~~rhgb quiet~~	+	module /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ro root=/dev/mapper/vg_01-lv_root rd_MD_UUID=4f48439e:39ec22e8:a6154857:c340036f rd_LVM_LV=vg_01/lv_root rd_LVM_LV=vg_01/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us
	module /initramfs-2.6.32.17-157.xendom0.fc12.x86_64.img		module /initramfs-2.6.32.17-157.xendom0.fc12.x86_64.img
	</source>		</source>
Line 66:		Line 71:

	<source lang="bash">		<source lang="bash">
-	~~vim /etc/fstab~~	+	echo "xenfs /proc/xen xenfs defaults 0 0" >> /etc/fstab
-	~~</source>~~	+
-	~~<source lang=~~"~~text">~~	+
-	xenfs /proc/xen xenfs defaults 0 0	+
	</source>		</source>

	= Build from source =		= Build from source =

-	This is required if you want [[dom0~~]] and [[DRBD~~]].	+	This is required if you want a more up to date [[dom0]].

	== Install Requirements ==		== Install Requirements ==
Line 134:		Line 136:
	</source>		</source>

-	'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work!	+	'''Note''': Copy and modify the entry created by the RPM. Simply copying this entry will almost certainly not work! Your <span class="code">root=</span> is likely different and your <span class="code">rd_MD_UUID=</span> will definitely be different, even on the same machine across installs. Generally speaking, what follows the <span class="code">kernel /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ...</span> entry made by the dom0 kernel can be copied after the <span class="code">module /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ...</span> entry in the example below.

	'''Xen 4.0.1''':		'''Xen 4.0.1''':
Line 142:		Line 144:
	root (hd0,0)		root (hd0,0)
	kernel /xen-4.0.1.gz dom0_mem=1024M		kernel /xen-4.0.1.gz dom0_mem=1024M
-	module /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ro root=/dev/mapper/vg_01-lv_root rd_MD_UUID=~~4f48439e:39ec22e8:a6154857:c340036f~~ rd_LVM_LV=vg_01/lv_root rd_LVM_LV=vg_01/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet	+	module /vmlinuz-2.6.32.17-157.xendom0.fc12.x86_64 ro root=/dev/mapper/vg_01-lv_root rd_MD_UUID=<your UUID> rd_LVM_LV=vg_01/lv_root rd_LVM_LV=vg_01/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us rhgb quiet
	module /initramfs-2.6.32.17-157.xendom0.fc12.x86_64.img		module /initramfs-2.6.32.17-157.xendom0.fc12.x86_64.img
	</source>		</source>
Line 155:		Line 157:
	</source>		</source>

-	= Make ~~cman and~~ xend play nice =	+	= Make xend play nice with clustering =

	By default under Fedora 13, <span class="code">cman</span> will start before <span class="code">xend</span>. This is a problem because <span class="code">xend</span> takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.		By default under Fedora 13, <span class="code">cman</span> will start before <span class="code">xend</span>. This is a problem because <span class="code">xend</span> takes the network down as part of it's setup. This causes [[totem]] communication to fail which leads to fencing.

-	To avoid this, edit <span class="code">/etc/init.d/~~cman~~</span> and tell it to start ~~after <span class="code">xend</span> by adding "<span class="code">xend</span>" to <span class="code">Required-Start</span> and <span class="code">Required-Stop</span> boot facility lists.~~	+	To avoid this, edit <span class="code">/etc/init.d/xend</span> and tell it to start earlier than position <span class="code">98</span>. This is done by changing the line <span class="code">chkconfig: 2345 98 01</span> to <span class="code">chkconfig: 2345 11 98</span>.
-		+
-	~~When done, the <span class="code">cman</span> LSB header should look like this:~~	+
-		+
-	~~<source lang="bash">~~	+
-	~~vim /etc/init.d/cman~~	+
-	~~</source>~~	+
-	~~<source lang="bash">~~	+
-	~~### BEGIN INIT INFO~~	+
-	~~# Provides: cman~~	+
-	~~# Required-Start: $network $time xend~~	+
-	~~# Required-Stop: $network $time xend~~	+
-	~~# Default-Start:~~	+
-	~~# Default-Stop:~~	+
-	~~# Short-Description: Starts and stops cman~~	+
-	~~# Description: Starts and stops the Cluster Manager set of daemons~~	+
-	~~### END INIT INFO~~	+
-	~~</source>~~	+
-		+
-	~~Next, we need to tell <span class="code">xend</span> not to default to start~~ position <span class="code">98</span>. Without this change, services we want to start after <span class="code">xend</span> (<span class="code">xman</span> -> <span class="code">drbd</span> -> <span class="code">clvmd</span> -> <span class="code">xendomains</span>) would all fall to start position <span class="code">99</span>. This ~~would mean that~~ the start order would become dependent on the daemon name, which is a bad idea. For example, this would cause <span class="code">clvmd</span> to start before <span class="code">cman</span>, which we don't want.	+
-		+
-	~~Change~~ <span class="code">chkconfig: 2345 98 01</span> to <span class="code">chkconfig: 2345 11 98</span>.	+

	<source lang="bash">		<source lang="bash">
-	vim /etc/init.d/~~drbd~~	+	vim /etc/init.d/xend
	</source>		</source>
	<source lang="bash">		<source lang="bash">
		+	#!/bin/bash
		+	#
		+	# xend Script to start and stop the Xen control daemon.
		+	#
		+	# Author: Keir Fraser <keir.fraser@cl.cam.ac.uk>
		+	#
	# chkconfig: 2345 11 98		# chkconfig: 2345 11 98
	# description: Starts and stops the Xen control daemon.		# description: Starts and stops the Xen control daemon.
	### BEGIN INIT INFO		### BEGIN INIT INFO
	# Provides: xend		# Provides: xend
-	# Required-Start: $syslog	+	# Required-Start: $syslog $remote_fs
-	# Should-Start: ~~$remote_fs~~	+	# Should-Start:
-	# Required-Stop: $syslog	+	# Required-Stop: $syslog $remote_fs
-	# Should-Stop: ~~$remote_fs~~	+	# Should-Stop:
	# Default-Start: 3 4 5		# Default-Start: 3 4 5
	# Default-Stop: 0 1 2 6		# Default-Stop: 0 1 2 6
Line 202:		Line 189:
	</source>		</source>

-	Lastly, remove and re-add the ~~<span class="code">cman</span> and~~ <span class="code">xend</span> daemon to the start list:	+	Lastly, remove and re-add the <span class="code">xend</span> daemon in the start list:
-		+
-	~~'''Note'''': This has extra entries to make sure all relevant entries are sorted when re-added. If you do not have some of the daemons listed below installed yet, simply skip them.~~	+

	<source lang="bash">		<source lang="bash">
-	chkconfig xend ~~off; chkconfig cman off; chkconfig drbd off; chkconfig clvmd off; chkconfig xendomains off; chkconfig gfs2~~ off	+	chkconfig xend off
-	chkconfig xend ~~on; chkconfig cman on; chkconfig drbd on; chkconfig clvmd on; chkconfig xendomains on; chkconfig gfs2~~ on	+	chkconfig xend on
	</source>		</source>

Line 219:		Line 204:
	= Creating domU VMs =		= Creating domU VMs =

-	~~Once~~ you ~~have ..~~.	+	If you are running on top of DRBD in a cluster, please see "[[DRBD on Fedora 13]]" before proceeding.
		+
		+	'''ToDo'''

	{{footer}}		{{footer}}

DRBD on Fedora 13

Digimer — Wed, 01 Sep 2010 19:15:57 GMT

Compile the DRBD module for Xen dom0:

← Older revision		Revision as of 19:15, 1 September 2010
Line 17:		Line 17:
	= Compile the DRBD module for Xen dom0 =		= Compile the DRBD module for Xen dom0 =

-	If you are running the [[Xen_on_Fedora_13#dom0\|custom Xen dom0]], you will need to build the DRBD module from the source RPM.	+	If you are running the [[Xen_on_Fedora_13#dom0\|custom Xen dom0]], you will probably need to build the DRBD module from the source RPM. You can try the following RPMs, but be aware that if the [[dom0]] kernel gets updated, you will need to rebuild these RPMs using the following steps.
		+
		+	* [http://wiki.alteeve.com/files/an-cluster/drbd-km-2.6.32.17_157.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm drbd-km-2.6.32.17_157.xendom0.fc12.x86_64-8.3.7-12.fc13.x86_64.rpm] - DRBD kernel module for myoung's 2.6.32.17_157 dom0 kernel (897 KiB)
		+	* [http://wiki.alteeve.com/files/an-cluster/drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm drbd-km-debuginfo-8.3.7-12.fc13.x86_64.rpm] - Debug info for DRBD kernel module for myoung's 2.6.32.17_157 dom0 kernel (3.2 KiB)
		+
		+	If the above RPMs don't work or if the dom0 kernel you are using in any way differs, please continue.

	Install the build environment:		Install the build environment:

Two Node Fedora 13 Cluster

Digimer — Wed, 01 Sep 2010 19:09:19 GMT

Setup cman:

← Older revision		Revision as of 19:09, 1 September 2010
(One intermediate revision not shown)
Line 468:		Line 468:
	</source>		</source>

-	If there are errors, fix them. Once it is formatted properly, the contents of you <span class="code">cluster.conf</span> file will be returned followed by "<span class="code">/etc/cluster/cluster.conf validates</span>". Once this is the case, ~~the last step is~~ to enable <span class="code">cman</span> to start at boot.	+	If there are errors, fix them. Once it is formatted properly, the contents of you <span class="code">cluster.conf</span> file will be returned followed by "<span class="code">/etc/cluster/cluster.conf validates</span>". Once this is the case, you can move on.
		+
		+	A note; Normally at this stage, you'd use <span class="code">chkconfig</span> to enable <span class="code">cman</span> to start at boot. However, depending on how you use your cluster, you may need to modify the boot order. For now, let's leave it off until we're got all the components enabled. This is particularly important if you will be using [[DRBD]], [[CLVM]] or [[Xen]].

	<source lang="bash">		<source lang="bash">
-	chkconfig cman on	+	chkconfig cman off
	</source>		</source>

-	=~~=== Make~~ cman ~~and xend Play Nice ==~~==	+	To see if it's working though, have a terminal open to both nodes and manually start <span class="code">cman</span>. I suggest have two terminal windows open on each node. In one, start <span class="code">tail</span>ing <span class="code">/var/log/messages</span>:

-	~~If you are using a [[Xen]] [[dom0]], you will need to modify the~~ <~~span class~~="~~code~~">/~~etc~~/~~init.d~~/~~cman~~</~~span~~> ~~file to make sure that <span class="code">cman</span> start '''after''' <span class="code">xend</span>. Please see the following article for detailed steps:~~	+	<source lang="bash">
		+	clear; tail -f -n 0 /var/log/messages
		+	</source>

-	* [[~~Xen_on_Fedora_13#Make_cman_and_xend_play_nice\|Make~~ cman and ~~xend play nice~~ in ~~Fedora 13]]~~.	+	Now, with both log files being watched, start <span class="code">cman</span> on both nodes.
		+
		+	'''Note''': You MUST execute the following on both nodes withing the time limit set by <span class="code">post_join_delay</span> (default it <span class="code">6</span> seconds). If you wait longer than this, the first node started will [[fence]] the other node, assuming you have fencing configured properly. If fencing is '''not''' configured properly, your first node will hang.
		+
		+	<source lang="bash">
		+	/etc/init.d/cman start
		+	</source>
		+
		+	Examine the log files to verify there were no errors. If there were, fix them. If there was not, fantastic!
		+
		+	'''Warning''': This last step will forcefully reboot your nodes. Do not do this on a production cluster! If it must be done on a production server, stop all non-essential services and be sure you have a good backup, ''just in case''.
		+
		+	Lastly, test that your fencing is configured and working properly. We do this by using a program called <span class="code">fence_node</span>. With both nodes up and running <span class="code">cman</span>, call a fence against the other node. Be sure to be watching the log files.
		+
		+	'''From''' <span class="code">an-node01.alteeve.com</span>
		+
		+	<source lang="bash">
		+	fence_node an-node02.alteeve.com
		+	</source>
		+	<source lang="bash">
		+	fence an-node02.alteeve.com success
		+	</source>
		+
		+	If it worked, when the node comes back up, repeat the process from <span class="code">an-node02.alteeve.com</span> against <span class="code">an-node01.alteeve.com</span>.

	=== Setup Corosync ===		=== Setup Corosync ===
		+
		+	'''Note''': In our cluster, we will not need to configure Corosync.

	When using <span class="code">cman</span>, as we are here, Corosync's configuration file is ignored.		When using <span class="code">cman</span>, as we are here, Corosync's configuration file is ignored.

-	Most of the settings available in <span class="code">[[Two-Node Fedora 13 corosync.conf\|corosync.conf]]</span> can be ~~ported directly into~~ the <span class="code">[[Two-Node Fedora 13 cluster.conf\|cluster.conf]]</span> file. Please see the <span class="code">cluster.conf</span> article for a comprehensive list of what is and is not supported.	+	Most of the settings available in <span class="code">[[Two-Node Fedora 13 corosync.conf\|corosync.conf]]</span> can be found in the <span class="code">[[Two-Node Fedora 13 cluster.conf\|cluster.conf]]</span> file. Please see the <span class="code">cluster.conf</span> article for a comprehensive list of what is and is not supported.

	If you want to configure corosync and not use cman, please see the following article:		If you want to configure corosync and not use cman, please see the following article:

← Older revision		Revision as of 17:53, 7 September 2010
Line 12:		Line 12:

	<source lang="text">		<source lang="text">
-	Node1 Node2	+	Node1 Node2
-	_____ _____ _____ _____	+	_____ _____ _____ _____
-	\| sda \| \| sdb \| \| sda \| \| sdb \|	+	\| sda \| \| sdb \| \| sda \| \| sdb \|
-	\|_____\| \|_____\| \|_____\| \|_____\|	+	\|_____\| \|_____\| \|_____\| \|_____\|
-	\|_______\| \|_______\|	+	\|_______\| \|_______\|
-	__\|__ __\|__	+	_______ ____\|___ _______ _______ ____\|___ _______
-	__\|__ __\|__ __\|__ __\|__	+	__\|__ __\|__ __\|__ __\|__ __\|__ __\|__ __\|__ __\|__
-	\| md0 \| ~~md1 \| \| md0~~ \| md1 \|	+	\| md0 \| \| md1 \| \| md2 \| \| md3 \| \| md3 \| \| md2 \| \| md1 \| \| md0 \|
-	\|~~_____~~\|~~_____~~\| \|~~_____\|_____\|~~	+	\|_____\| \|_____\| \|_____\| \|_____\| \|_____\| \|_____\| \|_____\| \|_____\|
-	\| \| \| \|	+	\| \| \| \| \| \| \| \|
-	~~___~~\|~~___~~ \| ~~___~~\|~~___~~ \|	+	___\|___ _\|_ ____\|____ \|___________\| ____\|____ _\|_ ___\|___
-	\| ~~/boot~~ \| \| \| ~~/boot~~ \| \|	+	\| /boot \| \| / \| \| <swap> \| \| \| <swap> \| \| / \| \| /boot \|
-	\|~~_______~~\| \| \|~~_______~~\| \|	+	\|_______\| \|___\| \|_________\| ______\|______ \|_________\| \|___\| \|_______\|
-	~~____~~\|~~____ ____~~\|~~____~~	+	\| /dev/drbd0 \|
-	\| ~~LVM PV \| \| LVM PV~~ \|	+	\|_____________\|
-	\|~~_________~~\| \|~~_________\|~~	+	\|
-	~~____~~\|~~____ ____~~\|~~____~~	+	____\|______
-	\| ~~vg_01~~ \| \| ~~vg_01 \|~~	+	\| clvm PV \|
-	~~\|_________\| \|_________\|~~	+	\|___________\|
-	~~\| \|~~	+	\|
-	___~~\|_______ ___________ ___________ _______~~\|___	+	_____\|_____
-	~~____~~\|~~____~~ ____\|____ ~~____~~\|~~____ ____~~\|~~____~~ ____\|____ ~~____~~\|~~____~~	+	\| drbd_vg0 \|
-	~~\| lv_swap \| \| lv_root \| \| lv_drbd \| \| lv_drbd \| \| lv_root \| \| lv_swap~~ \|	+	\|___________\|
-	\|~~_________~~\| \|~~_________~~\| ~~\|_________\| \|_________\| \|_________\| \|_________\|~~	+	\|
-	~~\| \| \| \| \| \|~~	+	_____\|_____ ___...____
-	~~____\|____ ____\|____ \| \| ____\|____ ____\|____~~	+	\| \| \|
-	\| <swap> \| \| ~~/ \| \| \|~~ \| <swap> \| \| / \|	+	___\|___ ___\|___ ___\|___
-	\|~~_________~~\| \|~~_________~~\| \| \| \|_________\| \|_________\|	+	\| lv_X \| \| lv_Y \| \| lv_N \|
-	\|~~________________~~\|	+	\|_______\| \|_______\| \|_______\|
-	\|	+
-	~~______~~\|~~______~~	+
-	\| /dev/drbd0 \|	+
-	\|_____________\|	+
-	\|	+
-	____\|______	+
-	\| clvm PV \|	+
-	\|___________\|	+
-	\|	+
-	_____\|_____	+
-	\| drbd_vg0 \|	+
-	\|___________\|	+
-	\|	+
-	_____\|_____ ___...____	+
-	\| \| \|	+
-	___\|___ ___\|___ ___\|___	+
-	\| lv_X \| \| lv_Y \| \| lv_N \|	+
-	\|_______\| \|_______\| \|_______\|	+
	</source>		</source>

← Older revision		Revision as of 15:17, 2 September 2010
(One intermediate revision not shown)
Line 8:		Line 8:

	[[DRBD]] is useful in small clusters as it provides real-time mirroring of data across two (or more) nodes. In [[Two Node Fedora 13 Cluster\|two-node clusters]], this can be used to host clustered [[LVM]] physical volumes. On these volumes you can create logical volumes to host [[GFS2]] partitions, virtual machines, [[iSCSI]] and so forth.		[[DRBD]] is useful in small clusters as it provides real-time mirroring of data across two (or more) nodes. In [[Two Node Fedora 13 Cluster\|two-node clusters]], this can be used to host clustered [[LVM]] physical volumes. On these volumes you can create logical volumes to host [[GFS2]] partitions, virtual machines, [[iSCSI]] and so forth.
		+
		+	== A Map of the Cluster's Storage ==
		+
		+	<source lang="text">
		+	Node1 Node2
		+	_____ _____ _____ _____
		+	\| sda \| \| sdb \| \| sda \| \| sdb \|
		+	\|_____\| \|_____\| \|_____\| \|_____\|
		+	\|_______\| \|_______\|
		+	__\|__ __\|__
		+	__\|__ __\|__ __\|__ __\|__
		+	\| md0 \| md1 \| \| md0 \| md1 \|
		+	\|_____\|_____\| \|_____\|_____\|
		+	\| \| \| \|
		+	___\|___ \| ___\|___ \|
		+	\| /boot \| \| \| /boot \| \|
		+	\|_______\| \| \|_______\| \|
		+	____\|____ ____\|____
		+	\| LVM PV \| \| LVM PV \|
		+	\|_________\| \|_________\|
		+	____\|____ ____\|____
		+	\| vg_01 \| \| vg_01 \|
		+	\|_________\| \|_________\|
		+	\| \|
		+	___\|_______ ___________ ___________ _______\|___
		+	____\|____ ____\|____ ____\|____ ____\|____ ____\|____ ____\|____
		+	\| lv_swap \| \| lv_root \| \| lv_drbd \| \| lv_drbd \| \| lv_root \| \| lv_swap \|
		+	\|_________\| \|_________\| \|_________\| \|_________\| \|_________\| \|_________\|
		+	\| \| \| \| \| \|
		+	____\|____ ____\|____ \| \| ____\|____ ____\|____
		+	\| <swap> \| \| / \| \| \| \| <swap> \| \| / \|
		+	\|_________\| \|_________\| \| \| \|_________\| \|_________\|
		+	\|________________\|
		+	\|
		+	______\|______
		+	\| /dev/drbd0 \|
		+	\|_____________\|
		+	\|
		+	____\|______
		+	\| clvm PV \|
		+	\|___________\|
		+	\|
		+	_____\|_____
		+	\| drbd_vg0 \|
		+	\|___________\|
		+	\|
		+	_____\|_____ ___...____
		+	\| \| \|
		+	___\|___ ___\|___ ___\|___
		+	\| lv_X \| \| lv_Y \| \| lv_N \|
		+	\|_______\| \|_______\| \|_______\|
		+	</source>

	= Install =		= Install =
Line 398:		Line 450:
	### END INIT INFO		### END INIT INFO
	</source>		</source>
-

	Change the start order by removing and re-adding all cluster-related daemons:		Change the start order by removing and re-adding all cluster-related daemons:
-
-	~~'''Note'''': This has extra entries to make sure all relevant entries are sorted when re-added. If you do not have some of the daemons listed below installed yet, simply skip them.~~

	<source lang="bash">		<source lang="bash">
-	chkconfig xend off; chkconfig cman ~~off; chkconfig drbd~~ off; chkconfig clvmd off; ~~chkconfig xendomains off~~	+	chkconfig xend off; chkconfig cman off; chkconfig clvmd off; chkconfig xend on; chkconfig cman on; chkconfig clvmd on
-	chkconfig xend on; chkconfig cman ~~on; chkconfig drbd~~ on; chkconfig clvmd ~~on; chkconfig xendomains~~ on	+
	</source>		</source>