Compiled from: CodeAddict’s Blog and http://pastebin.com/buBHcfxU
First: Install XenServer on first disk (/dev/sda) and NOT configure any local storage.
/dev/sda should contain three partitions, verify this with:
sgdisk -p /dev/sda
In some cases XenServer doesn’t create third partition, i.e. result of this command has only two partitions and looks like:
Number Start (sector) End (sector) Size Code Name
1 2048 8388641 4.0 GiB 0700
2 8390656 16777249 4.0 GiB 0700
In this case you need to create third partition manually:
sgdisk --new=3:16779264:DISK_SIZE-34 /dev/sda sgdisk --typecode=3:fd00
The second partition is used for backups during a XenServer upgrade. This partition won’t used in my XenServer upgrade method.
Now we are going to use /dev/sdb as the mirror disk. First clear the partition table.
sgdisk --zap-all /dev/sdb
and install a GPT table on it
sgdisk --mbrtogpt --clear /dev/sdb
The following commands are dependent on your installation. You must copy the start and end sectors from /dev/sda, but in practice, for Xenserver (checked on 6.1,6.2) you must only change DISK_SIZE with sectors on /dev/sdb (first string from output of command:
sgdisk -p /dev/sdb
sgdisk --new=1:2048:8388641 /dev/sdb sgdisk --typecode=1:fd00 /dev/sdb sgdisk --attributes=1:set:2 /dev/sdb sgdisk --new=2:8390656:16777249 /dev/sdb sgdisk --typecode=2:fd00 /dev/sdb sgdisk --new=3:16779264:DISK_SIZE-34 /dev/sdb sgdisk --typecode=3:fd00 /dev/sdb
Reboot server
After reboot, create the software RAID devices.
modprobe raid1 mdadm --create /dev/md0 --level=1 --raid-devices=2 --metadata=0.90 missing /dev/sdb1 mdadm --create /dev/md1 --level=1 --raid-devices=2 --metadata=0.90 missing /dev/sdb2 mdadm --create /dev/md2 --level=1 --raid-devices=2 --metadata=0.90 missing /dev/sdb3
Format the RAID root disk, mount it at /mnt and copy all to it
mkfs.ext3 /dev/md0 mount /dev/md0 /mnt cp -vxpR / /mnt
Modify /mnt/etc/fstab and change root filesystem to /dev/md0. (Do NOT include the LABEL=)
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab
Copy the RAID setup to /etc/mdadm.conf (for mkinitrd) and /mnt/etc/mdadm.conf
mdadm --detail --scan >> /mnt/etc/mdadm.conf cp /mnt/etc/mdadm.conf /etc/mdadm.conf
Create a new boot image:
mkdir /mnt/root/initrd-raid mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`
Copy the new ramdisk to the /mnt/boot folder and modify boot menu
rm /mnt/boot/initrd-3.10-xen.img cp /mnt/root/initrd-raid/initrd-`uname -r`-raid.img /mnt/boot/ cd /mnt/boot ln -s initrd-`uname -r`-raid.img initrd-3.10-xen.img
Replace in /mnt/boot/extlinux.conf string “root=LABEL=root-…” to “root=/dev/md0” in all menu entries.
sed -i 's/LABEL=[a-zA-Z\-]*/\/dev\/md0/' extlinux.conf
Set up MBR for GPT on /dev/sdb
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb cd /mnt extlinux --raid -i boot/
Reboot (IMPORTANT: Set your server to boot from the SECONDARY HDD before booting!)
When XenServer is up again, include /dev/sda in the array with the following commands:
sgdisk --attributes=1:set:2 /dev/sda sgdisk --typecode=1:fd00 /dev/sda sgdisk --typecode=2:fd00 /dev/sda sgdisk --typecode=3:fd00 /dev/sda mdadm -a /dev/md0 /dev/sda1 mdadm -a /dev/md1 /dev/sda2 mdadm -a /dev/md2 /dev/sda3
The array needs to complete its initial build/synchronisation. That is gonna take a while. Follow its progress with:
watch -n 1 cat /proc/mdstat
At last add /dev/md2 as a local SR to XenServer.
xe sr-create content-type=user device-config:device=/dev/md2 host-uuid= name-label="Local Storage" shared=false type=lvm
Also, type can be “ext”: type=ext
PS: Update for installed system: XenServer with software RAID1 update (6.1 to 6.2)
Hi,
It’s great article. I did everything as you said and RAID working perfectly. But, if I can ask about something…
Please, let me explain: a few days ago I had to replace one of my disk (sda), beacause it’s broken.
First of all, I did:
mdadm /dev/md0 –fail /dev/sda1.
mdadm /dev/md0 –remove /dev/sda1
(both command for every partition)
I turned off server, replace sda disk and I booted from second disk – sdb without any problem.
cat /proc/mdstat showed me that arrays works only on one disk sdb.
I copied: sgdisk -R /dev/sda /dev/sdb and sgdisk -G /dev/sda and after that I did:
cat /usr/share/syslinux/gptmbr.bin > /dev/sda – what I have red in the internet this command should install bootloader on the disk, right? (sgdisk -p /dev/sda showed me the same information what are on the second disk)
After that I added sda disk to arrays and RAID started to recovery – of course with successful. 😉
But I’m not sure about install bootloader. Is it enough what I did? Or do I have to install extlinux –install /boot? Or maybe I have to: cat /usr/share/syslinux/gptmbr.bin > /dev/sda after rebuild arrays?
I don’t have any guarantee, that system will UP from sda disk when the sdb will down.
If I understand well idea of RAID-1 bot of disks are 1:1, so MBR should be copied from one disk to second. Even that, I checked sha1sum (dd if=/dev/sda(sdb) of=some_file bs=512 count=1) and the sums are the same. It shoud,right?
Thank you for your help,
Simon
Hello,
Yes, idea of RAID1 is that one disk is a copy of another, but this is software RAID and it based on partitions. I.e. software RAID is a copy of partition, not disk
All that you need for raid – rebuild array and synchronize all partitions. You need not rewrite MBR again after rebuild arrays. For boot from raid:
cat /usr/share/syslinux/gptmbr.bin > /dev/sda
but you already done that.
Just in case – set in BIOS boot from first disk and try to boot
Pozdrawiam,
Walery
Hi,
Thank you for explanation! It was what I needed! I did exactly as you said: cat /usr/share/syslinux/gptmbr.bin > /dev/sda before rebuild – now I have to try to boot from sda.
Regards,
Simon
Hi,
Great post. Thank you!
I am stuck however at rebooting from the second hard drive. It displays “missing os” and then boots from the first (sda). Any ideas on how to proceed/what seems to not take?
Cheers!
1. Check the second disk:
sgdisk –attributes=1:show /dev/sdb
must be return: 1:2:1 (legacy BIOS bootable)
if not, it is not bootable
2. Do you set up MBR for GPT on the second disk?
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
etc?
I certainly did.
It turns out a leftover mdadm was causing problems. It would boot from sda but assemble the sdb2 raid device.
It’s very late and I probably don’t make much sense. But thank you again! (It’s working now after I dd if=/dev/zero of=/dev/sda & /dev/sdb to wipe everything! 🙂
Cheers!
Hej Walery (Czesc)!
Really good post. It works for Citrix Free XenServer 6.2.0. But I ran into a problem with
mdadm -a /dev/md2 /dev/sda3
that returned error:
mdadm: Cannot open /dev/sda3: Device or resource busy
Fix is here: http://blog.codeaddict.org/?p=5#comment-599 – see Remik’s post.
Cheers! / Pozdrawiam!
Mariusz
Hello Mariusz,
Thanks, this post really fixes this problem, but… the error arise, when XenServer prepares local disk for storage.
Have you created a local storage, during XenServer installation (any disk selected, like http://www.petri.co.il/wp-content/uploads/07-virtual-machine-storage.png)?
Regards,
Walery
You have to create a 3rd partion in the sda (use the following command same like for sdb)
sgdisk –new=3:16779264:DISK_SIZE-34 /dev/sda
Thanks to this note. In some cases XenServer really doesn’t create third partition and you need to create it manually. I change article considering this fact:
sgdisk –new=3:16779264:DISK_SIZE-34 /dev/sda
sgdisk –typecode=3:fd00
Thanks so much! I had the same problem:
mdadm: Cannot open /dev/sda3: Device or resource busy
and mdadm –zero-superblock /dev/sda3 did not help.
after long time searching I found your link and it fixed it for me!
–>sda3 now added /dev/md2 and bussy building :p *yay*
Thanks!
Cześć Mariusz!
Thank you for solution but I have question. I already have VM on local repository. Will I lose my VM and data using xe sr-forget and xe sr-create commands?
This is a solution from not working site http://blog.codeaddict.org/?p=5#comment-599:
To fix it I had to do these commands:
1) Find Your local storage uuid:
xe sr-list
The uuid might look like: dce9fc00-3cd9-92c7-be04-be29382dca7f
2) Find uuid of PDB connected to Your local storage (sr-uuid is Your local storage uuid):
xe pbd-list sr-uuid=dce9fc00-3cd9-92c7-be04-be29382dca7f
Look for PDB uuid, which is in the first line of the anwer.
3) Unplug PDB from local storage:
xe pbd-unplug uuid=PDB_UUID
4) Forget the local storage sr:
xe sr-forget uuid=dce9fc00-3cd9-92c7-be04-be29382dca7f
5) Now add the sda3 to md2 array:
mdadm -a /dev/md2 /dev/sda3
6) Recreate local storage on md2:
xe sr-create name-label=”Local Storage” type=lvm device-config:device=/dev/md2 shared=false
Founded in web archive
Anyone tested upgrade from 6.1 soft raid installation to 6.2?
What should we do to upgrade to 6.2?
See my another post: http://wysotsky.info/?p=145
Thank you very much for this great post! I Just installed 2 XenServer 6.2 without any problems. I tried allot of howto’s, but this was the only one witch worked with XenSever 6.2! Thanks allot!
Thanks very much! Worked great for XenServer 6.2!
Article is going good. But when it says leave out the “LABEL=”
Modify /mnt/etc/fstab and change root filesystem to /dev/md0. (Do NOT include the LABEL=)
sed -i ‘s/LABEL=[a-zA-Z\-]*\s\(.*\)/\/dev\/md0 \1/’ /mnt/etc/fstab
What do you insert at that point? Like everything except LABEL= or everything behind it?
After clean install of XenServer, the first string in /etc/fstab (/mnt/etc/fstab in article) look like
LABEL=root-eogfmwmf / ext3 defaults 1 1
but should be
/dev/md0 / ext3 defaults 1 1
Make this change manually or execute sed command
So that being said, do I enter whats below? Sorry for the questioning but this my first raid with xenserver.
sed -I ‘s/LABEL=/dev/md0’ /mnt/etc/fstab
Thanks for your help.
Almost, but not quite right, you should enter:
sed -i ‘s/LABEL=[a-zA-Z\-]*/\/dev\/md0/’ /mnt/etc/fstab
Detail:
1. LABEL=[a-zA-Z\-]*
Find string, that begin LABEL= and any latin letters or dash after it – you also need to remove word after LABEL=
2. \/dev\/md0
replace found string to /dev/md0
Because regexp in sed delimited by /, you need to escaping slashes in /dev/md0
Almost all code blocks from article may be used “as is”, without change – I’m lazy and simply copying the strings from blog article, when installing servers 🙂
Thank you, I was not thinking when I wrote article that I can use more short string in sed for change fstab – I have already changed the article.
Hi,
Having a bit of a problem. I’m using xenserver 6.2 and I can’t boot from the raid device. I have edit extlinuc.conf and removed the splash and quiet to see where it’s hanging and the issues is the he scans for md devices but doesnt find any. And so the swtichroot from init fails.
When booting back from the first disk, no md devices were found either. /etc/mdadm.conf was missing. So I did mdadm –scan –examine > /etc/mdadm.conf, rebooted and the md devices were there.
I ran the mkinitrd again, altered it again and also added the mdadm.conf file to mkinitrd image. After that, rebooted from the second disk, but my problem remains.
Maybe a inportant detail: this setup is our testsetup to be performed on the production servers. For testing I use vmware workstation (it supprts virtual intel-VT) and I have my succesfull xenserver test running in it. Now the issue with vmware workstation is that you have to specify the bootorder of ythe disks in the bios. Selecting the boot selection screen (BSS) only displays hd,cdrom,usb, … But you cant choose which of the HDs in the BSS. So therefor I have to switch them in bios. Thus sda becomes sdb and vice versa. I would not assume this would be an issue, since this whole setup is to provide a raid solution so when 1 disks fails, the xenserver still can boot.
So any ideas would be helpfull.
I Have a spare supermicro laying around, will test the setup on that machine now to see if things are different there.
cheers
I have found it. While replaying my steps on the physical machine, I noticed I had forgotten to change the partition type to linux raid. Changed it and now all works well. Sorry to bother 🙂
Добрый день! все делаю по мануалу, но при создании рэйда mdadm –create /dev/md0 –level=1 –raid-devices=2 missing /dev/sdb1
выскакивает ошибка хранилище при установке не ставил.
mdadm: another array by this name is already running.
Такое может быть после нескольких попыток установки ксена? разных версий пробывал ставить 6.0 потом 6.1, потом 6.2
За ранее спасибо!
Скорее всего на этих дисках уже пробовали когда-то ставить рейд, вот он при установке его и подхватывает. Проверить можно:
mdadm -D /dev/md0
Если там только второй диск (/dev/sdb), то можно остановить его и удалить массив:
mdadm -S /dev/md0
dd if=/dev/zero of=/dev/sdb bs=1M count=1024
пробовал останавливать, не останавливается. Решил акронисом очистить диски и на чистую поставить, действительно несколько раз пробовал запускать рэйд. Спасибо за помощь
добрый день вроде получилось, только вот неуверенность работает или нет рэйд, Как можно удостовериться в этом. Знаю что надо читать маны по рэйд, но я новичок в линуксе. Также не знаю как узнать в линуксе с какого диска я загрузился. Спасибо за ранее.
Ну для начала в консоли запустить:
df -h
если в в том, что оно выведет будет строка типа
/dev/md0 4.0G 2.4G 1.4G 64% /
т.е. работает с /dev/md0, значит рейд работает
Все получилось, спасибо! Пост супер.
re:
mdadm -D /dev/md0
Если там только второй диск (/dev/sdb), то можно остановить его и удалить массив:
mdadm -S /dev/md0
dd if=/dev/zero of=/dev/sdb bs=1M count=1024
re:
Ну для начала в консоли запустить:
df -h
если в в том, что оно выведет будет строка типа
/dev/md0 4.0G 2.4G 1.4G 64% /
т.е. работает с /dev/md0, значит рейд работает
Так можно проверить все ли получилось как надо.
у меня сейчас другой вопрос возникает как пользоваться локальным хранилищем, точнее как туда запихать образов?
Ставишь XenCenter, а потом – если у тебя уже есть какой-то есть образ, то import, если нет, то New VM и ставишь
Hello,
This is a great article and everything works for me including upgrade to 6.2. I wish, however to use only 2 partitions with md0=boot & OS and MD1 = Storage repo.
How do I change the following command for md0 and md1 only whilst ensuring both arrays assemble and run on reboot?
sed -i ‘s/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/’ init
Thank you.
Yes, you can do it, but Xenserver uses 2-nd partition during upgrade.
why DISK_SIZE-34 and not DISK_SIZE is that minus – 34 ?
Sorry, I forgot exact meaning of 34, but it is maximum for 3rd partition. Partition size greater than DISK_SIZE-34 cause error on my servers
when you cant do boot from secundary disk can we
mount /dev/md0 /mnt
cp /mnt/etc/fstab /etc/fstab
and then reboot and continue fixing sda.
Is there any pitfall ?
Because you need boot from secondary disk for using /dev/md0 as root. When you can’t boot from secondary disk, you also need to change extlinux on the first disk before reboot, for using raid as root partition… But I did not try it.
Я делаю второй раз уже по вашей инструкции, но третий раздел у меня не создает система. Есть только два по 4. Третий я уже создаю сам.
То есть изначально при установке на первый диск на /dev/sda всего два раздела?
В принципе это не страшно, все равно третий раздел можно настроить уже после настройки raid, но все равно странно.
Как оказалось XenServer действительно иногда не создает третий раздел. Я изменил текст статьи с учетом этого
[…] […]
Hello. After:
mkfs.ext3 /dev/md0
mount /dev/md0 /mnt
cp -vxpR / /mnt
I get:
cp: cannot create directory `/mnt/opt’: Read-only file system
cp: cannot create directory `/mnt/cli-rt’: Read-only file system
cp: cannot create directory `/mnt/usr’: Read-only file system
cp: cannot create directory `/mnt/dev’: Read-only file system
cp: cannot create directory `/mnt/media’: Read-only file system
`/Read_Me_First.html’ -> `/mnt/Read_Me_First.html’
cp: cannot create regular file `/mnt/Read_Me_First.html’: Read-only file system
cp: cannot create directory `/mnt/root’: Read-only file system
cp: cannot create directory `/mnt/boot’: Read-only file system
cp: cannot create directory `/mnt/sys’: Read-only file system
cp: cannot create directory `/mnt/srv’: Read-only file system
cp: cannot create directory `/mnt/home’: Read-only file system
cp: cannot create directory `/mnt/etc’: Read-only file system
cp: cannot create directory `/mnt/proc’: Read-only file system
cp: cannot create directory `/mnt/selinux’: Read-only file system
cp: preserving times for `/mnt/’: Read-only file system
Help please! 😀
1. Please check /dev/md0 is really mounted to /mnt
execute the “mount” command and see in output exist string like:
/dev/md0 on /mnt type ext3 (rw)
if this string doesn’t exist, the partition /dev/md0 is not mounted
2. if string exist, but you don’t see “(rw)” at the end, it may be mounted read-only, you must remount it
umount /mnt
mount -o rw /dev/md0 /mnt
3. check state of /dev/md0
cat /proc/mdstat
mdadm -D /dev/md0
Ok. I managed to pass the mkfs.ext3 /dev/md0;mount /dev/md0 /mnt;cp -vxpR / /mnt. A HDD was a piece of crap, I replaced the HDD and worked like a charm. Now I have another issue.
So I install XenServer 6.2, I update it to SP1, i follow your tutorial, and I stop at:
Reboot (IMPORTANT: Set your server to boot from the SECONDARY HDD before booting!)
Fact: I setup the server to boot from the second HDD. It appears that its loading files, etc. and then the white screen appear: Citrix XenServer, and it stops there. What is the problem?
What should I do???
Used the fallback command at boot. I get a kernel panic regarding that its unable to mount /dev/boot or /root…
I have the same problem.. Citrix is booting from second disk, but is hanging on the white loading screen with “Citrix” on it.
Press ESC after white screen have appeared, and watch boot process and errors in it
Hi,
I have followed the instructions above to raid1 for XenServer 6.2. Everything went ok but the “extlinux –raid -i boot/” command get “Segmentation fault” error. The working directory is /mnt of course, and the /dev/md0 is mounted well. I search relevant information but i cannot find it. What can i do?
Thanks your answer!
I answer my question. 🙂
I need a chrooted enviroment. After i mount /dev/md0 to the /mnt i mount the necessary stuffs for chroot:
mount -o bind /proc /mount/point/proc
mount -o bind /dev /mount/point/dev
mount -o bind /dev/pts /mount/point/dev/pts
mount -o bind /sys /mount/point/sys
chroot /mnt and then extlinux -r -i /boot
After the reboot everything works, the root partition is on the softraid array.
after: “Reboot (IMPORTANT: Set your server to boot from the SECONDARY HDD before booting!)”
i have this kernel panic:
http://zapodaj.net/a0c9512b38735.jpg.html
what went wrong?
Did you create new boot image after creating and mounting /dev/md0, before reboot? Boot on first disk, mount /dev/md0 to /mnt and check that in /mnt/boot you have new boot image.
Also check /mnt/boot/extlinux.conf for code “/root=/dev/mnt0”
Hi all,
After doing all steps describes in your post, the reboot of my machine failed in the same case of KK. I got an kernel panic with exactly same errors describe in this picture => http://zapodaj.net/a0c9512b38735.jpg.html
Have you succed the reboot after an particular manipulation or not ?
Thanks 🙂
Sorry, I don’t know, what is causing this problem 🙁
Today I had a try installing XenServer as described in this post. Now I don’t have extra server and install XenServer in VirtualBox. After boot from secondary disk system boots normally. Further all patitions on /dev/sda have included in md raid and on next reboot, system booted from first disk, but on /dev/md0 partition.
For test, physically disconnect first hard disk and boot from second.
I did the steps but when start booting from the second drive the splash screen remains fixed. With ESC nothing happens. I did the installation procedure twice and the second time when I put –create mdadm / dev / md0 –level = 1 –raid-devices = 2 missing / dev / sdb1 tells me “another array by This name is already running”. can be the problem there?
Sorry for my english!
Message about another array in the second installation is a normal behavior, it’s happened because you didn’t clear disks before second install. But it’s not a source of problem. When you boot from second drive, boot started, but can’t finished, I think that this may be due to:
1. new boot image was not created
2. fstab was not changed
So you need to boot from first disk, mount array:
mount /dev/md0 /mnt
and check:
1. in /mnt/boot/ file initrd-…-raid.img exists
2. in file /mnt/etc/fstab root filesystem is changed to /dev/md0
Excuse my english I am foreigner! 🙂
I had the issue where my RAID10 wont autostart after reboot. After many hours searching for fix i found that the module for RAID10 is not active in the kernel of the xenserver. So you add option to this command
mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`and it look something like this:
mkinitrd -v --with=raidx --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r`Where x is the number of the RAID in mu case it is 10. I don’t test it for all RAIDs but I thing it will work just fine.
Hi there!
When you say “NOT configure any local storage” – it’s mean that we dont need specify any storage (sda) for VM during installation? or what does it mean?
When i install XenServer on clean sda i see in XenCenter local storage, it’s right?
But when i start to configure softwate raid, in the step:
cat /mnt/usr/share/syslinux/gptmbr.bin > /dev/sdb
cd /mnt
extlinux –raid -i boot/
i see “Segmentation fault” error.
And when reboot – “Missing OS” on sdb
What i do wrong?
1. Local storage – yes, it’s mean you need not create local storage when you install XenServer. Because after installing XS on one disk, you have got a local storage on this disk (/dev/sda) and after move XS from single disk to RAID, XS still wrote on this disk, instead of RAID. Thus local storage must be created after moving to RAID and you should not see local storage in XenCenter before moving to RAID
2. When you have segmentation fault in extlinux, you don’t boot from secondary disk, but that extlinux behavior is wrong. Please check memory and second disk for errors.
It’s ok, i successfuly setup RAID thank you!
Have another question:
When i apply the hotfixes (XS62E014, XS62ESP1 and XS62ESP1002) after setting up RAID, do i need to update procedure from http://wysotsky.info/?p=145 ???? It’s not new installation it’s just updating via XenCenter console.
I instal that hotfixes and now i check cat /proc/mdstat
And i see this:
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[0] sda2[1]
4193216 blocks [2/2] [UU]
md2 : active raid1 sda3[1]
968372864 blocks [2/1] [_U]
md0 : active raid1 sdb1[0]
4193216 blocks [2/1] [U_]
unused devices:
Is it good?
I try to stop xenserver and disable first disk and it’s boot perfect from second disk, then i disable second disk and boot perfect from first disk.
No, it’s no good. If you use hot fixes you won’t update, except case when you have new version on CD and boot from this CD for update.
As I see in your message, you have unsynchronized RAID. Simply add lost partitions to arrays, e.g:
mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md2 /dev/sdb3
I cannot understand copy all to it. I have tried “cp -vxpR / /mnt” but doesn’t run.
Kindly can someone answer to me?
Format the RAID root disk, mount it at /mnt and copy all to it
mkfs.ext3 /dev/md0
mount /dev/md0 /mnt
cp -vxpR / /mnt
Do you have any error for “mkfs.ext3” or “mount”? Do “cp” cause error or exit without any messages and don’t copy?
Please check that after “mount” you really mounted /dev/md0 into /mnt (via command “df”)
Reboot (IMPORTANT: Set your server to boot from the SECONDARY HDD before booting!)
When XenServer is up again, include /dev/sda in the array with the following commands:
The server I was on was not configurable to boot from individual drives, just the SATA controller or other devices. My solution was to reverse the SATA cables ()disk order) so that the machine booted from md0. Then I had to add /dev/sdb (used to be sda) to the RAID array.
sgdisk --attributes=1:set:2 /dev/sdbsgdisk --typecode=1:fd00 /dev/sdb
sgdisk --typecode=2:fd00 /dev/sdb
sgdisk --typecode=3:fd00 /dev/sdb
mdadm -a /dev/md0 /dev/sdb1
mdadm -a /dev/md1 /dev/sdb2
mdadm -a /dev/md2 /dev/sdb3
I don’t know is that solution good or bad, because I don’t know, how to mdadm saves information about disks in raid. If you BIOS doesn’t allow you to change boot order, I recommend to boot from live USB/CD, like SystemRescueCD and add volumes to raid from it.
Thank you for this guide. It took me quite a few tries to get the sw raid going.
I too had trouble in booting from the 2nd hard disk so I went the Knoppix Live DVD way and syncronized the disks there.
Here are my additional notes:
1) /dev/md0 device is indeed created during XenServer 6.2.0 install so I used md1, md2 and md3 as my device names.
2) I didn’t need to reboot the machine after creating the partitions on the 2nd disk with sgdisk.
3) I used the following commands to assemble the raid after booting up Knoppix:
# mdadm –assemble –run /dev/md1 /dev/sdb1
# mdadm –assemble –run /dev/md2 /dev/sdb2
# mdadm –assemble –run /dev/md3 /dev/sdb3
(Then add the partitions of the 1st disk according to your guide.)
4) Reminder of writing gptmbr.bin to the 1st disk.
Even though /dev/md0 exists after plain XenServer install, you can still use mdraid device /dev/md0 for /dev/sda1 and /dev/sdb1. My bad.
eveorything is ok, just i have problem with md0
[root@localhost ~]# mdadm -a /dev/md0 /dev/sda1
mdadm: Cannot open /dev/sda1: Device or resource bus
i try to remove dmraid by using yum remove dmraid!! but i still got the problem
any help? thx
At first, check which device you boot from (df -h). If it’s not from /dev/sda, also check that you don’t have other MD RAID devices, which use /dev/sda1 (cat /proc/mdstat)
i get this output
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 4.0G 1.9G 2.0G 49% /
none 373M 24K 373M 1% /dev/shm
/opt/xensource/packages/iso/XenCenter.iso
52M 52M 0 100% /var/xen/xc-install
how to change it ?
You booted from first disk, not from second. Did you really set second disk for boot, as described in this article?
“Reboot (IMPORTANT: Set your server to boot from the SECONDARY HDD before booting!)”
Добрый день, подскажите, сделал все по вашей инструкции, все получилось, но после перезагрузки отваливается md2, причем в списке девайсов (/dev/) он присутствует, но не отзывается, хен центр говорит что локальное хранилище потеряно, при проверке говорит:
[root@XenServer ~]# mdadm -D /dev/md2
mdadm: cannot open /dev/md2: No such file or directory
По команде: mdadm –create /dev/md2 –level=1 –raid-devices=2 missing /dev/sdb3, стартует, далее добавляет второй диск, синхронизация на пол дня (диски по 3 Тб) и потом ребут и все заново. Думал что дело в этом sed -i ‘s/raidautorun \/dev\/md0/raidautorun \/dev\/md0\nraidautorun \/dev\/md1\nraidautorun \/dev\/md2/’ init, но посмотрел инит, там прописан авто старт всех трех разделов, не знаю уже куда копать. Буду очень признателен за помощь.
А после синхронизации всех массивов выполнялась команда
mdadm –detail –scan >> /etc/mdadm.conf
?
После этого в файле /etc/mdadm.conf должны быть как минимум три строки вида:
ARRAY /dev/md0 level=raid1 num-devices=2 …
ARRAY /dev/md2 level=raid1 num-devices=2 …
ARRAY /dev/md1 level=raid1 num-devices=2 …
Собственно отсутствие строки с ARRAY /dev/md2 и может быть причиной того, что он не стартует
Спасибо! Посмотрел файл, действительно нет записи о md2, поставил синхронизацию, как закончится, добавлю команду, надеюсь поможет.)
Благодарю, все получилось.
Вопрос, буквально 5 минут назад вышло обновление XS62ESP1008, какие рекомендации перед установкой? Не слетит ли рейд после апдейта?
Благодарю, все получилось.
Вопрос, буквально 5 минут назад вышло обновление XS62ESP1008, какие рекомендации перед установкой? Не слетит ли рейд после апдейта?
Не должно – ядро не меняется, поэтому с рейдом проблем не будет
Hi,
Can I start using XenServer with one disk, after some time connect second drive, set up md and save all the data? I do not understand if md can be created without deletion of data on the disk.
Yes, you can add second disk after any period of time. After adding second disk, you need to create md partitions on it (RAID1 on one disk) and copy all data from first disk to RAID. Only after boot from RAID and check that boot is normal, you must change partitions on first disk and add it to RAID.
Thank you very much. I thought the same that after some time with one disk (without md):
1. You can connect and prepare md for second disk
2. copy data from the first disk to second’s array (dd). At this stage you will have raid1 with only disk.
3. then prepare first disk for raid and synchronise.
Guys, I can’t see Remik’s post. This link is not working http://blog.codeaddict.org/?p=5#comment-599 .
I’m almost done in configuring raid 1 for xen and facing the same issue:
# mdadm -a /dev/md2 /dev/sda3
mdadm: Cannot open /dev/sda3: Device or resource busy
I tried to remove third partition on sda, then recreate, use partx -a /dev/sda. It can’t help.
SDA:
Number Start (sector) End (sector) Size Code Name
1 2048 8388641 4.0 GiB FD00
2 8390656 16777249 4.0 GiB FD00
3 16779264 312581741 141.0 GiB 0700
SDB:
1 2048 8388641 4.0 GiB FD00
2 8390656 16777249 4.0 GiB FD00
3 16779264 312581741 141.0 GiB FD00
MDSTAT:
md1 : active raid1 sda2[0] sdb2[1]
4193216 blocks [2/2] [UU]
md2 : active raid1 sdb3[1]
147901120 blocks [2/1] [_U]
md0 : active raid1 sda1[0] sdb1[1]
4193216 blocks [2/2] [UU]
BOOTED:
/dev/sdb
What about partition table, is md looks healthy? I’m not able to find on Google if this is first hard disk issue or md itself. Could you comment on this?
# fdisk -l
WARNING: GPT (GUID Partition Table) detected on ‘/dev/sda’! The util fdisk doesn’t support GPT. Use GNU Parted.
Disk /dev/sda: 200.0 GB, 200049647616 bytes
256 heads, 63 sectors/track, 24226 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 24227 195360983+ ee EFI GPT
WARNING: GPT (GUID Partition Table) detected on ‘/dev/sdb’! The util fdisk doesn’t support GPT. Use GNU Parted.
Disk /dev/sdb: 160.0 GB, 160041885696 bytes
256 heads, 63 sectors/track, 19381 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 19382 156290903+ ee EFI GPT
Disk /dev/md0: 4293 MB, 4293853184 bytes
2 heads, 4 sectors/track, 1048304 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
This doesn’t look like a partition table
Probably you selected the wrong device.
Device Boot Start End Blocks Id System
/dev/md0p1 ? 403062366 457171319 216435814+ 7 HPFS/NTFS
Partition 1 does not end on cylinder boundary.
/dev/md0p2 ? 409002618 653185122 976730017 16 Hidden FAT16
Partition 2 does not end on cylinder boundary.
/dev/md0p3 ? 1 1 0 6f Unknown
Partition 3 does not end on cylinder boundary.
/dev/md0p4 6275073 121817047 462167897 0 Empty
Partition 4 does not end on cylinder boundary.
Partition table entries are not in disk order
Disk /dev/md2: 151.4 GB, 151450746880 bytes
2 heads, 4 sectors/track, 36975280 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md2 doesn’t contain a valid partition table
Disk /dev/md1: 4293 MB, 4293853184 bytes
2 heads, 4 sectors/track, 1048304 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 doesn’t contain a valid partition table
I created /dev/sda4 and added to /dev/md2. It solved the issue.
Just note that /dev/sda4 block size must be the same as /dev/sdb3.
Hi,
I installed Xen dom again and found the correct way regarding:
mdadm: Cannot open /dev/sda3: Device or resource busy
Instruction (if you don’t need data from storage /dev/sda3 – you can backup always):
1) temporary remove raidautomount related with md2 device from /root/initrd-raid/init
2) remove this partition : sgdisk –delete=3 /dev/sda
3) sgdisk -p /dev/sda
4) reboot
After reboot:
1) recreate your storage (/dev/sda3)
sgdisk –new=3:16779264:DISK_SIZE-34 /dev/sda
sgdisk –typecode=3:fd00
2) update partition table : partx -a /dev/sda
3) add your device to raid md2 successfully:
mdadm -a /dev/md2 /dev/sda3
4) restore raidautomount md2 in /root/initrd-raid/init
5) you will see the progress about recovery using cat /proc/mdstat
Hi all,
I’ve just done this with XenServer 6.5 as a test and I got it to work. You have to take some extra steps as mdraid support is not loaded on boot by default in the install, hence you cannot start you md-devices.
You can do a simple modprobe raid1 (or whatever you preferences, you could add raid345 as well) to enable your md devices. Another thing I found is that this version of mdadm create the devices nodes in a different way should they be created with the default settings (auto=yes). To prevent this, create the devices as describbed in this tutorial, but before copieing the files, run the mdadm scan (eg. mdadm -Ds >> /etc/mdadm.conf). This way your file should also be includes in the mkinitrd (last line of the output from mkinitrd should say something about /etc/mdadm.conf). Then your md-devices will be created as md[0,1,2], otherwise they might be called /dev/md125 and /dev/md/0_0 (which is a symlink to the former).
Also _don’t_ chroot into the mountpoint before extracting the kernel! If you extract from the chrooted location, the mdraid driver will not be included and you will get some nice kernel panics when trying to boot :).
Last problem I had was that the xen volume group was not removed after changing the partition table, so I simply moved it to the /dev/md2
pvcreate /dev/md2
vgextend vgname /dev/md2
pvmove /dev/sda3 /dev/md2
vgreduce vgname /dev/sda3
After this, I could successfully add /dev/sda3 to /dev/md2
When creating /dev/md0, /dev/md1 and /dev/md2 you need to use option –metadata=0.90.
The default metadata version 1.2 is not compatible with syslinux and resulted in “missing os”
when trying to boot.
(The previous comment is regarding XenServer 6.5)
Editing the init file inside initrd-3.10-xen.img is not necessary anymore with XenServer 6.5.
The init file does not contain any raidautorun lines by default but instead uses mdadm.
Спасибо за статью!
Возможно-ли изменить размер корневого раздела md0? У меня xenserver 6.2 и корневой раздел заполнен на 100%. Как мне его увеличить за счет другого раздела?
[root@xenserver]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md0 4127360 3992952 0 100% /
none 381124 32 381092 1% /dev/shm
/opt/xensource/packages/iso/XenCenter.iso
52252 52252 0 100% /var/xen/xc-install
/dev/mapper/XSLocalEXT–e03f5334–129b–e547–8ee5–8aeb62271814-e03f5334–129b–e547–8ee5–8aeb62271814
953165724 525490856 379256872 59% /var/run/sr-mount/e03f5334-129b-e547-8ee5-8aeb62271814
Спасибо!
Здравствуйте,
Есть два варианта – увеличивать и не увеличивать 🙂
Увеличить можно только за счет второго раздела, который предназначен для обновления (еще 4гб), но я бы не советовал – после этого обновится на новую версию будет сложнее. Проще почистить существующий раздел – за счет старых логов и файлов от уже установленных обновлений около 1гб можно освободить… по крйней мере у меня на серверах где-то так и есть – свободно 0.8 – 1.2 гб.
Ну а если проблемы с обновлениями на новую версию не пугают, то загрузится с чего-нибудь типа gparted и увеличить первый раздел за счет второго… правда потом еще нужно будет увеличивать md0
Спасибо за ответ! Подскажите пожалуйста от куда можно удалить файлы уже установленных обновлений?
/var/patch – при этом каталог applied лучше не трогать, там только описания патчей
еще /var/log советую глянуть, в какой-то из версий XenServer был глюк – старые логи не удалялись
[…] Kolejna strona na jaką trafiłem to http://wysotsky.info/?p=106 […]
This may be resurrecting the dead, but my installation does not have the third partition, so upon creating I receive this error…
Could not create partition 2 from 16779264 to 0
Error encountered; not saving changes.
I am not sure where to go with this at the moment. Any help would be appreciated if this happens
There are two questions:
1. Are disk sizes identical (identical disk manufacturer)?
2. Which command generates error?
Hola buenas tardes,
Hay que ejecutar todos los comandos del post para un xenserver 6.2? O hay que parara cuando empieza a poner para xenserver 6.5?
Un saludo