Workshops for NTFS and Linux disk cloning and partition resizing
Workshop 1: clone NTFS and Linux partitions into a smaller disk
Workshop on disk partitions cloning and resizing
I am chronicling here the use cases I have actually performed successfully while experimenting with different scenarios of hard drive cloning and resizing. This is simply the result of many hours and attempts to get Windows and Linux to co-exist on the same machine. There must be numerous permutations in how these things can be done, but I hope the basics I am detailing here still apply in solving more complex partitioning problems.
Do you notice that no matter how recently you downloaded these Linux distribution CDs, you always find around 200 updates waiting for you post installation ? Is it not a pain to have to do this on every new disk ? On top of that, would you not have done certain customisation, personalisation etc. and wanted to have the exact same settings on each new clone (for both Windows and Linux) ? This is the case I make for disk cloning. You build yourself a master disk of Windows and Linux installed and fully updated and customised, then you simply clone the disk into new ones. It is straight forward if they are of identical size, geometry and dimensions. But that is no fun. The technique I developed in these workshops require a certain amount of hand holding, but will allow you to clone to targets of different size, geometry or vendor.
Why not use off-the-shelf software like Norton Ghost or Clonezilla etc. ? Well, I have. I found that these software are pretty good for straight forward cases, but then again, for these cases, I can simply use a Live CD and dd to achieve the same thing. It is not comforting for a DIY nut like me to hand control over to some automation tool, only to find out in the end that it does not work without any means to discover what has gone wrong. I once had to sit through a Clonezilla session to clone and resize a NTFS partition, had to answer loads of questions which require the kind of knowledge you would need to do it manually in the first place, only to discover that the newly cloned and resized partition refused to boot.
Is it also my imagination, or is it true, that the more automated and flashy the tool is, the more likely it is to barf at overlapping partitions ? These overlapping partitions happened to me recently after I installed Ubuntu 8.10, and after that one partitioning tool after another (both commercial and open source) gave up on the disk, until the point where sfdisk was the only tool which still managed to display the partition table.
People usually warn you that you have to be extremely careful when messing around with disk partitions, and that you might end up with an unusable disk and lose all your precious data. Well, I find that with enough preparations, most ballsup can be reverted. Granted, you have to keep your wits about you while doing this, as certain commands require that you pass in the source then the destination, while others need the opposite. If you get them the wrong way round then there is nothing in these workshops that will help you. This is why you should only do this once you have made backups of all your valuable data and while you are completely sober.
- There are particular surgical procedures which are simply too hair raising, or too technical to detail here, for fear of disrupting the flow of information, or distracting you from the point I am trying to put across. One example of such procedures is the manual editing of the offset to the boot sector in an NTFS partition, which is described in excellent details in the article titled "Forking a XP-installation" in the Reference section. I would urge you to read up on these articles if and when the need arises, not as a pre-requisite. The trick here is to do it in such a way that you do not need to perform such procedures
- Playing with the partition table is a very risky business, so only do it as the last resort (or if you have a pre-disposition to extreme pains and pleasures, aka adrenaline junkies in certain circles)
- Backup everything valuable, as chances are you could lose everything
- the procedures I describe have been tested with Windows XP. Your mileage will vary if you have Windows Vista. If in doubt, do not do it
- As usual, just because it works for me, it does not mean it works for you. I suggest you do not copy and paste the commands here line for line, but try to understand the rationale behind each command, then extrapolate to apply to your particular circumstances (this unfortunately requires a certain amount of Unix knowledge, but I do not want to cover the basics here where there are plenty of online resources that can help you
Please pay attention to the terminologies used below to avoid confusion between what I say and what you think I say. It could be the difference between jubilantly/triumphantly playing the air guitar and utter, suicidal desperation while staring at your totally wiped MP3 collection
- disk: the block device representing your whole disk, not the individual partitions e.g. /dev/sda, /dev/sdb, /dev/sdc etc. in Unix speak
- partition: the partition in which your operating system is installed e.g. /dev/sda1, /dev/sda2, /dev/sdb1 etc. in Unix speak
- Master Boot Record (MBR): the block of data on sector 0 of a disk which stores specific bootstrap code to jump to the partition where the selected OS resides
- Primary Partition table: the list of 16 byte entries which stores the offsets of the four primary partitions. This block is found immediately after the MBR block. You can see from the figure below that the first entry will be pointing to /dev/sda1, then the next points to /dev/sda2 etc.
The figure is not to scale. The purpose is to show you where things are organised relative to each other. The important point I try to highlight here is the blocks are not necessarily contiguous, as is evidence with the interleaved gray regions of unallocated disk. In practice, they might be closer to each other than shown, but will still not be contiguous due to rounding to the next nearest sector/head/cylinder boundary when advancing from one partition to the next
Explication and rationale
The most straightforward case of disk cloning will involve simply boot up your machine using the Ubuntu Live CD and then run the command below in a terminal window:
dd if=/dev/sda of=/dev/sdb bs=32M
This will only work if your drives (sda and sdb) are of similar size and symmetry. This technique is however wasteful because dd copies the contents of the source disk to the target disk byte by byte, even if each of your partitions (sda1 and sda2) is only partially filled. If the target disk is smaller than the source disk, the second partition will be truncated on the target drive, rendering it useless.
You can use tar or rsync to copy the contents of the partitions (this can only be done once you have mounted the partitions into their respective file systems). This will guarantee that only the contents of the partitions are copied, but it does not guarantee that the offsets stored in each partition will get re-calculated, and this means neither partition will boot. Both NTFS and ext3 (Linux file system) store certain information about the partition they are in, specially how far they are from sector 0.
NTFS makes use of the whole partition i.e. it has markers at the beginning, at the end, and even in the middle of the partition. This is the reason why you can not simply shrink the partition at the partition table level and expect NTFS to simply make use of the newly allocated size
ext2 and ext3 (which is ext2 with a journal) have superblocks scattered throughout the partition. These superblocks are meant to be for internal recovery in case of bad sectors. As is in the case of NTFS, you can not just shrink/enlarge the partition at the partition table level. You have to make the native file system (NTFS/EXT3) aware of the new size too. I find that it is usually much simpler to leave the NTFS partition where it is i.e. keep the offset to /dev/sda1 the same, then resize it to accomodate the Ext3 partition. It is a relatively simpler operation to relocate Ext3, but then again I might be a bit biased
I am sure you know this already, but it is worth re-iterating. The partition (e.g. /dev/sda1) is where you store the files. This area has a pointer to it in the Primary Partition Table. If you lose the pointer, you simply do not know where the start of the partition is, but it does not mean your data is gone. There are tools which can search your disk and look for OS signatures to try to guess this pointer if you happen to lose it, but it is best to make sure you do not lose the pointer in the first place. For disk cloning, it does not matter too much as the target disk has no partition on it to start with. However, if you ever want to play around with these pointers, it is best to dump your partition table to some place safe. I will detail that in another workshop
The purpose of cloning and resizing at the same time is to allocate the partition table on the target disk, then use native utilities to take care of the contents in each partition. There are extra things to do to make them each bootable. You will find these details in the actual workshop.
Workshop 1: clone NTFS and Linux partitions into a smaller disk