Tuesday, March 1, 2011

how to save bandwidth in git pull when you do a make world on Xen source

Those who have compiled xen from source must have faced this annoying situation.
When you did a make world then you see some output like this
install -m0644 -p /usr/src/xen-4.0.1/xen/xen-syms //boot/xen-syms-4.0.1
make[2]: Leaving directory `/usr/src/xen-4.0.1/xen'
make[1]: Leaving directory `/usr/src/xen-4.0.1/xen'
for i in  linux-2.6-pvops  ; do make $i-install || exit 1; done
make[1]: Entering directory `/usr/src/xen-4.0.1'
make -f buildconfigs/mk.linux-2.6-pvops build
make[2]: Entering directory `/usr/src/xen-4.0.1'
set -ex; \
        if ! [ -d linux-2.6-pvops.git ]; then \
                rm -rf linux-2.6-pvops.git linux-2.6-pvops.git.tmp; \
                mkdir linux-2.6-pvops.git.tmp; rmdir linux-2.6-pvops.git.tmp; \
                git clone -o xen -n git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-pvops.git.tmp; \
                (cd linux-2.6-pvops.git.tmp; git checkout -b xen/stable-2.6.32.x xen/xen/stable-2.6.32.x ); \
                mv linux-2.6-pvops.git.tmp linux-2.6-pvops.git; \
        fi
+ [ -d linux-2.6-pvops.git ]
+ rm -rf linux-2.6-pvops.git linux-2.6-pvops.git.tmp
+ mkdir linux-2.6-pvops.git.tmp
+ rmdir linux-2.6-pvops.git.tmp
+ git clone -o xen -n git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-pvops.git.tmp
Initialized empty Git repository in /usr/src/xen-4.0.1/linux-2.6-pvops.git.tmp/.git/
remote: Counting objects: 1941611, done.
remote: Compressing objects: 100% (319164/319164), done.
The above really takes long time because it is basically cloning the git repository from jeremy's tree to your local machine and you have to wait until the pull completes.
The size is approximately 875 Mb so you will have to wait till that much time.
After digging this problem for some time I finally found a remedy for the same.

You need to fix the Makefile for the same.



When you untar xen-4.0.1.tar then you see
/usr/src/xen-4.0.1  
(location /usr/src can change depending upon where you untarred it at your laptop I prefer /usr/src for all source builds)

Now before you begin any sort of compilation etc in your home directory
pull the git tree as above mentioned (since we will remove the lines from Makefile itself)

so have at some safe location
cd $HOME
git clone -o xen -n git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-pvops.git.tmp;
At this point you should be able to see $HOME/linux-2.6-pvops.git.tmp we will use this again and again.

Now /usr/src/xen-4.0.1/buildconfigs directory open file
src.git-clone and add a hash # at the beginning of following lines
               rm -rf $(LINUX_SRCDIR) $(LINUX_SRCDIR).tmp; \
               mkdir $(LINUX_SRCDIR).tmp; rmdir $(LINUX_SRCDIR).tmp; \
               $(GIT) clone -o $(XEN_GIT_ORIGIN) -n $(XEN_LINUX_GIT_URL) $(LINUX_SRCDIR).tmp; \
               (cd $(LINUX_SRCDIR).tmp; git checkout -b $(XEN_LINUX_GIT_LOCALBRANCH) $(XEN_LINUX_GITREV) ); \
               mv $(LINUX_SRCDIR).tmp $(LINUX_SRCDIR); \

so that they look as

#               rm -rf $(LINUX_SRCDIR) $(LINUX_SRCDIR).tmp; \
#               mkdir $(LINUX_SRCDIR).tmp; rmdir $(LINUX_SRCDIR).tmp; \
#               $(GIT) clone -o $(XEN_GIT_ORIGIN) -n $(XEN_LINUX_GIT_URL) $(LINUX_SRCDIR).tmp; \
#               (cd $(LINUX_SRCDIR).tmp; git checkout -b $(XEN_LINUX_GIT_LOCALBRANCH) $(XEN_LINUX_GITREV) ); \
#               mv $(LINUX_SRCDIR).tmp $(LINUX_SRCDIR); \
The above lines in src.git-clone are responsible for this time consuming pull.
Now when you need to compile again and again xen-tools you can simply extract
the xen-4.0.1.tar.gz
and
copy the $HOME/linux-2.6-pvops.git.tmp to /usr/src/xen-4.0.1/

now you can begin the compile process again.

make install xen-tools
and it should work fine.

Now here I would like to add some more points that I had also commented out all lines that begin with rm in file
/xen-4.0.1/buildconfigs/mk.linux-2.6-common 
since to uninstall the tools I had used
make uninstall and make clean command which is available as clear from the Makefile.
But then I again copied the

$HOME/linux-2.6-pvops.git.tmp to /usr/src/xen-4.0.1 and then again
make xen
make tools
make stubdom
make install xen
make install xen-tools

So your point is only the src.git-clone file.

No comments: