Virge - high level goals

There has been a lot of changes to my life recently, however there is one thing still in my mind. And it has been there for a while, but I never got any time to sit down, write a summary and start doing that. I use the blog mainly to create my opinion on various topics so I've decided to brainstorm it here.

With my friend we like technology, free technology. I always get quite quickly excited about what is possible to achieve with it. Gentoo, Ubuntu, GNOME, Apache foundation just to name a few. And I like to play with that. For a year or two we've been playing with gentoo Linux and Xen and we had a great time. Now there is one physical box running dozen of virtual boxes separated into various network silos, backuped, each with different functionality, monitored and there are several scripts that makes deployment of new machines quite easy. I take it as a proof of concept.

Idea is to have a Linux box where one can deploy virtual appliances with ease in a secure environment with advanced features for network, file systems, software packages and easy to use admin console. We would like to use Gentoo as (despite its recent problems) it is one of most advanced distributions of Linux out there, EVMS (for reliable data storage with possibility to do cluster EVMS), XEN to power virtualization and possibly www console for managing the machines, machines should use binary packages for quick setup. Usage? Home appliances, ISP machines, Datacenter in just one box. I short: the ultimate linux machine ;).

I'm aware of others doing the same (rPath,Redhat,VMware), but as I said, I like the technology so this is our try. Virge is simply Virtualized Gentoo.

Steps:

• create LiveUSB with latest Gentoo2007.0 (updated), that would install Virge on the new box (including the Xen enabled kernel, EVMS setup, some appliances)
• create admin console using Django+Python+libvirt for managing boxes
• merge this application into portage

During the POF we've ran into some issues with portage, for example we are aware that it is not easy to do binary packages with different USE flags (gentoo feature). But I'll leave those to be solved during the FUN.

Any help is welcomed of course.

Gentoo two cents

So after about a minute I replied to this post and suggested Daniel to return back to Gentoo I read his second post where he explains in details why he does not want to (or can't).

However I still have a feeling there is a lack of leadership in the Gentoo community. Community needs individuals with ideas, and because the way communities work at the moment I mean individuals who have ideas and make them reality. This is how I think of Gentoo, vision of technically advanced Linux distribution with features not seen anywhere else that became true.

Whether lack of leadership is the source of small innovation in gentoo recently I do not know. But as long as I know Gentoo it did not changed very much (that is not necessarily anything wrong). There is excellent idea of ports (borrowed from other OS), super package manager "emerge", technically one of the best distros out there. One would say THE distro.

But others have their point as well, Fedora being sandbox for RedHat (in good way of course), Ubuntu focusing on end users and laptops, Novell playing corporate games and many others.

As a user I feel there is no clear vision of where is Gentoo going to. "Meta" does not tell me much.

My two cents here would be:
Make servers the prime arena for Gentoo and get in touch with some big company. Aim for Google for example. This would bring commercial support. Many times I see enthusiastic admins running gentoo on a spare desktop box, leaving the big and interesting hardware for RedHat, Novell, and other players with "support" sticker included. I'm not aware of any commercial company providing large scale support for Gentoo linuxes. I wish LSB was more widely supported by every player in the field. Now we have every other Linux company tinkering with MS instead of pulling one string!

To do that several things need to be done

1. create package format specification (yet keep emerge as a reference implementation), this will lead to faster development of other packaging systems (faster, with better features) that can eventually replace emerge in some time
2. finish support for handling binary packages and different USE flags. as far as I know currently broken
   
3. create tools for managing vast amount of servers (different profiles compiled on one box,...), as Jonathan Shwartz says "The Network is the Computer"
   
4. support virtualization
5. perhaps run emerge as a daemon, machineXX will ask builder machine (emerge daemon running there) : "hey I'm using this profile and I would like to install Apache with those USE flags" and get the binary packages from the builder. But I think there is a long way to get there.
   

Just my two cents.

Ubuntu lovely, Gentoo cool

Since last month I'm using Ubuntu on my laptop. Beautiful distro. Fast boot, working sleep mode and a solid desktop experience. Reasonable amount of packages, yet installing some newer applications might get bit tricky as only security and update patches are back ported. Support is on very good level and it has good momentum. Compared to Vista and Xp is offers simple but very intuitive user experience across the whole desktop. Personally all I had to do to was to enable activation of windows by moving mouse button over them. You just then hover your mouse over you music player and scroll the middle mouse button to raise/lower the volume. Use same technique for seeking movies and it works in many other areas as well. I do not know if there is any real reason why this is not default. Put it in one sentece I would say Ubuntu is just great for end users.
But I miss Gentoo. Gentoo offers unmatchable flexibility in configuration and system maintenance (at least as far as I know). Powerful tools and a large catalog of applications ready just to be plugged into the system. Yet, still very clean design and philosophy. Ideal distribution for developers, advanced system administrators and those willing to try. Unfortunately Daniel Robbins has left Gentoo a while ago and it seems, although there are many great people involved in the project, not many new great features came out of it then. Still it is one of the best advanced Linux distros out there.

slashtime - handy and powerful

Due to nature of my work, time to time I need to know the current time in different timezones. I was scared when I saw various applications on my colleagues desktops consuming megabytes of memory, slowing down startup of laptops and the money its costs. Luckily for Unix likers there is simple perl script called slashtime. Get it here, save it to /usr/local/bin and make it executable. Then edit ~/.tzlist (sample file is on the webpage as well) to suit your needs and then anytime you need to know:Small but handy and powerful.

more on gentoo

This is reply to this post by Alexandre Buisse.

Hi, this is really good piece of vision. You are right that there is lack of direction. I have to say I agree to most of what you've written, let me get through a bit other point of view on gentoo.

I love gentoo, but I spend to much of my time just building it. What I miss is some rapid deployment of gentoo. Not everybody enjoys compiling from source especially when you have multiple gentoo boxes deployed and mostly you compile the same stuff all the time. Hence focus on binary packages is important. At the moment with my friend we built some scripts and processes around gentoo that allows us to have two compiling machines and many binary-only deployed machines. But it is not polished and cumbersome in many ways (and I'm sure others have done it better). For example we hit hard problems when we needed different flags per machine. No fun there.

This problem led me back to gentoo mailing lists and the discovery of paludis and all the not so cool status of current gentoo development at the moment. From my point of view, there are some devs that are not behaving accordingly to others view of nice a polite behavior. Ok. But I see they are also trying to do some nice things like write alternative to portage and fix current issues gentoo have. Btw complete rewrite of apps from zero once in a time is a good practice. I couldn't even finish reading the thread based on which Daniel decided to leave again, to me that thread was about arguing about small things. But lets skip this my incomplete picture of things.

Another thing I miss is virtualization in gentoo. Would it be possible to somehow project gentoo that some core system (lets say XEN or KVM enabled) exits and then there are binary stage4s (again XEN or KVM enabled) ready to be deployed with customized setup? This means webserver, intranet server ( apache+wiki+ldap ready), backend server (samba+printing+LDAP+MAIL......), you name it. I guess here it is almost perfectly aligned with overlays.

gentoo domination postponed

Gentoo has been my favorite distribution for some time (found my first post on gentoo forums around 18th March 2002). Before that I was switching between RedHat, Suse and Mandrake based on the current level of applications that came with it. Later, I started to compile packages from source (because the most recent version was not in the repository yet). This had led me to the LFS (Linux From Scratch) and I learned a lot there. I'm not sure about current status of LFS, but it was sometimes pain to have it up to date and not break when something basic needed update. Then I saw post from Daniel Robbins (the father of gentoo) and quickly realized Gentoo is all I need.
Great support, innovative, excellent documentation and perfect source of skills and knowledge for all Unix admins. Gentoo brings some BSD ideas (ports) to Linux. User compiles the whole system from scratch with possibility to customize the system on a level that I didn't see in any other distribution yet all that in fully automated form.
Gentoo is great for servers. Through past few years I've learned how to maintain Linux Desktop as well, but there are other distributions that focus on End user and they do their work much better (e.g. Ubuntu Linux). One thing I think is not finished in Gentoo is the handling of binary packages. Binary packages becomes handy when one have to maintain multiple servers. Why to compile the same package on all servers and waste CPU time (lets cool the planet!) on this redundant task? Sometimes (mainly when we talk about security) it is desired to have gcc removed and perform update of machine via pre-compiled packages. Emerge can handle this (sharing the PKGDIR over network) but with flaws.

• metada - when central machine creates new package and store it on the shared drive, there is no way to notify slave machines to update its metada file about packages. I believe we solved this by also installing http server on the core machine and using PORTAGE_BINHOST variable on the slave machines. We still utilize shared drive for /usr/portage and /srv/packages as PKGDIR (eliminating the download of binary packages).
  
• USE flags - this seems to be bigger issue we haven't find solution yet. Generally, packages can use this variable to determine additional functionality provided by the package (i.e. linking to Qt, Kde and not GNOME and vice versa). Sometimes it is handy to compile apache with LDAP USE flag and sometimes not. However both binary packages have very same name (e.g. apache-2.0.58-r2.tbz2) and what USE flags were used is determined after the package was downloaded and unpacked (too late) and then update is failing on machines based on their package.use settings.

I haven't found any clever idea how to solve the issue yet. Perhaps different naming convention for use flags in binary packages is needed and server daemon running on core machine. Client machine would then ask the Core machine for binary package with specific USE flags and Core would either look it up in its repository or compile. This would rock, really. Gentoo domination would be unstoppable, until then it is postponed.

Today when I was surfing the web to see whether paludis (C++ replacement for emerge) is handling things differently I discovered that Daniel Robbins is coming back to gentoo. Congratulations and welcome back, you've made my life a lot easier (maybe not easier, but definitely more enjoyable ). Rock on!