http://magazine.redhat.com/2007/08/07/fedora-7-community-remix/
Fedora 7: Community remix
by Mike McGrath
One of the popular features of Fedora 7 is the ability to remix Fedora and build your own custom version. Now don't get me wrong–building a new distribution is nice. But what about those who wish to create their own Fedora-based project? How do you grow a complete community in an enterprise environment or in the general public? Thanks to some of the lesser-known features, anyone can use the exact same tools that make Fedora, well… Fedora.
Fedora development is generally US-based and, as a result, must follow all applicable US law regarding intellectual property, copyrights, and patents. Additionally, Fedora prides itself on being free in every sense of the word. Not only are our developers free but we encourage our users to be free as well. And one part of being free is allowing our code to be used by others–even if they want to build something less free. Revisor and Pungi are perfect examples of Fedora allowing it's user base to customize Fedora simply and freely.
With Revisor, users can easily create any spin of Fedora imaginable. Add codecs for a media center, strip out games, or add additional training applications to build a specific corporate version of Fedora. This new custom version, however, isn't useful for very long without maintainence and distribution.
It's not only possible to create a custom operating system based off of Fedora's code base, but it's also possible to copy the Fedora Project's community methods and best practices. From source to build system to distribution and communication, everything the Fedora Project needs is 100% open source. Users and developers can tweak the operating system and leave everything else alone. Enterprises can create RPMs or rolll their own custom operating system–with everyone using the exact same methods as the Fedora Project.
The first thing someone needs to create their own community is a home. Typically, this home is a simple web page. In Fedora we have a MoinMoin wiki. The wiki provides a central place to collect all Fedora information. It houses packaging guidelines, download locations, and sometimes provides work-flow. It also documents Fedora's governance, provides information about various teams or groups, and gives developers a place to share a bit about themselves. More than anything, though, it is Fedora's home and gives the community a opportunity to speak up and say "we're here."
The next essential item is communication. Initially, straight email is a good way to get things going. When the community grows (even to just a few people), something more will be needed; most (like the Fedora group) will turn to a mailing list or even a forum. When people think about lists and open source software, they think mailman. Most communities try to separate email discussion into different relevant subjects. Two lists are typically sufficient, one for development and one for users. The number of lists may scale quickly, depending on the size and scope of your project. At present, Fedora has around 70 lists.
The Fedora Project also uses IRC for more immediate communications. It is not necessary to host IRC directly–in fact the Fedora community relies on Freenode. Other communities convene there and it's valuable (and convenient) to have all of these open source people in the same place. Since telephone communication is not always possible or desirable, IRC provides a great inexpensive alternative for real-time communication. Again, it's appropriate to create separate channels based on topic or need. Fedora has many different channels, with #fedora, #fedora-devel, and #fedora-meeting being among the most popular.
Once everyone has a place to go for information on the community and can communicate openly with one another, then it's time to get down to business. Most communities will wish to customize Fedora in some way. To do that, they'll need to compile RPMs. Fortunately, with Fedora 7 every part of the build process is 100% open and available for others to duplicate.
Fedora is built from RPMs. The idea behind an RPM is that a packager takes pristine source from upstream, adds patches, creates a spec file, and builds a binary RPM to be distributed. RPMs are aware of dependencies for both the build process and at install time. Building RPMs is a tricky and time-consuming process. Fedora has many package maintainers to help do the job right. Fortunately all of Fedora's official RPMs are maintained by Fedora so others don't have to worry about keeping them up-to-date.
The easiest way to determine if a developer's custom RPM is compatible with Fedora is to actually get it accepted by Fedora. Becoming a Fedora contributor is a well-documented process, but most people needing to create their own community based on Fedora probably can't get their packages into the official Fedora operating system. This is typically because of legal reasons. However, just because something is illegal in the United States doesn't mean its illegal in other areas of the world. This situation is a perfect use-case for customizing and creating a localized version of Fedora.
Even if a package cannot make it into the official Fedora, it's still a good idea to follow the Fedora Packaging Guidelines. These guidelines are open and all Fedora packages are subject to it. An interesting side effect of this is that many Red Hat® Enterprise Linux® packages are also packaged using these guidelines. They are an excellent guide for anyone packaging something for an operating system in the Red Hat family. They also generally translate well for other community projects as well as for applications in the enterprise.
Once a custom set of RPMs has been developed, it's time to set up a system so they can be created the exact same way over and over again by different people. Reproducability is an extremely important aspect of software development and is especially true where an entire distribution is concerned. RPM was designed so that pristine source plus added patches and a spec file creates an RPM. The catch is that some packages will compile differently depending on what's already on the host. To overcome this limitation Fedora uses mock.
In its simplest form, Mock takes an SRPM as an argument, builds a chroot with a set of minimal packages, then spits out an RPM. Using RPMs 'BuildRequires' function ensures that any SRPM passed to it will get built the same way every time. There are some caveats to that, but it's best that beginners think of mock in this way until they becomes more familiar with it.
Building RPM's can be a time-consuming process and many will find it annoying to use their workstation for this task. This is especially true in larger communities or in an enterprise environment. To overcome this hurdle, the Fedora Project started using a build system called Koji. At it's core, Koji is a build queue manager for mock. At its most complex, it is a complete build solution for all of Fedora's building needs. Using x86_64 and ppc builders, Koji keeps track of what's building on what builders and sends tasks to each builder based on available load.
Even with only two build machine architectures, Fedora can build i386, x86_64, and ppc RPMs. This is great for the developers since they may not have access to these architectures and their machines never get tied up in the build process. It is also a very secure method, since they're pulling everything out of cvs and lookaside cache[1], which prevents people from submitting malicious code by anonymous means. It also allows Fedora administrators to recreate any build at any time.
Because mock uses yum to build its chroot for every build, it is a good idea to keep a local mirror of Fedora. Using Koji can alleviate this need, but most people test new RPMs using a local mock install. Creating a Fedora mirror is fairly easy and it's also a well-documented process.
For those with high bandwidth, Fedora is always on the lookout for more mirrors. Fedora's new Mirror Manager makes it possible to set up new mirrors. Administrators can even use Mirror Manager to direct those in your organization to a local mirror using the netblock feature. This is very handy in the enterprise, as it's possible to use a stock Fedora install without any changes to the yum configuration and yet still direct users to a local mirror.
On the ticketing and bug side of Fedora's development lies bugzilla. Bug systems provide a mechanism to get metrics about the project and, of course, track issues with software. It also provides a means for users to comment and prevents developers from getting a thousand emails about the same bug. There is no perfect bug tracking system, but those serious about their community need to have one.
Now that all of those RPM's are being built and updated, it's a good idea to use some sort of updates system to pull and manage what updates go out and when. The Fedora Project created an updates system called Bodhi. The updates system is used for test cases–making sure RPMs don't break a good repository. It also allows some control over when a package gets pushed, as well as providing notes related to each update. The system itself is targeted at more mature communities and the enterprise.
Once the systems have been set up, it is just a matter of keeping everything together and making sure users and developers are informed about new events and continue to participate. By basing a custom operating system on Fedora's already-active code base, much of the difficult work of keeping the operating system up-to-date is complete, leaving community leaders to focus on development and allowing the community to grow on its own.
Everything about Fedora's process is now in the open and can be customized to any organization or community's needs. Those already running Fedora or any operating system in the Red Hat family can now build packages and communities in the exact same way that Fedora does.
[1] lookaside cache is basically the upstream tarballs that The fedora Project store's in a separate location from the CVS.
