Mini How-To

Where to begin?

I fre­quently get asked how I man­aged to end up work­ing on the Fedora ker­nel for liv­ing by peo­ple hop­ing to one day get into ker­nel hack­ing them­selves. One of the most com­mon things I hear is that the ker­nel is so big, how could any­one pos­si­bly under­stand it all? Truth is, there are very few peo­ple that really under­stand the whole ker­nel. The major­ity of the ‘big name’ ker­nel hack­ers got where they are today by spe­cial­is­ing in one thing, and branch­ing out. There are excep­tions to this of course, with a num­ber of peo­ple like Andrew Mor­ton, Alan Cox, and Linus who are ‘all rounders’, who have hacked on close to every­thing in the tree at some point. Whilst the ker­nel could always use more peo­ple like these super­heros, there is noth­ing wrong with becom­ing a spe­cial­ist in one area.

One thing that both the all-rounders, and spe­cial­ists have in com­mon how­ever, is an under­stand­ing of the com­mon ker­nel APIs. Things like ‘how to allocate/free mem­ory’ ‘how to cre­ate proc/sysfs files’ Most ‘how do ..’ ques­tions can be answered by tak­ing a look at how other parts of the ker­nel are already doing some­thing sim­i­lar. With enough expe­ri­ence of use of the com­mon APIs, higher level con­cepts are learned such as ‘how to cre­ate user­space inter­faces that don’t suck’.

There is no fast path to learn­ing ker­nel hack­ing. It comes down to a big time invest­ment on your part to read (and under­stand) code, learn from mis­takes you make (and you will make them!), and above all, real­is­ing that in the end, it’s just code. There may some addi­tional restric­tions for ker­nel hack­ing that you could get away with in user­space, but once you’ve grasped the basics, a lot of it just follows.

The basics:

• Make sure you under­stand how to com­pile and install a ker­nel before going any further.
• A sound knowl­edge of C is essen­tial. If you’re still strug­gling with pointer arith­metic and such con­cepts, stick with hack­ing stuff in user­space until you under­stand more. A crash in user­space caused by your mis­un­der­stand­ing is a lot eas­ier to debug, under­stand and learn from than one in the ker­nel which just causes your machine to lock up or reboot itself.

Must-read mate­r­ial:

• ker­nel­new­bies is a great resource for those start­ing out. It con­tains a lot of exam­ples, and pointers.
• Some good books on ker­nel hack­ing include Linux Ker­nel Devel­op­ment (2nd ed.) by Robert Love, Linux Ker­nel in a nut­shell by GregKH and the 3rd edi­tion of Linux Device Dri­vers by Jonathan Cor­bet.
• Jonathan also writes a really good con­cise sum­mary of the past week of ker­nel devel­op­ment each week at lwn.net. It’s well worth read­ing, even if you fol­low linux-kernel, as the rephras­ing and expla­na­tions are some­times a lot bet­ter in sum­maries than read­ing through a 200 email thread.
• The Linux ker­nel comes with a Documentation/ direc­tory which con­tains a num­ber of really use­ful doc­u­ments worth spend­ing some time reading.
• Finally the code itself. Find some­thing that inter­ests you, find out where in the ker­nel that is han­dled, and just start reading.

Use­ful tools:

• grep –r is invaluable.
• You’re going to be build­ing and rebuild­ing a lot. So install ‘ccache’. Most dis­tros have it pack­aged so that it auto­mat­i­cally sets itself up after installing. You may also want to look at ‘distcc’ if you’ve a lot of poten­tial build-cluster can­di­date machines local to you.
• Regard­less of whether you’re an emacs or vi per­son, ‘ctags’ are invalu­able for nav­i­gat­ing your way around the source tree. ‘make tags’ in the toplevel of the ker­nel tree will gen­er­ate an index. You can learn how to nav­i­gate with them in the man pages of your favorite edi­tor. (In vi, ctrl-] over a sym­bol jumps to that func­tion, ctrl-t takes you back. :ts will bring up a list of alter­na­tives if there are >1 hit for that func­tion name. You can also­vim –t func­tion­name­from the com­mand line)
• Some other folks find cscope really use­ful for the same pur­pose. ‘make cscope’ gen­er­ates the index, run­ning ‘cscope’ gets you an inter­face to jump to where func­tions are used/defined etc.

“I don’t know what to hack on!“
A great way of putting your newly learned skills to good use is to take a look at the open bugsin the ker­nel bug tracker, find some­thing, and try to help fix it. Whilst many dri­ver bugs need the hard­ware to really debug/test a solu­tion, a lot of prob­lems can still be found purely by code inspec­tion. There are no short­age of new bugs being filed all the time, and bug-fixing is a great way to learn about many dif­fer­ent areas of the ker­nel and how they inter­act. The days of dri­ver authors get­ting all the chicks and fast cars ended some years ago. Fix­ing bugs, that’s where the fun is at.

“I really just don’t get it“
Not every­one gets to be a space­man, rock­star, or ker­nel hacker when they grow up. It’s fine. Really. There are still a lot of things you can do to help out Linux.

• Test­ing. Even if hack­ing code isn’t your thing, build­ing and test­ing the lat­est snap­shots of Linus’ tree, or Andrew’s –mm tree if you’re feel­ing really adven­tur­ous is always use­ful. If it breaks, great! You get to con­tribute some­thing. A bug report to the linux-kernel list.
• related to test­ing — write test tools. A new syscall got added? Great, write an appli­ca­tion to use it in every way imag­in­able. Com­plain loudly when it breaks. Some of the sim­plest test tools have been the most use­ful to us. Filesys­tem stress tools like fsx have been so use­ful they become a ‘must-use’ tool for filesys­tem developers.
• Hack­ing user­space isn’t ‘uncool’. There are a *lot* of things still in need of a lot of love in user­space. Find some­thing that bugs you, and fix it. Can’t fix it? Get involved with the peo­ple who wrote it, maybe they’ll give you point­ers. Some of the larger projects like xorg are really in need of hack­ers (FWIW, if you have an incli­na­tion to work on hard­ware related code like dri­vers, X may be a bet­ter start­ing place than the ker­nel for rea­sons such as ease of debugging).
• Triage work. The ker­nel bugzilla is swamped with bugs. A lot of them really old ones that may even be fixed by now. No-one has the time to reg­u­larly go through them all, look­ing for patches that never got applied upstream, clos­ing dupli­cates, ping­ing reporters etc. Get involved!
• Doc­u­men­ta­tion. If along your jour­ney you find some­thing par­tic­u­larly hard to under­stand, and you found no doc­u­men­ta­tion on it, here’s your chance to be a documentation-writing-superhero! Ker­nel hack­ers hate writ­ing doc­u­men­ta­tion for some strange reason.

“What about jan­i­tor tasks?“
Whilst the jan­i­tor project has some use­ful infor­ma­tion, patches that do noth­ing but clean up code to com­ply with style guide­lines and other such triv­ial patches aren’t really a great way to learn. No-one ever learned any skills by chang­ing inden­ta­tion of a func­tion. Learn some of the ‘rules’ pro­posed there, but instead of focus­ing on them as ‘some­thing to do’, use those rules whilst doing some­thing more useful.

I add to this page peri­od­i­cally when I get new ques­tions from peo­ple, or new ideas that I think peo­ple could learn from. If you have ques­tions about begin­ning Linux ker­nel hack­ing, feel free to drop me an email, and I’ll try to point you in the right direc­tion. I don’t really have the time to help you debug why your code is crash­ing, but for higher level ques­tions, even if I can’t help you out, I prob­a­bly know some­one who can.

No related posts.