Internship wrap-up

My OPW internship officially ended on Monday. I’ll summarize what I did during the past three months.

I first got an overview of the drivers in staging – what kind of drivers there are and how close they are to being moved out of staging. I emailed the people responsible for some of the more “neglected” drivers and asked them about the status of the drivers. Most responded, some didn’t. In the end I deleted 14 drivers from the kernel, as no one was working on them anymore (and odds are no one is using them). The remaining 8 drivers either still had someone working on them or my emails (hopefully) got someone to work on them.

After that I got a touchscreen device, messed around with it a bit, and started working on reorganizing an existing analog-to-digital converter (ADC) driver on it (the ADC can be used for touchscreen input and battery monitoring, for example). To do that, I read a lot of code in the industrial input/output subsystem (IIO, the subsystem for ADCs), driver core (for things like driver binding) and boot code (mostly out of curiosity). I also began working on a simpler cleanup task for IIO.

That’s all I’ve managed to do the past three months (and I’ve really only just started with the ADC driver). I’m pretty disappointed myself, though at least Greg doesn’t seem to be. I guess I was hoping to achieve more than I did. Not sure why I didn’t. Part of it’s probably that for the first month I was still dealing with university things, which took a lot of time away from the internship. Another part is that when I started I was already kind of burnt out from working on my bachelor’s thesis, and I think that made me inefficient. Also, I think I probably started out with too little knowledge of the kernel, and had a lot of background to catch up on (and still do).

On the bright side, I intend to continue working with the kernel, starting from continuing with the reorganization of the ADC driver. My personal goal is to finish it by LinuxCon Europe in October. Which, by the way, I’m excited to be attending and presenting at (just a 5 minute talk as part of the OPW talk). But, before I do anything else, I’m taking a couple of weeks off to rest, since right now I could really use a break.

Anyway, that concludes the internship for me. I really enjoyed the summer, and I hope other interns did too. Thanks to all those who put everything together (program coordinators, mentors, etc.), and I hope all the participating organizations got a bunch of great new contributors!

A brief status update

Got a new device last week!

The device and peripherals.
Closeup of the device.

The device has an i.MX283 processor and a touchscreen attached to the other side. It booted from the image that came installed on the SD card:

It even has Doom installed on it! Which, I found, is actually easy to install with Buildroot. Who knew video games on embedded devices were so popular?

Since I’m going to be working on a kernel module (mxs-lradc) for i.MX28, I have to put a new kernel on the device. The device came with a manual that described a way to do that, which is really nice, since I have no experience with embedded devices.

The SD card the device boots from is formatted with three partitions. The first has some initial i.MX23/28 boot code on it (“bootlets”), the second has the bootloader (they use Barebox), and the third has the root filesystem, which also contains the kernel image (and device tree binaries) in the /boot directory. From what I understand, the bootlets run first (after the firmware) and start the bootloader, which in turn starts the kernel.

Updating all of this should be pretty easy using Buildroot. They provide a Github repository with a Buildroot configuration file for the device. By downloading it and just running make crystalfontz_cfa10036_defconfig and make, Buildroot will download and build all the necessary parts for the system. This includes things like the cross-compilation toolchain, the bootlets, Barebox and its configuration, and their version of the kernel. The locations of all of these are specified in the configuration file.

The result is two images and a compressed root filesystem, which can be placed on the three partitions. The manufacturer even provided a handy Python script that formats the SD card and puts the images on it.

Trouble is, after I did all this, the device wouldn’t boot. It just hung on a blank screen. Good thing I backed up the original image (with dd).

My initial reaction was to try to get some output from the system, using the debug UART on the processor (on the image above, the four golden headers (holes) to the right of the red plate). I spent over two days on it, so far unsuccessfully (more on this below).

At some point, however, I noticed that the working system was running kernel version 3.12.17, while the kernel being built was v3.15-rc5, and the configuration file used was generated for v3.9 (presumably the new options were set to the defaults). So I thought I’d build v3.12.17 instead. I got the configuration for it from /proc/config.gz on the working system, and modified Buildroot to build it.

I once again placed the images and rootfs on the SD card, inserted it in the device, and… it didn’t boot. Nothing on the screen again.

Eventually I came up with the idea of just taking the working image, mounting the rootfs on it on my computer, and simply replacing the zImage file in the /boot directory with the one I built. And that worked! The device booted again with no problems.

Next I tried the same with v3.13. It booted, but the touchscreen didn’t work. Well, okay. Then I tried 3.14. Blank screen again, didn’t boot. sigh

Haven’t tried to find the reason yet, but it’s a start. At least now I have a way to boot my own kernel.

But I’d also like to figure out what’s wrong with the bootlets and/or Barebox images I put on the card, and why they don’t boot. I can see from the bootlet code that it prints debug information out to the debug UART on the board. So it would be great if I could read from the UART.

Furthermore, the kernel also boots with the command line parameter console=/dev/ttyAMA0, which is the debug UART. That means system messages are printed to it, and reading it could also help me find out what’s wrong with v3.14. (Though I could probably also override the console parameter by finding where the command line is parsed in the kernel.)

I tried to read from the UART using the serial to USB converter that came with the device. It connects the UART to my computer’s USB port. I should be able to read from it using the ftdi-sio driver, by issuing “normal” serial commands to /dev/USB0. I tried it with both Minicom and a small program using termios (example code), but so far no luck. I’m just getting garbage:

Minicom output.

I got the serial communication parameters from sysfs and the tcgetattr() system call on the device (with a working kernel). Not sure if that’s the right way to do it. And I can see the AMBA PL011 driver running on the device (or at least it says so in the system log during bootup). And yet nothing I write to /dev/console on the device is arriving on my computer, or I’m not handling it right. I must be doing something wrong. There are also weird SysRq “help” messages in the system log on the device, which implies that the UART is getting something from my end or the converter.

I’ve spent several days messing with this thing, and while I’d really like to get it to work, I’ll probably leave it alone for the time being. It’s not really what I’m supposed to be working on (the mxs-lradc driver is), and I’d like to get something done by the end of the internship, so I’ll probably do some other things next, like delete some drivers, and do some IIO subsystem driver cleanup that doesn’t require any hardware.

Which reminds me, this isn’t all I’ve worked on during the past couple of weeks. I also removed one more driver, and talked to some driver “owners” about their drivers, either on the mailing lists or some of them emailed me personally. I never got a response about 7 drivers, and was going to ask the maintainers (or original authors) about them again, but Greg thought I should just delete them, so I’ll do that instead.

In addition, I spent a lot of time learning about the IIO subsystem, by reading the documentation and the code of the IIO “core”, a “dummy” driver, and userspace examples.

I also subscribed to a bunch more kernel mailing lists related to the device and the mxs-lradc driver (input, pm (power management), ARM, and devicetree). It’s interesting since I get to see what kind of work goes on in the kernel. Even though 99% of the time I don’t understand what they’re talking about… And the volume is pretty high, so I can’t read everything anyway, but I at least try to scan the subject lines, and read the email only if it seems interesting.

It’s also nice to see people making mistakes :) Makes me worry less about messing up myself, since I see it’s usually not a big deal. Oh, and I also found that the LKML archives go back to 1996, which is pretty cool (and humbling).

I’ve also found myself in various “detours” reading code from parts of the kernel not strictly related to what I’m working on, like the module infrastructure, device trees, how devices and drivers are initialized, or how the kernel starts running. I should probably avoid going off track like this.

To finish up, I’d say it’s been a fun couple of weeks. Despite a few frustrating days with the UART port, I’m still very happy to have an actual device to work with and test my changes on, and I feel like I’ve learned a lot about Linux and embedded devices, which is nice.

End of June

Well, I said I’d write more frequently, yet two weeks have passed again. Time goes fast.

The past couple of weeks I’ve sent out emails to the maintainers of all the old staging drivers but two. Of the 20 I sent out, I’ve so far gotten responses to 13. It looks like 2 of the drivers will hopefully have some final issues fixed soon and be moved out of staging, 3 drivers still have someone working on them, 6 drivers I deleted since no one is working on them anymore, one I plan to delete, and one is waiting to be reviewed. For the drivers that have been deleted, people say they don’t have time to work on the driver anymore, or they don’t have access to the hardware anymore, or that the hardware is really old and/or no one uses it anymore, so there’s no point in cleaning the driver up. It’s unfortunate since all the drivers work and could be used by people.

I should be able to finish with the old drivers soon, and finally move on to IIO stuff. I got a bunch of ideas for what to work on from the IIO list. Some simpler ideas don’t require the hardware but anything more involved does. Despite half the internship being over, Greg agreed to get me one device (thank you!) so that I can work on and test one of the proposed tasks, and hopefully get the driver out of staging.

The driver is called mxs-lradc. It’s a driver for low-resolution analog-to-digital converters (“lradc”) on MXS based i.MX application processors. (I still haven’t figured out what MXS is, but apparently i.MX23 and i.MX28 are based on it.) According to the reference manual, the processor is intended to be used for various panels, control displays, graphical remote controls, handheld scanners and printers, point-of-sale terminals, portable medical devices, “automotive infotainment”, and other places. It has a host of hardware features, one of which are the analog-to-digital converters (ADC). They can be used for touchscreen input, to read the chip’s temperature or battery voltage, as general purpose analog-to-digital signal converters, and some other things.

I thought the reference manual was very informative. The LRADC stuff starts on page 2659 (chapter 38). I spent a pleasant Sunday reading the manual, the code for the driver, and lots of other sources on the net to try to understand exactly what the processor and the driver are about.

Most of the code for the driver already exists in the IIO subsystem, but since the ADC channels have a lot of functions (they currently provide touchscreen support, battery life/voltage information, die temperature readings, and general purpose analog-to-digital signal conversion), it makes sense to split those different parts up into different subsystems. For this I’ll need to 1) create a new power subsystem (drivers/power/) driver, which will probably also have to handle battery charging, a new feature, 2) move the touchscreen stuff to its own driver in drivers/input/touchscreen/, 3) only leave the general purpose ADC stuff in drivers/iio/, and 4) write a “multifunction device” driver in drivers/mfd/ to manage them all. (At least this is how I understand it at the moment.)

So I’ll get to learn a little about four subsystems, which will be nice. It seems like there’s already one ADC/touchscreen driver and one power/touchscreen type of driver in drivers/mfd/, so hopefully studying how they work and are structured will be helpful. Even better, I noticed that an i.MX25 ADC and touchscreen mfd driver was recently submitted to the IIO mailing list, so that could definitely be useful. But I haven’t looked into any of these yet. I’m also not sure at all about what the power subsystem driver will have to do exactly, but hopefully I’ll figure it out.

As I already mentioned, I’ll also be getting a device with the processor to test my changes. I’m really grateful to Greg for paying for it from his own Linux devices budget. I think it will be really cool to test my changes on the actual hardware, can’t wait. I’ve never worked with a device like this before, so it will definitely be a new experience.

That’s it for now. I can’t believe I’m half way through the internship already. Hopefully I’ll finish with the driver deletions soon. Right now I need to delete one more driver, send out two more emails about two drivers, and potentially respond to some emails. Then I’ll move on to IIO tasks that don’t require the hardware, and once the device arrives, I’ll start working on that.

The last 3 weeks

I haven’t written here in a while, mostly because of some final university-related things I had to do the past few weeks. On the bright side, as of yesterday I have a bachelor’s degree (with honors) in computer science. On the other hand, I haven’t had as much time as I’d like for the internship. I feel pretty bad about it, but hopefully I’ll be able to make up for it now that I’m done with school.

So I’ll describe what I have managed to do in the internsip the past 3 weeks.

3 weeks ago I went over all the drivers in the kernel’s staging tree (drivers/staging/). Currently that’s 118 drivers total, but the number changes all the time, as new drivers are added and current ones are either cleaned up and moved to their proper place in the kernel, or deleted as no one’s working on them. I had two goals: 1) to see which drivers haven’t been worked on in over a year, and are therefore candidates for removal, and 2) to see which drivers I’d like to work on to clean them up.

I used the git history for the first goal. Mostly I used the following command for each driver:

git log --format="%<(8)%C(yellow)%h %<(22)%Cgreen%ar %<(22)%Cblue%an %Creset%s" --no-merges driverdir/

The format looks a bit messy, but it just displays the commit ID, author, subject line, and how long ago the change was made, in a nicely aligned, readable and colored way. For each driver, I tried to check if it looked like someone had actually tried to move the driver out of staging in a year, by either completing things in the driver’s TODO list or doing other “substantial” improvements. This means that fixes to checkpatch, sparse, and Coccinelle warnings didn’t count.

In the end I found 48 drivers that seemed to be abandoned. 20 of those were in staging/iio/ (the “Industrial I/O” subsystem), and 4 in staging/media/ (the media subsystem).

As for the second goal, I mostly used the driver’s description in the Kconfig file, checked the TODO file (if it had one), and occasionally superficially read the code (function names etc.) to get an idea of what kinds of things the driver does. For most of the drivers I had to search and read about them online too, because their descriptions were pretty uninformative, and also because I don’t know that much about different types of devices. It was a great learning experience though.

In the end I came up with 5 “categories” of drivers I thought I would most enjoy working on. When choosing these, I focused on the types of drivers that were in staging but hadn’t been worked on too much. In no particular order:

  • network – this included drivers for 5 network cards in staging, 4 WiFi and one ethernet
  • media & video – including drivers for a TV tuner, an FM radio receiver, infrared remote controls, a video processing frontend, and a video decoder
  • display & graphics – including drivers for a text-based LCD panel, a One Laptop per Child display controller, and a framebuffer for a graphics card
  • sensors – drivers in the IIO subsystem, including drivers for accelerometers, gyroscopes, impact sensors, light & proximity sensors, and analog to digital converters
  • other – a couple of other interesting drivers for a card reader, an nVidia embedded controller, and a NAND flash device

Out of these, Greg (my mentor) thought that sensors/IIO would be my best long-term bet, as most of the other drivers were one-off types and their hardware would be difficult to acquire (except media drivers, he isn’t familiar with them). I think this is a great idea, sensors sound like a lot of fun to experiment with.

So that’s what I did 3 weeks ago. Then 2 weeks ago I took a week off to deal with school things. I had an exam, prepared for a final presentation in front of a tough committee, dealt with feedback for my bachelor’s thesis, things like that.

Last week I returned to continue work in the internship. I’d expected Greg to contact the maintainers of the abandoned drivers, but instead he thought I should do it. I wrote a form letter to send them all, and got the names and email addresses of the maintainers from either the TODO and MAINTAINERS files, or if they didn’t have anyone in there then I just got the person who first added the driver to the kernel. To be honest, it’s pretty scary to be contacting these people, as some of them have been involved with Linux for a very long time, and I’m still so new here.

Greg also contacted the IIO maintainer (Jonathan Cameron) to ask what still needs to be done with the IIO drivers left in staging, so that I could help clean them up. Jonathan had some initial ideas that I will be starting work with soon, and other people in the IIO mailing list (which was Cc’d on the initial email) have offered their ideas too.

As a sidenote, I found it interesting that the entire IIO mailing list was Cc’d on the email. I’m not used to this kind of transparency, and I’m liking it a lot. In general, I feel like everything is done very “publicly” in the kernel, and I think that’s really cool. It contrasts with several other projects that I’ve worked on (at university and at jobs) where lots of things that weren’t done right were “hidden” from others, and denied, and sometimes even outright lied about. And I was always expected to do the same and I hated it.

This week I put together all the emails I’ll be sending out to the maintainers of the abandoned drivers. I sent 5 out yesterday, and got some responses, and will send the rest out soon. Also, the discussion about which IIO tasks I’ll be working on is still ongoing.

Maybe the main thing that worries me about the internship right now is that with some of the responses I’m getting from either the IIO people or the maintainers of the abandoned drivers, I don’t really know what they’re talking about, and more importantly, if I’m supposed to know. I mean, if they didn’t clarify, then it must mean that they expect me to know, right? So I end up wondering if I should go and spend hours trying to find out, or just ask them what they meant and risk sounding like an idiot. And some of this stuff feels like it should be done by a longtime contributor who is known in the community, not a newb like me. Well, at least it’s challenging.

As for the internship as a whole, it’s been pretty different than what I expected. I thought I’d mostly be writing code, but I haven’t done that at all so far. There’s a lot of people interaction right now, which honestly I think is more difficult for me, but it also builds confidence (I hope), so that’s good.

And that’s it for now. This post turned out much longer than I expected, but I guess that’s what happens when you don’t write for several weeks. I’ll try to be more frequent from now on.

A new blog appears!

Hi! I’m Kristina. I’m a 23-year-old computer science student from Estonia. I was recently accepted into the Outreach Program for Women (OPW) as a Linux kernel intern. I’ve never had a blog before, but since blogging is part of OPW, I’m starting this one here. Initially I’ll probably talk about my work for OPW, but in time I may branch out to other subjects, e.g. projects I’m working on in my spare time.

I guess I should say a few words about myself. I’m from Estonia, and currently live here in the city of Tartu (but will move back to Tallinn next month). I’m studying computer science at an undergraduate level at the University of Tartu, and will graduate next month. I’ve also worked as a software developer, and once organized a TEDx event (TEDxTartu). When I manage to find time off from school, I like to read (fantasy, non-fiction), play board and video games, watch movies, and I also love to travel.

I want to go to graduate school at one point, but not before I have a clearer picture of what I’d like to do research in. The past couple of years I’ve worked with some research groups at the university (mostly in the fields of computational neuroscience and bioinformatics) but wasn’t really excited about the work. Lately I’ve instead been curious about computer hardware, electronics, and low-level software development. I’ve started reading some books on the subjects and have found them really interesting.

Which is also where OPW comes in! A couple of years ago I read about Google Summer of Code and thought it would be very cool to participate. I haven’t contributed to any open source projects though, mostly because of lack of time. As I was looking through the list of participating projects, I immediately noticed some related to hardware. One of them was the Linux kernel (which is the part of the Linux operating system that deals with hardware). I’ve used Linux for over three years, so I like the idea of working on something that I’m familiar with. The kernel manages a lot of different kinds of hardware, so I think the opportunity to learn about them while also giving back to the community is just perfect.

My project is about cleaning up one or more device drivers in the kernel’s staging directory. (A driver is just code that controls a hardware device.) The staging directory contains drivers that have been included in the kernel, but haven’t been brought up to the kernel’s usual quality level. This could mean, for example, that they have actual bugs, or duplicate some code already present in the kernel, or are missing some essential features. Or maybe they just need more testing, or their coding style doesn’t match the rest of the kernel’s. My goal will be to improve a driver (or several) so that it can be moved out of staging and into the kernel proper.

This week was the first week of my internship. I wasn’t sure which driver to work on, so my mentor (Greg Kroah-Hartman) suggested I go over all the drivers in staging to find which ones interest me the most. So far I’ve only gone over about a third of them, since I wasn’t able to work as much as I should have this week (mostly because of some final university stuff I had to do). Hopefully I’ll make up for it in the coming weeks. Plus I think I was being way too thorough at first. Anyway, as I go I’m also looking at the git history to see which drivers aren’t being worked on by anyone, and should therefore be removed from the kernel. The rule is that if someone puts a driver in staging then they (or somebody else) should also work to get it cleaned up. If nobody works on a driver for more than a year (or so) then it gets kicked out.

Well, that’s all I have to say for now. Good, means I can get back to work :) I feel like I’ve already learned a lot this week, it’s amazing the kinds of devices that exist out there, and can be controlled by just a little driver (and some medium). I hope it continues to be this interesting (and fun) throughout the internship.