A brief status update

Got a new device last week!

The device and peripherals.
Closeup of the device.

The device has an i.MX283 processor and a touchscreen attached to the other side. It booted from the image that came installed on the SD card:

It even has Doom installed on it! Which, I found, is actually easy to install with Buildroot. Who knew video games on embedded devices were so popular?

Since I’m going to be working on a kernel module (mxs-lradc) for i.MX28, I have to put a new kernel on the device. The device came with a manual that described a way to do that, which is really nice, since I have no experience with embedded devices.

The SD card the device boots from is formatted with three partitions. The first has some initial i.MX23/28 boot code on it (“bootlets”), the second has the bootloader (they use Barebox), and the third has the root filesystem, which also contains the kernel image (and device tree binaries) in the /boot directory. From what I understand, the bootlets run first (after the firmware) and start the bootloader, which in turn starts the kernel.

Updating all of this should be pretty easy using Buildroot. They provide a Github repository with a Buildroot configuration file for the device. By downloading it and just running make crystalfontz_cfa10036_defconfig and make, Buildroot will download and build all the necessary parts for the system. This includes things like the cross-compilation toolchain, the bootlets, Barebox and its configuration, and their version of the kernel. The locations of all of these are specified in the configuration file.

The result is two images and a compressed root filesystem, which can be placed on the three partitions. The manufacturer even provided a handy Python script that formats the SD card and puts the images on it.

Trouble is, after I did all this, the device wouldn’t boot. It just hung on a blank screen. Good thing I backed up the original image (with dd).

My initial reaction was to try to get some output from the system, using the debug UART on the processor (on the image above, the four golden headers (holes) to the right of the red plate). I spent over two days on it, so far unsuccessfully (more on this below).

At some point, however, I noticed that the working system was running kernel version 3.12.17, while the kernel being built was v3.15-rc5, and the configuration file used was generated for v3.9 (presumably the new options were set to the defaults). So I thought I’d build v3.12.17 instead. I got the configuration for it from /proc/config.gz on the working system, and modified Buildroot to build it.

I once again placed the images and rootfs on the SD card, inserted it in the device, and… it didn’t boot. Nothing on the screen again.

Eventually I came up with the idea of just taking the working image, mounting the rootfs on it on my computer, and simply replacing the zImage file in the /boot directory with the one I built. And that worked! The device booted again with no problems.

Next I tried the same with v3.13. It booted, but the touchscreen didn’t work. Well, okay. Then I tried 3.14. Blank screen again, didn’t boot. sigh

Haven’t tried to find the reason yet, but it’s a start. At least now I have a way to boot my own kernel.

But I’d also like to figure out what’s wrong with the bootlets and/or Barebox images I put on the card, and why they don’t boot. I can see from the bootlet code that it prints debug information out to the debug UART on the board. So it would be great if I could read from the UART.

Furthermore, the kernel also boots with the command line parameter console=/dev/ttyAMA0, which is the debug UART. That means system messages are printed to it, and reading it could also help me find out what’s wrong with v3.14. (Though I could probably also override the console parameter by finding where the command line is parsed in the kernel.)

I tried to read from the UART using the serial to USB converter that came with the device. It connects the UART to my computer’s USB port. I should be able to read from it using the ftdi-sio driver, by issuing “normal” serial commands to /dev/USB0. I tried it with both Minicom and a small program using termios (example code), but so far no luck. I’m just getting garbage:

Minicom output.

I got the serial communication parameters from sysfs and the tcgetattr() system call on the device (with a working kernel). Not sure if that’s the right way to do it. And I can see the AMBA PL011 driver running on the device (or at least it says so in the system log during bootup). And yet nothing I write to /dev/console on the device is arriving on my computer, or I’m not handling it right. I must be doing something wrong. There are also weird SysRq “help” messages in the system log on the device, which implies that the UART is getting something from my end or the converter.

I’ve spent several days messing with this thing, and while I’d really like to get it to work, I’ll probably leave it alone for the time being. It’s not really what I’m supposed to be working on (the mxs-lradc driver is), and I’d like to get something done by the end of the internship, so I’ll probably do some other things next, like delete some drivers, and do some IIO subsystem driver cleanup that doesn’t require any hardware.

Which reminds me, this isn’t all I’ve worked on during the past couple of weeks. I also removed one more driver, and talked to some driver “owners” about their drivers, either on the mailing lists or some of them emailed me personally. I never got a response about 7 drivers, and was going to ask the maintainers (or original authors) about them again, but Greg thought I should just delete them, so I’ll do that instead.

In addition, I spent a lot of time learning about the IIO subsystem, by reading the documentation and the code of the IIO “core”, a “dummy” driver, and userspace examples.

I also subscribed to a bunch more kernel mailing lists related to the device and the mxs-lradc driver (input, pm (power management), ARM, and devicetree). It’s interesting since I get to see what kind of work goes on in the kernel. Even though 99% of the time I don’t understand what they’re talking about… And the volume is pretty high, so I can’t read everything anyway, but I at least try to scan the subject lines, and read the email only if it seems interesting.

It’s also nice to see people making mistakes :) Makes me worry less about messing up myself, since I see it’s usually not a big deal. Oh, and I also found that the LKML archives go back to 1996, which is pretty cool (and humbling).

I’ve also found myself in various “detours” reading code from parts of the kernel not strictly related to what I’m working on, like the module infrastructure, device trees, how devices and drivers are initialized, or how the kernel starts running. I should probably avoid going off track like this.

To finish up, I’d say it’s been a fun couple of weeks. Despite a few frustrating days with the UART port, I’m still very happy to have an actual device to work with and test my changes on, and I feel like I’ve learned a lot about Linux and embedded devices, which is nice.


End of June

Well, I said I’d write more frequently, yet two weeks have passed again. Time goes fast.

The past couple of weeks I’ve sent out emails to the maintainers of all the old staging drivers but two. Of the 20 I sent out, I’ve so far gotten responses to 13. It looks like 2 of the drivers will hopefully have some final issues fixed soon and be moved out of staging, 3 drivers still have someone working on them, 6 drivers I deleted since no one is working on them anymore, one I plan to delete, and one is waiting to be reviewed. For the drivers that have been deleted, people say they don’t have time to work on the driver anymore, or they don’t have access to the hardware anymore, or that the hardware is really old and/or no one uses it anymore, so there’s no point in cleaning the driver up. It’s unfortunate since all the drivers work and could be used by people.

I should be able to finish with the old drivers soon, and finally move on to IIO stuff. I got a bunch of ideas for what to work on from the IIO list. Some simpler ideas don’t require the hardware but anything more involved does. Despite half the internship being over, Greg agreed to get me one device (thank you!) so that I can work on and test one of the proposed tasks, and hopefully get the driver out of staging.

The driver is called mxs-lradc. It’s a driver for low-resolution analog-to-digital converters (“lradc”) on MXS based i.MX application processors. (I still haven’t figured out what MXS is, but apparently i.MX23 and i.MX28 are based on it.) According to the reference manual, the processor is intended to be used for various panels, control displays, graphical remote controls, handheld scanners and printers, point-of-sale terminals, portable medical devices, “automotive infotainment”, and other places. It has a host of hardware features, one of which are the analog-to-digital converters (ADC). They can be used for touchscreen input, to read the chip’s temperature or battery voltage, as general purpose analog-to-digital signal converters, and some other things.

I thought the reference manual was very informative. The LRADC stuff starts on page 2659 (chapter 38). I spent a pleasant Sunday reading the manual, the code for the driver, and lots of other sources on the net to try to understand exactly what the processor and the driver are about.

Most of the code for the driver already exists in the IIO subsystem, but since the ADC channels have a lot of functions (they currently provide touchscreen support, battery life/voltage information, die temperature readings, and general purpose analog-to-digital signal conversion), it makes sense to split those different parts up into different subsystems. For this I’ll need to 1) create a new power subsystem (drivers/power/) driver, which will probably also have to handle battery charging, a new feature, 2) move the touchscreen stuff to its own driver in drivers/input/touchscreen/, 3) only leave the general purpose ADC stuff in drivers/iio/, and 4) write a “multifunction device” driver in drivers/mfd/ to manage them all. (At least this is how I understand it at the moment.)

So I’ll get to learn a little about four subsystems, which will be nice. It seems like there’s already one ADC/touchscreen driver and one power/touchscreen type of driver in drivers/mfd/, so hopefully studying how they work and are structured will be helpful. Even better, I noticed that an i.MX25 ADC and touchscreen mfd driver was recently submitted to the IIO mailing list, so that could definitely be useful. But I haven’t looked into any of these yet. I’m also not sure at all about what the power subsystem driver will have to do exactly, but hopefully I’ll figure it out.

As I already mentioned, I’ll also be getting a device with the processor to test my changes. I’m really grateful to Greg for paying for it from his own Linux devices budget. I think it will be really cool to test my changes on the actual hardware, can’t wait. I’ve never worked with a device like this before, so it will definitely be a new experience.

That’s it for now. I can’t believe I’m half way through the internship already. Hopefully I’ll finish with the driver deletions soon. Right now I need to delete one more driver, send out two more emails about two drivers, and potentially respond to some emails. Then I’ll move on to IIO tasks that don’t require the hardware, and once the device arrives, I’ll start working on that.