Small Technical Blog
EHCI Debug
The EHCI Debug Port is an optional capability of EHCI controllers which can be used for early debugging for hardware which does not have a serial port. All USB2 host controllers are EHCI controllers.
Since I installed Coreboot on my Librem, I experienced some issues. In order to investigate - and since the Librem has no serial port - I aimed to do some Coreboot and kernel debugging with a Raspberry Pi Zero.
Tasks:
- Edit the kernel configuration to include the print ehci and xhci output options.
- Activate USB Debug output on Coreboot
- Set up a Raspberry Pi Zero to use it as EHCI client (USB Gadget).
Goal: enable USB (EHCI) kernel debug output.
We need a self-built Debian kernel where EHCI is enabled in the kernel configuration.
On the Librem
First find out the version of the installed kernel:
eramon@caipirinha:~$ uname -a
Linux caipirinha 4.19.0-2-amd64 #1 SMP Debian 4.19.16-1 (2019-01-17) x86_64 GNU/Linux
Download and unpack debian kernel source:
sudo apt-get install linux-source-4.19
tar xaf /usr/src/linux-source-4.19.tar.xz
Install kernel build dependencies:
sudo apt build-dep linux-image-4.19.0-2-amd64
linux-image-amd64
Get current configuration and generate defaults for missing properties:
cd linux-source-4.19
cp /boot/config-4.19.0-2-amd64 .config
make olddefconfig
Run makemenuconfig and activate the kernel config option for EHCI Debug (by default not enabled on the Debian kernel):
make menuconfig
> Search (CONFIG_EARLY_PRINTK_DBGP) > Early printk
--- Early printk
[*] Early printk via EHCI debug port
...
[*] Earl printk via the xHCI debug port
Save & Exit.
When first trying to build the kernel, I got a certificate error. Researching a little I found a thread explaining the problem and the solution (see references). In the .config file copied from /boot, I had to find and comment out following properties:
CONFIG_SYSTEM_TRUSTED_KEYS
CONFIG_MODULE_SIG_ALL
CONFIG_MODULE_SIG_KEY
In this way the custom kernel will be able to get modules signed by a one-time key.
NOTE: is there eventually a better way to get the .config so we can avoid the certificate issue?
Invoking make afterwards prompted me to configure some kernel options related to the ones I edited before. I chose the defaults in all cases:
eramon@caipirinha:~/debian/linux-source-4.19$ make deb-pkg -j8
scripts/kconfig/conf --syncconfig Kconfig
*
* Restart config...
*
*
* Certificates for signature checking
*
File name or PKCS#11 URI of module signing key (MODULE_SIG_KEY) [certs/signing_key.pem] (NEW)
Provide system-wide ring of trusted keys (SYSTEM_TRUSTED_KEYRING) [Y/?] y
Additional X.509 keys for default system keyring (SYSTEM_TRUSTED_KEYS) [] (NEW)
Reserve area for inserting a certificate without recompiling (SYSTEM_EXTRA_CERTIFICATE) [N/y/?] n
Provide a keyring to which extra trustable keys may be added (SECONDARY_TRUSTED_KEYRING) [N/y/?] n
Provide system-wide ring of blacklisted keys (SYSTEM_BLACKLIST_KEYRING) [N/y/?] n
The kernel built succeeded and following .deb files were generated by the build process:
cd ..
eramon@caipirinha:~/debian$ ls -la linux-image*deb
-rw-r--r-- 1 eramon eramon 46189016 Mar 25 22:25 linux-image-4.19.16_4.19.16-1_amd64.deb
-rw-r--r-- 1 eramon eramon 669497020 Mar 25 22:43 linux-image-4.19.16-dbg_4.19.16-1_amd64.deb
Install the new kernel:
sudo dpkg -i linux-image-4.19.16_4.19.16-1_amd64.deb
After the installation, following files have been installed under /boot:
eramon@caipirinha:~/debian$ ls -la /boot |grep 4.19.16
-rw-r--r-- 1 root root 206398 Mar 25 21:06 config-4.19.16
-rw-r--r-- 1 root root 32961834 Mar 26 07:43 initrd.img-4.19.16
-rw-r--r-- 1 root root 3346540 Mar 25 21:06 System.map-4.19.16
-rw-r--r-- 1 root root 5187456 Mar 25 21:06 vmlinuz-4.19.16
Now the kernel should print both ehci and xhci output, which we can see using a USB gadget - for example the Raspberry Pi Zero. Keep reading to find out more.
Goal: activate USB Debug Output in the Coreboot configuration
Find the USB debug port:
sudo lshw
...
*-usb:1
description: USB controller
product: Wildcat Point-LP USB EHCI Controller
vendor: Intel Corporation
physical id: 1d
bus info: pci@0000:00:1d.0
version: 03
width: 32 bits
clock: 33MHz
capabilities: pm debug ehci bus_master cap_list
configuration: driver=ehci-pci latency=0
resources: irq:23 memory:b221a000-b221a3ff
*-usbhost
product: EHCI Host Controller
vendor: Linux 4.19.16 ehci_hcd
physical id: 1
bus info: usb@1
logical name: usb1
version: 4.19
capabilities: usb-2.00
configuration: driver=hub slots=2 speed=480Mbit/s
*-usb
description: USB hub
vendor: Intel Corp.
physical id: 1
bus info: usb@1:1
version: 0.03
capabilities: usb-2.00
configuration: driver=hub slots=8 speed=480Mbit/s
To get a USB debug console, enable both CONFIG_USBDEBUG and CONFIG_CONSOLE_USB (menu option USB 2.0 EHCI debug dongle support) in coreboot’s kconfig.
/home/eramon/dev/coreboot-working/.config - coreboot configuration
> Debugging > Search (CONFIG_USBDEBUG)
Symbol: USBDEBUG [=y] │
│ Type : boolean │
│ Prompt: USB 2.0 EHCI debug dongle support │
│ Location: │
│ (1) -> Generic Drivers │
│ Defined at src/mainboard/msi/ms9652_fam10/Kconfig:63 │
│ Depends on: VENDOR_MSI [=n] && BOARD_MSI_MS9652_FAM10 [=n]
/home/eramon/dev/coreboot-working/.config - coreboot configuration
> Debugging > Search (CONFIG_USBDEBUG) > Generic Drivers
[*] USB 2.0 EHCI debug dongle support
Goal: set up a Raspberry Pi Zero as USB Device (a.k.a USB Gadget)
On any computer running Linux
Insert a sdcard.
Download latest Raspbian image (see references for Downloads link).
Unzip and copy the image to the sdcard:
unzip 2018-11-13-raspbian-stretch-lite.zip
sudo dd if=2018-11-13-raspbian-stretch-lite.img of=/dev/sdb bs=4M conv=fsync
sudo sync
Extract the sdcard and insert it again.
Two new drives should be automatically mounted: boot and rootfs:
eramon@caipirinha:~/dev/techblog$ mount |grep sdb
/dev/sdb1 on /media/eramon/boot type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)
/dev/sdb2 on /media/eramon/rootfs type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2)
On the boot partition, edit following file:
vi /media/eramon/boot/config.txt
Add these lines at the end, in order to enable serial port output and device tree overlay.
# Enable serial
enable_uart=1
# Device tree overlay and DesignWare Core driver
dtoverlay=dwc2
On the same partition, enable ssh by creating an empty ssh file:
touch /media/eramon/boot/ssh
Sync:
sudo sync
Extract the sdcard.
Insert the sdcard on the Raspberry Pi Zero.
Connect a serial/USB debug cable to the corresponding GPIO pins on the board (see references to find out how). Connect the USB end of the cable to the laptop and start minicom:
sudo minicom -D /dev/ttyUSB0
Log in via serial. The default username is pi and the password is raspberry.
Raspbian GNU/Linux 9 raspberrypi ttyS0
raspberrypi login: pi
Password:
Linux raspberrypi 4.14.79+ #1159 Sun Nov 4 17:28:08 GMT 2018 armv6l
On the Raspberry Pi Zero
Configure wifi and find out IP address:
sudo raspi-config
2. Network options Configure network settings
N2 Wi-fi Enter SSID and passphrase
pi@raspberrypi:~$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:27:eb:d8:e2:69 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.198/24 brd 192.168.81.255 scope global wlan0
valid_lft forever preferred_lft forever
Now you can connect to the board directly via SSH.
NOTE: alternatively, instead of connecting via serial (and afterwards via SSH), you can use a mini-HDMI to HDMI adapter for connecting a display to the board.
In order for the Raspberry Pi to behave as an USB Device, we need to modify the raspbian kernel to include the corresponding modules and to apply some patches.
On any computer running Linux
First of all, log in to the raspberry pi:
ssh pi@192.168.1.198
On the Raspberry Pi Zero
Update sources and upgrade the system:
sudo apt-get update
sudo apt-get upgrade
Install kernel build dependencies and get the kernel source:
sudo apt-get install git bc libncurses5-dev flex bison libsssl-dev
git clone --depth=1 https://github.com/raspberrypi/linux
Generate Raspberry Pi Zero default kernel configuration:
cd linux
KERNEL=kernel
make bcmrpi_defconfig
Run make menuconfig to activate the module we need:
make menuconfig
Enable g_dbgp in menuconfig as a module:
– Device Drivers -> USB Support -> USB Gadget Support -> EHCI Debug Device Gadget
Build kernel, modules and device tree:
make -j4 zImage modules dtbs
Download patches to fix known issues (see references below):
wget https://johnlewis.ie/debug_patches.tar.xz
tar -xJf debug_patches.tar.xz
Apply the patches:
pi@raspberrypi:~/dev/linux $ patch -p0 < dbgp.patch
patching file drivers/usb/gadget/legacy/dbgp.c
Hunk #1 succeeded at 211 (offset 1 line).
Hunk #2 succeeded at 375 (offset 1 line).
pi@raspberrypi:~/dev/linux $ patch -p0 < gadget.patch
patching file drivers/usb/dwc2/gadget.c
Hunk #1 succeeded at 1687 (offset 504 lines).
pi@raspberrypi:~/dev/linux $ patch -p0 < u_serial.patch
patching file drivers/usb/gadget/function/u_serial.c
Hunk #1 succeeded at 1373 (offset -145 lines).
Build and install the kernel, modules and device tree:
make -j4 zImage modules dtbs
sudo make modules_install
sudo cp arch/arm/boot/dts/*.dtb /boot/
sudo cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm/boot/zImage /boot/kernel.img
Note: this commands are extracted from the official Raspberry Pi documentation (kernel build) - see references. What I do not understand: why copy and not just make install?_
Reboot.
Run modprobe:
sudo modprobe g_dbgp
Check if the modules we need are loaded and running:
pi@raspberrypi:~ $ sudo lsmod |grep dbg
g_dbgp 2987 0
libcomposite 50203 1 g_dbgp
u_serial 10969 1 g_dbgp
udc_core 39660 4 u_serial,dwc2,libcomposite,g_dbgp
pi@raspberrypi:~ $ sudo lsmod |grep dwc2
dwc2 156750 0
udc_core 39660 4 u_serial,dwc2,libcomposite,g_dbgp
Install minicom:
sudo apt-get install minicom
Connect to /dev/ttyGS0 with minicom
sudo minicom -D /dev/ttyGS0
Now connect the EHCI-enabled device via the micro USB cable and boot it. With a micro USB/USB cable, connect the EHCI host (the Librem) to the USB Gadget (the Raspberry Pi). Reboot.
Done. I was able to see the Coreboot output :)
TODO: I should also be able to see the kernel debug output
TODO: extend Coreboot documentation in order to add the Raspberry Pi Zero as an official supported USB device.
References and links:
Debian Handbook Kernel Compilation
Raspberry Pi Uart Configuration