while not keypressed (Posts about ubuntu)

Subpixel rendering in Fedora 22

Roman Yepishev — Thu, 17 Sep 2015 06:16:14 GMT

I've spent last week trying to get subpixel rendering working with fedora's freetype package, and even though it had the patch...

--- freetype-2.5.2/include/config/ftoption.h 2013-12-08 19:40:19.000000000 +0100
+++ freetype-2.5.2-enable-spr/include/config/ftoption.h 2014-01-17 19:17:34.000000000 +0100
@@ -92,7 +92,7 @@
   /* This is done to allow FreeType clients to run unmodified, forcing     */
   /* them to display normal gray-level anti-aliased glyphs.                */
   /*                                                                       */
-  /* #define FT_CONFIG_OPTION_SUBPIXEL_RENDERING */
+  #define FT_CONFIG_OPTION_SUBPIXEL_RENDERING


   /*************************************************************************/

... the fonts still looked horribly (e.g. this web page)

Today I actually read the spec file, and it started with...

# Patented subpixel rendering disabled by default.
# Pass '--with subpixel_rendering' on rpmbuild command-line to enable.
%{!?_with_subpixel_rendering: %{!?_without_subpixel_rendering: %define _without_subpixel_rendering --without-subpixel_rendering}}

insert a minute of swearing

Everybody seem to be all hyped about infinality patches, and they do make the fonts look great, but these were already merged.

rpmbuild  -bb freetype.spec --with subpixel_rendering

Even Cantarell font looks almost decent now.

ARGH.

AppArmor config for Yandex.Disk

Roman Yepishev — Wed, 04 Dec 2013 11:05:02 GMT

I've been using Yandex.Disk for quite a while for data synchronization and file sharing because of its simplicity - a standard WebDAV implementation (+ extensions for file sharing, all documented, although in Russian) allowing for different clients to be used, a nice web UI and they have introduced a Linux client recently which marries WebDAV and XMPP which is used for OOB notifications.

Being a bit cautious I decided to run it on my system but restrict access to everything else, so I wrote the following apparmor config and now it is only allowed to touch the files within ~/Yandex.Disk directory and it's configuration, put it to /etc/apparmor.d/usr.bin.yandex-disk:

usr.bin.yandex-disk (Source)

# vim:syntax=apparmor
# Author: Roman Yepishev <roman.yepishev@yandex.com>
#include <tunables/global>

/usr/bin/yandex-disk {
    #include <abstractions/base>
    #include <abstractions/nameservice>
    #include <abstractions/user-tmp>
    
    owner @{HOME}/.config/yandex-disk/ rw,
    owner @{HOME}/.config/yandex-disk/** rw,

# yandex-disk can add itself to autostart configuration
    owner @{HOME}/.config/autostart/Yandex.Disk.desktop rw,

# default sync location
    owner @{HOME}/Yandex.Disk/ rw,
    owner @{HOME}/Yandex.Disk/** rw,

# or change the location like this:
#   owner @{HOME}/OtherFolder/ rw,
#   owner @{HOME}/OtherFolder/** rw,

    /usr/bin/yandex-disk ixr,

    /bin/dash ixr,
}

Yandex.Disk supports symlinks, so if symlinks to other locations are used, these all should be described in the configuration file. Otherwise, the access checks will fail and you will get this in kernel log:

[77109.846335] type=1400 audit(1386173505.778:116): apparmor="DENIED" ↩
operation="open" parent=2000 profile="/usr/bin/yandex-disk" ↩
name="/home/rye/_wine.tar.gz" pid=2869 comm="yandex-disk" ↩
requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

P.S. Yeah, I've stopped using Ubuntu One for the time being.

Fixing Tizen SDK IDE Crash on Project Creation

Roman Yepishev — Sat, 20 Jul 2013 19:19:13 GMT

So recently I installed Tizen SDK on my Ubuntu 13.04 and found that I can't create a project - the java process crashes in some native library and it was not obvious where.

After some extensive googling I found a report about this particular problem - TIZEN 2.0 MAGNOLIA SDK Crashes while creating a new project where daviderosa found the reason: Google Talk Plugin.

Uninstalling the plugin allowed me to continue creating the project.

Unfortunately the mozilla process launched by eclipse tries to load all the plugins possible automatically, and there is no switch to disable that. The best solution would be to specify Mozilla profile in your eclipse.ini, but Tizen SDK 2.1 is based on a SWT version which does not yet have this feature.

Uninstalling google-talkplugin completely or moving the installed plugin links (/usr/lib/firefox/plugins and /usr/lib/mozilla/plugins) and putting them back when needed seem to be the solutions for now.

Bisecting Kernel Using PXE Boot

Roman Yepishev — Tue, 19 Mar 2013 15:01:50 GMT

Recently I found that an Atheros AR5BXB64 wireless card in my Acer Aspire One AOA150 netbook stopped working properly in 13.04 with 3.8 kernel. The problem was that any file download over the wireless interface had a high chance of being corrupted. I was not able to download updates - every time something big (like a kernel upgrade) got downloaded, I got the file, but the checksum did not match. The card was able to inject copies of previous buffers into the stream in a way that still passed IP and TCP checksum verification.

This was exactly the same symptom we had when Active State Power Management (ASPM) L0s was enabled for that PCI-E device, when the hardware itself was not actually able to handle this state properly. It was fixed in 2009 but suddenly got broken in 3.8 again.

I filed a bug (LP:1137513) and, since I had a workaround, I just continued using the netbook with a setpci call in rc.local until I was asked whether any prior Raring kernels exhibited this behavior.

I booted 12.10 from a Flash drive and found that the card was working properly there. Then I checked 3.7 kernels and found that the issue did not exist there and appeared only in 3.8.0-rc1, which suggested there was indeed a regression in the kernel. And the difference was that 3.7 and earlier kernels had LinkCtl: ASPM Disabled while 3.8+ had LinkCtl: ASPM L0s L1 Enabled in the lspci -vv output.

Since I was not able to find any relevant commits that looked like they were deliberately doing something to ASPM handling, the only way to proceed was to run a binary search. Since it was possible to check whether the bug was there or not just by checking lspci, it could and should have been automated.

Bisect works by splitting the commits between known good and known bad revision in half with every iteration and testing the commit in between. Rinse and repeat until you hit a commit that made the switch from good to bad and that's where the things broke.

I had to test 5840 commits, with roughly 13 steps (2¹³=8192, 2¹²=4096). 13 manual reboots are 12 reboots too many so I went on to set up an environment that would do everything for me. Additionally I did not want to build the kernels on that netbook itself as it is pretty slow. The build host and the netbook should have been separated if I wanted to finish the task in a reasonable amount of time.

So, here's the plan:

Set up a PXE environment.
Find the minimal kernel config that exhibits the issue.
Create a custom initrd containing all the tools needed to detect whether an issue is there.
Create a script that can be reliably used by git bisect to make it all run automatically.

PXE

PXE is a Pre-boot Execution Environment which allows network boot without any local disk installation. It works by starting a DHCP client early in the boot process, obtaining the address of the TFTP server that hosts the boot images, downloading the image and passing the control over. TFTP is a Trivial File Transfer Protocol, a really simple connection-less and authentication-less protocol that's running over UDP.

In my case I already had a DHCP server which was running on my OpenWRT-based router. I set up a virtual machine and had my DHCP server direct the queries for the boot images to that virtual machine. This was added to my /etc/dnsmasq.conf on the router:

# PXE boot
dhcp-boot=pxelinux.0,raring-server-i386.lappyfamily.net,192.168.100.102

Then I switched to the "raring-server-i386" and installed atftpd.

Ubuntu 12.04 Note

The version in 12.04 suffers from a bug that prevents rlinetd from being set up properly. If you happen to hit the bug, you can create the configuration manually by executing the following:

. /etc/default/atftpd
sudo update-inetd --group BOOT --add \
"tftp dgram udp wait nobody /usr/sbin/tcpd /usr/sbin/in.tftpd $OPTIONS"
unset OPTIONS

In order to run Linux with PXE you will need to use PXELINUX. The required image, pxelinux.0 can be obtained after installing syslinux package from /usr/lib/syslinux/pxelinux.0. By default atftpd uses /srv/tftp as a root for the shared filesystem, so pxelinux.0 should go to /srv/tftp and be readable by nobody user.

Then I set up the netbook to boot from PXE. This will vary between different BIOS setup interfaces, but it boils down to enabling integrated LAN adapter, enabling boot ROM on that adapter and setting the network device as the first one to boot from. A wired connection to an IPv4 network is needed.

With pxelinux.0 in place I booted the netbook:

Intel UNDI, PXE-2.1 (build 082)
Copyright (C) 1997-2000  Intel Corporation

This Product is covered by one or more of the following patents:
US5,307,459, US5,434,872, US5,732,094, US6,570,804, US6,115,776 and
US6,327,625

Realtek PCI Express Fast Ethernet Controller Series v1.15 (090224)

CLIENT MAC ADDR: 00 1E 68 E2 10 AE  GUID: 40709CDA-A8E0-D411-8C6C-001E68E210AE
CLIENT IP: 192.168.1.14  MASK: 255.255.255.0  DHCP IP: 192.168.1.1
GATEWAY IP: 192.168.1.1



!PXE entry point found (we hope) at 9D17:0109 via plan A
UNDI code segment at 9D17 len 19CA
UNDI data segment at 930A len A0D0
Getting cached packet  01 02 03
My IP address seems to be C0A8010E 192.168.1.14
ip=192.168.1.14:192.168.100.102:192.168.1.1:255.255.255.0
BOOTIF=01-00-1e-68-e2-10-ae
SYSUUID=40709cda-a8e0-d411-8c6c-001e68e210ae
TFTP prefix:
Unable to locate configuration file

Boot failed: press a key to retry, or wait for reset...

The line starting with "My IP address" is our pxelinux.0 working. Now we need to put configuration and kernel in place.

You can just grab the kernel and initrd from existing installation and put them in /srv/tftp. In my case I put kernel to /srv/tftp/vmlinuz and initrd to /srv/tftp/initrd.img. If you check /var/log/syslog on the TFTP server, you will see that pxelinux wanted to get the configuration from the following locations:

pxelinux.cfg/40709cda-a8e0-d411-8c6c-001e68e210ae
pxelinux.cfg/01-00-1e-68-e2-10-ae
pxelinux.cfg/C0A8010E
pxelinux.cfg/C0A8010
pxelinux.cfg/C0A801
pxelinux.cfg/C0A80
pxelinux.cfg/C0A8
pxelinux.cfg/C0A
pxelinux.cfg/C0
pxelinux.cfg/C
pxelinux.cfg/default

If you are wondering, C0A8010E is 192.168.1.14 in hexadecimal notation.

So we need to put configuration at some location it is querying. I decided to use the MAC address, 01:00:1e:68:e2:10:ae and put the following configuration to /srv/tftp/pxelinux.cfg/01-00-1e-68-e2-10-ae (yes, the folder is named pxelinux.cfg):

DEFAULT linux
LABEL linux
KERNEL vmlinuz
APPEND initrd=initrd.img panic=10

You can reboot your device now and you will see that the kernel is loaded, initrd.img is loaded too but the boot breaks since the root device is undefined. While you can put the root information from your grub configuration, we don't actually need to do that. As we are going to build our own kernel with minimal number of features, we will run everything from the intitial RAM drive, initrd. I put the panic=10 option so that the kernel reboots after 10 seconds in case it fails so that I would not need to shut the netbook down manually.

Kernel

Well, that was easy. I grabbed the git repository of Linus, git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git, checked out v3.8-rc1 and ran make menuconfig

Since I was interested in ath5k module only I disabled loadable module support and made ath5k a built-in module. I left quite a few features I did not know/care about enabled which I believe has somehow increased the build time, but in the end it took 3 minutes to build a kernel on my VM with 4 concurrent builds (make -j 4, the VM had 4 CPUs allocated)

initrd

When kernel boots with initrd, it mounts that image as a root device and runs init as PID 1 trying /sbin/init, /etc/init, /bin/init and /bin/sh. In case /bin/sh is handled by busybox and it is running as PID 1, it attempts to run /init and /sbin/init. So I needed to create a new initramfs that had everything inside the /init script itself:

Setting up /dev, /proc, /sys and /tmp in case I need that for debugging.
Configure the network interface.
Execute the required test.
Wait for a reboot command.

Here's what I came up with, the first part is stolen from /init that ships with Ubuntu:

#!/bin/sh

[ -d /dev ] || mkdir -m 0755 /dev
[ -d /root ] || mkdir -m 0700 /root
[ -d /sys ] || mkdir /sys
[ -d /proc ] || mkdir /proc
[ -d /tmp ] || mkdir /tmp
mkdir -p /var/lock
mount -t sysfs -o nodev,noexec,nosuid sysfs /sys
mount -t proc -o nodev,noexec,nosuid proc /proc
# Some things don't work properly without /etc/mtab.
ln -sf /proc/mounts /etc/mtab

ifconfig eth0 192.168.1.14
route add default gw 192.168.1.1

sleep 5

lspci -vv -d 168c:001c | nc 192.168.100.102 5555

nc -l -p 5555 && reboot -f

So this init script configures the wired interface, sends the output of lspci to the host running the bisect script and listens to an incoming TCP connection on port 5555 so that it would reboot the host when connection is terminated. There is no need for anything fancier.

The initramfs contained the minimal set of applications and libraries, but you will want to grab busybox from a working system as it has a reboot applet which is missing from the version shipped with initrd by default on Ubuntu systems. I needed to do a bit of tweaking before I got everything working as I wanted so I had /bin/sh (symlink to /bin/busybox) originally in place of all these lspci calls. Also I used nc from netcat-traditional since it had less dependencies.

ubuntu@raring-server-i386:~/minroot$ find
./init
./bin
./bin/sh
./bin/lspci
./bin/nc
./bin/bash
./bin/busybox
./lib
./lib/ld-linux.so.2
./lib/libresolv.so.2
./lib/libc.so.6
./lib/libz.so.1
./lib/libpci.so.3
./dev
./dev/ram0
./dev/console
./dev/null
ubuntu@raring-server-i386:~/minroot$ find | cpio --format=newc -o | gzip > /srv/tftp/initrd.img
7806 blocks

Bisect script

git-bisect run expects the script to return 0 when this build is considered a success or 1 when the build is considered a failure, so in our case a successful build will have "ASPM Disabled" in the output. grep returns 0 when a string is found, 1 when not found and 2 when error, so we can just use that.

I called the script 'bisect-helper' and put it in my home directory on the VM.

#!/bin/bash

cp ~/kernel.config ~/linux/.config
cd ~/linux
make olddefconfig
make clean
make -j 4
cp arch/i386/boot/bzImage /srv/tftp/vmlinuz

# reboot netbook
echo "reboot" | nc 192.168.1.14 5555

# when it goes up, it will send us a message
nc -l 5555 | grep 'LnkCtl:\s*ASPM Disabled'
exit $?

The script needs to be tested manually first to check whether it is actually producing the results expected:

ubuntu@raring-server-i386:~/linux$ git checkout v3.7
Checking out files: 100% (11706/11706), done.
Previous HEAD position was 19f949f... Linux 3.8
HEAD is now at 2959440... Linux 3.7
ubuntu@raring-server-i386:~/linux$ ~/bisect-helper; echo $?
...
Kernel: arch/x86/boot/bzImage is ready  (#40)
        LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
0
ubuntu@raring-server-i386:~/linux$ git checkout v3.8
Checking out files: 100% (11706/11706), done.
Previous HEAD position was 2959440... Linux 3.7
HEAD is now at 19f949f... Linux 3.8
ubuntu@raring-server-i386:~/linux$ ~/bisect-helper; echo $?
...
Kernel: arch/x86/boot/bzImage is ready  (#41)
1

Yep, looks like it is working. Now let the madness begin!

ubuntu@raring-server-i386:~/linux$ git bisect start v3.8-rc1 v3.7
Bisecting: 5840 revisions left to test after this (roughly 13 steps)
[6be35c700f742e911ecedd07fcc43d4439922334] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
ubuntu@raring-server-i386:~/linux$ time git bisect run ../bisect-helper
running ../bisect-helper
...
(an hour later)
...
8c33f51df406e1a1f7fa4e9b244845b7ebd61fa6 is the first bad commit
commit 8c33f51df406e1a1f7fa4e9b244845b7ebd61fa6
Author: Taku Izumi <izumi.taku@jp.fujitsu.com>
Date:   Tue Oct 30 15:27:13 2012 +0900

    PCI/ACPI: Request _OSC control before scanning PCI root bus

    This patch moves up the code block to request _OSC control in order to
    separate ACPI work and PCI work in acpi_pci_root_add().

    Signed-off-by: Taku Izumi <izumi.taku@jp.fujitsu.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>

:040000 040000 ac58db59f5eaf541d5cb36197aaaf5dfe319ea37 d25e645d59da508e3a1bcb01509cbfb5c7a8239b M  drivers
bisect run success

real    63m39.674s
user    155m30.732s
sys 27m33.772s

It took ~64 minutes to run 13 kernel builds and the offending commit was discovered. I filed a bug, it got picked up by Yinghai Lu, who found that this commit triggered an issue in another function that differentiated between ASPM handling for non-hotplug and hotplug cases. This was affecting iwlwifi as well as other modules which were using pci_disable_link_state call, so it was a real issue.

Yinghai Lu has posted the patch to the kernel-pci mailing list and it should be merged at some point.

Summary

I've always wanted to do git bisect on something, but I had no idea what to start with. Bisecting is not specific to git, there's a Bazaar plugin as well and it is fairly easy to use this technique with any other VCS.

Minimizing the kernel build time is important. Originally I spent nearly 2 hours fiddling with the kernel configuration until I removed most of the things that were completely irrelevant to the issue being debugged. Having a powerful machine to build the software is a must in case the actual test needs to be performed on a slower machine.

Booting linux with PXE involves quite a few moving parts (DHCP, TFTP, boot image, kernel and initrd) that may take a while to configure, but all the time invested in this configuration will eventually pay off.

Getting rid of bash annoyances

Roman Yepishev — Wed, 09 Jan 2013 12:28:00 GMT

$ zcat /boot/initrd.<tab><tab><TAB!>

I know the file is there and I know it’s a file an application can handle, but bash autocompletion tries to be smart by refusing to provide the file name.

This gets really annoying if you happen to work with files that don’t have an extension the autocompletion scripts expect or you are running a command using sudo and want to pass the filename.

There is a dedicated shortcut for filename completion – M-/ by default, but I’ve never used tab completion for options and that means that I can get rid of it.

There is no need to uninstall anything, adding complete -r to .bashrc will remove all the completion functions.

To completely disable programmatic completion, add shopt -u progcomp to .bashrc.

$ zcat /boot/initrd.img-3.<tab>
initrd.img-3.2.0-29-generic         initrd.img-3.7.0-7-generic
initrd.img-3.2.0-35-generic         initrd.img-3.8.0-030800rc2-generic

Nice.

Another thing that was annoying me for a while is command-not-found package. It is extremely helpful at discovering what package the application I want is in. However, there were quite a few times when I made a typo, pressed Enter, noticed it right away but had to wait for a second or so before I got the “command not found” and the prompt back. When disk is really busy, making a typo costs me another 30 seconds of disk trashing before command-not-found comes up with a friendly suggestion that killal is better spelled as Command 'killall' from package 'psmisc'.

Having command-not-found installed and available but not kicking in on every occasion is preferred. The function bash runs in case it can’t find the command is command_not_found_handle so we simply need to unset this function in .bashrc and add an alias (in my case packages-providing after LP:486716, but it can be anything) which will execute the real command-not-found script:

unset command_not_found_handle
alias packages-providing='/usr/lib/command-not-found --no-failure-msg'

So now I will be given the package name only when I want it:

$ sl
bash: sl: command not found

$ packages-providing sl
The program 'sl' is currently not installed. You can install it by typing:
sudo apt-get install sl

r8169 module with DKMS in Ubuntu 12.04 LTS

Roman Yepishev — Tue, 18 Dec 2012 13:04:00 GMT

When testing network issues with different cards it is a good idea to make sure they actually have different chipsets.

Update: I found no difference between the NIC operating with built-in r8169 module and the version that is available on the RealTek’s web site. You will want to use the vendor-provided module only if your card is not supported by the driver shipped with the linux kernel.

Update: I packaged the source-only dkms – r8169-6.017.00-source-only.dkms.tar.gz, which can be installed with:

sudo dkms ldtarball r8169-6.017.00-source-only.dkms.tar.gz
sudo dkms build r8169/6.017.00
sudo dkms install r8169/6.017.00

Update: See the end of the post, patch is required for DGE-528T to be picked by this module.

I had a complex VM network setup with separate vlan on the router for virtual machines and second network card in the server to prevent connection hanging when network topology was changing on the VM startup. Then I decided to simplify the network and use one NIC only, in my case that would be DLink DGE-528T.

This card is driven by realtek 8169 chip and for some reason these like to drop network connection. I’ve found quite a few topics on poor performance of 8169 with in-kernel driver, but the built-in r8168 chip running under r8169 driver included in the kernel performs way worse than r8169.

There is a driver update on the Realtek web site so I decided to give it a try.

I started with dkms.conf from Fixing RTL8111/8168B kernel module on Debian/Ubuntu post by Randy, but for some reason r8169 module kept overwriting the system wide one. It turns out that Makefile installs module automatically when called with no specific target.

So here’s how I made it work:

Download the tarball from Realtek web site.
Unpack the resulting r8169-6.017.00.tar.bz2 to /usr/src.

Add the dkms.conf file to /usr/src/r8169-6.017.00/:

PACKAGE_NAME="r8169"
PACKAGE_VERSION="6.017.00"
CLEAN="make clean"
BUILT_MODULE_NAME[0]="r8169"
BUILT_MODULE_LOCATION[0]="src"
DEST_MODULE_LOCATION[0]="/updates"
MAKE[0]="'make' 'modules'"
AUTOINSTALL="YES"

Build and install the module:

$ sudo dkms install r8169/6.017.00
...
 - Installation
   - Installing to /lib/modules/3.2.0-35-generic/updates/dkms/
...

At the moment due to the bug in dkms script it is not possible to create the dkms tarball directly from realtek sources but you can create one after you perform the steps above.

$ sudo dkms mktarball r8169/6.017.00
Marking modules for 3.2.0-35-generic (x86_64) for archiving...

Marking /var/lib/dkms/r8169/6.017.00/source for archiving...

Tarball location: /var/lib/dkms/r8169/6.017.00/tarball//r8169-6.017.00-kernel3.2.0-35-generic-x86_64.dkms.tar.gz

Afterwards you can load this tarbal on a different machine:

$ sudo dkms ldtarball r8169-6.017.00-kernel3.2.0-35-generic-x86_64.dkms.tar.gz

Loading tarball for r8169-6.017.00
Loading /var/lib/dkms/r8169/6.017.00/3.2.0-35-generic/x86_64...

DKMS: ldtarball completed.

Creating symlink /var/lib/dkms/r8169/6.017.00/source ->
                /usr/src/r8169-6.017.00

DKMS: add completed.

$ sudo dkms install r8169/6.017.00

Creating symlink /var/lib/dkms/r8169/6.017.00/source ->
                /usr/src/r8169-6.017.00

DKMS: add completed.

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area....
'make' 'modules'....
cleaning build area....

DKMS: build completed.

r8169.ko:
Running module version sanity check.
- Original module
- Installation
- Installing to /lib/modules/3.2.0-35-generic/updates/dkms/

depmod....

DKMS: install completed.

DGE-528T

So I left this module running for about a week and yesterday I could not make the card work after a reboot. It turns out that while the updated r8169 module is installed, it is not being used for D-Link DGE-528T card.

r8169_n.c contains the following:

static struct pci_device_id rtl8169_pci_tbl[] = {
        { PCI_DEVICE(PCI_VENDOR_ID_REALTEK,     0x8167), 0, 0, RTL_CFG_0 },
        { PCI_DEVICE(PCI_VENDOR_ID_REALTEK,     0x8169), 0, 0, RTL_CFG_0 },
        { PCI_VENDOR_ID_DLINK, 0x4300, PCI_VENDOR_ID_DLINK, 0x4c00, 0, 0, RTL_CFG_0 },
        {0,},
};

So the module does not match PCI_VENDOR_ID_DLINK:0x4c00 subsystem (lspci -vnn):

02:06.0 Ethernet controller [0200]: D-Link System Inc DGE-528T Gigabit Ethernet Adapter [1186:4300] (rev 10)
    Subsystem: D-Link System Inc DGE-528T Gigabit Ethernet Adapter [1186:4300]
    Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 21
    I/O ports at e800 [size=256]
    Memory at febffc00 (32-bit, non-prefetchable) [size=256]
    Expansion ROM at febc0000 [disabled] [size=128K]
    Capabilities: [dc] Power Management version 2
    Kernel driver in use: r8169
    Kernel modules: r8169

1186:4c00 is actually a DGE-T530 card, but it looks like there are some DGE-T528 cards with 1186:4300 subsystem and some have 1186:4c00 one. The original driver from D-Link has PCI_ANY_ID for subsystem fields so it looks like these are compatible.

I was so sure the updated module would fix the issues I was having with the network card that it actually stopped misbehaving.

However, I decided to patch the module to make it work with my card too:

--- r8169_n.c.ori   2012-05-03 15:23:12.000000000 +0300
+++ r8169_n.c   2012-12-30 19:42:14.818322918 +0200
@@ -115,7 +115,8 @@
static struct pci_device_id rtl8169_pci_tbl[] = {
    { PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x8167), 0, 0, RTL_CFG_0 },
    { PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x8169), 0, 0, RTL_CFG_0 },
-   { PCI_VENDOR_ID_DLINK, 0x4300, PCI_VENDOR_ID_DLINK, 0x4c00, 0, 0, RTL_CFG_0 },
+   { PCI_DEVICE(PCI_VENDOR_ID_DLINK,       0x4300), 0, 0, RTL_CFG_0 },
+   //{ PCI_VENDOR_ID_DLINK, 0x4300, PCI_VENDOR_ID_DLINK, 0x4c00, 0, 0, RTL_CFG_0 },
    {0,},
};

And now the module prints the following when loaded:

[18014.031288] r8169 Gigabit Ethernet driver 6.017.00-NAPI loaded
[18014.031344] r8169 0000:02:06.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
[18014.033537] r8169: This product is covered by one or more of the following patents: US5,307,459, US5,434,872, US5,732,094, US6,570,884, US6,115,776, and US6,327,625.

So now I am really testing the module provided by Realtek.

When localhost is not

Roman Yepishev — Thu, 06 Dec 2012 15:07:00 GMT

/etc/hosts is a file where these entries should never be touched:

127.0.0.1       localhost
::1             ip6-localhost ip6-loopback

However, during my last trip to my VPS to fix my mail system after opendkim update in Ubuntu 10.04 I found something interesting in netstat:

$ sudo netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address        ... PID/Program name
...
tcp        0      0 173.212.238.58:8891  ... 15410/opendkim

Basically, when I specified localhost in opendkim configuration, it was listening on a public interface instead. Pinging localhost revealed it is actually a non-loopback address:

$ ping localhost
PING yankee.lappyfamily.net (173.212.238.58) 56(84) bytes of data.
...

It looks like it’s been this way since the very begninning, as my /etc/hosts had the following:

# Auto-generated hostname. Please do not remove this comment.
173.212.238.58 yankee.lappyfamily.net  yankee localhost 204538 localhost.localdomain

And this was clearly a misconfiguration (I am sure 204538 is a good hostname).

I looked at my local Ubuntu installation and updated the VPS so that the hosts file became:

127.0.0.1 localhost
::1 ip6-localhost ip6-loopback
# Auto-generated hostname. Please do not remove this comment.
173.212.238.58 yankee.lappyfamily.net  yankee

After this I restarted all the applications that were supposed to listen on the loopback interface and verified the fix with netstat again.

First of all, you need to have a firewall configured on your servers and allow only trusted incoming connections to trusted applications. This is what prevented my opendkim installation from accepting the incoming requests from the internet.

Second, you need to verify that localhost actually refers to the loopback interface and does not resolve to your public one, as you have a fully qualified name for that purpose.

I found that now the control panel for the VPS I am using correctly generates the hostname line, but it may not have been the case a year ago when I got the VPS first configured.

Importing disqus comments into WordPress

Roman Yepishev — Tue, 04 Dec 2012 21:43:00 GMT

Long story short: this is not trivial.

Disqus is a popular service that provides commenting functionality. All you need to do for your HTML page to have embedded discussion is to add a bit of javascript.

They have awesome importers from various blog engines. This blog was originally hosted on Blogspot, migration to static blog generated by Octopress required me to search for alternative commenting facilities and I decided to use disqus (is there anything else, really?) Now I switched to WordPress and I wanted my comments back.

Disqus does provide the ability to export all the comments in a XML file. Quick Google search told me that it was possible to get the comments quite easily into WordPress by converting the disqus dump into WXR, but it is not really that easy. WordPress will ignore the post if it is already there and will not import comments. There’s a plugin that imports disqus comments, but I wanted full support of nested comments and little to no PHP coding.

I dumped all the wordpress database locally and hacked up a script that reads the comments.xml file and puts the necessary data into the database. The script needs access to some sort of a database in order to figure out the post_id for each post and generate the comment identifiers for correct nesting.

#!/usr/bin/python """Quick and dirty hack to get disqus comments into WordPress DB""" import sys from datetime import datetime from xml.etree import ElementTree as ET import MySQLdb ANONYMOUS_EMAIL = 'nobody@example.net' # Database configuration DATABASE = { 'host': 'lab.lappyfamily.net', 'user': 'rtg', 'name': 'rtginua6_wp1', } # Admin information ADMIN_INFO = { 'comment_author': 'Roman Yepishev', 'comment_author_email': 'roman.yepishev@yandex.ua', 'user_id': 1 } # Names used in disqus that represent administrator ADMIN_ALIASES = set(['rtg', 'Roman', 'rye']) class DisqusImporter(object): """ Imports Disqus XML into MySQL database for WordPress """ def __init__(self): self.wpdb = MySQLdb.connect(host=DATABASE['host'], user=DATABASE['user'], db=DATABASE['name'], charset='utf8') self.wp_post_url_to_id = {} def make_wordpress_url_map(self): """Creates URL->ID map for WordPress URLS""" cursor = self.wpdb.cursor() cursor.execute(""" SELECT guid, id FROM wp_posts WHERE post_type = 'post' """) for row in cursor: self.wp_post_url_to_id[row[0]] = row[1] def parse_disqus_comments(self, path): """Parse comments creating WP-like structure""" NS = '{http://disqus.com}' NS_DI = '{http://disqus.com/disqus-internals}' tree = ET.parse(path) root = tree.getroot() comments = {} thread_id_to_url = {} # Gathering post threads identifiers. # Each thread corresponds to a blog post for thread in root.findall(NS + 'thread'): dsq_id = thread.attrib[NS_DI + 'id'] link = thread.find(NS + 'link') thread_id_to_url[dsq_id] = link.text # Parsing posts for post in root.findall(NS + 'post'): dsq_id = post.attrib[NS_DI + 'id'] thread_id = post.find(NS + 'thread').attrib[NS_DI + 'id'] # If we don't have the mapping from post to # WordPress post ID, we can't # proceed with this comment thread_url = thread_id_to_url[thread_id] if thread_url not in self.wp_post_url_to_id: print "Skipping comment for {}".format(thread_url) continue created_at = post.find(NS + 'createdAt').text author = post.find(NS + 'author') email = author.find(NS + 'email').text name = author.find(NS + 'name').text parent = post.find(NS + 'parent') if parent is not None: parent_id = parent.attrib[NS_DI + 'id'] else: parent_id = None # MySQL issues a warning if we stuff data in YYY-mm-ddTHH:MM:SSZ comment_date = datetime.strptime(created_at, '%Y-%m-%dT%H:%M:%SZ' ).strftime('%Y-%m-%d %H:%M:%S') comment_post_id = self.wp_post_url_to_id[thread_url] post_data = { 'comment_post_ID': comment_post_id, 'comment_content': post.find(NS + 'message').text, 'comment_date': comment_date, 'comment_date_gmt': comment_date, 'comment_author': name, 'comment_author_IP': post.find(NS + 'ipAddress').text, 'comment_author_email': email if email else ANONYMOUS_EMAIL, 'user_id': 0, 'parent_id': parent_id, 'children': [] } # Fixup for my own comments if name in ADMIN_ALIASES: post_data.update(ADMIN_INFO) comments[dsq_id] = post_data # First pass - creating comment tree for comment in comments.values(): if comment['parent_id']: parent_comment = comments[comment['parent_id']] parent_comment['children'].append(comment) # Second pass - dropping posts that are not toplevel # They are already in 'children' for comment_id in comments.keys(): # If it is still here (we could have deleted it) if comment_id in comments: comment = comments[comment_id] else: continue if comment['parent_id']: del comments[comment_id] return comments def add_comment(self, comment, parent_id): """Add comment and all the child comments to the DB""" cursor = self.wpdb.cursor() comment['comment_parent'] = parent_id cursor.execute(""" INSERT INTO wp_comments ( comment_post_ID, comment_author, comment_author_email, comment_author_IP, comment_date, comment_date_gmt, comment_content, comment_parent, user_id) VALUES ( %(comment_post_ID)s, %(comment_author)s, %(comment_author_email)s, %(comment_author_IP)s, %(comment_date)s, %(comment_date_gmt)s, %(comment_content)s, %(comment_parent)s, %(user_id)s) """, comment ) parent_id = cursor.lastrowid for item in comment['children']: self.add_comment(item, parent_id) def update_comment_count(self): """ Synchronizes cached comment count with the actual number of comments """ cursor = self.wpdb.cursor() # Update post counts cursor.execute(""" UPDATE wp_posts AS p LEFT JOIN ( SELECT comment_post_ID, count(comment_post_ID) as comment_count FROM wp_comments WHERE comment_approved = '1' GROUP BY comment_post_ID ) as c ON p.id = c.comment_post_ID SET p.comment_count = c.comment_count WHERE p.id = c.comment_post_ID """) def main(self, path): """Entry point""" self.make_wordpress_url_map() comments = self.parse_disqus_comments(path) for item in comments.values(): self.add_comment(item, 0) self.update_comment_count() if __name__ == "__main__": importer = DisqusImporter() importer.main(sys.argv[1])

...
ANONYMOUS_EMAIL = 'nobody@example.net'

# Database configuration
DATABASE = {
    'host': 'lab.lappyfamily.net',
    'user': 'rtg',
    'name': 'rtginua6_wp1',
}

# Admin information
ADMIN_INFO = {
    'comment_author': 'Roman Yepishev',
    'comment_author_email': 'roman.yepishev@yandex.ua',
    'user_id': 1
}

# Names used in disqus that represent administrator
ADMIN_ALIASES = set(['rtg', 'Roman', 'rye'])
...

And the script itself runs on an uncompressed disqus XML. The script will skip all pages that it could not find the URLs for:

$ python wp-import-disqus.py comments.xml
Skipping comment for http://rtg.in.ua/app/acer-exif-fixup/index.html

After script finished I dumped the comments table and uploaded it via the phpmyadmin interface.

$ mysqldump -h lab rtginua6_wp1 wp_comments > wp_comments.sql

This resulted in all 140 comments being correctly imported in a correct encoding and properly nested.

If you happen to know a better way to import comments, feel free to provide the links to alternative solutions.

opendkim update in Lucid 10.04 LTS

Roman Yepishev — Thu, 22 Nov 2012 03:44:00 GMT

A few days ago I decided to upgrade the packages on my Lucid VPS and today I noticed that there is a queue of unsent messages on my VM host with “Service unavailable” response from my mail relay on the VPS host.

Upon closer examination I found that the mail log was full of:

Nov 18 06:45:25 yankee postfix/smtpd[25869]: warning: connect to Milter
service inet:localhost:8891: Connection refused
Nov 18 06:45:25 yankee postfix/smtpd[25869]: NOQUEUE: milter-reject: CONNECT
from unknown[2a01:d0:801a:1::12]: 451 4.7.1 Service unavailable - try again
later; proto=SMTP

What happened is I got a new opendkim package (2.6.8-0ubuntu1~ubuntu10.04.1) which listens on a unix domain socket by default, and not inet interface which I had configured for the filters.

The fix was to update /etc/opendkim.conf and add this:

Socket          inet:8891@localhost

Alternatively, this can be configured in /etc/default/opendkim:

# Uncomment to specify an alternate socket
# Note that setting this will override any Socket value in opendkim.conf
#SOCKET="local:/var/run/opendkim/opendkim.sock" # default
#SOCKET="inet:54321" # listen on all interfaces on port 54321
#SOCKET="inet:12345@localhost" # listen on loopback on port 12345
#SOCKET="inet:12345@192.0.2.1" # listen on 192.0.2.1 on port 12345

This took a while since I could not recall how I configured opendkim to listen to tcp socket in the first place. At least now it is obvious.

Securing wordpress admin with SSL – The hard way

Roman Yepishev — Tue, 20 Nov 2012 10:31:00 GMT

I have since moved away from WordPress back to a static generator, so everything below can be considered an outdated information.

Having performed initial migration from Octopress to a hosted WordPress installation I found that I became in charge of login security. The hosting provider I am currently using provides neither dedicated IP addresses nor shared SSL certificates (update: it does provide a dedicated IP address upon request, but requires a support ticket and a small fee for that) it means that my WordPress password and session information is transmitted in clear text. Of course, when I am at home it’s not a big deal, only the ISP may intercept it. But when I am anywhere else, my devices can connect to some unprotected Wi-Fi access point and accidentally surrender my credentials to somebody running tcpdump nearby…

Update: Since I found out that the hosting provider actually provides a dedicated IP upon request for a quite small fee (25₴ at the moment, around $3), the value of the article has decreased. However, it is perfectly usable when nginx and wordpress are installed on the same host/datacenter but nginx is fronting apache.

I have a VPS which is used for occasional VPN access (using OpenVPN). One option was to always start OpenVPN client before accessing the blog. I could easily forget to do that. My browser may open the tab where I am already logged in and then my session cookie can be easily intercepted. Also I can’t use this VPS for WordPress hosting itself because it has only 512Mb of RAM and bringing up a SQL server, HTTP server (as well as keeping existing mail server setup, bip and OpenVPN running) may cause something to fall outside the available memory. Having said that, it looks like it is possible to run a WordPress blog in a similar configuration with whopping 10 million hits a day.

I decided to make this VPS act as a proxy server, which listens to HTTP and HTTPS traffic and forwards the requests to the hosting provider. In this case everything will pass through the VPS.

WordPress already has the options to force https scheme on wp-login.php and wp-admin directory.

Since this VPS is a low-profile one, a slim HTTP server is required. While I am already familiar with Lighttpd, I decided to try nginx.

You will need a SSL certificate. StartSSL is a good place to get one for free and it is recognized by major browsers out of the box. Approving new accounts takes time so I decided to use the old proven self signed certificate first and then change it to a proper one.

Here I would like to remind how important StartSSL client certificate is. You get the client certificate upon account approval and then use it to log into StartSSL control panel. This certificate will eventually expire and if you fail to renew it, you will never be able to get back to your account. Don’t repeat my mistake – when StartSSL tells you the certificate is about to expire, then get a new one immediately.

The VPS is running Ubuntu 10.04 LTS and (naturally) nginx version is quite old (0.7.65, February 2010). Nginx team has a PPA that builds packages for 10.04 too, so installation was simple:

sudo apt-add-repository ppa:nginx/stable
sudo apt-get update
sudo apt-get install nginx

Nginx comes with a proxy cache module. At first I started with a simple configuration that cached everything, then I found that the pages that were rendered for some commenter were also served to other people disclosing the previous commenter’s email. This is why preventing serving the cached content to the owners of our cookies is so important.

nic.ua also uses nginx as a proxy in front of apache. Therefore it is overwriting the X-Real-IP header which my proxy is setting. In order to work around that I used X-Upstream-Real-IP header, which I use in .htaccess to set UPSTREAM_PROXY_TRUSTED variable. The latter is then checked in wp-config.php to decide whether X-Forwarded-Proto and X-Upstream-Real-IP headers can be trusted. These X- headers can easily be forged and X-Forwarded-Proto should only be used if it comes from the trusted sources, or else the MITM attacks become extremely easy.

/etc/nginx/sites-available/rtg.in.ua:

sites-available-rtg-in-ua.apache (Source)

server {
  listen [::]:80;
  server_name www.rtg.in.ua blog.rtg.in.ua;
  rewrite ^ http://rtg.in.ua$request_uri? permanent;
}

upstream wordpress {
  server 91.209.206.62:80;
}

proxy_cache cache;

server {
  include proxy_params;
  
  # sites-available/default should also have listen directive changed
  # for IPv6 & IPv4 to work to 
  #   listen   [::]:80 default_server;
  listen [::]:80;
  server_name  rtg.in.ua;

  access_log  /var/log/nginx/rtg-in-ua.access.log;

  location / {
    proxy_pass http://wordpress;
    proxy_cache_valid 404 1m;

    # we should not serve cached version if we have one of these cookies
    if ($http_cookie ~* "wordpress|comment_author|wp-postpass_") {
      set $bypass_cache 1;
    }

    proxy_cache_bypass $bypass_cache;
  }

  # These are completely static and can be shared between
  # HTTP and HTTPS virtual hosts
  location ~* \.(jpg|jpeg|png|gif|css|js|mp3|wav|swf|ogg|txt) {
    proxy_cache_key $host$request_uri;
    proxy_cache_valid 200 120m;
    # 30 days
    expires 2592000;
    proxy_pass http://wordpress;
  }

  location ~* (^|\/)feed\/ {
    proxy_cache_valid 200 60m;
    proxy_pass http://wordpress;
  }

}

# HTTPS server
# We allow all interaction to happen over HTTPS too.
server {
  listen   [::]:443;
  server_name  rtg.in.ua;
  include proxy_params;

  ssl  on;
  ssl_certificate     /etc/ssl/certs/rtg-in-ua.crt;
  ssl_certificate_key /etc/ssl/private/server.key;

  ssl_session_timeout  5m;

  ssl_protocols  SSLv2 SSLv3 TLSv1;
  ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
  ssl_prefer_server_ciphers   on;

  access_log  /var/log/nginx/rtg-in-ua.access.log;

  location / {
    proxy_cache cache;
    proxy_cache_valid 404 1m;

    if ($http_cookie ~* "wordpress|comment_author|wp-postpass_") {
       set $bypass_cache 1;
    }

    proxy_cache_bypass $bypass_cache;
    proxy_pass  http://wordpress;
  }

  location ~* ^(wp-admin|wp-login) {
    proxy_pass  http://wordpress;
  }

  location ~* \.(jpg|jpeg|png|gif|css|js|mp3|wav|swf|ogg|txt) {
    proxy_cache_key $host$request_uri;
    proxy_cache_valid 200 120m;
    expires 2592000;
    proxy_pass http://wordpress;
  }

  location ~* (^|\/)feed\/ {
    proxy_cache_valid 200 60m;
    proxy_cache cache;
    proxy_pass http://wordpress;
  }
}

/etc/nginx/conf.d/proxy-cache.conf:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=cache:30m max_size=1G;
proxy_temp_path /var/lib/nginx/proxy 1 2;
proxy_cache_key $scheme$host$request_uri;

/etc/nginx/proxy_params:

proxy_set_header Host $host;
# This gets overwritten:
proxy_set_header X-Real-IP $remote_addr;
# This is our own header:
proxy_set_header X-Upstream-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

While it looks like I am just re-setting proxy_cache_key to the default value, nginx is actually using backend URL by default (contrary to the docs!), thus you will be serving a document full of generated HTTP links on a HTTPS connection and vice versa.

I decided to create a new directory for the cache:

sudo mkdir -m 0700 /var/cache/nginx
sudo chown www-data:root /var/cache/nginx

Enabled the virtual host and restarted the server:

sudo ln -s /etc/nginx/sites-available/rtg.in.ua /etc/nginx/sites-enabled
sudo /etc/init.d/nginx restart

Nginx is configured, the proxy server is ready to serve.

Now back to the WordPress installation, setting the UPSTREAM_PROXY_TRUSTED variable:

.htaccess:

# Trusted IPs
RewriteCond %{REMOTE_ADDR} =173.212.238.58 [OR]
RewriteCond %{REMOTE_ADDR} =2607:f878:1:654:0:25:3078:1 [OR]
RewriteCond %{REMOTE_ADDR} =178.159.236.29 [OR]
RewriteCond %{REMOTE_ADDR} =2a01:d0:801a:1::14
RewriteRule . - [E=UPSTREAM_PROXY_TRUSTED:1]

And now we need WordPress to redirect all the login and admin activities to https:// urls as well as trust X-Forwarded-Proto HTTP header by adding the following to wp-config.php:

...
/*
* http://codex.wordpress.org/Administration_Over_SSL
*/
define('FORCE_SSL_ADMIN', true);
define('FORCE_SSL_LOGIN', true);

/*
* This variable is set in .htaccess in case
* REMOTE_ADDR is one of our upstream proxies
*/
if ($_ENV['UPSTREAM_PROXY_TRUSTED']) {

    if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
    $_SERVER['HTTPS']='on';

    /*
    * X-Real-IP is overwritten by nginx proxy @ nic.ua
    * so we are using our own header.
    */
    $x_real_ip = $_SERVER['HTTP_X_UPSTREAM_REAL_IP'];

    // IPv4 in IPv6 notation
    if (substr($x_real_ip, 0, 7) == "::ffff:")
        $x_real_ip = substr($x_real_ip, 7);

    $_SERVER['REMOTE_ADDR'] = $x_real_ip;

}

/* That's all, stop editing! Happy blogging. */
...

That’s pretty much it. Self-signed, but encrypted:

Chromium SSL popup

This approach has several flaws. First of all, I am increasing the number of points of failure. Since the data has to travel from Ukraine to the US even when request comes from Ukraine, the latency is increased a bit. CDN usage also gets more complicated (unless static files are served on a different domain). Neither of these issues is currently bothering as I am still in the process of migrating my content.

Please note that I am not a nginx guru, so if you see any error in the configuration, let me know.