Fonera port

There has been some demand to get NNTPGrab operational on Fonera 2.0 devices. This page documents the current progress of this port

Preparing the build environment

Download and prepare the fon-ng repository:

svn co http://svn.fonosfera.org/fon-ng/trunk fon-ng
cd fon-ng
./install.sh
cd openwrt

Add the NNTPGrab packages to the Fonera/OpenWRT buildsystem

echo "src-svn nntpgrab http://svn.openftd.org/svn/openwrt" >> feeds.conf
./scripts/feeds update nntpgrab
./scripts/feeds install -a -p nntpgrab

All the dependencies of NNTPGrab (gettext, libiconv and GLib 2) are bundled in this repository, so you don't need to add the official OpenWRT repositories (it probably even conflicts for now). The plan is to get the changes which are required in the dependencies upstream (OpenWRT) as soon as everything runs stable

Now you can execute make menuconfig and create a configuration for the Fonera. NNTPGrab is split in 2 packages at the moment (libnntpgrab and nntpgrab-server) and can be found in the section Network -> Usenet. All other options can be left at their defaults. If you want to create a package for the Fonena 2.0n you need to indicate so

When the configuration has been created you can use make V=99 to build everything

All built packages can now be found in the folder bin/packages/mips (or bin/packages/mipsel if you're compiling for the Fonera 2.0n). You can now copy these files to the Fonera device using (for example) scp or by copying them to an USB stick:

glib2_2.20.3-1_mips.ipk
libiconv_1.11-1_mips.ipk
libintl_0.16.1-4_mips.ipk
libnntpgrab_*_mips.ipk
nntpgrab-server_*_mips.ipk

You can now install these packages using opkg. Please note that the Fonera itself doesn't have enough disk space to install everything, so you're recommended to install the packages to (for example) an USB stick.

Starting the NNTPGrab Server

When all the files have been compiled and installed on the Fonera, you can start the NNTPGrab Server with this command:

LD_LIBRARY_PATH=$PREFIX/usr/lib \
NNTPGRAB_LIBDIR=$PREFIX/usr/lib/nntpgrab \
NNTPGRAB_CONFIG_DIR=/tmp/run/mountd/sda1 \
NNTPGRAB_DEFAULT_DL_DIR=/tmp/run/mountd/sda1 \
NNTPGRAB_WWWDIR=$PREFIX/usr/share/nntpgrab/web \
$PREFIX/usr/bin/nntpgrab_server

The $PREFIX is the location where the files have been installed using opkg.
The /tmp/run/mountd/sda1 is the path to a connected hard drive where all settings and downloads can be saved
The --background argument can be used to start the program in the background

You can now connect to the NNTPGrab Server by using NNTPGrab 0.6.90 or later from your desktop environment. On startup, you're asked for an IP address of an NNTPGrab Server. This is the IP address of your Fonera (192.168.10.1 by default)

The webinterface can be reached on http://192.168.10.1:5423. When asked for an username and password, use admin/admin

Configuration

The location of the NNTPGrab configuration can be influenced with the environment variable NNTPGRAB_CONFIG_DIR (see above). If this environment variable isn't set, the default path will be used: ~/.config/NNTPGrab. On Fonera devices this means /tmp/.config/NNTPGrab. This folder is part of a RAM-disk, so it's contents will get lost on a reboot. It's really recommended to use this environment variable.

The default location for all downloads can be set with NNTPGRAB_DEFAULT_DL_DIR (as can be seen above). The environment variable only needs to be set the first time the program is used. Once the initial configuration has been created, the download paths can be set using the NNTPGrab GUI from your desktop environment. This can be done in the configuration tab in the GUI. You need to choose a folder which isn't part of the internal memory of the Fonera as this is too limited

Another thing which needs to be done in the configuration is to disable the options 'Automatically perform a PAR2 repair' and 'Automatically unpack files'. This is because the Fonera hardware is too limited for these features to operate and requires some packages which currently aren't packaged yet (par2 and unrar)

After this initial configuration is done you can import files in the program and the download will start automatically

Dependency tree

The entire dependency tree looks like this:

nntpgrab-server
     |
     +----------> libnntpgrab
                      |
                      +--------> glib2
                      |            |
                      |            +-------> libiconv (for character encodings)
                      |            |
                      |            +-------> gettext (for translations)
                      |             
                      +--------> cyaSSL (for connecting to NNTP servers over SSL)

Found issues

  • The GLib2 package from the OpenWRT repository doesn't compile with the Fonera buildsystem
    • The tarball with the source code cannot be downloaded as @GNOME isn't known to to buildsystem
    • When compiling the package, the wrong version of the glib-genmarshal tool is used (the version for the MIPS architecture is used instead of one which runs on the native environment)
  • The libxml2 package from the OpenWRT repository doesn't compile with the Fonera buildsystem
    • The command make install is never executed so the resulting binaries/libraries can't be found. A workaround for this was added
  • A bug in the uClibc function sscanf was detected. The testcase at http://ftd4linux.nl/contrib/fonera_sscanf_testcase.c returns '0' on uClibc environment while on GLibc environments, the value '1' is returned. A workaround to this bug was added to the NNTPGrab package

TODO

The TODO list of the Fonera port of NNTPGrab has been split in two sections, generic task and Fonera-specific tasks.
The tasks mentioned in the generic list benefit users of NNTPGrab on all environments and are more or less already part of the Roadmap

Generic tasks:
  • Rewrite the decoder plugin to be more efficient done!
  • Make the server plugin more CPU efficient (get rid of the various regular expressions) done!
  • Profile the memory usage of NNTPGrab and identify spots which can be improved
  • Reduce the number of threads involved (most ideal would be one thread for all downloads/server connections) done!
  • Implement a plugin infrastructure which makes it easy to improve integration in specific environments done!
Fonera specific tasks:
  • Create a LUA-binding which can communicate with NNTPGrab
  • Create a WebUI which can communicate with NNTPGrab using the LUA-binding
  • Replace the OpenSSL dependency in the nntp code with a SSL library which is already part of the Fonera firmware (axTLS or CyaSSL) done!
  • Rewrite the configuration backend so that UCI is used to maintain the settings
  • Change the default configuration so that all downloads will be saved to an external drive by default instead of to the limited Fonera memory done!

Binaries

Binaries of NNTPGrab for the Fonera and it's dependencies are published at http://snapshots.nntpgrab.nl/fonera. This is all a work-in-progress with known issues, so there are no stability guarantees yet

Also available in: HTML TXT