If all information is collected, type the command quit in the terminal to close the debugger and NNTPGrab

Developer documentation¶

If you want to help the development of NNTPGrab or extend it, documentation can be found here. If something is not clear enough on these pages, feel free to post a message on the forum

API Documentation
Control NNTPGrab using JSON-RPC
Control NNTPGrab using PHP
Control NNTPGrab using a custom TCP protocol (obsoleted)
Developing plugins
Developing on Mac OS X
Fonera port
eTrayz port
Help wanted

Developing on Mac OS X¶

This page will document the steps required to get started on developing NNTPGrab on Mac OS X environments

Requirements¶

• Mac OS X 10.5 or higher
• XCode
• Qt SDK

Environment¶

NNTPGrab requires some external dependencies which are not bundled with XCode or the Qt SDK. These can be installed by performing these commands in the Terminal:

mkdir -p ~/Development/NNTPGrab
cd ~/Development
svn co https://svn.openftd.org/svn/build_env/osx NNTPGrab

When using Qt Creator, some environment variables need to be set to allow the build scripts to detect all the external dependencies which are required for NNTPGrab. Please perform these commands in the Terminal:

echo ~/Development/NNTPGrab/deps/bin > ~/tmp
sudo mv ~/tmp /etc/paths.d/NNTPGrab

cp /etc/profile ~/tmp
chmod 644 ~/tmp
echo "export PKG_CONFIG_LIBDIR=~/Development/NNTPGrab/deps/lib/pkgconfig" >> ~/tmp
sudo mv ~/tmp /etc/profile
sudo chown root:wheel /etc/profile
source /etc/profile

cd ~/Development/NNTPGrab/deps/share/aclocal
sudo cp glib-2.0.m4 glib-gettext.m4 intltool.m4 nls.m4 pkg.m4 /usr/share/aclocal

cd ~/Development/NNTPGrab/deps/lib
sed -i '' "s|PREFIX|`echo ~`/Development/NNTPGrab/deps|g" libgio-2.0.la
sed -i '' "s|PREFIX|`echo ~`/Development/NNTPGrab/deps|g" libglib-2.0.la
sed -i '' "s|PREFIX|`echo ~`/Development/NNTPGrab/deps|g" libgmodule-2.0.la
sed -i '' "s|PREFIX|`echo ~`/Development/NNTPGrab/deps|g" libgobject-2.0.la
sed -i '' "s|PREFIX|`echo ~`/Development/NNTPGrab/deps|g" libgthread-2.0.la
sed -i '' "s|PREFIX|`echo ~`/Development/NNTPGrab/deps|g" libintl.la
sed -i '' "s|PREFIX|`echo ~`/Development/NNTPGrab/deps|g" libsoup-2.4.la

cd ~/Development/NNTPGrab/deps/lib/pkgconfig
sed -i '' "s|PREFIX|`echo ~`|" gio-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" gio-unix-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" glib-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" gmodule-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" gmodule-export-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" gmodule-no-export-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" gobject-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" gthread-2.0.pc
sed -i '' "s|PREFIX|`echo ~`|" libsoup-2.4.pc

Restart the Terminal application now to allow the new environment variables to be set

Retrieving and preparing the source code¶

Retrieve the NNTPGrab source code and prepare it with these commands:

cd ~/Development/NNTPGrab
svn co https://svn.openftd.org/svn/nntpgrab/trunk src
cd src
./autogen.sh
./configure \
   --prefix=`echo ~`/Development/NNTPGrab/dest \
   --disable-gui \
   --disable-webinterface \
   --disable-dependency-tracking \
   CPPFLAGS="-I`echo ~`/Development/NNTPGrab/deps/include" \
   CFLAGS="-arch ppc -arch i386 -arch x86_64" \
   DYLD_LIBRARY_PATH="`echo ~`/Development/NNTPGrab/deps/lib" 
make

Qt Creator¶

Now everything's ready we can open the project in Qt Creator. Start Qt Creator and select ''File'' -> ''Open File or Project''. Now navigate to the folder Development/NNTPGrab/src and select the file ''NNTPGrab.creator''

The Qt version of the NNTPGrab Server is currently bundled as a seperate project. To work on this, open the project file ''server_qt/server_qt.pro''

The Qt version of the NNTPGrab GUI is currently bundled as a seperate project. To work on this, open the project file ''client/gui_qt/gui_qt.pro''

Downloads¶

The latest released version of NNTPGrab is version 0.7.2 which is released on Monday May 28 2012

The download mirrors are provided by the following persons / companies:

Test versions and Subversion¶

All the files mentioned on this page are stable releases. Looking for the latest test versions and snapshot releases of NNTPGrab instead? Go here
Information about the Subversion repository where all development takes place can be found at the Subversion page

Windows¶

NNTPGrab requires Windows 2000 or higher.
The setup can be found at nntpgrab_0_7_2_Setup.exe
A x64 version of NNTPGrab is also available: nntpgrab_0_7_2_Setup64.exe

Mac OS X¶

NNTPGrab requires Mac OS X 10.5 (Leopard) or higher.
The standalone package can be found at NNTPGrab-0.7.2.zip
The Server package can be found at NNTPGrab-Server-0.7.2.zip
The (deprecated) Gtk frontend can be found at NNTPGrab-Gtk-0.7.2.zip

Fedora Linux¶

You can install NNTPGrab by using the command yum install nntpgrab or by using the graphical software manager to install the 'nntpgrab' package. No additional configuration is necessary.

Future releases of NNTPGrab will automatically be installed when you perform the command yum update or use the graphical update program from Fedora.

If you want to have faster access to newer versions of NNTPGrab, you need to install this package

CentOS / Red Hat Enterprise Linux¶

First you need to have access to the EPEL-repository. Instructions how to do so can be found here

Afterwards you can install NNTPGrab by using the command yum install nntpgrab or by using the graphical software manager to install the 'nntpgrab' package. No additional configuration is necessary.

Future releases of NNTPGrab will automatically be installed when you perform the command yum update or use the graphical update program from CentOS

If you want to have access to test versions of NNTPGrab, you need to install this package

Gentoo Linux¶

First make sure you have installed the package layman.
Now perform the following commands:

layman -o https://www.nntpgrab.nl/releases/gentoo/overlay.xml -f
layman -o https://www.nntpgrab.nl/releases/gentoo/overlay.xml -a nntpgrab

If you already have OpenFTD installed you can skip the above commands

You can now install NNTPGrab by using the command:

emerge -av nntpgrab

If you receive a error mentioning the package 'nntpgrab' isn't found, you need
to add the following line to the file /etc/make.conf:

source /usr/local/portage/layman/make.conf

If this does not work, add the following line to /etc/make.conf:

PORTAGE_OVERLAY="/usr/local/portage/layman/nntpgrab $PORTAGE_OVERLAY"

To retrieve updates, you need to use the following commands:

layman -S
emerge -av nntpgrab

OpenSuSE 12.1 / 12.2 / 12.3 / 13.1 - SuSE Linux Enterprise 11¶

To prepare your environment, start YaST by clicking on it under System in your menu or by typing yast in the run command box (press Alt+F2). Select Installation Source. Click on Add, Select enter URL.

For OpenSuSE 13.1: Use the following address for the repository URL https://www.nntpgrab.nl/releases/openSUSE_13.1

For OpenSuSE 12.3: Use the following address for the repository URL https://www.nntpgrab.nl/releases/openSUSE_12.3

For OpenSuSE 12.2: Use the following address for the repository URL https://www.nntpgrab.nl/releases/openSUSE_12.2

For OpenSuSE 12.1: Use the following address for the repository URL https://www.nntpgrab.nl/releases/openSUSE_12.1

For SuSE Linux Enterprise 11.0: Use the following address for the repository URL https://www.nntpgrab.nl/releases/SLE_11

Next you can install the NNTPGrab package by starting YaST. Select Software. Now search for the package 'nntpgrab' and install it.

Future releases of NNTPGrab will now automatically be installed when they are available

Ubuntu¶

Read this page for instructions how to install NNTPGrab on Ubuntu

Other / Source Code¶

The source code can be found at nntpgrab-0.7.2.tar.bz2

Old versions¶

NNTPGrab 0.1: Windows Mac OS X Source code
NNTPGrab 0.2: Windows Mac OS X Source code
NNTPGrab 0.2.1: Windows Mac OS X Source code
NNTPGrab 0.2.2: Windows Mac OS X Source code
NNTPGrab 0.2.3: Windows Mac OS X Source code
NNTPGrab 0.2.4: Windows Mac OS X Source code
NNTPGrab 0.2.5: Windows Mac OS X Source code
NNTPGrab 0.3.0: Windows Mac OS X Standalone Mac OS X Networked Source code
NNTPGrab 0.3.1: Windows Mac OS X Standalone Mac OS X Networked Source code
NNTPGrab 0.3.2: Windows Mac OS X Standalone Mac OS X Networked Source code
NNTPGrab 0.3.3: Windows Mac OS X Standalone Mac OS X Networked Source code
NNTPGrab 0.4.0: Windows Mac OS X Standalone Mac OS X Networked Mac OS X Server Source code
NNTPGrab 0.4.1: Windows Mac OS X Standalone Mac OS X Networked Mac OS X Server Source code
NNTPGrab 0.4.2: Windows Mac OS X Standalone Mac OS X Networked Mac OS X Server Source code
NNTPGrab 0.5.0: Windows Mac OS X Mac OS X Server Source code
NNTPGrab 0.5.1: Windows Mac OS X Mac OS X Server Source code
NNTPGrab 0.6.0: Windows Mac OS X Mac OS X Server Source code
NNTPGrab 0.6.1: Windows Mac OS X Mac OS X Server Source code
NNTPGrab 0.6.2: Windows 32bit Windows 64bit Mac OS X Mac OS X Server Source code
NNTPGrab 0.7.0: Windows 32bit Windows 64bit Mac OS X Mac OS X Server Mac OS X GTK Source code
NNTPGrab 0.7.1: Windows 32bit Windows 64bit Mac OS X Mac OS X Server Mac OS X GTK Source code

Download testing and unstable releases¶

The latest unstable version of NNTPGrab is version 0.6.92 (0.7 Beta 3) which is released on Wednesday July 20 2011

The download mirrors are provided by the following persons / companies:

Snapshot builds¶

Looking for experimental builds of NNTPGrab? These are published at the following addresses:

NNTPGrab 0.7 branch (testing)
Windows (x86): https://snapshots.nntpgrab.nl/win32/stable-0.7
Windows (x64): https://snapshots.nntpgrab.nl/win64/stable-0.7
Mac OS X: https://snapshots.nntpgrab.nl/osx/stable-0.7

NNTPGrab trunk branch (experimental)
Windows (x86): https://snapshots.nntpgrab.nl/win32/unstable-trunk
Windows (x64): https://snapshots.nntpgrab.nl/win64/unstable-trunk
Mac OS X: https://snapshots.nntpgrab.nl/osx/unstable-trunk

Other snapshots
All other available snapshots can be found at: https://snapshots.nntpgrab.nl

Please note that the snapshots from the trunk branch can be highly experimental from time to time

The links mentioned below are all testing releases which should be reasonable stable

Windows¶

NNTPGrab requires Windows 2000 or higher. The setup can be found at nntpgrab_0_6_92_Setup.exe. The setup for x64 versions of Windows can be found at nntpgrab_0_6_92_Setup64.exe

Mac OS X¶

NNTPGrab requires Mac OS X 10.5 (Leopard) or higher. The standalone package can be found at NNTPGrab-0.6.92.zip. The Server package can be found at NNTPGrab-Server-0.6.92.zip. The (deprecated) GTK-based package can be found at NNTPGrab-Gtk-0.6.92.zip

Fedora Linux¶

To enable the test versions of NNTPGrab, start the Add/remove software program (which can be found in System -> Administration), click on System -> Software Sources and enable the repository NNTPGrab for Fedora X - xxxxx - Testing.

New test versions will now be automatically pulled in with the regular updates

CentOS / Red Hat Enterprise Linux 5 and 6¶

Please download and install this package

To enable the test versions of NNTPGrab, open the file /etc/yum.repos.d/nntpgrab.repo in an editor and change the option enabled=0 to enabled=1 in the [nntpgrab-testing] section.

New test versions will now be automatically pulled in with the regular updates

Afterwards you can install NNTPGrab by using the command yum install nntpgrab or by using the graphical software manager to install the 'nntpgrab' package. No additional configuration is necessary.

Future releases of NNTPGrab will automatically be installed when you perform the command yum update or use the graphical update program from CentOS.

Gentoo Linux¶

First follow the regular Gentoo installation instructions

Add the following line to the file /etc/portage/package.keywords:

net-nntp/nntpgrab ~*
www-apps/nntpgrab-web ~*

Perform an emerge --update to install the latest unstable version

OpenSuSE 12.1 / 12.2 / 12.3 /13.1 - SuSE Linux Enterprise 11¶

To prepare your environment, start YaST by clicking on it under System in your menu or by typing yast in the run command box (press Alt+F2). Select Installation Source. Click on Add, Select enter URL.

For OpenSuSE 13.1: Use the following address for the repository URL https://www.nntpgrab.nl/releases/testing/openSUSE_13.1

For OpenSuSE 12.3: Use the following address for the repository URL https://www.nntpgrab.nl/releases/testing/openSUSE_12.3

For OpenSuSE 12.2: Use the following address for the repository URL https://www.nntpgrab.nl/releases/testing/openSUSE_12.2

For OpenSuSE 12.1: Use the following address for the repository URL https://www.nntpgrab.nl/releases/testing/openSUSE_12.1

For SuSE Linux Enterprise 11.0: Use the following address for the repository URL https://www.nntpgrab.nl/releases/testing/SLE_11

Next you can install the NNTPGrab package by starting YaST. Select Software. Now search for the package 'nntpgrab' and install it.

Future test releases of NNTPGrab will now automatically be installed when they are available

Ubuntu¶

Read this page for instructions how to install NNTPGrab on Ubuntu. Please note that the testing repository uses a different address than the stable releases!

Other / Source Code¶

The source code can be found at nntpgrab-0.6.92.tar.bz2

eTrayz port¶

The eTrayz is a NAS which runs on Linux and has a number of interesting features. As it contains the Linux operating system it should be possible to get NNTPGrab operational on it. This wiki describes how NNTPGrab can be made operational on eTrayz devices.

Luckily enough, the default eTrayz installation contains all the dependencies of NNTPGrab (like GLib2 en libxml2) and contains all development tools so that makes things much more easy than other embedded devices.

Installation¶

A script has been created to make compilation of NNTPGrab as easy as possible. First you need to log in to the eTrayz using SSH

When you're logged in using SSH, perform these commands:

mkdir /home/sysadmin/nntpgrab
cd /home/sysadmin/nntpgrab
wget https://www.nntpgrab.nl/contrib/build_nntpgrab.sh
chmod +x build_nntpgrab.sh
./build_nntpgrab.sh

The last command can take a while to complete. It should end with this message:

NNTPGrab is successfully installed in /home/sysadmin/nntpgrab !
You can start the NNTPGrab Server with the command /home/sysadmin/nntpgrab/bin/start_nntpgrab_server

If you receive any other message, feel free to contact us using the forum

Eventually, this script should get replaced by a proper package, but as I (Alguno) am not familiar with that I'll leave to up to somebody else

Running¶

Now it should be possible to start the NNTPGrab Server using the command mentioned by the script: /home/sysadmin/nntpgrab/bin/start_nntpgrab_server
This should start NNTPGrab in the foreground. If you wish to run NNTPGrab in the background, add the argument --background to it

Issues¶

Right now there's one known issue. Automatic importing of .nzb files may cause the NNTPGrab Server to crash. Unfortunately I (Alguno) don't have the proper hardware here or access to one to diagnose this. If you happen to stumble across this bug, please try to create a backtrace of gdb. If you need help with this, please contact us in the forum

Feature requests¶

At the old website of NNTPGrab it was possible to request features using a special wiki page. For this new website things are slightly different.

Feature requests can be done by creating a new issue and using the 'Tracker: Feature'.

People who are interested in features which are requested by somebody else can indicate this by opening the feature request and using the vote function.

When creating a new feature request, please leave the 'Version' field empty as that will be used by the developers to indicate the version of NNTPGrab in which the feature will be implemented

Fonera port¶

There has been some demand to get NNTPGrab operational on Fonera 2.0 devices. This page documents the current progress of this port

Preparing the build environment¶

Download and prepare the fon-ng repository:

svn co https://svn.fonosfera.org/fon-ng/trunk fon-ng
cd fon-ng
./install.sh
cd openwrt

Add the NNTPGrab packages to the Fonera/OpenWRT buildsystem

echo "src-svn nntpgrab https://svn.openftd.org/svn/openwrt" >> feeds.conf
./scripts/feeds update nntpgrab
./scripts/feeds install -a -p nntpgrab

All the dependencies of NNTPGrab (gettext, libiconv and GLib 2) are bundled in this repository, so you don't need to add the official OpenWRT repositories (it probably even conflicts for now). The plan is to get the changes which are required in the dependencies upstream (OpenWRT) as soon as everything runs stable

Now you can execute make menuconfig and create a configuration for the Fonera. NNTPGrab is split in 2 packages at the moment (libnntpgrab and nntpgrab-server) and can be found in the section Network -> Usenet. All other options can be left at their defaults. If you want to create a package for the Fonena 2.0n you need to indicate so

When the configuration has been created you can use make V=99 to build everything

All built packages can now be found in the folder bin/packages/mips (or bin/packages/mipsel if you're compiling for the Fonera 2.0n). You can now copy these files to the Fonera device using (for example) scp or by copying them to an USB stick:

glib2_2.20.3-1_mips.ipk
libiconv_1.11-1_mips.ipk
libintl_0.16.1-4_mips.ipk
libnntpgrab_*_mips.ipk
nntpgrab-server_*_mips.ipk

You can now install these packages using opkg. Please note that the Fonera itself doesn't have enough disk space to install everything, so you're recommended to install the packages to (for example) an USB stick.

Starting the NNTPGrab Server¶

When all the files have been compiled and installed on the Fonera, you can start the NNTPGrab Server with this command:

LD_LIBRARY_PATH=$PREFIX/usr/lib \
NNTPGRAB_LIBDIR=$PREFIX/usr/lib/nntpgrab \
NNTPGRAB_CONFIG_DIR=/tmp/run/mountd/sda1 \
NNTPGRAB_DEFAULT_DL_DIR=/tmp/run/mountd/sda1 \
NNTPGRAB_WWWDIR=$PREFIX/usr/share/nntpgrab/web \
$PREFIX/usr/bin/nntpgrab_server

The $PREFIX is the location where the files have been installed using opkg.
The /tmp/run/mountd/sda1 is the path to a connected hard drive where all settings and downloads can be saved
The --background argument can be used to start the program in the background

You can now connect to the NNTPGrab Server by using NNTPGrab 0.6.90 or later from your desktop environment. On startup, you're asked for an IP address of an NNTPGrab Server. This is the IP address of your Fonera (192.168.10.1 by default)

The webinterface can be reached on https://192.168.10.1:5423. When asked for an username and password, use admin/admin

Configuration¶

The location of the NNTPGrab configuration can be influenced with the environment variable NNTPGRAB_CONFIG_DIR (see above). If this environment variable isn't set, the default path will be used: ~/.config/NNTPGrab. On Fonera devices this means /tmp/.config/NNTPGrab. This folder is part of a RAM-disk, so it's contents will get lost on a reboot. It's really recommended to use this environment variable.

The default location for all downloads can be set with NNTPGRAB_DEFAULT_DL_DIR (as can be seen above). The environment variable only needs to be set the first time the program is used. Once the initial configuration has been created, the download paths can be set using the NNTPGrab GUI from your desktop environment. This can be done in the configuration tab in the GUI. You need to choose a folder which isn't part of the internal memory of the Fonera as this is too limited

Another thing which needs to be done in the configuration is to disable the options 'Automatically perform a PAR2 repair' and 'Automatically unpack files'. This is because the Fonera hardware is too limited for these features to operate and requires some packages which currently aren't packaged yet (par2 and unrar)

After this initial configuration is done you can import files in the program and the download will start automatically

Dependency tree¶

The entire dependency tree looks like this:

nntpgrab-server
     |
     +----------> libnntpgrab
                      |
                      +--------> glib2
                      |            |
                      |            +-------> libiconv (for character encodings)
                      |            |
                      |            +-------> gettext (for translations)
                      |             
                      +--------> cyaSSL (for connecting to NNTP servers over SSL)

Found issues¶

• The GLib2 package from the OpenWRT repository doesn't compile with the Fonera buildsystem
  • The tarball with the source code cannot be downloaded as @GNOME isn't known to to buildsystem
  • When compiling the package, the wrong version of the glib-genmarshal tool is used (the version for the MIPS architecture is used instead of one which runs on the native environment)
• The libxml2 package from the OpenWRT repository doesn't compile with the Fonera buildsystem
  • The command make install is never executed so the resulting binaries/libraries can't be found. A workaround for this was added
• A bug in the uClibc function sscanf was detected. The testcase at https://ftd4linux.nl/contrib/fonera_sscanf_testcase.c returns '0' on uClibc environment while on GLibc environments, the value '1' is returned. A workaround to this bug was added to the NNTPGrab package

TODO¶

The TODO list of the Fonera port of NNTPGrab has been split in two sections, generic task and Fonera-specific tasks.
The tasks mentioned in the generic list benefit users of NNTPGrab on all environments and are more or less already part of the Roadmap

• Rewrite the decoder plugin to be more efficient done!
• Make the server plugin more CPU efficient (get rid of the various regular expressions) done!
• Profile the memory usage of NNTPGrab and identify spots which can be improved
• Reduce the number of threads involved (most ideal would be one thread for all downloads/server connections) done!
• Implement a plugin infrastructure which makes it easy to improve integration in specific environments done!

• Create a LUA-binding which can communicate with NNTPGrab
• Create a WebUI which can communicate with NNTPGrab using the LUA-binding
• Replace the OpenSSL dependency in the nntp code with a SSL library which is already part of the Fonera firmware (axTLS or CyaSSL) done!
• Rewrite the configuration backend so that UCI is used to maintain the settings
• Change the default configuration so that all downloads will be saved to an external drive by default instead of to the limited Fonera memory done!

Binaries¶

Binaries of NNTPGrab for the Fonera and it's dependencies are published at https://snapshots.nntpgrab.nl/fonera. This is all a work-in-progress with known issues, so there are no stability guarantees yet

Help Wanted¶

NNTPGrab is an open source project which means that anybody can help with the development of the program.
We're always open for external contributions.

• Translations (no programming experience required)
• QT frontend
• Native Mac OS X frontend
• C#/.NET frontend (already in development, but more help is appreciated)
• Web interface
• And of course people who are willing to implement new features or fix bugs in the program

If you think you can help us, don't hesitate to contact us!
The fastest way to get in contact with us is through IRC or through the forum on this website

JSON-RPC Documentation¶

As of NNTPGrab 0.6 it is possible to control NNTPGrab using JSON-RPC. This requires the JSON-RPC plugin to be loaded in the program.

• JSON-RPC Documentation
• Installation
• Handshake
• JSON-RPC over HTTP
• JSON-RPC over TCP
• Authentication
• JSON-RPC over HTTP
• JSON-RPC over TCP
• Functions provided by the NNTPGrab JSON-RPC plugin
• nntpgrab_get_version_info
• nntpgrab_config_get_avail_servers
• nntpgrab_config_get_server_info
• nntpgrab_config_add_server
• nntpgrab_config_del_server
• nntpgrab_config_edit_server
• nntpgrab_config_get_opts
• nntpgrab_config_set_opts
• nntpgrab_schedular_start
• nntpgrab_schedular_stop
• nntpgrab_schedular_get_state
• nntpgrab_schedular_add_file_to_queue
• nntpgrab_schedular_del_file_from_queue
• nntpgrab_schedular_restart_file
• nntpgrab_schedular_save_queue
• nntpgrab_schedular_get_all_tasks
• nntpgrab_schedular_move_file
• nntpgrab_schedular_move_collection
• nntpgrab_schedular_mark_task_optional
• nntpgrab_plugins_get_avail_plugins
• nntpgrab_plugins_get_plugin_info
• nntpgrab_plugins_load_plugin
• nntpgrab_plugins_unload_plugin
• nntpgrab_plugins_set_persistent
• nntpgrab_utils_strip_subject
• nntpgrab_utils_calculate_file_size
• nntpgrab_utils_calculate_estimated_time_remaining
• nntpgrab_utils_get_readable_time_remaining
• nntpgrab_utils_get_readable_finished_time
• nntpgrab_utils_get_folder_listing
• Events provided by the NNTPGrab JSON-RPC plugin
• config_changed
• part_download_start
• part_done
• part_failed
• traffic_monitor_update
• part_progress_update
• collection_added
• collection_removed
• collection_modified
• file_added
• file_removed
• file_download_state_update
• file_state_changed
• connection_connecting
• connection_connected
• connection_disconnect
• schedular_state_changed
• log_message
• task_moved
• collection_moved
• all_downloads_completed
• plugin_loaded
• plugin_unloaded
• plugin_event
• Plugin events
• PAR2 plugin
• par2_begin_verify
• par2_load_progress_update
• par2_recovery_file_loaded
• par2_file_loaded
• par2_repair_progress_update
• par2_repair_failure
• par2_repair_success
• par2_no_repair_required
• Unpack plugin
• unpack_progress_update
• unpack_message_received
• unpack_working_archive_changed

Installation¶

JSON-RPC support is provided by the JSON-RPC plugin which is installed along with the program by default. To activate the JSON-RPC server, you need to go to the configuration tab in NNTPGrab, select Plugins and make sure that the option Enable embedded webserver is checked. It's also possible to indicate the port number of which the JSON-RPC server needs to listen.

After the JSON-RPC server has been activated you can send JSON-RPC commands to the URL https://localhost:5423/jsonrpc (where localhost is the hostname or IP address of the host where NNTPGrab is running and 5423 is the port number which has been set in the NNTPGrab configuration.

Handshake¶

JSON-RPC over HTTP¶

When using JSON-RPC no special handshake is needed. It's sufficient to send a regular HTTP POST request containing JSON-RPC data to the server. The connection will automatically be closed by the server after every request as Mongoose (the embedded webserver library used by NNTPGrab) doesn't support HTTP keep-alive.

JSON-RPC over TCP¶

If possible it's recommended to use JSON-RPC over TCP. This is because NNTPGrab can emit events (notifications) at any time. When using JSON-RPC over TCP the connection doesn't need to be re-established after every request so this saves time and resources.

For JSON-RPC over TCP the same port as JSON-RPC over HTTP can be used, but instead of a HTTP request, a custom request needs to be sent by the client to the server. This way the webserver knows what method needs to be used.

To activate JSON-RPC over TCP the client needs to send this message to the server:

NNTPGrab - API version 20110625\r\n\r\n

The value 20110625 is the current API version as defined in the file nntpgrab_core/nntpgrab_plugins.h and the \r\n\r\n sequence are two newlines

In response, the server can send one of these replies:

OK\r\n
API mismatch, expected 20091228\r\n

If the server sends the OK response then the handshake is completed and JSON-RPC request can be sent to the server. The server can also send any events (notifications) back to the client at any moment

If the server sends the API mismatch response then a different version of the API is expected. The value 20091228 mentioned here is just an example, it can be different depending on the version of NNTPGrab installed. The connection will be closed automatically by the server if this response is emit

Authentication¶

JSON-RPC over HTTP¶

Authentication is required to use JSON-RPC over HTTP. This is realized by the HTTP authentication method Digest. The default username and password for NNTPGrab 0.6.x is admin/admin.

JSON-RPC over TCP¶

Authentication isn't implemented at the moment for JSON-RPC over TCP, but it will be in future versions. Check back later

Functions provided by the NNTPGrab JSON-RPC plugin¶

nntpgrab_get_version_info¶

nntpgrab_get_version_info - Returns several version numbers belonging to NNTPGrab

object nntpgrab_get_version_info

Returns an object with the following fields:

(
    [glue_version]
    [major_version]
    [minor_version]
    [micro_version]
    [version]
)

Available since NNTPGrab 0.6

nntpgrab_config_get_avail_servers¶

nntpgrab_config_get_avail_servers - Retrieve a list of the available usenet servers

array nntpgrab_config_get_avail_servers()

Returns an array containing the servername's of all the available usenet servers

Available since NNTPGrab 0.6

nntpgrab_config_get_server_info¶

nntpgrab_config_get_server_info - Retrieve the configuration details about a specific usenet server

object nntpgrab_config_get_server_info(string servername)

When the given servername isn't known an error is returned, otherwise
an object containing the following elements is returned

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [send_group_command]
    [use_ssl]
    [enabled]
)

The contents of the field priority has one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.6

nntpgrab_config_add_server¶

nntpgrab_config_add_server - Add a new server to the NNTPGrab configuration

boolean nntpgrab_config_add_server(array serverconfig)

Returns TRUE when the server was successfully added to the NNTPGrab configuration
or FALSE when the given servername is already known.

The serverconfig array needs to have the following fields:

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [send_group_command]
    [use_ssl]
    [enabled]
)

The contents of the field priority needs to have one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.6

nntpgrab_config_del_server¶

nntpgrab_config_del_server - Remove a server from the NNTPGrab configuration

boolean nntpgrab_config_del_server(string servername)

Returns TRUE when the server was successfully removed from the NNTPGrab configuration
or FALSE when the given servername isn't known.

Available since NNTPGrab 0.6

nntpgrab_config_edit_server¶

nntpgrab_config_edit_server - Change the settings of a server

boolean nntpgrab_config_edit_server(string original_servername, object serverconfig)

Returns TRUE when the change was successfully performed or FALSE when the given original_servername isn't known.

The serverconfig object needs to have the following fields:

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [send_group_command]
    [use_ssl]
    [enabled]
)

The contents of the field priority needs to have one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.6

nntpgrab_config_get_opts¶

nntpgrab_config_get_opts - Retrieve the basic configuration information from NNTPGrab

object nntpgrab_config_get_opts()

Returns an object containing the following fields:

(
    [download_directory]
    [temp_directory]
    [enable_intelligent_par2_downloading]
    [enable_par2_repair]
    [enable_auto_import]
    [auto_import_directory]
    [move_file_after_auto_import]
    [enable_auto_unpack]
    [enable_bandwidth_shaping]
    [max_bandwidth]
    [enable_webserver]
    [webserver_port]
    [enable_logger]
)

Available since NNTPGrab 0.6

nntpgrab_config_set_opts¶

nntpgrab_config_set_opts - Change the basic configuration of NNTPGrab

boolean nntpgrab_config_set_opts(object options)

Returns TRUE when the changes have been performed or FALSE when there was an error

The options object needs to have the following fields:

(
    [download_directory]
    [temp_directory]
    [enable_intelligent_par2_downloading]
    [enable_par2_repair]
    [enable_auto_import]
    [auto_import_directory]
    [move_file_after_auto_import]
    [enable_auto_unpack]
    [enable_bandwidth_shaping]
    [max_bandwidth]
    [enable_webserver]
    [webserver_port]
    [enable_logger]
)

Available since NNTPGrab 0.6

nntpgrab_schedular_start¶

nntpgrab_schedular_start - Start the NNTPGrab schedular

boolean nntpgrab_schedular_start()

Returns TRUE when the schedular has been successfully started or FALSE when
the schedular was already running or in the stopping state

Available since NNTPGrab 0.6

nntpgrab_schedular_stop¶

nntpgrab_schedular_stop - Stop the NNTPGrab schedular

nntpgrab_schedular_stop(boolean wait)

If wait is FALSE, a request will be sent to the NNTPGrab Server to
stop the schedular, but the function will not wait for the schedular to
be really stopped. If wait is TRUE, this function will wait
until the schedular is really stopped.

Available since NNTPGrab 0.6

nntpgrab_schedular_get_state¶

nntpgrab_schedular_get_state - Retrieve the current state of the NNTPGrab schedular

int nntpgrab_schedular_get_state()

This function can return one of the following values:

0 - Schedular is running
1 - Schedular is stopping
2 - Schedular is stopped

Available since NNTPGrab 0.6

nntpgrab_schedular_add_file_to_queue¶

nntpgrab_schedular_add_file_to_queue - Add a new file to the download queue

boolean nntpgrab_schedular_add_file_to_queue(string collection_name, string subject, string poster, int timestamp, int file_size, array groups, array parts)

Returns TRUE when the file has been successfully added or FALSE when there already is a file with
the same subject in the same collection_name in the download queue

The file_size needs to be in bytes

The array groups contains a list of usenet groups in which this file is published

The array parts contains an array which has the following fields:

(
    [0] => Object -- This block is representing one part. This block can occur multiple times
        (
            [message_id]
            [part_num]
            [size]
        )
)

Available since NNTPGrab 0.6

nntpgrab_schedular_del_file_from_queue¶

nntpgrab_schedular_del_file_from_queue - Remove a file from the download queue

boolean nntpgrab_schedular_del_file_from_queue(string collection_name, string subject)

Returns TRUE when the file has been successfully removed or FALSE when the given collection_name and subject aren't known

Available since NNTPGrab 0.6

nntpgrab_schedular_restart_file¶

nntpgrab_schedular_restart_file - Restart a file

boolean nntpgrab_schedular_restart_file(string collection_name, string subject)

Returns TRUE when the file has been successfully restarted or FALSE when the given collection_name and subject aren't known

Available since NNTPGrab 0.6

nntpgrab_schedular_save_queue¶

nntpgrab_schedular_save_queue - Save the download queue

boolean nntpgrab_schedular_save_queue()

This function should be called after several new tasks have been added to the download queue

Returns TRUE when the save has been successful or FALSE when there was an error

Available since NNTPGrab 0.6

nntpgrab_schedular_get_all_tasks¶

nntpgrab_schedular_get_all_tasks - Retrieve all the tasks in the download queue

array nntpgrab_schedular_get_all_tasks()

The array returned from this function contains a lot of data:

(
    [0] => Object -- This block is representing one collection. This block can occur multiple times
        (
            [collection_name]
            [total_size]
            [total_size_remaining]
            [files] => Array
                (
                    [1] => Object -- This is representing one file. This block can occur multiple times
                        (
                            [subject]
                            [poster]
                            [stamp]
                            [file_size]
                            [file_size_remaining]
                            [num_parts_total]
                            [num_parts_downloaded]
                            [num_parts_failed]
                            [status]
                            [groups] => Array -- This is representing a list of usenet groups in which this file is published
                                (
                                    [0]
                                )
                        )
                )
        )
)

The fields total_size and file_size mention the size in KB (kilobytes, 1024 bytes)

Possible values for the status are:

0 - Waiting for download
1 - Downloading
2 - Waiting for decoding
3 - Decoding
4 - Finished - complete
5 - Finished - incomplete
6 - Finished - no parts available
7 - Skipped

Available since NNTPGrab 0.6

nntpgrab_schedular_move_file¶

nntpgrab_schedular_move_file - Change the position and/or location of a file in the download queue

boolean nntpgrab_schedular_move_file(string collection_name_src, string subject_src, string collection_name_dest, int position_dest)

collection_name_src can be the same as collection_name_dest. If position_dest is -1, the file will be added to the end of the queue.
If subject is empty, then position_dest will mean the position of all the collections in the download queue.

Available since NNTPGrab 0.6

nntpgrab_schedular_move_collection¶

nntpgrab_schedular_move_collection - Change the position of a collection in the download queue

boolean nntpgrab_schedular_move_collection(string collection_name, int new_position)

If new_position is -1, the file will be added to the end of the queue.

Available since NNTPGrab 0.6

nntpgrab_schedular_mark_task_optional¶

nntpgrab_schedular_mark_task_optional - Mark a task optional/non-optional in the download queue

boolean nntpgrab_schedular_mark_task_optional string collection, string subject, boolean is_optional)

Available since NNTPGrab 0.7

nntpgrab_plugins_get_avail_plugins¶

nntpgrab_plugins_get_avail_plugins - Get a list of all the available plugins

array nntpgrab_plugins_get_avail_plugins()

Returns an array containing string's with the name of the plugin

Available since NNTPGrab 0.6

nntpgrab_plugins_get_plugin_info¶

nntpgrab_plugins_get_plugin_info - Retrieve information about a specific plugin

object nntpgrab_plugins_get_plugin_info(string plugin_name)

The returned object contains these fields:

(
    [name]
    [version]
    [author]
    [url]
    [description]
    [is_loaded]
    [is_persistent]
)

Available since NNTPGrab 0.6

nntpgrab_plugins_load_plugin¶

nntpgrab_plugins_load_plugin - Load a plugin

boolean nntpgrab_plugins_load_plugin(string plugin_name)

Returns TRUE if the plugin was successfully loaded. In case of an error, an JSON error will be returned containing details

Available since NNTPGrab 0.6

nntpgrab_plugins_unload_plugin¶

nntpgrab_plugins_unload_plugin - Unload a plugin

boolean nntpgrab_plugins_unload_plugin(string plugin_name)

Returns TRUE if the plugin was successfully unloaded. In case of an error, an JSON error will be returned containing details

Available since NNTPGrab 0.6

nntpgrab_plugins_set_persistent¶

nntpgrab_plugin_set_persistent - Change the persistent flag of a plugin

void nntpgrab_plugins_set_persistent(string plugin_name, boolean is_persistent)

In case of an error, an JSON error will be returned containing details

Available since NNTPGrab 0.6

nntpgrab_utils_strip_subject¶

nntpgrab_utils_strip_subject - Split a subject in various useable fields

array nntpgrab_utils_strip_subject(string subject)

The array returned contains the following fields:

(
    [subject]
    [subject_without_partnum]
    [file_num]
    [total_files]
    [filename]
    [extension]
    [file_type]
    [par2_startnum]
    [num_par2_blocks]
    [part_num]
    [total_parts]
)

Note that some fields can be missing. If a field is missing, it can't be detected in the given subject

Available since NNTPGrab 0.6

nntpgrab_utils_calculate_file_size¶

nntpgrab_utils_calculate_file_size - Returns the given file size in a human readable notation

string nntpgrab_utils_calculate_file_size(long filesize)

The filesize should be in KB (kilobytes)

Available since NNTPGrab 0.6

nntpgrab_utils_calculate_estimated_time_remaining¶

nntpgrab_utils_calculate_estimated_time_remaining - Make an estimation when a file with the given size will be completed

int nntpgrab_utils_calculate_estimated_time_remaining (array speed_history, int filesize)

The array speed_history should be the history part of the array returned by the function nntpgrab_connections_get_speed_history
The filesize is in KB (kilobytes)
The value returned is the number of seconds necessary to complete the given file size

Available since NNTPGrab 0.6

nntpgrab_utils_get_readable_time_remaining¶

nntpgrab_utils_get_readable_time_remaining - Returns the human readable notation of a given number of seconds

string nntpgrab_utils_get_readable_time_remaining (int time_remaining)

The value time_remaining is returned by the function nntpgrab_utils_calculate_estimated_time_remaining

Available since NNTPGrab 0.6

nntpgrab_utils_get_readable_finished_time¶

nntpgrab_utils_get_readable_finished_time - Returns the human readable notation of a given number of seconds relative to the current time

string nntpgrab_utils_get_readable_finished_time (int time_remaining)

The value time_remaining is returned by the function nntpgrab_utils_calculate_estimated_time_remaining

Available since NNTPGrab 0.6

nntpgrab_utils_get_folder_listing¶

nntpgrab_utils_get_folder_listing - Retrieves a list containing all the sub-folders which are in the given folder

mixed nntpgrab_utils_get_folder_listing(string parent)

Returns FALSE when the given parent isn't known by the NNTPGrab Server or an
array containing object's with these values:

(
    [folder]
    [has_subfolders]
)

Available since NNTPGrab 0.6

Events provided by the NNTPGrab JSON-RPC plugin¶

When connected to NNTPGrab using JSON-RPC over TCP events can be emit at any moment. When using JSON-RPC over HTTP, events (or in JSON-RPC terms: notifications) can be sent along with responses to requests. These are the possible events which the NNTPGrab Core can send:

config_changed¶

Something in the configuration has changed

config_changed

part_download_start¶

The download of a part has just started

part_download_start string servername, int conn_id, string collection_name, string subject, int part_num

part_done¶

The download of a part has completed successfully

part_done string servername, int conn_id, string collection_name, string subject, int part_num, int size

part_failed¶

The download of a part has failed (probably because the part isn't available on the server

part_failed string servername, int conn_id, string collection_name, string subject, int part_num, int size, boolean all_servers_tried

traffic_monitor_update¶

Every second, traffic statistics about the last 10 seconds are given

traffic_monitor_update int bytes_received1, int  bytes_received2, int  bytes_received3, int bytes_received4, int bytes_received5, int bytes_received6. int bytes_received7, int bytes_received8, int bytes_received9, int bytes_received10, int stamp, double average

part_progress_update¶

Every 1/10th second of a part download, this message will be sent

part_progress_update string servername, int conn_id, string collection_name, string subject, int part_num, int bytes_downloaded, int bytes_total

collection_added¶

A new collection was added

collection_added string collection_name, string poster

collection_removed¶

A collection was removed

collection_removed string collection_name

collection_modified¶

The poster of a collection was modified

collection_modified string collection_name, string poster

file_added¶

A file was added to a already existing collection

file_added string collection_name, string subject, string poster, int stamp, int file_size, int total_size, int total_size_remaining, int status, int num_parts, array groups

The groups argument is a list of newsgroups (string's)

NOTE: The fields file_size, total_size and total_size_remaining contain the number of KB's instead of bytes

file_removed¶

A file was removed from the download queue

file_removed string collection_name, string subject, int total_size, int total_size_remaining

NOTE: The fields total_size and total_size_remaining contain the number of KB's instead of bytes

file_download_state_update¶

The state of a file in the download queue has changed

file_download_state_update string collection_name, string subject, int num_parts_total, int num_parts_done, int num_parts_failed, int file_size, int file_size_remaining, int total_size, int total_size_remaining

NOTE: The fields file_size, file_size_remaining, total_size and total_size_remaining contain the number of KB's instead of bytes

file_state_changed¶

A file in the download queue was restarted

file_state_changed string collection_name, string subject, string real_filename, int old_state, int new_state

connection_connecting¶

A new connection is being made to a usenet server

connection_connecting string servername, int conn_id

connection_connected¶

A connection attempt has succeeded

connection_connected string servername, int conn_id, string welcome_msg

connection_disconnect¶

A connection to a usenet server was disconnected

connection_disconnect string servername, int conn_id, int disconnect_type, string reason

disconnect_type can be any of the following values:

0:  normal
1:  hostname could not be resolved
2:  connection refused
3:  too many connections
4:  connection timeout
5:  read error
6:  read timeout
7:  write error
8:  idle timeout
9:  invalid message received from server
10: login failure
11: unable to initialize SSL
12: unexpected error occured

schedular_state_changed¶

The state of the schedular has changed

schedular_state_changed int new_state, string reason

log_message¶

A log message has been emitted

log_message string component, int log_level, string msg

log_level can contain one of the following values:

1: info
2: warning
3: error
4: fatal
5: debug

task_moved¶

The position of a task in the download queue has changed

task_moved string orig_collection_name, string subject, string new_collection_name, int old_position, int new_position

collection_moved¶

The position of a collection in the download queue has changed

collection_moved string collection_name, int old_position, int new_position

all_downloads_completed¶

This message is emit when all the items in the download queue are completed

all_downloads_completed

plugin_loaded¶

A plugin has been loaded

plugin_loaded string plugin_name, boolean is_persistent

plugin_unloaded¶

A plugin has been unloaded

plugin_unloaded string plugin_name

plugin_event¶

A plugin has emit an event

plugin_event string plugin_name, string event_name, array values

The values parameter is an array containing string's

Plugin events¶

Certain plugin can emit events which are interesting to frontends.
This is a list of all known events which can be emit by plugins (as part of the plugin_event event)

PAR2 plugin¶

The name of this plugin is PAR2

par2_begin_verify¶

A PAR2 verification (and if necessary repair) has just started

values[0]: collection_name
values[1]: active_par2_filename

par2_load_progress_update¶

A file is now being verified

values[0]: collection_name
values[1]: active_par2_filename
values[2]: filename
values[3]: progress

par2_recovery_file_loaded¶

A PAR2 recovery file has been loaded completely

values[0]: collection_name
values[1]: active_par2_filename
values[2]: filename
values[3]: num_new_packets
values[4]: num_blocks_found

par2_file_loaded¶

A regular file has been loaded completely

values[0]: collection_name
values[1]: active_par2_filename
values[2]: filename
values[3]: state
values[4]: num_blocks_found
values[5]: num_blocks_expected

The field state can be one of the following values:

MISSING
FOUND
DAMAGED
NO_NEW_BLOCKS_FOUND

par2_repair_progress_update¶

A PAR2 repair is now going on

values[0]: collection_name
values[1]: active_par2_filename
values[2]: progress

par2_repair_failure¶

The PAR2 repair has failed because there aren't enough recovery blocks available or some other error

values[0]: collection_name
values[1]: active_par2_filename
values[2]: error message
values[3]: num_blocks_more_required

par2_repair_success¶

The PAR2 repair has completed successfully

values[0]: collection_name
values[1]: active_par2_filename

par2_no_repair_required¶

All the files in the PAR2 set have been verified and no repair is necessary

values[0]: collection_name
values[1]: active_par2_filename

Unpack plugin¶

The name of this plugin is Unpack

unpack_progress_update¶

The unpack component has made some progress

values[0]: collection_name
values[1]: filename
values[2]: progress

unpack_message_received¶

The unpack component has emit a message. This message mostly is the file which is currently extracted

values[0]: collection_name
values[1]: filename
values[2]: message

unpack_working_archive_changed¶

The unpack component is now unpacking from a new archive file

values[0]: collection_name
values[1]: filename
values[2]: new archive filename

License¶

NNTPGrab is licensed under the GPL2+ license:

Copyright (C) 2005-2010 Erik van Pienbroek

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

PHP documentation¶

As of NNTPGrab 0.3 it is possible the create a website on which you can communicate with NNTPGrab. This is made possible by using a PHP module, which is written in the C programming language.

Installation¶

To enable this PHP module, you need to make sure you have the PHP development files installed on your Linux environment (mostly this is done by installing a package called php-devel). When you install NNTPGrab, you have to call the ./configure script with the argument --enable-php-module. This will automatically detect your PHP installation and compile/install the PHP module. A final step is necessary before you can use the functions provided by this module. You need to open your php.ini in a editor and add the line extension=nntpgrab.so to it. It is a good idea to restart your webserver at this point.

Functions provided by the NNTPGrab PHP module¶

nntpgrab_connect¶

nntpgrab_connect - Connect to the NNTPGrab Server

mixed nntpgrab_connect(string $hostname, int $port, string $username, string $password, bool $use_ssl)

The parameters username, password and use_ssl are ignored in NNTPGrab version 0.3, but these will be used in a future version of NNTPGrab

Returns TRUE on success or a string containing an error message

Available since NNTPGrab 0.3

nntpgrab_get_is_connected¶

nntpgrab_get_is_connected - Retrieve the status of the connection to the NNTPGrab Server

boolean nntpgrab_get_is_connected

Returns TRUE when there is an active connection to the NNTPGrab Server

Available since NNTPGrab 0.3

nntpgrab_get_connect_errmsg¶

nntpgrab_get_connect_errmsg - Returns the error message belonging to the last connection attempt done by nntpgrab_connect()

string nntpgrab_get_connect_errmsg

Returns the last known error message or the message "No error occured" if the last connection attempt was successfull

Available since NNTPGrab 0.3

nntpgrab_get_version_info¶

nntpgrab_get_version_info - Returns several version numbers belonging to NNTPGrab

array nntpgrab_get_version_info

Returns an array with the following fields:

(
    [glue_version]
    [major_version]
    [minor_version]
    [micro_version]
    [version]
)

Available since NNTPGrab 0.3

In version 0.5, the field api_version has been renamed to glue_version

nntpgrab_config_get_avail_servers¶

nntpgrab_config_get_avail_servers - Retrieve a list of the available usenet servers

array nntpgrab_config_get_avail_servers()

Returns an array containing the servername's of all the available usenet server

Available since NNTPGrab 0.3

nntpgrab_config_get_server_info¶

nntpgrab_config_get_server_info - Retrieve the configuration details about a specific usenet server

array nntpgrab_config_get_server_info(string $servername)

When the given servername isn't known, the value NULL is returned, otherwise an array containing the following elements is returned

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [use_ssl]
    [enabled]
)

The contents of the field priority has one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.3

nntpgrab_config_add_server¶

nntpgrab_config_add_server - Add a new server to the NNTPGrab configuration

boolean nntpgrab_config_add_server(array $serverconfig)

Returns TRUE when the server was successfully added to the NNTPGrab configuration or FALSE when the given servername is already known.

The serverconfig array needs to have the following fields:

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [use_ssl]
    [enabled]
)

The contents of the field priority needs to have one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.3

nntpgrab_config_del_server¶

nntpgrab_config_del_server - Remove a server from the NNTPGrab configuration

boolean nntpgrab_config_del_server(string $servername)

Returns TRUE when the server was successfully removed from the NNTPGrab configuration or FALSE when the given servername isn't known.

Available since NNTPGrab 0.3

nntpgrab_config_edit_server¶

nntpgrab_config_edit_server - Change the settings of a server

boolean nntpgrab_config_edit_server(string $original_servername, array $serverconfig)

Returns TRUE when the change was successfully performed or FALSE when the given original_servername isn't known.

The serverconfig array needs to have the following fields:

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [use_ssl]
    [enabled]
)

The contents of the field priority needs to have one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.3

nntpgrab_config_get_opts¶

nntpgrab_config_get_opts - Retrieve the basic configuration information from NNTPGrab

array nntpgrab_config_get_opts()

Returns an array containing the following fields:

(
    [download_directory]
    [temp_directory]
    [enable_intelligent_par2_downloading]
    [enable_auto_import]
    [auto_import_directory]
    [move_file_after_auto_import]
)

Available since NNTPGrab 0.3

The fields enable_auto_import, auto_import_directory and move_file_after_auto_import have been added in NNTPGrab 0.4

nntpgrab_config_set_opts¶

nntpgrab_config_set_opts - Change the basic configuration of NNTPGrab

boolean nntpgrab_config_set_opts(array $options)

Returns TRUE when the changes have been performed or FALSE when there was an error

The options array needs to have the following fields:

(
    [download_directory]
    [temp_directory]
    [enable_intelligent_par2_downloading]
    [enable_auto_import]
    [auto_import_directory]
    [move_file_after_auto_import]
)

Available since NNTPGrab 0.3

The fields enable_auto_import, auto_import_directory and move_file_after_auto_import have been added in NNTPGrab 0.4

nntpgrab_config_get_folder_listing¶

nntpgrab_config_get_folder_listing - Retrieves a list containing all the sub-folders which are in the given folder

mixed nntpgrab_config_get_folder_listing(string $parent)

Returns FALSE when the given parent isn't known by the NNTPGrab Server or an array containing all the sub-folders which are in the given folder

Available since NNTPGrab 0.3

nntpgrab_schedular_start¶

nntpgrab_schedular_start - Start the NNTPGrab schedular

boolean nntpgrab_schedular_start()

Returns TRUE when the schedular has been successfully started or FALSE when the schedular was already running or in a stopping state

Available since NNTPGrab 0.3

nntpgrab_schedular_stop¶

nntpgrab_schedular_stop - Stop the NNTPGrab schedular

nntpgrab_schedular_stop(boolean $wait)

If wait is FALSE, a request will be sent to the NNTPGrab Server to stop the schedular, but the function will not wait for the schedular to be really stopped. If wait is TRUE, this function will wait until the schedular is really stopped.

Available since NNTPGrab 0.3

nntpgrab_schedular_get_state¶

nntpgrab_schedular_get_state - Retrieve the current state of the NNTPGrab schedular

int nntpgrab_schedular_get_state()

This function can return one of the following values:

0 - Schedular is running
1 - Schedular is stopping
2 - Schedular is stopped

Available since NNTPGrab 0.3

nntpgrab_schedular_add_task_to_queue¶

nntpgrab_schedular_add_task_to_queue - Add a new task (/file) to the download queue

boolean nntpgrab_schedular_add_task_to_queue(string $collection_name, string $subject, string $poster, int $timestamp, int $file_size, array $groups, array $parts)

Returns TRUE when the task has been successfully added or FALSE when there already is a task with the same subject in the same collection_name in the download queue

The file_size needs to be in bytes

The array groups contains a list of usenet groups in which this file is published

The array parts contains an array which has the following fields:

(
    [0] => Array -- This block is representing one part. This block can occur multiple times
        (
            [message_id]
            [partnum]
            [size]
        )
)

Available since NNTPGrab 0.3

nntpgrab_schedular_del_task_from_queue¶

nntpgrab_schedular_del_task_from_queue - Remove a task from the download queue

boolean nntpgrab_schedular_del_task_from_queue(string $collection_name, string $subject)

Returns TRUE when the task has been successfully removed or FALSE when the given collection_name and subject aren't known

Available since NNTPGrab 0.3

nntpgrab_schedular_restart_task¶

nntpgrab_schedular_restart_task - Restart a task

boolean nntpgrab_schedular_restart_task(string $collection_name, string $subject)

Returns TRUE when the task has been successfully restarted or FALSE when the given collection_name and subject aren't known

Available since NNTPGrab 0.3

nntpgrab_schedular_save_queue¶

nntpgrab_schedular_save_queue - Save the download queue

boolean nntpgrab_schedular_save_queue()

This function should be called after several new tasks have been added to the download queue

Returns TRUE when the save has been successful or FALSE when there was an error

Available since NNTPGrab 0.3

nntpgrab_schedular_get_all_tasks¶

nntpgrab_schedular_get_all_tasks - Retrieve all the tasks in the download queue

array nntpgrab_schedular_get_all_tasks()

The array returned from this function contains a lot of data:

(
    [0] => Array -- This block is representing one collection. This block can occur multiple times
        (
            [collection_name]
            [total_size]
            [total_size_remaining]
            [files] => Array
                (
                    [1] => Array -- This is representing one file. This block can occur multiple times
                        (
                            [subject]
                            [poster]
                            [stamp]
                            [file_size]
                            [file_size_remaining]
                            [num_parts_total]
                            [num_parts_downloaded]
                            [num_parts_failed]
                            [status]
                            [groups] => Array -- This is representing a list of usenet groups in which this file is published
                                (
                                    [0]
                                )
                        )
                )
        )
)

The fields total_size and file_size mention the size in KB (kilobytes, 1024 bytes)

Possible values for the status are:

0 - Waiting for download
1 - Downloading
2 - Waiting for decoding
3 - Decoding
4 - Finished - complete
5 - Finished - incomplete
6 - Finished - no parts available
7 - Skipped

Available since NNTPGrab 0.3

The status'es Finished - incomplete and Finished - no parts available have been added in NNTPGrab 0.4

nntpgrab_schedular_move_task¶

nntpgrab_schedular_move_task - Change the position and/or location of a task in the download queue

boolean nntpgrab_schedular_move_task(string $collection_name_src, string $subject_src, string $collection_name_dest, int $position_dest)

collection_name_src can be the same as collection_name_dest. If position_dest is -1, the file will be added to the end of the queue. If subject is empty, then position_dest will mean the position of all the collections in the download queue.

Available since NNTPGrab 0.3

nntpgrab_schedular_move_collection¶

nntpgrab_schedular_move_collection - Change the position of a collection in the download queue

boolean nntpgrab_schedular_move_collection(string $collection_name, int $new_position)

If new_position is -1, the file will be added to the end of the queue.

Available since NNTPGrab 0.3

nntpgrab_debug_get_messages¶

nntpgrab_debug_get_messages - Retrieve the last 100 received debug messages

array nntpgrab_debug_get_messages()

Any debug messages that the NNTPGrab Server or the Glue layer can send will be buffered in the background by the PHP module. With this function it is possible to retrieve all the debug messages which are buffered. Only the last 100 received debug messages will be kept in the buffer

Available since NNTPGrab 0.3

nntpgrab_connections_get_speed_history¶

nntpgrab_connections_get_speed_history - Retrieve the number of bytes received in the last 10 seconds

array nntpgrab_connections_get_speed_history()

The array returned contains the following data:

(
    [history] => Array
        (
            [0]
            [1]
            [2]
            [3]
            [4]
            [5]
            [6]
            [7]
            [8]
            [9]
        )

    [timestamp]
    [average]
)

The timestamp is an unix timestamp

Available since NNTPGrab 0.4

nntpgrab_utils_strip_subject¶

nntpgrab_utils_strip_subject - Split a subject in various usable fields

array nntpgrab_utils_strip_subject(string $subject)

The array returned contains the following fields:

(
    [subject]
    [subject_without_partnum]
    [file_num]
    [total_files]
    [filename]
    [extension]
    [file_type]
    [par2_startnum]
    [num_par2_blocks]
    [part_num]
    [total_parts]
)

Note that some fields can be missing. If a field is missing, it can't be detected in the given subject

Available since NNTPGrab 0.4

nntpgrab_utils_calculate_file_size¶

nntpgrab_utils_calculate_file_size - Returns the given file size in a human readable notation

string nntpgrab_utils_calculate_file_size(long filesize)

The filesize should be in KB (kilobytes)

Available since NNTPGrab 0.4

nntpgrab_utils_calculate_estimated_time_remaining¶

nntpgrab_utils_calculate_estimated_time_remaining - Make an estimation when a file with the given size will be completed

int nntpgrab_utils_calculate_estimated_time_remaining (array speed_history, int filesize)

The array speed_history should be the history part of the array returned by the function nntpgrab_connections_get_speed_history. The filesize is in KB (kilobytes). The value returned is the number of seconds necessary to complete the given file size

Available since NNTPGrab 0.4

nntpgrab_utils_get_readable_time_remaining¶

nntpgrab_utils_get_readable_time_remaining - Returns the human readable notation of a given number of seconds

string nntpgrab_utils_get_readable_time_remaining (int time_remaining)

The value time_remaining is returned by the function nntpgrab_utils_calculate_estimated_time_remaining

Available since NNTPGrab 0.4

nntpgrab_utils_get_readable_finished_time¶

nntpgrab_utils_get_readable_finished_time - Returns the human readable notation of a given number of seconds relative to the current time

string nntpgrab_utils_get_readable_finished_time (int time_remaining)

The value time_remaining is returned by the function nntpgrab_utils_calculate_estimated_time_remaining

Available since NNTPGrab 0.4

Plugin development¶

As of NNTPGrab 0.6.0 the program is fully plugin based. This makes it very easy to add new functionality to the program. New functionality can be added in the form of seperate plugins. This documentation describes the steps needed to create a plugin suitable for NNTPGrab.

Lifecycle of a plugin¶

All plugins need to implement a number of functions. These functions need to be exported before the plugin can be loaded from the NNTPGrab Core:

void nntpgrab_plugin_initialize(NGPlugin *plugin_data) ngboolean nntpgrab_plugin_load(NGPlugin *plugin_data, char **errmsg) ngboolean nntpgrab_plugin_can_unload(NGPlugin *plugin_data, char **reason) void nntpgrab_plugin_unload(NGPlugin *plugin_data) void nntpgrab_plugin_destroy(NGPlugin *plugin_data) int nntpgrab_plugin_get_version(void) 

The NGPlugin structure¶

The NGPlugin structure is defined in the file nntpgrab_core/nntpgrab_plugin.h. Plugins must include this file. The NGPlugin structure looks like this:

typedef struct _ng_plugin { GObject parent; NGPluginCoreData *core_data; NGPluginCoreFuncs core_funcs; gpointer priv; } NGPlugin; 

This structure contains 4 fields.
The first field is GObject parent. This is used internally by GLib/GObject and must not be touched by plugins
The second field is NGPluginCoreData *core_data. This is a pointer to a structure containing internal plugin information which is maintained by the NNTPGrab Core. This structure also must not be touched by plugins
The third field is NGPluginCoreFuncs core_funcs. This structure contains pointers to various functions in the NNTPGrab Core. Plugins can call these functions to change the behaviour of the program
The fourth field is gpointer priv. Plugins can put any kind of data in this field

nntpgrab_plugin_initialize¶

On startup of NNTPGrab, the program searches in the plugin folder for all the available plugins. This is done by searching files which implement all the functions which were listed above.

To retrieve information about the plugin, this function is called automatically. Must of the time, this function is implemented like this:

void nntpgrab_plugin_initialize(NGPlugin *plugin_data) { ng_plugin_set_name(plugin_data, "Example"); ng_plugin_set_version(plugin_data, "1.0"); ng_plugin_set_author(plugin_data, "Erik van Pienbroek"); ng_plugin_set_url(plugin_data, "https://www.nntpgrab.nl"); ng_plugin_set_description(plugin_data, "Example plugin which prints a message in the console every time a file is added"); } 

If the plugin want to offer new functionality (functions) to the NNTPGrab Core or other plugins then the function ng_plugin_register_function() can be used here
If the plugin wants to use functionality (functions) from other plugins, then this can be indicated by using the function ng_plugin_set_required_function()

This is also the only place where the functions ng_plugin_register_function() and ng_plugin_set_required_function() can be used

nntpgrab_plugin_load¶

This function will be called by the NNTPGrab Core when the plugin needs to be loaded and get ready to used. The function ng_plugin_connect_event() can be used here to connect to events.

If wanted, the plugin_data->priv field can be initialized here to something useful

nntpgrab_plugin_can_unload¶

NNTPGrab frontends can call this function to find out whether the plugin is performing a task at the moment.
If this is the case, the parameter char **reason can be set to a small comment about the active task and the value FALSE can be returned.
If the plugin isn't doing anything at the moment, the value TRUE can be returned

nntpgrab_plugin_unload¶

Whenever a plugin needs to be unloaded (because of program shutdown or on the user's request) this function will be called.
It's basically the reverse of the nntpgrab_plugin_load function. If the plugin_data->priv field was set, then it can be cleared here
It isn't necessary to unregister any event connections (done by ng_plugin_connect_event()) as that'll be automatically done by the NNTPGrab Core

nntpgrab_plugin_destroy¶

On program shutdown, this function will be automatically called. Most plugins don't need to add any implementation to this function

nntpgrab_plugin_get_version¶

The implementation of this function must be the same across all plugins. All that needs to happen is a return NNTPGRAB_PLUGIN_API_VERSION;. With this information, the NNTPGrab Core can find out if the plugin is using the same API version of the NNTPGrab Core is expecting. When an API mismatch is detected, the plugin will be ignored

Executing functions of other plugins¶

Whenever a plugin want to use functionality from another plugin, a special function has to be used: ng_plugin_call()

This function expects a variable number of arguments. The first parameters (which are always required) are a pointer to an instance of the NGPlugin structure (which is used across all plugin functions) and a function name. Depending on the function which needs to be executed, zero or more additional arguments are needed.

If the function which needs to be executed has a return value, then it needs to be added as the last parameter to the ng_plugin_call() call.

To help the NNTPGrab Core in dependency resolving, any external function which will be used by the plugin needs to be registered with the ng_plugin_set_required_function() function in the [Plugin_development#nntpgrab_plugin_initialize|nntpgrab_plugin_initialize()]] function

Every function expects a different set of parameters. To find out what the right expected parameters are see the API documentation section

Connecting to events of other plugins¶

Plugins can emit events whenever something has occured. Other plugins can connect to these event by using the ng_plugin_connect_event() function
This function expects a pointer to an instance of the NGPlugin structure (which is used across all plugin functions), an event name and the name of the function which needs to be invoked whenever the event occurs.

Every event has a different set of arguments. To find out what the right expected parameters are see the API documentation section

API documentation¶

For a list of functions which NNTPGrab plugins can use to communicate with other plugins, see the documentation for nntpgrab_plugin.h

This is a list of events which are exported by the NNTPGrab Core and other plugins. Plugins can connect to these events with the function ng_plugin_connect_event().

config_changed¶

Something in the configuration has changed

void config_changed (NGPlugin *plugin_data, void *user_data)

part_download_start¶

The download of a part has just started

void part_download_start (NGPlugin *plugin_data, const char *servername, int conn_id, const char *collection_name, const char *subject, int part_num, void *user_data)

part_done¶

The download of a part has completed successfully

void part_done (NGPlugin *plugin_data, const char *servername, int conn_id, const char *collection_name, const char *subject, int part_num, int size, void *user_data)

part_failed¶

The download of a part has failed (probably because the part isn't available on the server)

void part_failed (NGPlugin *plugin_data, const char *servername, int conn_id, const char *collection_name, const char *subject, int part_num, int size, ngboolean all_servers_tried, void *user_data)

traffic_monitor_update¶

Every second, traffic statistics about the last 10 seconds are given

void traffic_monitor_update (NGPlugin *plugin_data, int bytes_received1, int bytes_received2, int bytes_received3, int bytes_received4, int bytes_received5, int bytes_received6, int bytes_received7, int bytes_received8, int bytes_received9, int bytes_received10, guint64 stamp, double average, void *user_data)

part_progress_update¶

Every 1/10th second of a part download, this message will be sent

void part_progress_update (NGPlugin *plugin_data, const char *servername, int conn_id, const char *subject, int part_num, int bytes_downloaded, int bytes_total, void *user_data)

collection_added¶

A new collection was added

void collection_added (NGPlugin *plugin_data, const char *collection_name, const char *poster, void *user_data)

collection_removed¶

A collection was removed

void collection_removed (NGPlugin *plugin_data, const char *collection_name, void *user_data)

collection_modified¶

The poster of a collection was modified

void collection_modified (NGPlugin *plugin_data, const char *collection_name, const char *poster, void *user_data)

file_added¶

A file was added to a already existing collection

void file_added (NGPlugin *plugin_data, const char *collection_name, const char *subject, const char *poster, nguint64 stamp, nguint64 file_size, nguint64 total_size, nguint64 total_size_remaining, NGTaskState status, int num_parts, NGList *groups, void *user_data)

The groups argument is a list of newsgroups (const char*'s)

file_removed¶

A file was removed from the download queue

void file_removed (NGPlugin *plugin_data, const char *collection_name, const char *subject, nguint64 total_size, nguint64 total_size_remaining, void *user_data)

file_download_state_update¶

The state of a file in the download queue has changed

void file_download_state_update (NGPlugin *plugin_data, const char *collection_name, const char *subject, int num_parts_total, int num_parts_done, int num_parts_failed, nguint64 file_size, nguint64 file_size_remaining, nguint64 total_size, nguint64 total_size_remaining, void *user_data)

file_state_changed¶

A file in the download queue was restarted

void file_state_changed (NGPlugin *plugin_data, const char *collection_name, const char *subject, const char *real_filename, NGTaskState old_state, NGTaskState new_state, void *user_data)

connection_connecting¶

A new connection is being made to a usenet server

void connection_connecting (NGPlugin *plugin_data, const char *servername, int conn_id, void *user_data)

connection_connected¶

A connection attempt has succeeded

void connection_connected (NGPlugin *plugin_data, const char *servername, int conn_id, const char *welcome_msg, void *user_data)

connection_disconnect¶

A connection to a usenet server was disconnected

void connection_disconnect (NGPlugin *plugin_data, const char *servername, int conn_id, NNTPDisconnectType disconnect_type, const char *msg, void *user_data)

schedular_state_changed¶

The state of the schedular has changed

void schedular_state_changed (NGPlugin *plugin_data, NGSchedularState new_state, const char *reason, void *user_data)

log_message¶

A log message has been emitted

void log_message (NGPlugin *plugin_data, const char *component, NGLogLevel log_level, const char *msg, void *user_data)

task_moved¶

The position of a task in the download queue has changed

void task_moved (NGPlugin *plugin_data, const char *orig_collection_name, const char *subject, const char *new_collection_name, int old_position, int new_position, void *user_data)

collection_moved¶

The position of a collection in the download queue has changed

void collection_moved (NGPlugin *plugin_data, const char *collection_name, int old_position, int new_position, void *user_data)

all_downloads_completed¶

This message is emit when all the items in the download queue are completed

void all_downloads_completed (NGPlugin *plugin_data, void *user_data)

plugin_loaded¶

A plugin has been loaded

void plugin_loaded (NGPlugin *plugin_data, const char *plugin_name, ngboolean is_persistent, void *user_data)

plugin_unloaded¶

A plugin has been unloaded

void plugin_unloaded (NGPlugin *plugin_data, const char *plugin_name, void *user_data)

plugin_event¶

A plugin has emit an event

void plugin_event (NGPlugin *plugin_data, const char *plugin_name, const char *event_name, const char **values)

The values parameter is a NULL-terminated array containing const char*'s

Example plugin¶

For a simple example plugin, see the file plugins/example/example.c. This is a very basic plugin which waits for the event file_added to occur. Whenever the event occurs, some parameters belonging to that specific event are printed in the console

Screenshots¶

This page contains several screenshots of NNTPGrab. If you've got more screenshots which you would like to add, feel free to send us an e-mail

Windows screenshots¶

Version 0.6.90

Version 0.6.2

Linux screenshots¶

Mac OS X screenshots¶

Version 0.6.90

Version 0.6.2

Subversion¶

All development of NNTPGrab takes places in a Subversion repository. The address of this repository is https://svn.openftd.org/svn/nntpgrab

In the repository the following layout is used:

/trunk - development of the next major version of NNTPGrab
/branches - bugfixes for stable releases of NNTPGrab
/tags - contains all releases of NNTPGrab

The code can be browsed by using the Repository link.
There's also a svnweb available at https://svn.openftd.org/viewvc/NNTPGrab/

Protocol¶

NOTE: This documentation is obsolete and doesn't apply anymore to NNTPGrab 0.6.0 and higher. For documentation about remote controlling NNTPGrab 0.6.0 and higher, please see the JSON-RPC documentation

This document describes how any frontend can communicate to the backend over raw sockets. This makes it possible to control the NNTPGrab backend from another host.

The NNTPGrab Core listens on port 5423/tcp.

After making a connection to this port, the NNTPGrab Core replies with the following welcome message:

NNTPGrab Server - API version 20090505

The documentation on this wiki is all based on API version 20090505

The NNTPGrab protocol is a line based protocol. Every command should end with the newline character (\n).

When connecting, the client should send one of the following commands before further communication is possible:

MODE LISTENER

MODE COMMAND

OK

After this command, the socket is either broadcasting all the signals coming from the NNTPGrab Core or awaiting commands.

When an invalid command is entered, the server will send the response:

INVALID COMMAND

MODE COMMAND¶

CONFIG_GET_AVAIL_SERVERS¶

Returns a list of all the available usenet servers.

Notation:

CONFIG_GET_AVAIL_SERVERS

Response:

NULL

Example:

> CONFIG_GET_AVAIL_SERVERS
< NewsZilla
< NULL

CONFIG_GET_SERVER_INFO¶

Retrieve information about a specific usenet server. Returns NULL when the given servername isn't found

Notation:

CONFIG_GET_SERVER_INFO 

Response:

NULL

NULL

Example:

> CONFIG_GET_SERVER_INFO NewsZilla
< NewsZilla
< newszilla.xs4all.nl
< 119
< 
< 
< 3
< 1
< FALSE
< TRUE
< NULL

CONFIG_ADD_SERVER¶

Add a new usenet server
Returns FALSE when the servername already exists

Notation:

CONFIG_ADD_SERVER \t\t\t\t\t\t\t\t

Response:

TRUE

FALSE

Example:

> CONFIG_ADD_SERVER NewsZilla    newszilla.xs4all.nl    119            3    1    FALSE    TRUE
< TRUE

CONFIG_DEL_SERVER¶

Remove a usenet server
Returns FALSE when the servername isn't known

Notation:

CONFIG_DEL_SERVER 

Response:

TRUE

FALSE

Example:

> CONFIG_DEL_SERVER NewsZilla
< TRUE

CONFIG_EDIT_SERVER¶

Changes the settings of a usenet server
Returns FALSE when the servername isn't known

Notation:

CONFIG_EDIT_SERVER \t\t\t\t\t\t\t\t\t

Response:

TRUE

FALSE

Example:

> CONFIG_EDIT_SERVER NewsZilla    NewsZilla_IPV6    newszilla6.xs4all.nl    119    my_username    my_pass    3    1    FALSE    TRUE
< TRUE

CONFIG_GET_OPTS¶

Retrieves the global settings of the NNTPGrab Core

Notation:

CONFIG_GET_OPTS

Response:

NULL

Example:

> CONFIG_GET_OPTS
< /home/test/NNTPGrab/Downloads
< /home/test/NNTPGrab/Temp
< FALSE
< TRUE
< TRUE
< /home/test/NNTPGrab/NZB
< TRUE
< TRUE
< NULL

CONFIG_SET_OPTS¶

Changes the global settings of the NNTPGrab Core
Returns FALSE when one of the new settings are invalid

Notation:

CONFIG_SET_OPTS \t\t\t\t\t\t\t

Response:

TRUE

FALSE

Example:

> CONFIG_SET_OPTS /home/test/NNTPGrab/Downloads    /home/test/NNTPGrab/Temp    FALSE    TRUE    TRUE    /home/test/NNTPGrab/NZB    FALSE    TRUE
< TRUE

CONFIG_GET_FOLDER_LISTING¶

Retrieves a list containing all the sub-folders which are in the given folder.
Each result also contains a TRUE or FALSE mentioning if the sub-folders itself contains sub-folders

Notation:

CONFIG_GET_FOLDER_LISTING 

Response:

FALSE

NULL

folder1\tTRUE
folderX\tFALSE
NULL

Example:

> CONFIG_GET_FOLDER_LISTING /
< bin    FALSE
< etc    TRUE
< proc    TRUE
< sbin    FALSE
< usr    TRUE
< var    TRUE
< NULL

SCHEDULAR_START¶

Start the NNTPGrab schedular
Returns FALSE when the schedular is already running

Notation:

SCHEDULAR_START

Response:

TRUE

FALSE

Example:

> SCHEDULAR_START
< TRUE

SCHEDULAR_STOP¶

Stop the NNTPGrab schedular
Returns FALSE when the schedular is already stopped

Notation:

SCHEDULAR_STOP 

Response:

TRUE

FALSE

Example:

> SCHEDULAR_STOP TRUE
< TRUE

SCHEDULAR_GET_STATE¶

Retrieves the current state of the schedular

Notation:

SCHEDULAR_GET_STATE

Response:

RUNNING

STOPPING

STOPPED

Example:

> SCHEDULAR_GET_STATE
< STOPPED

SCHEDULAR_ADD_TASK_TO_QUEUE¶

Add a new task (file) to the download queue
When a file contains multiple groups or multiple parts, these need to be split using the | character. A part needs to be in the notation ,,. The file size is mentioned in bytes and the stamp is an unix timestamp

Notation:

SCHEDULAR_ADD_TASK_TO_QUEUE \t\t\t\t\t\t

Response:

TRUE

FALSE 

Example:

> SCHEDULAR_ADD_TASK_TO_QUEUE my collection    my subject    some poster    1199389869    123456789    alt.binaries.boneless|alt.binaries.nl    ,1,500|,2,250
< FALSE The file already exists in the download queue

SCHEDULAR_DEL_TASK_FROM_QUEUE¶

Removes a task from the download queue

Notation:

SCHEDULAR_DEL_TASK_FROM_QUEUE \t

Response:

TRUE

FALSE 

Example:

> SCHEDULAR_DEL_TASK_FROM_QUEUE my collection    my subject
< TRUE

SCHEDULAR_RESTART_TASK¶

Restarts a task already present in the download queue

Notation:

SCHEDULAR_RESTART_TASK \t

Response:

TRUE

FALSE 

Example:

> SCHEDULAR_RESTART my collection    some other subject
< FALSE File is not present in the given collection

SCHEDULAR_SAVE_QUEUE¶

Saves the download queue. This needs to be done after added new files to the download queue

Notation:

SCHEDULAR_SAVE_QUEUE

Response:

TRUE

FALSE 

Example:

> SCHEDULAR_SAVE_QUEUE
< TRUE

SCHEDULAR_FOREACH_TASK¶

Returns all the collections and tasks present in the download queue

Notation:

SCHEDULAR_FOREACH_TASK

Response:

COLLECTION \t\t\t
\tFILE \t\t\t\t\t\t\t\t\t\t
(this list can be returned multiple times)
NULL

Possible status'es:

WAITING_FOR_DOWNLOAD
DOWNLOADING
WAITING_FOR_DECODE
DECODING
FINISHED_COMPLETE
FINISHED_INCOMPLETE
FINISHED_NO_PARTS_AVAIL
SKIPPED

Example:

> SCHEDULAR_FOREACH_TASK
< COLLECTION my collection    12345678    100    3
<     FILE my_subject    some_poster    1199389869    0    1234567    1    100    10    0    WAITING_FOR_DOWNLOAD    alt.binaries.boneless|alt.binaries.nl
<     FILE some_other_subject    some poster    1199389900    100    565432    2    50    0    0    alt.binaries.x
< COLLECTION some other collection    626832    200
<     FILE another subject    another poster    1199331624    1583632    200    1    10    0    0    alt.binaries.x
< NULL

SCHEDULAR_MOVE_TASK¶

Change the position and/or location of a task in the download queue.
The collection_name_src can be the same as collection_name_dest. If the position is -1, the file will be added to the end of the queue. If subject is empty, then the position will mean the position of all the collections in the download queue.

Notation:

SCHEDULAR_MOVE_TASK \t\t\t

Response:

TRUE

FALSE

Example, change the position of a task within the same collection:

> SCHEDULAR_MOVE_TASK my collection    my subject    my collection    5
< TRUE

Example, move the task to another collection

> SCHEDULAR_MOVE_TASK my collection    my subject    another collection    -1
< FALSE

SCHEDULAR_MOVE_COLLECTION¶

Change the position of a collection in the download queue.
If the new_position is -1, the collection will be moved to the end of the queue.

Notation:

SCHEDULAR_MOVE_COLLECTION \t

Response:

TRUE

FALSE

Example

> SCHEDULAR_MOVE_COLLECTION my collection    -1
< TRUE

MODE LISTENER¶

While in the listener mode, all signals coming from the NNTPGrab Core are automatically broadcasted on the socket. These are the possible messages which the NNTPGrab Core can send.

CONFIG_CHANGED¶

Something in the configuration has changed

CONFIG_CHANGED

PART_DOWNLOAD_START¶

The download of a part has just started

PART_DOWNLOAD_START \t\t\t\t

PART_DONE¶

The download of a part has completed successfully

PART_DONE \t\t\t\t\t

PART_FAILED¶

The download of a part has failed (probably because the part isn't available on the server

PART_FAILED \t\t\t\t\t\t

TRAFFIC_MONITOR_UPDATE¶

Every second, traffic statistics about the last 10 seconds are given

TRAFFIC_MONITOR_UPDATE \t\t\t\t\t\t\t\t\t\t\t

PART_PROGRESS_UPDATE¶

Every 10KB of a part download, this message will be sent

PART_PROGRESS_UPDATE \t\t\t\t\t

COLLECTION_ADDED¶

A new collection was added

COLLECTION_ADDED \t

COLLECTION_REMOVED¶

A collection was removed

COLLECTION_REMOVED 

COLLECTION_MODIFIED¶

The poster of a collection was modified

COLLECTION_MODIFIED \t

FILE_ADDED¶

A file was added to a already existing collection. The argument is a list of newsgroups, separated by the | character

FILE_ADDED \t\t\t\t\t\t\t

FILE_REMOVED¶

A file was removed from the download queue

FILE_REMOVED \t\t\t

FILE_DOWNLOAD_STATE_UPDATE¶

The state of a file in the download queue has changed

FILE_DOWNLOAD_STATE_UPDATE \t\t\t\t\t\tt\t\t

FILE_STATE_CHANGED¶

A file in the download queue was restarted

FILE_STATE_CHANGED \t\t\t\t

CONNECTION_CONNECTING¶

A new connection is being made to a usenet server

CONNECTION_CONNECTING \t

CONNECTION_CONNECTED¶

A connection attempt has succeeded

CONNECTION_CONNECTED \t\t

CONNECTION_DISCONNECT¶

A connection to a usenet server was disconnected

CONNECTION_DISCONNECT \t\t\t

disconnect_type can be any of the following values:

NORMAL
NO_SUCH_HOST
CONNECTION_REFUSED
TOO_MANY_CONNECTIONS
CONNECT_TIMEOUT
READ_ERROR
READ_TIMEOUT
WRITE_ERROR
IDLE_TIMEOUT
INVALID_MSG
LOGIN_FAILURE
ERROR_SSL_INITIALISE
UNEXPECTED

SCHEDULAR_STATE_CHANGED¶

The state of the schedular has changed

SCHEDULAR_STATE_CHANGED \t

FATAL_ERROR¶

A fatal error has occured

FATAL_ERROR 

WARNING_MESSAGE¶

A warning has occured

WARNING_MESSAGE 

DEBUG_MESSAGE¶

A debug message has been emitted

DEBUG_MESSAGE 

TASK_MOVED¶

The position of a task in the download queue has changed

TASK_MOVED \t\t\t\t

COLLECTION_MOVED¶

The position of a collection in the download queue has changed

COLLECTION_MOVED \t\t

PAR2_REPAIR_MESSAGE¶

The PAR2 repair component has emit a message which can be shown to the user

PAR2_REPAIR_MESSAGE 

ALL_DOWNLOADS_COMPLETED¶

This message is emit when all the items in the download queue are completed

ALL_DOWNLOADS_COMPLETED

UNPACK_PROGRESS_UPDATE¶

The unpack component has made some progress

UNPACK_PROGRESS_UPDATE \t\t

UNPACK_MESSAGE_RECEIVED¶

The unpack component has emit a message. This message mostly is the file which is currently extracted

UNPACK_MESSAGE_RECEIVED \t\t

UNPACK_WORKING_ARCHIVE_CHANGED¶

The unpack component is now unpacking from a new archive file

UNPACK_WORKING_ARCHIVE_CHANGED \t\t

PAR2_BEGIN_VERIFY¶

A PAR2 verification (and if necessary repair) has just started

PAR2_BEGIN_VERIFY \t

PAR2_LOAD_PROGRESS_UPDATE¶

A file is now being verified

PAR2_LOAD_PROGRESS_UPDATE \t\t\t

PAR2_RECOVERY_FILE_LOADED¶

A PAR2 recovery file has been loaded completely

PAR2_RECOVERY_FILE_LOADED \t\t\t\t

PAR2_FILE_LOADED¶

A regular file has been loaded completely

PAR2_FILE_LOADED \t\t\t\t\t

The field state can be one of the following values:

MISSING
FOUND
DAMAGED
NO_NEW_BLOCKS_FOUND

PAR2_REPAIR_PROGRESS_UPDATE¶

A PAR2 repair is now going on

PAR2_REPAIR_PROGRESS_UPDATE \t\t

PAR2_REPAIR_FAILURE¶

The PAR2 repair has failed because there aren't enough recovery blocks available

PAR2_REPAIR_FAILURE \t\t

PAR2_REPAIR_SUCCESS¶

The PAR2 repair has completed successfully

PAR2_REPAIR_SUCCESS \t

PAR2_NO_REPAIR_REQUIRED¶

All the files in the PAR2 set have been verified and no repair is necessary

PAR2_NO_REPAIR_REQUIRED \t

Ubuntu¶

As of NNTPGrab version 0.3.2 there is a Ubuntu software repository available
containing NNTPGrab packages for the following Ubuntu releases:

• Ubuntu 10.04 (Lucid, LTS)
• Ubuntu 12.04 (Precise, LTS)
• Ubuntu 12.10 (Quantal)
• Ubuntu 13.04 (Raring)
• Ubuntu 13.10 (Saucy)
• Ubuntu 14.04 (Trusty)
• Ubuntu 14.10 (Utopic)

This repository also contains several NNTPGrab subpackages so that you can
decide for yourself whether you want to use the webinterface or not.

Preparations¶

Before you start, please download the file https://www.nntpgrab.nl/releases/ubuntu/gpg.key on your computer. The location doesn't matter

Configuration of the repository¶

Start by opening the Synaptic Package Manager. This program can be found at System -> Administration.

Once Synaptics is started, choose Settings -> Repositories. Next go to the tab called Third-Party-Software and press the Add button.
Enter one of the following line is the dialog presented:

For Ubuntu 10.04 (Lucid, LTS)

deb https://www.nntpgrab.nl/releases/ubuntu lucid main

deb https://www.nntpgrab.nl/releases/ubuntu precise main

deb https://www.nntpgrab.nl/releases/ubuntu quantal main

deb https://www.nntpgrab.nl/releases/ubuntu raring main

deb https://www.nntpgrab.nl/releases/ubuntu saucy main

deb https://www.nntpgrab.nl/releases/ubuntu trusty main

deb https://www.nntpgrab.nl/releases/ubuntu utopic main

Press the Add source button now to add the repository.

Next go to the tab called Authentication and press the Import Key File button.
Select the file you downloaded at the beginning of this document and press OK.

You can now close the Software sources window by clicking the Close button.

If everything is okay, you now get a message mentioning it is recommended to reload now.
Do so by pressing the Reload button.

Installation¶

As soon as the Reload process is finished you can press the Search button and search for nntpgrab.

You should see several results now.

Normally it is sufficient to mark the package nntpgrab for installation, but
if you want you can also choose to only install some components (like the NNTPGrab Server).

When you've marked one or more packages ready for installation, press the Apply button. The program will now be installed.

You can now remove the gpg.key file which you've downloaded at the beginning of this document.

After the installation, you can close Synaptics and there should be one or more shortcuts for NNTPGrab in your Application -> Internet menu.

Updates¶

Future updates of NNTPGrab wil automatically be presented to you as an update as soon as they're published.
The instructions above only need to be done once.

Test releases¶

If you want to help out testing unstable releases of NNTPGrab, you can repeat the above process, but use one of the following repository addresses:

For Ubuntu 10.04 (Lucid, LTS)

deb https://www.nntpgrab.nl/releases/ubuntu/testing lucid main

deb https://www.nntpgrab.nl/releases/ubuntu/testing precise main

deb https://www.nntpgrab.nl/releases/ubuntu/testing quantal main

deb https://www.nntpgrab.nl/releases/ubuntu/testing raring main

deb https://www.nntpgrab.nl/releases/ubuntu/testing saucy main

deb https://www.nntpgrab.nl/releases/ubuntu/testing trusty main

deb https://www.nntpgrab.nl/releases/ubuntu/testing utopic main

It isn't necessary to import the gpg.key file again

User documentation¶

Our friends at Binaries4All have created a HOWTO explaining how the NNTPGrab program can be used. At the moment this HOWTO is only available in the Dutch language and can be found at https://www.binaries4all.nl/nntpgrab/

Version history¶

The latest stable version of NNTPGrab is version 0.7.2

Version 0.7.2¶

• Allow the user to kill the NNTPGrab Server using the Qt frontend
• Allow frontends to shutdown the Qt-based NNTPGrab Server
• The move to top/up/down/bottom buttons in the download queue context menu in the Qt frontend were broken. Fixed. Bug #90
• Translated the Linux desktop files
• Make sure all messages are translated in the Qt frontend on Mac OS X
• Saving the download queue on win32 and win64 could result in a 'Permission denied' error. Fixed. Resolves the issue mentioned at https://www.nntpgrab.nl/boards/2/topics/369
• When importing a collection using the Qt frontend from the Online Search the collection name would get stripped unnecessary. Resolves the issue mentioned at https://www.nntpgrab.nl/boards/1/topics/446
• The automatic shutdown button wasn't visible in the Gtk frontend any more as of version 0.7.0. Fixed. Resolves the issue mentioned at https://www.nntpgrab.nl/boards/1/topics/371
• Connecting to a NNTPGrab Server in the Gtk3 frontend could result in an assertion failure. Fixed. Resolves the issue mentioned at https://www.nntpgrab.nl/boards/1/topics/434
• Fixed a small rendering issue in the Gtk3 frontend when using Gtk 3.4.0 or higher
• Fixed a crash in the Qt frontend which could occur when a plugin event was triggered without any parameters
• Automatic PAR2 repair and automatic unpack wasn't working in the Gtk frontend on Mac OS X. Fixed
• Fixed a possible crash in the NetworkManager plugin
• Fix compile failure in the Qt frontend when Subversion 1.7 is installed
• Fix compile failure in the Gtk frontend when using Gtk3 on win32 and win64 environments
• Fix compile failure in the PHP module when using PHP 5.4

Version 0.7.1¶

• Added Turkish translation
• Updated the French translation
• Various messages coming from the NNTPGrab Core didn't get translated in the Qt frontend. Fixed
• When an error occurs while a NZB file is imported an incomplete message would get shown in the Qt frontend. Fixed
• Added support to check the maintenance status of the Online Search service so that user-friendly messages can be shown when the Online Search service is down for maintenance
• When trying to unpack an password-protected archive let the user know that they have to manually unpack the archive as NNTPGrab doesn't support unpacking password-protected archives yet
• The NNTPGrab Qt Server could crash when a file changed state. Fixed. Bug #87
• Clicking on the 'Connect' button in the startup dialog could cause an assertion error when using GTK3. Fixed

Version 0.7.0¶

• Some files could get stuck in the download queue when a connection with a server was interrupted unexpectedly. Fixed
• Collections with a total size of 4GB or more were shown incorrectly in the Online Search tab. Fixed
• Require Online Search queries to be at least 4 characters long
• Made the columns in the Online Search tab resizable
• On the first time startup, always save a default configuration file to disk instead of waiting for the first change or proper shutdown as was mentioned at https://www.nntpgrab.nl/boards/2/topics/257
• The text "Now verifying files" wasn't translated in the Qt frontend. Fixed
• GNUTLS support should work again on Mac OS X. Bug #67
• Fixed an issue in the GTK frontend where moving the last file or collection in the download queue down would result in an assertion error. Bug #83
• The Qt frontend didn't remember the last used host/port which was used to connect to a NNTPGrab Server. Thanks to chrashoverraid for implementing this! Bug #84
• When switching to the configuration tab in the Qt frontend, no selection was initially shown in the list of configuration categories. Fixed
• Event handling was broken on RHEL5/CentOS5 (as mentioned at https://www.nntpgrab.nl/boards/2/topics/279). Fixed
• Don't show the automatic unpack options in the configuration tab on RHEL5/CentOS5 as these features aren't available on those environments
• Importing NZB files (regular import, not the quick import) was broken on RHEL5/CentOS5. Fixed
• Resolved various harmless glade warnings on environments where GtkBuilder isn't available
• When adding files the download progress percentage could be shown incorrect when the collection contains .par2 files. Fixed
• The intelligent PAR2 downloader, automatic PAR2 repair and automatic unpack wouldn't get triggered on collections which contain one or more files which are fully unavailable on all the usenet servers. Fixed
• Check if there are any files which are ready to be decoded every time the download queue is restarted. Previously, this only happened once when the download queue was started for the first time. Bug #85
• When the schedular is stopped due to an error, it wasn't possible to restart the schedular anymore in the Qt frontend. Fixed

Version 0.6.92 (0.7 Beta 3)¶

• Implemented a new option: Automatically remove collections from the download queue once the download has completed. Feature #37
• Implemented a new option: Send GROUP command before downloading. Some buggy usenet servers seem to require such a feature (such as Ziggo)
• Updated the Online Search code to be much more efficient then before. From now on information about files which are in a collection will only be retrieved on demand
• Improved the testsuite by porting all testcases to the GTester framework and adding various new testcases
• Various bugfixes and optimalisations in the download queue logic
• Always print debugging messages when we're using Win32/Win64 or Mac OS X. On all other platforms, check whether the environment variable NNTPGRAB_DEBUG is set before enabling debugging messages in the console
• Don't try to import old .nzb files on Mac OS X when a new .nzb file is detected in the auto import folder

• Downloading using SSL wasn't working anymore when using GNUTLS 2.8.0 or higher due to API changes. Bug #64
• Added a small optimalization to the nntpconnection_read_msg function by dropping some duplicate code
• The download queue could get stuck in the suspended state when all login attempts failed. Fixed
• As of GLib 2.27.1, the behaviour of g_get_user_data_dir() was changed to return a different value. Provide an upgrade path for Win32/Win64 users so that their configuration won't get lost due to this change. Bug #66
• When making changes to the download queue, write back those changes to a tempory file first before replacing the original file. This prevents the possibility of the download queue becoming corrupt when the disk is full
• Added support for multiple connection backends
• Added a new connection backend called 'gio' which is based on GLib-GIO and can optionally use glib-networking for SSL/TLS support
• Optimzed the retrieval of tasks from the download queue by caching the last known task per server. This eliminates the need to navigate through the entire download queue for every time that a new task is requested by the download thread

• Processing signals while being connected to a NNTPGrab Server was broken as of GLib 2.27.4. Fixed

• Fixed an issue in the Online Search where files with a size between 2GB and 4GB weren't displayed correctly

• When moving files/collections in the download queue to the top or up which are already at the top a 'beginMoveRows failed' message could get shown. Fixed. The same also applies to files/collections which are already at the bottom
• Allow the Qt frontend to build when Qt 4.5.0 or older is installed
• Added support for 'shadow builds' in the Qt frontend
• Use a bundled copy of the QtLockedFile and QtSingleApplication classes which will be used by the Qt frontend when no system-wide installed version can be found
• Prevent the user being spammed with 'buffer too small' messages. Patch by swsnyder
• Improved selecting files to be used during a NZB import. Bug #62. Patch by slayeriq
• Save and restory geometry of the program window. Feature #65. Patch by tg1311
• Show the correct timestamp for collections in the download queue
• Convert all strings coming from the NNTPGrab Core to the QString native character encoding (which isn't UTF-8 on Windows). This fixes an issue where names of collections and files could get malformed if they contain non-UTF8 characters. Bug #72
• Fixed a possible assertion error on Win32/Win64. Bug #75
• The configuration settings of the Qt frontend were saved in the wrong location on Win32/Win64. Fixed
• Fixed an issue where the window could disappear on Win32/Win64 when the tray icon is disabled

• Added support for GTK 3.x
• Showing the debug window in standalone mode was broken. Fixed
• If auto-import gets triggered when the GTK frontend is running, a harmless error could get shown indicating that the import failed (even though it succeeded). Fixed

• When importing files with special characters, some invalid JSON-RPC messages could get sent to external frontends. Fixed
• Add a Content-Length header to HTTP requests. Patch by chrashoverraid. Bug #74
• When connecting to a NNTPGrab Server using the 32bit Windows frontend, the server could crash. Fixed. Bug #70
• Don't crash when in invalid jsonrpc request is being parsed. Bug #79

Version 0.6.91 (0.7 Beta 2)¶

• Made the Qt frontend the default frontend for Win32, Win64 and OSX environments
• Added a new ./configure option: --disable-par2cmdline-check
• Only show the filename without the complete path in the GUI while repairing and unpacking files
• Added support for GNUTLS instead of OpenSSL
• Made the various libraries compatible with recent versions of gobject-introspection
• Added support to shut down the system when using KDE
• Reduced some overhead between the glue layer and the frontends
• Allow frontends to indicate whether they want to receive log messages or not (disabled by default). This reduces the load in regular environments

• When the user changes the 'Temp directory' in the configuration all downloads didn't get decoded properly until the program was restarted. Fixed
• On some environments the program could give an 'Unable to resolve dependency' error on startup of the program. Fixed. Bug #49
• A harmless error message could get shown on startup on Win32/Win64 due to an older version of the example plugin being bundled in the NNTPGrab 0.6.90 setup. Fixed
• Automatic shutdown didn't work when the download queue contained 'Skipped' tasks. Fixed
• Refuse to accept configuration changes when the download folder, temp folder or automatic import folder refer to invalid paths
• A regression got introduced in NNTPGrab 0.6.2 which caused configuration changes in the port number of usenet servers to be ignored on Win32/Win64. Fixed
• Fixed an compile error which could occur when using the ./configure argument --disable-ssl
• Single file archives didn't get deleted automatically after an unpack. Fixed
• Added the characters '<' and '>' to the list of forbidden characters in collection names
• A regression was introduced in NNTPGrab 0.6.90 which causes the PAR2 repair to fail when using par2cmdline (instead of par2_tbb). Fixed. Bug #52
• Optimized the measuring of the download speed by using atomic operations instead of mutex operations
• Fixed a race condition in the core library which could cause some warnings
• Fixed an issue which could cause the initialisation of the NNTPGrab Core to succeed even if there was an error
• Fixed a possible schedular deadlock which could be triggered when an error during the decoding has occured
• Don't emit a log message whenever a throttle action occurs

• Implemented importing NZB files using a HTTP POST request in the embedded webserver. Feature #10
• Added a new JSON-RPC function called 'schedular_foreach_task' which is much more scalable than the original 'schedular_get_all_tasks' function. Fixes #53
• From now on an error message will be returned when a JSON-RPC method is called which doesn't exist
• Marking files optional/non-optional using the JSON-RPC interface was broken. Fixed
• Increased the recv buffer for requests in the JSON-RPC plugin from 64K to 512K. Should reduce the chance that clients get disconnected with a 'Too long request received' error

• Added support to connect to a NNTPGrab Server
• Implemented the traffic monitor in the Qt frontend
• Made the "Don't show this dialog again" checkbox in the connect screen operational
• Added support for the Online Search
• Implemented the Automatic PAR2 and Unpack tab
• Implemented everything in the Configuration tab
• Completed the implementation of the 'Manage Plugins' window
• Automatically load all 'auto-load' plugins on startup
• Added the --debug command line parameter
• Implemented the tray icon
• Implemented automatic import
• Give the window a nice icon
• Hide the configuration parts for the JSON-RPC and Logger plugins when they aren't loaded
• Automatically save the download queue after a successfull NZB import
• Renamed the toolbar button 'Enable automatic shutdown' to 'Automatic shutdown' to save some space
• Implemented the 'Automatic Shutdown' feature
• Automatically close the debug window (if it's visible) when the main window is closed
• Implemented showing/hiding columns in the download queue
• Implemented the 'hide inactive connections' configuration option
• Only allow the Qt frontend to be running once at the same time
• If another instance of the Qt frontend is detected, automatically give it focus
• Implemented opening NZB files from the command line
• Enabled showing the estimated time remaining and time to finish for all individual files in the download queue
• Show the download queue using a QTreeView instead of a QTreeWidget for bettter performance
• Fixed a bug where the calculation of the estimated time required didn't work correctly when having multiple collections
• A crash could occur in the Qt frontend when disabling one of the configured servers while a download is still active. Fixed
• Sanitize the collection name before importing files
• When marking a file which is currently being downloaded as 'skipped', the file size remaining wouldn't get reset. Fixed
• The 'Enable intelligent PAR2 downloader' configuration option was broken. Fixed

• Re-added support to kill the NNTPGrab Server from a connected frontend
• Added support for libnotify 0.7
• Hide the configuration parts for the JSON-RPC and Logger plugins when they aren't loaded
• Fixed a deadlock which occured in the GTK GUI when the NNTPGrab Server disappears or when a protocol error occured
• When marking a file which is currently being downloaded as 'skipped', the file size remaining wouldn't get reset. Fixed
• When the connection to a NNTPGrab Server is lost, the program would disappear immediately. Changed this behaviour by showing an error dialog and waiting for the user to close it before closing the entire program
• Fixed some Gtk-Critical warnings in the GTK GUI which could occur on configuration changes
• Notifications were broken. Fixed
• Fixed a possible segfault while starting the GTK frontend
• When the configuration option 'hide inactive connections' is set in the GTK GUI then some invalid data could get shown in the connections overview (like messages belonging to another connection). Fixed

Version 0.6.90 (0.7 Beta 1)¶

• Added a new feature: Only download PAR2 files when needed. People upgrading from older versions of NNTPGrab may need to enable this feature in the configuration first. Feature #1
• Files can be marked optional/non-optional in the GUI by right-clicking on them and selecting the appropriate option
• Improved the performance of download queue manipulations by replacing the SQLite-based download queue with a custom file format. Download queues created with older versions of NNTPGrab won't get converted so make sure to complete your download queue before upgrading to this version. Feature #3
• Added a preview of a new NNTPGrab frontend written in Qt. This will replace the original GTK frontend on Win32/Win64 and Mac OS X environments as it provides a more native look and feel for those platforms. Thanks go out to chrashoverraid and tg1311 for helping out with this frontend! Feature #2
• The integrated search engine is operational again
• Added 'Select all' / 'Select none' buttons to the integrated search engine
• Show all possible connections in the connections overview instead of only the active ones (a configuration option has been added to restore the old behaviour)
• Dropped the GtkHTML dependency by rewriting the update check in the GTK GUI. Feature #19
• Added support for GTK 3 (and prefer it over GTK 2 when both are available)
• Added GObject-Introspection support which makes it easier to control NNTPGrab using different programming languages (like Javascript and Python). See https://live.gnome.org/GObjectIntrospection for more details
• Improved developer documentation by using gtk-doc. This can be found at https://www.nntpgrab.nl/embedded/nntpgrab/docs/index.html
• Dropped (optional) libgnome dependency
• Improved the handling of startup failures
• Added two new environment variables which can be used to influence some default settings: NNTPGRAB_CONFIG_DIR and NNTPGRAB_DEFAULT_DL_DIR. These environment variables are needed for environments like the Fonera. Feature #4
• Allow features to be disable in the ./configure script. Feature #30
• Fixed a compile error when using ./configure --disable-ssl
• Dropped the libxml2 dependency from the NNTPGrab Core to avoid bloat for embedded environments. Feature #30
• Before decoding files, make sure that the filename is in the UTF-8 charset. If this isn't the case try to convert the filename from Windows-1252 to UTF-8 first. Fixes #34
• Ensure that the output of par2cmdline is UTF-8 before processing it. Should fix 'unknown state for PAR2 message' notification which could occur on files which are in the Windows-1252 character encoding
• Read data from SSL connections in 16KB blocks. Should stabilize the connection a bit
• In some situations connections could hang at 99%. Fixed
• When disk space is low, the message 'nntpconnection.c:995 Unable to write article data to file: Success' could get shown. Fixed
• Fixed various memory leaks in the NNTPGrab Core which occured while importing files
• Fixed a possible race condition when using libproxy for the first time
• Added a workaround for a Win32-specific bug in par2cmdline which could cause the PAR2 repair progress bar to keep moving infinitely
• When par2cmdline exited unexpectedly, the progressbar in the GUI could keep going on. Fixed
• Don't automatically remove the PAR2 files when par2cmdline was aborted unexpectedly
• The program could crash on Linux environments when the user closes the program while a PAR2 repair is still active. Fixed
• The unrar program could hang infinitely when an error occured while unpacking. Fixed
• Only show messages in the terminal when the command line argument --debug is used or the environment variable NNTPGRAB_DEBUG is set

Version 0.6.2¶

• Added support for Windows x64
• When connecting to servers, keep the results of all DNS lookups in a cache so reconnecting can be done faster. This should also lower the possibility of the error message 'No hostname could be found for servername x' from occuring
• Automatically retry connecting to usenet servers when a connection error has occured. This should fix a bug where the download queue could hang when the network connection is unstable
• Very small files could get stuck in the download queue. Fixed
• In a rare situation, the program could deadlock while trying to remove a file from the download queue. Fixed
• Really disable activating the automatic import on Mac OS X
• Don't use g_module_close() on Win32/Win64 as it causes a crash when the program is shut down
• When a network disconnect is discovered by the NetworkManager plugin, the download queue would always get stopped, even when the download queue was inactive. Improved the handling of this situation
• Don't wait for active decode actions to be completed before applying configuration changes. This fixes a temporary hang or a deadlock which could occur when changing something in the configuration while a file is being decoded
• The download queue could get in an undefined state when applying multiple configuration changes very quick after each other. Fixed
• When using collection names > 128 characters some error messages could pop up while adding files or performing a PAR2 repair or an unpack. Fixed
• Renamed the configuration option 'Estimated end time' to 'Estimated time remaining' to avoid confusion. Fixes #31
• Added some more logging to improve the diagnosing of incomplete downloads
• When a collection name contains the text '.par2', all files get marked as par2 recovery files by par2cmdline. This resulted in the automatic remove function to remove all files in the collection. Added a workaround for this bug
• Compile fix for environments without GTK 2.12.0 or higher
• NZBCreator: Automatically deselect the checkbox of a collection when there are no items selected in it
• NZBCreator: Don't show expanders for collections with only one file
• NZBCreator: Show the poster for each collection if all the files in a collection were done by the same poster
• NZBCreator: When importing files using the NZBCreator, use a more generic collection name instead of the name of the first file
• NZBCreator: Perform the importing of files in a seperate thread to avoid that the GUI gets frozen for a few seconds
• NZBCreator: Fixed a memory leak which occured while importing files

Version 0.6.1¶

• Prevent a crash on startup which several Mac OS X users were having
• An error message could get shown when trying to remove both an collection and a file within that specific collection at the same time. Fixed
• Files which are downloaded, but not yet decoded could remain stuck in the 'Waiting for decoding' state. Fixed
• When adding files using the NNTPGrab Server which are smaller than 1024 bytes, an error message could get shown in the frontend. Fixed
• The download queue could get suspended when having multiple servers and all the items in the download queue aren't available on one of those servers. Fixed
• When an connection error occurs while downloading, the download queue could end up in an undefined state. Fixed
• When a collection doesn't contain any PAR2 files, no automatic unpack would happen. Fixed
• Also try to unpack .001 files when these are RAR archives
• Fixed a possible crash or strange behaviour when trying to remove files from the download queue while other files are being downloaded
• When a read timeout occurs on all active connections at the same time, the download queue could get suspended even when there still are tasks remaining. Fixed
• Disabled the automatic import options in the GUI for Mac OS X as this feature isn't implemented yet for Mac OS X environments
• Fixed various small memory leaks

Version 0.6.0¶

• Added support to automatically remove PAR2 files after a successfull recovery
• Added a preview of the integrated webinterface created by Flex1986. This can be reached on the address https://localhost:5423 after the JSON-RPC plugin has been loaded and the 'Enable embedded webserver' option has been enabled in the configuration. If you're asked for a username and password, use admin/admin
• When trying to download files on Mac OS X the program could crash. Fixed
• Added support for downloading using a proxy server on Win32 environments
• Suspend the download thread and abort all open connections immediately when there's nothing to download
• Don't try to auto-load plugins which are already loaded. This resolves error messages which could be shown when connecting to an NNTPGrab Server
• Don't build the auto_import plugin when GLib 2.16.0 or higher isn't available
• Don't build the PHP module by default anymore as it's obsoleted by the embedded webserver
• When adding files using the JSON-RPC plugin/NNTPGrab Server, the message-id's got malformed. Fixed
• The JSON-RPC event 'connection_disconnect' didn't send a correct 'reason' field. Fixed
• Fixed a reference count issue which could cause a segfault when executing the JSON-RPC method 'system.events'
• Fixed an invalid free in the JSON-RPC method 'system.events'
• The JSON-RPC event 'traffic_monitor_update' was broken on non-English locales. Fixed
• When removing all items from a collection but not the collection itself could result in the error message 'cannot commit - no transaction is active'. Fixed
• Fixed a possible deadlock while removing items from the download queue
• Small optimalisation while removing items from the download queue
• Improved the resize behaviour of the 'Manage plugins' window

Version 0.5.91 (0.6 Beta 2)¶

• Added a Qt version of the NNTPGrab Server. This makes to program look more native on Win32 and Mac OS X environments
• Added a new feature: Automatically remove files after unpack (disabled by default)
• Implemented JSON-RPC over TCP
• Implemented the remaining JSON-RPC methods
• Ported the NNTPGrab Server and the Glue layer to use JSON-RPC over TCP
• Start the NNTPGrab Server (console) in the foreground by default
• Added the -h / --help arguments to the NNTPGrab Server (console)
• Listen on both IPv4 and IPv6 for the embedded webserver/JSON-RPC server
• Added a new plugin (created by crashoverraid) which prevents Win32 environments from going to sleep while the program is running
• Fixed a bug in the JSON-RPC parser which could lead to 100% CPU usage
• Fixed a possible crash in the JSON parser which could occur when trying to parse invalid data
• Fixed a warning in the NNTPGrab Server about the 'par2_repair_message' signal being unknown
• Always start the embedded webserver when using the NNTPGrab Server (even if it's disabled in the configuration)
• Automatically escape the characters " and \ in JSON requests
• Only load plugins which are really required by default
• Show a notification in the GUI when a NZB file is imported by the NNTPGrab Server
• Automatically save any configuration changes which were done using the JSON-RPC plugin
• Downloading using a proxy server wasn't working correctly. Fixed
• Added support to download using a proxy server on Mac OS X environments
• If a log message with log_level NG_LOG_LEVEL_WARNING occurs then show a popup in the GTK GUI
• While decoding files an error message could be overwritten by another one resulting in an invalid error message. Fixed
• If files from a NZB containing invalid data is being decoded an invalid error message could occur. Fixed
• Fixed a bug where downloading could stop after one part
• Fixed a possible bug where the download queue wouldn't revive anymore when suspended
• Enable the tray icon by default on new installations (except on OSX)
• Fixed a small GTK warning in the GUI which could occur when viewing the list of available auto-import folders
• Only try to unpack files when a collection has been completely verified using PAR2

Version 0.5.90 (0.6 Beta 1)¶

• Improved the overall performance of the program by reducing the number of threads involved while downloading files
• Configuration changes are applied much faster than before
• The decoder has been rewritten for improved performance
• Saving of downloaded files is now more efficient because the need for a temporary dump file has been eliminated
• Don't bother checking the download queue when no servers are enabled
• The download thread will now automatically suspend itself when no new queued items have been found in the last 30 seconds. As soon as some activity happens in the download queue (adding or restarting tasks) the download thread will automatically be resurrected
• Several other optimalisations have been applied in the schedular, the download thread and the GTK GUI
• Make downloading using SSL more reliable
• Bandwidth limiting has been implemented
• The reliability of the speed monitor has been improved
• Support for downloading using proxy servers has been implemented
• Use GNOME's proxy settings for the update check and NZBCreator when they're available (Linux only)
• A plugin architecture has been implemented which makes it much easier to add new functionality to the program
• Automatically pause/resume the download queue when a network connection has been disconnected/resumed (Linux only)
• Added a plugin which automatically logs all debug messages to disk
• Added a plugin which allows NNTPGrab to be controlled using JSON-RPC
• The program can now also run on Fonera environments
• Added French translation
• Optimized the NNTPGrab Server by eliminating several regular expressions
• Fixed a possible deadlock caused by the PAR2 code
• Added a new ./configure option: --disable-ssl
• Don't search for libcurl when running ./configure --disable-gui
• Added a workaround for a bug in par2cmdline (the index file MUST have the extension .par or .par2)
• Added some code to prevent the database being opened by multiple threads at the same time as this can cause 'Database is locked' errors on Win32
• Moving items in the download queue to the bottom was broken with GTK 2.18.0 or higher. Fixed

Version 0.5.1¶

• Readded PPC support for the Mac OS X version
• In some rare situations, the schedular could get paused. This should be resolved now
• Fixed a possible crash in the unpack plugin
• When performing a PAR2 verify, an assertion error could appear. Fixed
• The progress bar in the PAR2 tab could get stuck when an unexpected error occurs while performing a PAR2 recovery
• Don't freeze anymore while manipulating the download queue or configuration when a connection error has just occured
• Prevent some memory corruption
• Prevent an invalid free

Version 0.5.0¶

• Improved the automatic unpack GUI by showing the files which are currently being extracted
• Improved error handling in the automatic unpack
• Rewritten several parts of the Automatic Unpack plugin which are needed to prevent a console window being shown while unpacking on Win32 environments
• When a collection contains multiple PAR2 sets, the GUI could show a lot of assertion errors. Fixed
• When checking PAR2 files which belong to another PAR2 set, the progressbar in the GUI could get stuck. Fixed
• Don't bail out when decoding invalid yEnc messages. Solves the 'Unknown decode error occured, No error (0)' messages
• Reduce the required disk space on Win32 environments by using a small wrapper program for the executable nntpgrab_gui_debug.exe
• Dropped the cURL dependency for environments where libsoup 2.4 (or higher) is installed. This reduces the size of the installer on Win32 and Mac OS X environments
• The performance of decoding files should be slightly improved
• The NNTPGrab Server could freeze on old Linux environments when run in the background. Fixed
• Made the columns in the Repair and Unpack tab resizeable and repositionable
• Renamed the toolbar button 'Enable automatic shutdown' to 'Automatic shutdown' to prevent wrapping
• Automatic shutdown wasn't working on Mac OS X environments. Fixed
• Introduced an automatic snapshot generator which builds Mac OS X installers after each change done in the Subversion repository. These snapshots are published at https://www.nntpgrab.nl/snapshots
• Rewritten the speed monitor in the GUI to be more efficient
• The program could crash when performing a NZBCreator search with the minimum or maximum file size options set. Fixed
• Rewritten the processing of signals in the glue layer to be more efficient
• Made the download queue manipulation code more thread safe
• Several changes required for Win64 support
• Several memory leaks fixed

Version 0.4.91 (0.5 Beta 2)¶

• Removing collections was broken. Fixed
• Don't allow the PAR2 repair and the automatic unpack to happen at the same time as both processes cause high I/O load
• Implemented the GUI for the PAR2 repair and automatic unpack
• Added an extra checkbox to the connect dialog in which the user can indicate whether they want to see the connect dialog the next time they run the program. This option can be reset from the configuration tab if wanted
• Allow the automatic unpack to be configurable from the GUI
• Added some optimalisations in the code responsible for moving files and collections
• The NNTPGrab Server can now be run in the background
• Added a new option to the main menu: Kill the NNTPGrab Server. This option can only be activated when connected to a NNTPGrab Server
• The thread which is responsible for unpacking files could deadlock when corrupt files were being unpacked. This also caused the side-effect of the automatic shutdown not being triggered anymore
• Translations weren't working on Mac OS X. Fixed
• Opening .nzb files while the program wasn't running was broken on Mac OS X. Fixed
• Fixed a possible crash when pausing the download queue while one or more files are waiting to be decoded
• When the program was started, all files with the status 'Waiting to be decoded' would remain on that state forever. Fixed
• Simplified the installation of the PHP module on Win32 environments by eliminating all the external .DLL dependendencies
• In some situations, the program could crash on startup because of a race-condition in the libxml2 initialisation. Fixed
• Fixed a small resource leak
• Compile fix for Linux environments where DBus isn't installed
• Compile warning in base/utils.c fixed
• The program didn't start when having a version of GTK+ older than 2.12.0. Fixed
• When starting the Standalone version of NNTPGrab on Linux, load the file libnntpgrab.so.0 instead of libnntpgrab.so
• The position of collections and files in the database could become inconsistent. Fixed
• When decoding, some filenames could get mangled. From now on, only really forbidden characters may get mangled in the filename
• When all the configured usenet servers are marked 'disabled', all remaining downloads would get changed to the status 'Not available'. Fixed
• Files in the download queue which are no longer on the usenet server anymore were always retried on startup on NNTPGrab while that shouldn't be happening. Fixed
• Added code to prevent the warning "passing a child setup function to the g_spawn functions is pointless on Windows and it is ignored"
• The files nntpgrab_gui.exe and nntpgrab_gui_debug.exe on Win32 were missing a icon and version information. This is now added. Should also solve the 'missing hicolor icon theme' message

Version 0.4.90 (0.5 Beta 1)¶

• Implemented automatic unpack which is performed after a collection is completely downloaded (and after the PAR2 verification/repair if there are PAR2 files in the collection)
• Added support to shut down the computer after all files are downloaded. For Linux, this requires a recent Linux distribution (less than one year old) containing support for PolicyKit
• Merged the Networked GUI and the Standalone GUI in one program
• Improved the processing of output from the PAR2 process. For this version, this isn't visible yet in the NNTPGrab GUI, but this will be done in the next version of NNTPGrab
• Changed the name of the 'PAR2 Repair' tab to 'Repair and Unpack'
• Reduced the CPU usage when using large download queues
• When using large download queues, the memory usage could get very high. This was caused by in-efficient behaviour of libxml2. From now on the download queue will be maintained in a SQLite database. Download queues created using NNTPGrab 0.4.2 or below can't be used with this version anymore, so please complete your downloads with the old version of NNTPGrab before updating to this version
• Reduced the disk space required to install NNTPGrab on Win32 environments by stripping out unnecessary dependencies and unnecessary debug information. This process isn't completed yet, so the required disk space will even be more reduced in later versions
• If a collection only contains files from the same poster, show this poster in the collection row in the download queue
• Servers can now be quickly enabled/disabled by clicking on the 'Enabled' checkbox in the Configuration tab without opening the 'Edit Server' window
• When double clicking on an already existing server in the Configuration tab the 'Edit Server' window will automatically be shown
• From now on there are two different NNTPGrab Server applications: The application 'nntpgrab_server' is always a console application (no GUI stuff involved) while the application 'nntpgrab_server_gui' is an GUI application (which won't be build when the ./configure --disable-gui option is given)
• Made it easier for application developers to create NNTPGrab frontends by removing the dependency on GLib in the public headers and libraries
• Improved the documentation in the public headers
• Show the download queue context menu when the right mouse button or the combination CTRL + left mouse button is used (Mac OS X only). Bugzilla #174
• Added the character ':' to the list of forbidden characters for collection names
• When performing an auto-import, also filter any forbidden characters in the collection name
• Small fix in the Win32 installer: the file libgmodule-2.0-0.dll is also required for the webinterface to work on Win32 environments
• When the system is under high load, the message 'Code should not be reached' could still popup when trying to import NZB files. This is caused by https://bugzilla.gnome.org/show_bug.cgi?id=538458 To work around this, the timeout is increased from 3 seconds to 10 seconds
• Strip out any non-UTF8 characters coming from par2cmdline before presenting it to the frontends
• The program would keep on retrying downloading parts which contain no data instead of ignoring these empty parts. Fixed
• Merged the 'NZB Import' and 'Quick NZB Import' toolbar buttons in one with a drop-down list
• All the download queue manipulation is now done using a 'reference count' system. This makes it possible to reduce the number of locks which should improve the responsiveness of download queue manipulations
• Added support for compiling the program with GCC 4.4
• Added support for cross-compiling the program
• Introduced an automatic snapshot generator which builds Win32 installers after each change done in the Subversion repository. These snapshots are published at https://www.nntpgrab.nl/snapshots
• When using snapshots/subversion builds, the Subversion revision and build number are now shown in the Information tab

Version 0.4.2¶

• The most annoying bug of NNTPGrab up-to-date is the bug where importing a second NZB or importing a NZB from an external source (like a webbrowser) frequently fails with the message 'This code should not be reached'. This is caused by two bugs in GTK: https://bugzilla.gnome.org/show_bug.cgi?id=538458 and https://bugzilla.gnome.org/show_bug.cgi?id=355623 These bugs aren't fixed yet in GTK, but a proper workaround has been found which shouldn't have any side effects
• Improve the reliability of the automatic NZB import by queueing up events triggered by the directory monitor and only process these events when they haven't changed in the last 3 seconds
• Disabling/enabling the automatic NZB import could cause a crash on Win32. Fixed
• While decoding, don't reopen the resulting file again for each part. This should improve compatibility with several virus scanners on Win32 environments
• The schedular could get automatically stopped with the message 'No such file or directory' when the decoder tried to decode a file which lacks the '=yend' tag. Fixed
• The download queue context menu items 'open download folder' and 'execute downloaded file' were broken on Mac OS X. Fixed
• The NZB import screen was showing a raw unix timestamp and the file size in bytes. This has been changed to a human-readable timestamp and a better readable filesize
• Some texts didn't got translated. Fixed
• Don't link against NSS when NSS is found, but nss_compat_ossl isn't found
• Fixed a memory leak in the NZB loading code

Version 0.4.1¶

• The automatic PAR2 wouldn't start when the subject of the file is malformed. This is fixed by checking the file extension after each decode instead of relying only on the subject for the filename
• Before starting to decode files, test if the resulting directory can be opened. If this isn't possible, return an error message and pause the download queue to avoid files from getting lost
• As of NNTPGrab 0.3.0, connecting to IPv6 servers triggered a 'stack smashing attack' after which the program was automatically killed. Fixed
• The decoder could generate several empty files when there's no diskspace available. Fixed
• Improve error handling when saving files on async NFS shares
• Fixed a crash which occured when more than 10 servers are added in the configuration
• Raised the maximum number of servers from 10 to 25 servers
• Fixed a possible compile failure
• Fixed a possible crash in the webinterface and the networked GUI

Version 0.4.0¶

• Added a new option to the configuration tab: enable PAR2 repair
• Only try to auto-import files with the .nzb extension
• When files were moved, they weren't detected by the auto-import. Fixed
• The program would crash while decoding multipart uu-encoded files. From now on, all multipart uu-encoded files will always be incomplete (only the first part gets decoded). This is due to limitations in yydecode where a proper fix requires more time
• When the schedular was paused, active files could get stuck in the 'now downloading' state. Fixed
• Renamed the file plugins/par2/plugin_par2.cpp to plugins/par2/plugin_par2.c as it was causing compile problems on pardus linux
• Visual Studio compile fix
• Use a custom variant of the popen function on Win32 environments. This should hide the par2repair command line window during repair
• Several memory leaks in the NZBCreator have been fixed upstream in libsoup, so remove the unnecessary free's from the NZBCreator code to prevent warnings like GLib-CRITICAL: g_hash_table_unref: assertion `hash_table->ref_count > 0' failed
• The 'estimated time remaining' and 'estimed time to finish' could be shown for collections which were already fully downloaded. Fixed
• The generation of .rpt files containing crash reports on Windows was broken as of NNTPGrab 0.3.0. Fixed
• When the decoding of a file is done, notify the frontend about the resulting filename. Fixes bugzilla #166
• After each decode, change the current directory to the download folder itself (instead of the folder in which all the decoded files were placed) in order to prevent the NNTPGrab process from keeping the directory with the decoded files 'occupied' on Windows environments.
• On Windows environments, the decoding would always fail when the collection name contained special characters. Fixed
• If errors occur during the decode (due to corrupt or incomplete data), ignore these
• Small fix in the Linux shortcuts in order to conform better to the 'Freedesktop Desktop Entry Specification'

Version 0.3.93 (0.4 Beta 4)¶

• The program now shows whether downloaded files are fully complete, incomplete or not available at all
• Add the characters '(' and ')' to the list of forbidden characters in collection names due to a bug in par2cmdline
• Prevent the decoder from replacing any spaces with '_'
• Improved error processing in the decoder
• Prevent some false errors during the decode process
• The par2 repair wasn't started automatically anymore. Fixed
• Don't close the 'Adjust search parameters' section in the NZBCreator when no results are found during the search
• Fix an innocent warning which occured on GTK 2.14.0+ environments
• The decoder could mark the decoding as succeeded even when there was no disk space available. This problem happened on (async) NFS environments. Fixed
• The temporary files were deleted even if the decoding failed. This resulted in files having the 'Done' state. Fixed
• On decode errors, the error message 'Unknown decode error occured' was shown. This is now replaced with a more verbose message

Version 0.3.92 (0.4 Beta 3)¶

• Rewritten the decoder by using yydeview instead of uulib. This fixes the bug where small files weren't decoded right
• If the decoder detects an incomplete files, the extension '.broken' is added to the filename
• On some Windows environments, the downloaded files would always be corrupt. Fixed
• Set the read timeout for NNTP connections to 30 seconds instead of 10 seconds. This should fix some unexpected disconnects on slow responding servers
• If a connection could not be made to the usenet server, reset the status of the file first before sleeping for 30 seconds so other download threads can retry downloading the file
• Compile fix for environments without GTK 2.10.0 or higher
• Don't verify the certificate in NNTP-over-SSL connections as several usenet providers (like Eweka) use self-signed certificates which breaks the verification
• Use the GTK function gtk_show_uri() to open the download folder and to execute downloaded files on environments with GTK 2.14.0 or higher
• Integrated the use_nss_compat_ossl patch in the regular build
• The Windows setup.exe was missing some files required for the webinterface. Fixed

Version 0.3.91 (0.4 Beta 2)¶

• Delete the selected files from the download queue when the button is pressed
• When importing NZB files, remember the directory from which a NZB file was imported and automatically go to this directory the next time the user wants to import a NZB file
• Don't hide the window when the tray icon isn't shown
• On connection errors, sleep the download thread for 30 seconds and push the part back in the queue
• Don't mark parts as being downloaded on connection errors
• The decoder couldn't decode some files on Win32 environments (this resulted in 'No such file or directory' error message). Fixed
• Prevent the user from using illegal characters like *, !, " and ' for collection names as they cause issues when used as a path name
• Reverted commits 948 and 952 as they caused a regression with importing .nzb files from external sources (like a webbrowser) while using GTK 2.13.6 or higher
• The GUI of the NNTPGrab Server could freeze due to the use of some GTK functions in a different thread. Fixed
• Hide the NNTPGrab Server GUI on startup problems (unable to bind to port). Previously the GUI would stay in a frozen state for a few seconds
• Start the 'par2.exe' command from the same directory as the nntpgrab executable on Win32. This should fix the problem where sometimes the 'par2.exe' command could not be found
• Catch any error messages (stderr) from the 'par2' process and show them in the frontend
• Fixed a possible compilation failure which occured when libsoup isn't detected
• Fixed a possible crash when pressing the 'Cancel' button in the NZB Import dialog
• If the operating system supports both IPv4 and IPv6, listen on both in the NNTPGrab Server
• Initialisation errors in the NNTPGrab Server GUI could get lost. Fixed
• On Fedora and Win32, the NSS library is now used instead of OpenSSL for communication with usenet servers over SSL
• Collections containing non-UTF8 characters weren't repaired on Win32. Fixed
• Opening the download folder of collections and executing downloaded files containing non-UTF8 characters wasn't working on Win32. Fixed
• The menu items and the download queue context menu needed 2 clicks before they were shown on Win32. Fixed

Version 0.3.90 (0.4 Beta 1)¶

• Added support for automatic NZB import. A folder can be monitored by the program and if an .nzb file is placed in this folder, the file will automatically be read and added to the download queue. Folders can be monitored on both the local computer and on the NNTPGrab Server. This feature requires GLib 2.16.0 or higher, but is unsupported on Mac OS X for the moment
• Added support for the NZBCreator service. This is a usenet search service which is fully integrated in the program. It is possible to search for files on the usenet using several filters and to import search results directly into the NNTPGrab download queue
• Added support for automatic PAR2 repair. After a collection is fully downloaded all the available PAR2 sets in the collection are verified using the external program 'par2cmdline'. Support for this feature is very basic at the moment due to various reasons, but will be improved in later versions of NNTPGrab
• Several new functions added to the PHP module
• Added a GUI for the NNTPGrab Server (won't be used when ./configure --disable-gui is used or no graphical environment is running)
• In the NNTPGrab Server, when the program could not listen on port 5423, an popup dialog is now shown (when the GUI is enabled)
• Lowered total memory usage by using GSLice instead of malloc
• Reorganised the code infrastructure of the project in preparation for future features
• Use GIO functions to read NZB files on environments with GLib 2.16.0 or higher as this is more efficient

• Started work on a traffic monitor in the webinterface. This isn't visible yet in the webinterface

Version 0.3.3¶

• The list of available download folders wasn't generated when the 'config_changed' signal was already emitted. Fixed
• When generating the list of available download folder, the 'please wait' dialog could be blank. Fixed
• Fix a small memory leak
• Fix a possible deadlock which could occur when the schedular was paused and more than one file were waiting to be decoded
• When decoded files can't be saved, always stop the schedular
• When the state of the schedular changes (with an error message), show a notification balloon
• When the program is minimized and a NZB file is opened from an external program, deiconify the program first before handling the NZB file. This might fix the error message 'file nzb.c: line 644 (on_filechooserNZB_file_set): should not be reached'
• The message 'The list of available download folders is now being updated. Please wait a few seconds' could remain visible after changing the download or the temp folder. Fixed
• Show a popup message when the user tries to add a server with a servername which is already used
• Disable 'type-ahead-find' in the configuration tab as it could cause an assertion failure
• NZB files opened from external sources sometimes didn't get shown. Added an ugly workaround for it

Version 0.3.2¶

• Created an (experimental) version of NNTPGrab for Windows x64
• The NNTPGrab Server would always give an error message indicating the function 'freeaddrinfo' could not be found on Windows 2000 environments. Fixed
• When closing the 'Connect to NNTPGrab Server' dialog from the Networked GUI, the program would keep running. Fixed
• On connection errors in the Networked GUI, shut down the program properly instead of triggering the 'error reporting' facility
• The opening of .nzb files sometimes fails. On Mac OS X the program could even crash when opening an .nzb file (using the Finder) while the program isn't running yet. NNTPGrab Bugzilla #161
• Replaced the use of GStaticRWLock in the schedular by GStaticMutex. This should solve some locking problems
• Added a start menu shortcut for the NNTPGrab Server (Linux only)
• Several compilation fixes which are required to run NNTPGrab on older Linux environments
• The program wouldn't compile using Visual Studio without running the ./configure script. Fixed
• The calculation of the total_file_size and total_file_size_remaining in the webinterface was sometimes miscalculated. Reported by Chrashoverraid. Fixed
• When adding new files using the Networked GUI or the Webinterface, a crash could occur. Fixed
• The traffic monitor didn't work in the networked GUI and the webinterface on environments where GLib 2.14.0 or higher isn't installed. Fixed
• Add support for using NNTP-over-SSL with Astraweb
• NNTP-over-SSL connection problems were handled incorrectly. Fixed
• When decoding files, convert the filename to the utf-8 character encoding before saving it. This prevents the NNTPGrab download queue from getting corrupt. Bugzilla #163
• Updated the Configuration GUI so that everything in the interface is more consistent
• If a collection name contains invalid UTF-8 data, replace it with a space
• NZB files containing special characters couldn't get imported on Windows. Fixed
• Improved translations
• Some texts didn't got translated, Fixed

Version 0.3.1¶

• On Windows, the program remained visible on the taskbar when using the tray icon functionality. Fixed, https://bugzilla.gnome.org/show_bug.cgi?id=537183
• When NNTPGrab 0.3.0 is run for the second time, all the download columns in the download queue could be hidden. Fixed
• The estimated time remaining and estimated time to finish of entire collections became invalid on large collections (4GB+) or large download queues. Fixed
• There was a possible race condition where multiple NNTPConnection objects could get initialized at the same time, resulting in strange error messages like "GLib-GObject-ERROR: g_type_plugin_*() invalidly modified type 'NNTPConnection'". This race condition is now fixed
• Small optimalisation in the generation of the download folder listing
• Don't build the list of available download folders everytime the program starts, only do this when the user actives this configuration - download folders tab.
• Show a small popup dialog while the list of download folders is being built (on Mac OS X this can take up to 10 seconds due to automount). Bugzilla #158
• When opening .nzb files, add them to the 'recently used files' list in the filechooser and the desktop environment (the latter for Linux only)
• Produce more sane error messages on Windows when a connect timeout occurs (fixes the 'No error' message)
• Fixed a small memory leak in the NNTP plugin
• Connections could keep hanging in the 'Idle' state. Fixed
• The 'connection_disconnect' signals wasn't emitted on all connection errors. Fixed
• On Windows, several error messages could get lost (or be 'No error'). Fixed
• Lowered the read timeout to 10 seconds. This should tackle usenet servers which take a very long time to respond on authentication failures, like Eweka
• The behaviour of the setsockopt function (to set timeouts) is different on Windows.. Fixed
• On startup, several 'There is no disk in the drive' dialogs could pop up on Windows. Fixed
• On uninstall of the Win32 version some files remained installed. Fixed

Version 0.3.0¶

• On connection failures, stop trying after 5 seconds. This change also causes the schedular to pause more quickly and the server configuration to process changes faster
• Added a filter to the NZB import filechooser component, so that by default only .nzb files are shown. An option to show all the files is also added
• On fresh installations, the download and temp directory weren't set up correctly. Fixed
• On fresh installations, no columns/contents were shown in the download queue. Fixed
• In the download and temp folder chooser component, perform a scroll and expand only if it was triggered by another frontend
• Opening .nzb files on Mac OS X while NNTPGrab wasn't running yet caused to NZB Import dialog to be blank. Fixed
• Small memory leaks fixed

Version 0.2.91 (0.3 Beta 2)¶

• Added support for marking usenet servers as enabled/disabled
• Improved the installation directory discovery for Win32 environments
• The .nzb file associaton was broken for Win32. Fixed
• Several Visual Studio fixes
• The configuration of the download and temp folders should now work fine on Windows
• Disable drag&drop in the download queue as it is broken
• Added some hacks to get the PHP module to compile on OSX
• In some webserver environments, the webinterface could show some PHP code to the client. Fixed
• Search for the NNTPGrab plugins in the right folder (fixes 64bit /usr/lib64 compatibility)
• Use sane default values when the subject of a file could not be parsed successfully during the NZB import
• Prevent the user from running both the NNTPGrab Server and the Standalone GUI at the same time
• Create crash reports on Win32 when the NNTPGrab Server crashes
• On Win32, if the error WSAETIMEDOUT is triggered, keep trying again until a real timeout has occured
• Moved the Winsock initialisation to the main.c of the GUI and the NNTPGrab Server
• The NZB Import dialog could cause an (GTK internal) assertion error. Fixed
• Destroy the tray icon on shutdown of the GUI
• Removed a lot of unused code
• Lowered the memory usage by using fixed-size structures for all the signal processing
• Check for program updates from a seperate thread. Bugzilla #158
• Only show the notification-bubbles configuration when it is supported

Version 0.2.90 (0.3 Beta 1)¶

• Added support for downloading from multiple servers with multiple threads
• Each server can be assigned a priority (high, normal, low), so that parts are downloaded first from servers with a high priority before servers with a lower priority are tried
• Priority of items in the download queue can be changed
• Addition of network glue layer so frontend and backend can be seperated. Due to this layer it is now possible to perform the downloads on another computer than the computer where the GUI is running
• Added the NNTPGrab Server. This program runs in the background and executes commands from the Networked GUI and the webinterface
• Added a Networked GUI, which is the same as the normal GUI, but can be used to control a NNTPGrab Server (running on a different computer)
• Added a PHP module with which the NNTPGrab server can be controlled
• Added a (basic) webinterface which uses the PHP module to communicate with the NNTPGrab Server
• Added support for a tray icon
• File association with .nzb files
• The file size remaining, estimated time remaining and estimated time to finish are now shown in the GUI
• The columns shown in the download queue can now be turned on and off
• Added support to drag-drop the columns in the download queue
• Improved Mac OS X integration
• New ./configure option added: --disable-gui. With this option set, the GUI won't be build and the GUI-specific configure tests will be skipped
• The plugins are now installed in $prefix/lib/nntpgrab instead of $prefix/lib
• The configuration tab of the GUI is fully redesigned
• Small optimalisation in the speed monitor, only redraw the graph when the GUI is shown. Ignore speed monitor updates when the GUI is hidden/minimized
• Added Visual Studio 2008 project files
• When importing NZB files, select all the files by default
• Detect if the NNTPGrab GUI is already running and if so, give focus to this already running instance
• If a file is being decoded and the resulting filename already exists, then the resulting file will be renamed so it contains a suffix and the original file remains intact
• Provide more user-friendly error messages on possible initialisation errors
• Validate the input in the server configuration dialog before saving it
• Don't use recv() calls with the MSG_PEEK flag anymore as this was causing problems (like kernel warnings about race conditions). Instead, use a buffer for all the received data and only read new data when necessary
• Don't keep the traffic monitor lock longer than necessary. Should better stabilize the traffic monitor while decoding files

• slazh, creating the webinterface
• Icheb, adding ajax support to the webinterface
• PascalW, providing code for .nzb file association under OSX
• Dr3am3r, bugreports, testing of the PHP module and NZB import for the webinterface

Version 0.2.5¶

• The program could crash while decoding files. Fixed
• The total size of collections didn't got updated when a task was removed. Fixed
• The program could crash when there was no free disk space any more to save downloaded parts. Fixed
• Fixed a small compiler warning

Version 0.2.4¶

• The program would always crash when using GTK 2.12.7 or higher. Fixed
• Fix a off-by-one error which could cause the program to crash while downloading
• Fix possible deadlock
• Don't send 'traffic_monitor_update' signals when there hasn't been any traffic for the last 10 seconds. Lowers the CPU usage when there aren't any downloads active
• During the program update check, the program will now check if there are new unstable versions available and notify the user about this
• When the program crashes under Windows, a backtrace is automatically created and saved as the filename 'nntpgrab_gui.rpt'

Version 0.2.3¶

• In some situations, the decoder component could crash. Fixed
• Only send a 'part_progress_update' every 1/10th second instead of for every 10KB of data received to lower the CPU usage
• Only write every 128KB of received data to the disk instead of for every chunk of data received
• Catch possible errors which can occur while saving the downloaded parts
• Several small memory leaks solved
• On Windows Vista the decoding would always fail with the message 'Invalid argument'. Fixed

Version 0.2.2¶

• In the Windows installation script, check if NNTPGrab is still running when the (un)installation program is started
• Drastically lower the memory usage in the NNTP code by using mmap'ed files
• Possible race condition during the initialisation of the program solved
• When the decoding is finished, try to rename (move) the decoded file first instead of copying
• Menu items didn't get translated when using GTK 2.12.0 or higher. Fixed
• The program would crash when the servername of an already existing server was changed. Fixed
• Possible endless loop fixed in the NNTP download code
• Strip any newline characters from messages which need to be pushed to the frontends
• Errors during the downloading of the parts could get lost by some compilers. Fixed
• Prevent a double free and a small memory leak
• Possible crash while changing the configuration fixed
• Another endless loop in the NNTP plugin should be solved. This should solve the 100% CPU usage and the freezes when trying to shutdown NNTPGrab
• Incorporated fixes and optimalisations from the PAN project to the uulib decoder component
• Possible crash which could occur on decoding errors fixed
• When there are decode errors and the users wants to remove the file which triggered the decode error, a deadlock would occur. Fixed
• Under Linux environments, the File Chooser component in the configuration part of the program could show wrong data. Hopefully fixed (might be a GTK bug)

Version 0.2.1¶

• Possible crash in the NNTP plugin solved
• On Windows, NNTP connections were often reset while no errors had occured. Fixed
• Small bug in the update notification component fixed

Version 0.2¶

• Quick NZB Import button added
• Context menu added to the download queue with the options 'Remove item(s) from download queue', 'Restart selected item(s)', 'Open download folder' and 'Execute selected file'
• CPU usage lowered by using a hash table in the list with active connections (Bugzilla #145)
• Traffic monitor implemented. This traffic monitor shows the traffic of the last 10 seconds and the average speed of the last 3 seconds
• Donate button added
• NNTP-over-SSL implemented
• When importing a NZB file, you can now select to which collection the NZB should be added
• Connection-timeout-detection added
• Assertion error in the function 'part_download_start' solved
• Assertion error in the function 'connection_disconnect' solved
• Connection-reset-detection added. This fixes the behaviour where all the tasks in the download queue could be marked as 'done', while these aren't downloaded yet
• On decoding errors, a popup message is shown mentioning what went wrong
• Possible crashes during initialization of the program fixed
• Possible crash in the configuration object solved
• Program/start-menu icon added. Created by ArmiDuctor
• Some interface improvements in the NZB Import window
• Default position of all the windows changed to 'center on parent'
• Error messages occuring on read and write errors could get lost. Fixed
• When importing a NZB file for a second time without closing the window, all the previous NZB data would remain visible. Fixed
• When NNTPGrab was shut down while a decode still needed to take place, the decode would not occur the next time NNTPGrab was started. Fixed
• Don't remove the part-files on decode errors
• Added a seperate configure-check for the program 'msgfmt' to work around a bug in intltool 0.36.2 (or lower) which could cause compile problems
• Some 64bit platform fixes
• Possible deadlock while trying to remove items from the queue fixed. Bugzilla #152
• The program wasn't able to connect to some usenet servers like 'usenetserver'. Fixed
• When importing NZB files, the .nzb extension is stripped from the filename and this name is used (by default) for the collection name.

Version 0.1¶

• First release

Wiki¶

This is a wiki for the NNTPGrab project. It is split in three main sections: a generic part, an user part and an developer part

Generic pages¶

Version history
Feature requests
License
Contact

User documentation¶

User documentation
Using XAMPP and the NNTPGrab PHP module
Debugging crashes in NNTPGrab

Developer documentation¶

If you want to help the development of NNTPGrab or extend it, documentation can be found here. If something is not clear enough on these pages, feel free to post a message on the forum

API Documentation
Control NNTPGrab using JSON-RPC
Control NNTPGrab using PHP
Control NNTPGrab using a custom TCP protocol (obsoleted)
Developing plugins
Developing on Mac OS X
Fonera port
eTrayz port
Help wanted

Using XAMPP and the NNTPGrab PHP module¶

As of version 0.3 of NNTPGrab it is possible to control the program through a web-interface. To enable this web-interface you need to set up an webserver with PHP support. This document describes how to enable the web-interface using XAMPP.

• Apache
• PHP
• MySQL
• Perl

First, you need to go to the XAMPP website and download the latest version of XAMPP. Next, install it.

After the installation is complete, you can download and install NNTPGrab. If you choose for the custom installation, you need to make sure the 'Webinterface' component gets installed.

Next, open the Windows Explorer and navigate to the folder where you've installed NNTPGrab (this is C:\Program Files\NNTPGrab by default). In it you will find several folders.

• Copy the contents of the folder 'webinterface' to the XAMPP htdocs folder (this is by default C:\xampp\htdocs, you can delete all the files which are already there or create a new subfolder)
• Copy the contents of the folder 'php_module' to the XAMPP PHP extensions folder (C:\xampp\php\ext)
• Copy the contents of the folder 'php_deps' to the XAMPP Apache folder (C:\xampp\apache\bin) (This step is only necessary when using NNTPGrab 0.4.90 or older)

Next, you need to make a small adjustment to the PHP configuration file. This file can be found at C:\xampp\apache\bin\php.ini. Open this file in an editor (notepad for example) and search for a block of text containing 'extension=xxxx' rules. This should be around line number 600. You need to add a new line of text below the rest of the extension lines containing :

extension=php_nntpgrab.dll

Finally, start the XAMPP control panel and start/restart the Apache service. You're now ready to go and use your web browser to navigate to https://localhost to use the NNTPGrab webinterface