History

Wiki » Developer documentation »

JSON-RPC Documentation

As of NNTPGrab 0.6 it is possible to control NNTPGrab using JSON-RPC. This requires the JSON-RPC plugin to be loaded in the program.

Installation

JSON-RPC support is provided by the JSON-RPC plugin which is installed along with the program by default. To activate the JSON-RPC server, you need to go to the configuration tab in NNTPGrab, select Plugins and make sure that the option Enable embedded webserver is checked. It's also possible to indicate the port number of which the JSON-RPC server needs to listen.

After the JSON-RPC server has been activated you can send JSON-RPC commands to the URL https://localhost:5423/jsonrpc (where localhost is the hostname or IP address of the host where NNTPGrab is running and 5423 is the port number which has been set in the NNTPGrab configuration.

Handshake

JSON-RPC over HTTP

When using JSON-RPC no special handshake is needed. It's sufficient to send a regular HTTP POST request containing JSON-RPC data to the server. The connection will automatically be closed by the server after every request as Mongoose (the embedded webserver library used by NNTPGrab) doesn't support HTTP keep-alive.

JSON-RPC over TCP

If possible it's recommended to use JSON-RPC over TCP. This is because NNTPGrab can emit events (notifications) at any time. When using JSON-RPC over TCP the connection doesn't need to be re-established after every request so this saves time and resources.

For JSON-RPC over TCP the same port as JSON-RPC over HTTP can be used, but instead of a HTTP request, a custom request needs to be sent by the client to the server. This way the webserver knows what method needs to be used.

To activate JSON-RPC over TCP the client needs to send this message to the server:

NNTPGrab - API version 20110625\r\n\r\n

The value 20110625 is the current API version as defined in the file nntpgrab_core/nntpgrab_plugins.h and the \r\n\r\n sequence are two newlines

In response, the server can send one of these replies:

OK\r\n
API mismatch, expected 20091228\r\n

If the server sends the OK response then the handshake is completed and JSON-RPC request can be sent to the server. The server can also send any events (notifications) back to the client at any moment

If the server sends the API mismatch response then a different version of the API is expected. The value 20091228 mentioned here is just an example, it can be different depending on the version of NNTPGrab installed. The connection will be closed automatically by the server if this response is emit

Authentication

JSON-RPC over HTTP

Authentication is required to use JSON-RPC over HTTP. This is realized by the HTTP authentication method Digest. The default username and password for NNTPGrab 0.6.x is admin/admin.

JSON-RPC over TCP

Authentication isn't implemented at the moment for JSON-RPC over TCP, but it will be in future versions. Check back later

Functions provided by the NNTPGrab JSON-RPC plugin

nntpgrab_get_version_info

nntpgrab_get_version_info - Returns several version numbers belonging to NNTPGrab

object nntpgrab_get_version_info

Returns an object with the following fields:

(
    [glue_version]
    [major_version]
    [minor_version]
    [micro_version]
    [version]
)

Available since NNTPGrab 0.6

nntpgrab_config_get_avail_servers

nntpgrab_config_get_avail_servers - Retrieve a list of the available usenet servers

array nntpgrab_config_get_avail_servers()

Returns an array containing the servername's of all the available usenet servers

Available since NNTPGrab 0.6

nntpgrab_config_get_server_info

nntpgrab_config_get_server_info - Retrieve the configuration details about a specific usenet server

object nntpgrab_config_get_server_info(string servername)

When the given servername isn't known an error is returned, otherwise
an object containing the following elements is returned

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [send_group_command]
    [use_ssl]
    [enabled]
)

The contents of the field priority has one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.6

nntpgrab_config_add_server

nntpgrab_config_add_server - Add a new server to the NNTPGrab configuration

boolean nntpgrab_config_add_server(array serverconfig)

Returns TRUE when the server was successfully added to the NNTPGrab configuration
or FALSE when the given servername is already known.

The serverconfig array needs to have the following fields:

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [send_group_command]
    [use_ssl]
    [enabled]
)

The contents of the field priority needs to have one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.6

nntpgrab_config_del_server

nntpgrab_config_del_server - Remove a server from the NNTPGrab configuration

boolean nntpgrab_config_del_server(string servername)

Returns TRUE when the server was successfully removed from the NNTPGrab configuration
or FALSE when the given servername isn't known.

Available since NNTPGrab 0.6

nntpgrab_config_edit_server

nntpgrab_config_edit_server - Change the settings of a server

boolean nntpgrab_config_edit_server(string original_servername, object serverconfig)

Returns TRUE when the change was successfully performed or FALSE when the given original_servername isn't known.

The serverconfig object needs to have the following fields:

(
    [servername]
    [hostname]
    [port]
    [username]
    [password]
    [max_threads]
    [priority]
    [send_group_command]
    [use_ssl]
    [enabled]
)

The contents of the field priority needs to have one of the following values:

0 - High priority
1 - Normal priority
2 - Low priority

Available since NNTPGrab 0.6

nntpgrab_config_get_opts

nntpgrab_config_get_opts - Retrieve the basic configuration information from NNTPGrab

object nntpgrab_config_get_opts()

Returns an object containing the following fields:

(
    [download_directory]
    [temp_directory]
    [enable_intelligent_par2_downloading]
    [enable_par2_repair]
    [enable_auto_import]
    [auto_import_directory]
    [move_file_after_auto_import]
    [enable_auto_unpack]
    [enable_bandwidth_shaping]
    [max_bandwidth]
    [enable_webserver]
    [webserver_port]
    [enable_logger]
)

Available since NNTPGrab 0.6

nntpgrab_config_set_opts

nntpgrab_config_set_opts - Change the basic configuration of NNTPGrab

boolean nntpgrab_config_set_opts(object options)

Returns TRUE when the changes have been performed or FALSE when there was an error

The options object needs to have the following fields:

(
    [download_directory]
    [temp_directory]
    [enable_intelligent_par2_downloading]
    [enable_par2_repair]
    [enable_auto_import]
    [auto_import_directory]
    [move_file_after_auto_import]
    [enable_auto_unpack]
    [enable_bandwidth_shaping]
    [max_bandwidth]
    [enable_webserver]
    [webserver_port]
    [enable_logger]
)

Available since NNTPGrab 0.6

nntpgrab_schedular_start

nntpgrab_schedular_start - Start the NNTPGrab schedular

boolean nntpgrab_schedular_start()

Returns TRUE when the schedular has been successfully started or FALSE when
the schedular was already running or in the stopping state

Available since NNTPGrab 0.6

nntpgrab_schedular_stop

nntpgrab_schedular_stop - Stop the NNTPGrab schedular

nntpgrab_schedular_stop(boolean wait)

If wait is FALSE, a request will be sent to the NNTPGrab Server to
stop the schedular, but the function will not wait for the schedular to
be really stopped. If wait is TRUE, this function will wait
until the schedular is really stopped.

Available since NNTPGrab 0.6

nntpgrab_schedular_get_state

nntpgrab_schedular_get_state - Retrieve the current state of the NNTPGrab schedular

int nntpgrab_schedular_get_state()

This function can return one of the following values:

0 - Schedular is running
1 - Schedular is stopping
2 - Schedular is stopped

Available since NNTPGrab 0.6

nntpgrab_schedular_add_file_to_queue

nntpgrab_schedular_add_file_to_queue - Add a new file to the download queue

boolean nntpgrab_schedular_add_file_to_queue(string collection_name, string subject, string poster, int timestamp, int file_size, array groups, array parts)

Returns TRUE when the file has been successfully added or FALSE when there already is a file with
the same subject in the same collection_name in the download queue

The file_size needs to be in bytes

The array groups contains a list of usenet groups in which this file is published

The array parts contains an array which has the following fields:

(
    [0] => Object -- This block is representing one part. This block can occur multiple times
        (
            [message_id]
            [part_num]
            [size]
        )
)

Available since NNTPGrab 0.6

nntpgrab_schedular_del_file_from_queue

nntpgrab_schedular_del_file_from_queue - Remove a file from the download queue

boolean nntpgrab_schedular_del_file_from_queue(string collection_name, string subject)

Returns TRUE when the file has been successfully removed or FALSE when the given collection_name and subject aren't known

Available since NNTPGrab 0.6

nntpgrab_schedular_restart_file

nntpgrab_schedular_restart_file - Restart a file

boolean nntpgrab_schedular_restart_file(string collection_name, string subject)

Returns TRUE when the file has been successfully restarted or FALSE when the given collection_name and subject aren't known

Available since NNTPGrab 0.6

nntpgrab_schedular_save_queue

nntpgrab_schedular_save_queue - Save the download queue

boolean nntpgrab_schedular_save_queue()

This function should be called after several new tasks have been added to the download queue

Returns TRUE when the save has been successful or FALSE when there was an error

Available since NNTPGrab 0.6

nntpgrab_schedular_get_all_tasks

nntpgrab_schedular_get_all_tasks - Retrieve all the tasks in the download queue

array nntpgrab_schedular_get_all_tasks()

The array returned from this function contains a lot of data:

(
    [0] => Object -- This block is representing one collection. This block can occur multiple times
        (
            [collection_name]
            [total_size]
            [total_size_remaining]
            [files] => Array
                (
                    [1] => Object -- This is representing one file. This block can occur multiple times
                        (
                            [subject]
                            [poster]
                            [stamp]
                            [file_size]
                            [file_size_remaining]
                            [num_parts_total]
                            [num_parts_downloaded]
                            [num_parts_failed]
                            [status]
                            [groups] => Array -- This is representing a list of usenet groups in which this file is published
                                (
                                    [0]
                                )
                        )
                )
        )
)

The fields total_size and file_size mention the size in KB (kilobytes, 1024 bytes)

Possible values for the status are:

0 - Waiting for download
1 - Downloading
2 - Waiting for decoding
3 - Decoding
4 - Finished - complete
5 - Finished - incomplete
6 - Finished - no parts available
7 - Skipped

Available since NNTPGrab 0.6

nntpgrab_schedular_move_file

nntpgrab_schedular_move_file - Change the position and/or location of a file in the download queue

boolean nntpgrab_schedular_move_file(string collection_name_src, string subject_src, string collection_name_dest, int position_dest)

collection_name_src can be the same as collection_name_dest. If position_dest is -1, the file will be added to the end of the queue.
If subject is empty, then position_dest will mean the position of all the collections in the download queue.

Available since NNTPGrab 0.6

nntpgrab_schedular_move_collection

nntpgrab_schedular_move_collection - Change the position of a collection in the download queue

boolean nntpgrab_schedular_move_collection(string collection_name, int new_position)

If new_position is -1, the file will be added to the end of the queue.

Available since NNTPGrab 0.6

nntpgrab_schedular_mark_task_optional

nntpgrab_schedular_mark_task_optional - Mark a task optional/non-optional in the download queue

boolean nntpgrab_schedular_mark_task_optional string collection, string subject, boolean is_optional)

Available since NNTPGrab 0.7

nntpgrab_plugins_get_avail_plugins

nntpgrab_plugins_get_avail_plugins - Get a list of all the available plugins

array nntpgrab_plugins_get_avail_plugins()

Returns an array containing string's with the name of the plugin

Available since NNTPGrab 0.6

nntpgrab_plugins_get_plugin_info

nntpgrab_plugins_get_plugin_info - Retrieve information about a specific plugin

object nntpgrab_plugins_get_plugin_info(string plugin_name)

The returned object contains these fields:

(
    [name]
    [version]
    [author]
    [url]
    [description]
    [is_loaded]
    [is_persistent]
)

Available since NNTPGrab 0.6

nntpgrab_plugins_load_plugin

nntpgrab_plugins_load_plugin - Load a plugin

boolean nntpgrab_plugins_load_plugin(string plugin_name)

Returns TRUE if the plugin was successfully loaded. In case of an error, an JSON error will be returned containing details

Available since NNTPGrab 0.6

nntpgrab_plugins_unload_plugin

nntpgrab_plugins_unload_plugin - Unload a plugin

boolean nntpgrab_plugins_unload_plugin(string plugin_name)

Returns TRUE if the plugin was successfully unloaded. In case of an error, an JSON error will be returned containing details

Available since NNTPGrab 0.6

nntpgrab_plugins_set_persistent

nntpgrab_plugin_set_persistent - Change the persistent flag of a plugin

void nntpgrab_plugins_set_persistent(string plugin_name, boolean is_persistent)

In case of an error, an JSON error will be returned containing details

Available since NNTPGrab 0.6

nntpgrab_utils_strip_subject

nntpgrab_utils_strip_subject - Split a subject in various useable fields

array nntpgrab_utils_strip_subject(string subject)

The array returned contains the following fields:

(
    [subject]
    [subject_without_partnum]
    [file_num]
    [total_files]
    [filename]
    [extension]
    [file_type]
    [par2_startnum]
    [num_par2_blocks]
    [part_num]
    [total_parts]
)

Note that some fields can be missing. If a field is missing, it can't be detected in the given subject

Available since NNTPGrab 0.6

nntpgrab_utils_calculate_file_size

nntpgrab_utils_calculate_file_size - Returns the given file size in a human readable notation

string nntpgrab_utils_calculate_file_size(long filesize)

The filesize should be in KB (kilobytes)

Available since NNTPGrab 0.6

nntpgrab_utils_calculate_estimated_time_remaining

nntpgrab_utils_calculate_estimated_time_remaining - Make an estimation when a file with the given size will be completed

int nntpgrab_utils_calculate_estimated_time_remaining (array speed_history, int filesize)

The array speed_history should be the history part of the array returned by the function nntpgrab_connections_get_speed_history
The filesize is in KB (kilobytes)
The value returned is the number of seconds necessary to complete the given file size

Available since NNTPGrab 0.6

nntpgrab_utils_get_readable_time_remaining

nntpgrab_utils_get_readable_time_remaining - Returns the human readable notation of a given number of seconds

string nntpgrab_utils_get_readable_time_remaining (int time_remaining)

The value time_remaining is returned by the function nntpgrab_utils_calculate_estimated_time_remaining

Available since NNTPGrab 0.6

nntpgrab_utils_get_readable_finished_time

nntpgrab_utils_get_readable_finished_time - Returns the human readable notation of a given number of seconds relative to the current time

string nntpgrab_utils_get_readable_finished_time (int time_remaining)

The value time_remaining is returned by the function nntpgrab_utils_calculate_estimated_time_remaining

Available since NNTPGrab 0.6

nntpgrab_utils_get_folder_listing

nntpgrab_utils_get_folder_listing - Retrieves a list containing all the sub-folders which are in the given folder

mixed nntpgrab_utils_get_folder_listing(string parent)

Returns FALSE when the given parent isn't known by the NNTPGrab Server or an
array containing object's with these values:

(
    [folder]
    [has_subfolders]
)

Available since NNTPGrab 0.6

Events provided by the NNTPGrab JSON-RPC plugin

When connected to NNTPGrab using JSON-RPC over TCP events can be emit at any moment. When using JSON-RPC over HTTP, events (or in JSON-RPC terms: notifications) can be sent along with responses to requests. These are the possible events which the NNTPGrab Core can send:

config_changed

Something in the configuration has changed

config_changed

part_download_start

The download of a part has just started

part_download_start string servername, int conn_id, string collection_name, string subject, int part_num

part_done

The download of a part has completed successfully

part_done string servername, int conn_id, string collection_name, string subject, int part_num, int size

part_failed

The download of a part has failed (probably because the part isn't available on the server

part_failed string servername, int conn_id, string collection_name, string subject, int part_num, int size, boolean all_servers_tried

traffic_monitor_update

Every second, traffic statistics about the last 10 seconds are given

traffic_monitor_update int bytes_received1, int  bytes_received2, int  bytes_received3, int bytes_received4, int bytes_received5, int bytes_received6. int bytes_received7, int bytes_received8, int bytes_received9, int bytes_received10, int stamp, double average

part_progress_update

Every 1/10th second of a part download, this message will be sent

part_progress_update string servername, int conn_id, string collection_name, string subject, int part_num, int bytes_downloaded, int bytes_total

collection_added

A new collection was added

collection_added string collection_name, string poster

collection_removed

A collection was removed

collection_removed string collection_name

collection_modified

The poster of a collection was modified

collection_modified string collection_name, string poster

file_added

A file was added to a already existing collection

file_added string collection_name, string subject, string poster, int stamp, int file_size, int total_size, int total_size_remaining, int status, int num_parts, array groups

The groups argument is a list of newsgroups (string's)

NOTE: The fields file_size, total_size and total_size_remaining contain the number of KB's instead of bytes

file_removed

A file was removed from the download queue

file_removed string collection_name, string subject, int total_size, int total_size_remaining

NOTE: The fields total_size and total_size_remaining contain the number of KB's instead of bytes

file_download_state_update

The state of a file in the download queue has changed

file_download_state_update string collection_name, string subject, int num_parts_total, int num_parts_done, int num_parts_failed, int file_size, int file_size_remaining, int total_size, int total_size_remaining

NOTE: The fields file_size, file_size_remaining, total_size and total_size_remaining contain the number of KB's instead of bytes

file_state_changed

A file in the download queue was restarted

file_state_changed string collection_name, string subject, string real_filename, int old_state, int new_state

connection_connecting

A new connection is being made to a usenet server

connection_connecting string servername, int conn_id

connection_connected

A connection attempt has succeeded

connection_connected string servername, int conn_id, string welcome_msg

connection_disconnect

A connection to a usenet server was disconnected

connection_disconnect string servername, int conn_id, int disconnect_type, string reason

disconnect_type can be any of the following values:

0:  normal
1:  hostname could not be resolved
2:  connection refused
3:  too many connections
4:  connection timeout
5:  read error
6:  read timeout
7:  write error
8:  idle timeout
9:  invalid message received from server
10: login failure
11: unable to initialize SSL
12: unexpected error occured

schedular_state_changed

The state of the schedular has changed

schedular_state_changed int new_state, string reason

log_message

A log message has been emitted

log_message string component, int log_level, string msg

log_level can contain one of the following values:

1: info
2: warning
3: error
4: fatal
5: debug

task_moved

The position of a task in the download queue has changed

task_moved string orig_collection_name, string subject, string new_collection_name, int old_position, int new_position

collection_moved

The position of a collection in the download queue has changed

collection_moved string collection_name, int old_position, int new_position

all_downloads_completed

This message is emit when all the items in the download queue are completed

all_downloads_completed

plugin_loaded

A plugin has been loaded

plugin_loaded string plugin_name, boolean is_persistent

plugin_unloaded

A plugin has been unloaded

plugin_unloaded string plugin_name

plugin_event

A plugin has emit an event

plugin_event string plugin_name, string event_name, array values

The values parameter is an array containing string's

Plugin events

Certain plugin can emit events which are interesting to frontends.
This is a list of all known events which can be emit by plugins (as part of the plugin_event event)

PAR2 plugin

The name of this plugin is PAR2

par2_begin_verify

A PAR2 verification (and if necessary repair) has just started

values[0]: collection_name
values[1]: active_par2_filename

par2_load_progress_update

A file is now being verified

values[0]: collection_name
values[1]: active_par2_filename
values[2]: filename
values[3]: progress

par2_recovery_file_loaded

A PAR2 recovery file has been loaded completely

values[0]: collection_name
values[1]: active_par2_filename
values[2]: filename
values[3]: num_new_packets
values[4]: num_blocks_found

par2_file_loaded

A regular file has been loaded completely

values[0]: collection_name
values[1]: active_par2_filename
values[2]: filename
values[3]: state
values[4]: num_blocks_found
values[5]: num_blocks_expected

The field state can be one of the following values:

MISSING
FOUND
DAMAGED
NO_NEW_BLOCKS_FOUND

par2_repair_progress_update

A PAR2 repair is now going on

values[0]: collection_name
values[1]: active_par2_filename
values[2]: progress

par2_repair_failure

The PAR2 repair has failed because there aren't enough recovery blocks available or some other error

values[0]: collection_name
values[1]: active_par2_filename
values[2]: error message
values[3]: num_blocks_more_required

par2_repair_success

The PAR2 repair has completed successfully

values[0]: collection_name
values[1]: active_par2_filename

par2_no_repair_required

All the files in the PAR2 set have been verified and no repair is necessary

values[0]: collection_name
values[1]: active_par2_filename

Unpack plugin

The name of this plugin is Unpack

unpack_progress_update

The unpack component has made some progress

values[0]: collection_name
values[1]: filename
values[2]: progress

unpack_message_received

The unpack component has emit a message. This message mostly is the file which is currently extracted

values[0]: collection_name
values[1]: filename
values[2]: message

unpack_working_archive_changed

The unpack component is now unpacking from a new archive file

values[0]: collection_name
values[1]: filename
values[2]: new archive filename

Also available in: HTML TXT