How-To: Data logging to your own server

use your own server to collect monitored data from your node(s)

From Node versions 1.12 and upwards, the node software offers the possibility of configuring data logging to an ad-hoc server, in addition to logging data to this site.
This can be useful in a number of use-cases, for example:


The node software will log data in CSV format, where all monitored and logged fields are sent to your server in a comma separated list, once a minute.
Would you have multiple nodes in your network, you can configure your own node identifier for each node, that will be sent in the HTTP request header.

The following brief guide should explain the steps to perform in order to get your node(s) logging to your custom logging server.

The server parts are backed with some PHP sample code, as PHP is a well-spread language in the web programming world. We assume you possess the skills to get the sample code running, tweak it to your liking or adapt it to another programming language of your choice.

In order to get your node(s) /running software version >=1.12/ logging to your own server, running on your PC, or better, a low-power-consumption, maintenance and noiseless SBC (Single Board Computer) capable of running linux, such as the Raspberry Pi or the BeagleBoard (and there are some more available on the market).

Key steps

This is advanced functionality and so we assume you are familiar with setting up PC's or SBC's to run a supported OS of your choice (we recommend Linux) and a web server and virtual host dedicated to logging. Would you however need help with such services and do not want to Do It Yourself, we recommend you to ask for help of an IT professional.

Setup your server
  1. Setup your PC or SBC, including TCP/IP networking
  2. Install and configure your web server
    We assume you are familiar with setting up a web server, such as Apache or another one, of your choice.
  3. Write and deploy your logger
    A sample PHP code is provided below, that is a simple but functional piece of software that can give you a kick-start. For more details, please see the source code comments, below.
    Code: Download
    <!--
    logger.php
    ==========
    A simple data logger sample that can be used with the "APDuino Project" nodes'
    dedicated log server feature.
    
    It receives Comma Separated Values (CSV) from the node, sent periodically
    to the dedicated logging service and stores it to files in a log directory.
    
    Filenames are generated from the unique Node ID (you should set in the NodeUI
    for each node you want the feature to be enabled) and the actual date.
    
    Again, this is just a simple example, that however works. Feel free to tweak
    to your liking.
    Enjoy! :-)
    
    Requirements:
    * a web server with PHP5
    * "APDuino Project" node software (version >= 1.13)
      with 'Custom Logging' options configured (to point to the web server above)
    * web server user must have write access to the log directory ($logdir)
      see below...
    
    (C) 2015 Gyorgy Schreiber
    
    This is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.
    
    This code is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details: <http://www.gnu.org/licenses/>.
    
    -->
    <?php
    
    $mydir = dirname($_SERVER['SCRIPT_FILENAME']);    // get the directory of this script
    $logdir = $mydir."/log";    // web server (user) must have write access
    $timestamp = date("Y-m-d H:i:s");    // create a sortable timestamp
    $datestamp = date("Ymd");        // get a sortable date stamp
    
    // returning a very simple page for GET requests
    if($_SERVER['REQUEST_METHOD'] == 'GET') {
    ?>
    <html>
    <head><title>Logger</title></head>
    <body>
    This is just a sample data logger.<br/>
    Logs into <?php echo $logdir."/node_" ?>&lt;NodeID&gt;.log
    </body>
    </html>
    <?php
    // storing the CSV data for PUT/POST requests
    } elseif($_SERVER['REQUEST_METHOD'] == 'PUT' || $_SERVER['REQUEST_METHOD'] == 'POST') {
      $headers = apache_request_headers();        // request headers contain nodeid
      $rawdata = file_get_contents("php://input");    // request body contains CSV data
      foreach ($headers as $header => $value) {    // iterate the headers
        if($header == 'X-NodeId') {                // to find out the Node ID
          $nodeid = $value;                    // and save it in $nodeid
          break;                        // break loop when found
        }
      }
      if (isset($nodeid)) {        // if got node id, use it in the logfile
        $logfile=$logdir."/node_".$nodeid."_".$datestamp.".log";
      } else {            // otherwise lets use the IP in the filename
        $logfile=$logdir."/node_".$_SERVER['REMOTE_ADDR']."_".$datestamp.".log";
      }
      // and save the raw CSV sent by the node, prepended with the timestamp
      file_put_contents($logfile, $timestamp.$rawdata, FILE_APPEND );
      // you could do a lot more things, split the fields, store to a DB, etc.
      // those things are up to you, as it is just a simple (but working) example
      // to give you an idea
    }
    ?>


Configure your Node
Yet again, to use this functionality, you must have APDuino Node software version 1.12 or greater running on your Arduino™ Mega hardware.
  1. Open the Node's LAN console
    You must be in the same local network with your browser than the node, when configuring this feature. Access the Node Console via the 'My Nodes' menu of this site.
  2. Follow the steps outlined on this image, in order to get logging configured.
    System / Settings / Logging / Enter configuration / Save settings Configure logging to a dedicated server using the Node Console
    Important note on the 'Hostname' and 'IP' address: this feature is not doing any name resolution, therefore it is key that you enter the IP address of your server. However, hostname is offered and is used in the request, in case you would be running different virtual hosts on the server and would prefer to distinguish between them based on hostname.
  3. Dedicated logging settings will be effective upon saving, meaning that next time the Node is sending logs to APDuino, it should follow that request with another request to send CSV data to your configured server.