Data Logging¶
liveserial provides support for logging multiple sensor streams from a single serial port. To differentiate the streams, serial expects a key to be specified in the global configuration file (see Multiple Sensor Configuration File). Once the data lines can be differentiated by the sensor key, the data is split in the plots and log files.
In addition to just logging the data to csv files, the logging module also
handles the periodic querying of the data queues that the serial monitor threads
continuously write to (see Serial Port Monitoring). Thus, even in the event that the user
chooses no file logging, a liveserial.log.Logger instance is still
needed to help the Data Plotting and Serial Port Monitoring to talk to each other.
There are two different intervals configured for the logging:
- buffertime: how often (in milliseconds) the logger aggregates the serial data queues to produce a single value.
- logfreq: how often (in seconds) the logger appends the latest data points to the csv files for persistent storage.
Because the logger implements threading.Timer to handle the aggregation
and saving, the main thread doesn’t experience latency for large datasets being
written to disk, which allows the plotting to continue smoothly.
Methods for grabbing data and logging it to CSV files.
-
class
liveserial.log.Logger(interval, dataqs, livefeed, method='last', logdir=None, logfreq=10, plotting=False, config=None, aggregate=False)[source]¶ Logs data points from the serial port to CSV. The arguments to this class constructor are also available as attributes on the class instance.
Parameters: - interval (int) – how often (in milliseconds) to read data from the serial buffer.
- dataqs (list) – of
multiprocessing.Queue; stores the data read in from the serial port. - errorqs (list) – of
multiprocessing.Queue; stores any error raised during serial port reading. - livefeed (monitor.LiveDataFeed) – feed for storing the latest data points obtained from the serial port.
- method (str) – one of [‘average’, ‘last’] specifies how to aggregate multiple data points in the buffer.
- logdir (str) – directory to place log files in for the sensors. If None, then data will not be logged to disk.
- logfreq (int) – how often (in seconds) to write the accumulated data points to CSV.
- plotting (bool) – when False, the values being read off will be printed if logging is not enabled.
- config (ConfigParser) – global sensor configuration parser; if the argument is a str, then the file path.
- aggregate (bool) – when True, the logger should check sensor config for aggregate ports to be handled; otherwise, aggregate ports are ignored.
-
timer¶ threading.Timer – executes calls to the serial port reader to get the latest data and push it to the live feed.
-
lastsave¶ float – timestamp indicating the last time the CSV file was appended to
-
csvdata¶ dict – lists of sensor time and value readings, keyed by sensor identifier.
-
config¶ ConfigParser or str – global sensor configuration parser. If str, then the config is loaded from the specified file path.
-
aggregate¶ dict – keys are aggregate sensor names; values are functions that accept a dict of the latest physical sensor values, and return single, aggregated values for plotting.
-
ready(delay=None, wait=1.0)[source]¶ Returns True once we have accumulated a few timer calls of data. This ensures that we know how many sensors are running on the same COM port.
Parameters: delay (float) – fraction of a second to wait before checking if the data is there.
-
save()[source]¶ Saves the logger’s buffered points to a CSV file. If the file exists, then the data points are appended.
-
sensor_option(sensor, option, default=None, cast=None)[source]¶ Returns the specified sensor option if available.
Parameters: