Difference between revisions of "OLA developer info"

Latest revision as of 05:55, 31 December 2017

This attempts to describe the code structure of OLA, in particular the core C++ framework, the olad server and the plugins. Be sure to read Using OLA first to understand the Port, Device, Universe & Plugin terminology.

A Brief Tour

Let's quickly cover the layout, you can browse the code online to follow along.

include - contains the header files (.h) that are installed on the system

ola - headers for the OLA framework. If the header is used by both the client and server code it belongs here

olad - headers specifically for the olad server

common - code for the ola framework, contains the implementations of everything in include/ola/

olad - the olad server code

ola - the ola C++ client code

plugins - headers and code for all the olad plugins

javascript - the client side code used for the web UI

java - the Java OLA client

python - the Python OLA client

tools - Various utilities like the RDM sniffer, the RDM Responder Tests, DMX Trigger etc.

include/ola is further broken into modules. There are modules for RDM (include/ola/rdm), HTTP (include/ola), IO, Network etc.

Core Framework Classes

The headers can all be found in include/ola, the implementations are in common/. While some of these just wrap the native library calls, we abstract them so that it's easier to port OLA to new platforms. Ideally there would be no platform-dependent code outside of common.

DmxBuffer

include/ola/DmxBuffer.h

The DmxBuffer class allows DMX data to be passed around the code, while avoiding unnecessary copying.

DmxBuffer data;  // new empty buffer, starts with a size of 0
data.Size(); // size of the buffer, 0 to 512
data.Blackout();  // set all channels to 0
data.Set(data, length); // set the buffer from a uint8_t*
data.Get(data, &length); // copy the data buffer into the memory pointed to by data

Dmxbuffer data2 = data; // no copy

DmxBuffers are very similar to the boost shared_ptr

Callbacks

Callbacks are used extensively throughout OLA. They provide asynchronous notification when an operation completes, and reduce the coupling between modules.

Basic Callbacks

include/ola/Callback.h

Callbacks are similar to function pointers, they allow both functions and methods to be invoked at a later time with data from either / both the time the Callback is constructed and the time the Callback is executed.

All Callbacks have a Run() method, which is how the Callback is executed.

Callbacks come on two varieties, Persistent and SingleUse. SingleUse callbacks delete themselves after Run() is called so you don't have to.

// wrap a function that takes no args and returns a bool
SingleUseCallback<bool> *callback1 = NewSingleCallback(&Function0);

// some time later
bool result = callback1->Run();
// callback1 has deleted itself at this point

// create a Callback for Method1 of the Object class and bind TEST_VALUE as the first arg
Callback<void> *callback2 = NewCallback(object, &Object::Method1, TEST_VALUE);

// this will call object->Method1(TEST_VALUE)
callback2->Run();
// this wasn't a SingleUse Callback, so callback is still around and needs to be deleted manually.
delete callback2;

// create a Callback for a method that takes 1 arg and returns void
BaseCallback1<void, unsigned int> *callback3 = NewCallback(object, &Object::Method1);

// Call object->Method1(TEST_VALUE)
callback3->Run(TEST_VALUE);
// callback3 is still around at this stage
delete callback3;

// create a callback for a method that takes 2 args and returns void
  BaseCallback2<void, int, int> *callback4 = NewSingleCallback(
      object,
      &Object::Method2,
      TEST_VALUE);

// This calls object->Method2(TEST_VALUE, TEST_VALUE2);
callback4->Run(TEST_VALUE2);
// callback4 is still around
delete callback4;

CallbackRunner

include/ola/CallbackRunner.h

Sometimes you need to ensure that a callback is always executed when a method returns.

void Foo(MyCallback *on_complete) {
  CallbackRunner runner(on_complete);
  // do work here, which may contain return statements, on_complete will always be executed.
}

MultiCallback

include/ola/MultiCallback.h

Multicallback is a callback that executes another callback after it has been called N times.

/**
 * Calls DoSomething() for each Port and runs the on_complete callback once each port's callback has run.
 */
void DoSomethingForAllPorts(const vector<OutputPort> &ports,
                            SomethingCalback *on_complete) {
  // This will call on_complete once it itself has been Run ports.size() times.
  BaseCallback0<void> *multi_callback = NewMultiCallback(                                                                                                       
      ports.size(),
      NewSingleCallback(this, &SomethingComplete, on_complete));

  vector<OutputPort*>::iterator iter;
  for (iter = output_ports.begin(); iter != output_ports.end(); ++iter) {
    (*iter)->DoSomething(multi_callback);
  }
}

Memory Buffers and Streams

OLA has a collection of classes for reading and writing data from memory. If you've used the C++ streams classes a lot of this will look familiar.

Abstract classes have dotted borders.

Of all the classes, the ones you probably care about are the IOQueue, BigEndianOutputStream & BigEndianInputStream. Skip down to those classes for an example.

Input & Output Buffers

include/ola/io/InputBuffer.h
include/ola/io/OutputBuffer.h

These define the abstract classes for reading and writing from/to memory buffers.

IOQueue

include/ola/io/IOQueue.h

The IOQueue implements both the InputBuffer and OutputBuffer interfaces. They are used in all of the new network code to construct & parse packets.

IOQueues are nice because they grow as data is written. Normally you don't write to IOQueues directly but instead use an OutputStream.

IOQueues are also integrated with the Socket classes, so they avoid a memory copy. For instance when you call UDPSocket::SendTo() it uses Vector I/O to avoid a copy.

MemoryBuffer

include/ola/io/MemoryBuffer.h

A MemoryBuffer is useful if you have a const block of memory, and want to extract typed data sequentially using an InputStream.

void ExtractData(const uint8_t *data, unsigned int length) {
  MemoryBuffer input(data, length);
  // can now use input with an InputStream.
  // ...
}

Input & Output Streams

include/ola/io/InputStream.h
include/ola/io/OutputStream.h

These allow you to read / write typed data to Input/Output Buffers.

BigEndianInputStream & BigEndianOutputStream

include/ola/io/BigEndianInputStream.h
include/ola/io/BigEndianOutputStream.h

Sending example:

IOQueue packet;
BigEndianOutputStream output(&packet);

output << (int) 42; // automatically converts to big endian
output << true;
output.Write(raw_data, raw_data_size);

// now packet contains our binary data.
socket->SendTo(packet, target_address); // target_address is a IPV4SocketAddress.

Receiving Example (we don't support IOQueues for input yet):

bool HandlePacket(const uint_t *data, unsigned int length) {
  MemoryBuffer packet(data, length);
  BigEndianInputStream input(&packet);

  int32_t foo;
  if (!(input >> foo))
    return false;  // not enough data
  bool bar;
  if (!(input >> bar))
    return false;  // not enough data
  // do something with foo and bar here
  return true;
}

Clock

include/ola/Clock.h

Contains the TimeInterval, TimeStamp and Clock classes for managing time. It also defines a MockClock class which can be useful for testing.

// get the current time
TimeStamp timestamp, timestamp2;
Clock::CurrentTime(&timestamp);

// sleep for a bit
usleep(10000);

// print the duration we slept for
Clock::CurrentTime(&timestamp2);
TimeInterval interval = timestamp2 - timestamp;
cout << interval << endl;

Credentials

include/ola/base/Credentials.h

Functions to get user and group information.

uid_t uid = GetUID();
PasswdEntry passwd;
if (!GetPasswdUID(uid, &passwd))
  return;
cout << "Username is " << passwd.pw_name << endl;

Initialization

include/ola/base/Init.h

Helper functions that should be called on startup. They do things like install SEGV stack trace handlers, initialize the random number generator etc.

AppInit(argc, argv);

HTTP Server

include/ola/HTTPServer.h

The HTTP server is provided by microhttpd. The HTTPServer class provides a C++ wrapper around the microhttpd code.

Logging

include/ola/Logging.h

Contains logging macros which behave like streams:

OLA_FATAL << "foo";
OLA_ERROR << "bar";
OLA_INFO << "baz";
OLA_DEBUG << "bat";

Logging is initialized with a call to InitLogging(level, output). i.e.

// Send INFO and above to STDERR
InitLogging(OLA_LOG_INFO, OLA_LOG_STDERR);

// or

// Send WARNING and above to SYSLOG
InitLogging(OLA_LOG_WARN, OLA_LOG_SYSLOG);

Note you can't send different levels to different destinations (so you need one or the other of the above examples). Calls to InitLogging() overwrite the previous logging configuration.

String Utils

include/ola/StringUtils.h

While not a class, this defines a number of helper functions for dealing with Strings. If you need to split strings, convert ints to strings and back, escape, trim or capitalize strings use these functions.

STL Utils

include/stl/STLUtils.h

Various helper methods for dealing with STL containers like vector and map. Try to use this as much as possible to reduce code (and the change of introducing a bug!).

vector<Foo*> foos;
STLDeleteValues(&foos);  // delete all objects in foo

map<int, string> our_map;
vector<int> keys;
vector<string> values;
STLKeys(our_map, &keys);
STLValues(our_map, &values);

Filesystem Utils

include/ola/file/Util.h

A collection of functions for working with files.

vector<string> usb_devices;
// return all files that match /dev/ttyUSB*
FindMatchingFiles("/dev/", "ttyUSB", &files);

Random Numbers

include/ola/math/Random.h

A simple uniform random number generator. You need to call InitRandom() before using this. Do not use for anything security related.

InitRandom();
int r = Random(1, 10);  // returns an int in the range 1 .. 10

Backoff Policies

include/ola/util/Backoff.h

Backoffs are an important part of writing reliable software. Backoff.h defines various BackoffPolicies and a BackoffGenerator.

BackoffGenerator generator(
  new ola::ExponentialBackoffPolicy(TimeInterval(1, 0), TimeInterval(64, 0));

// when an event fails
TimeInterval backoff = generator.Next();
// schedule a retry...

// and when it fails a second time
TimeInterval backoff = generator.Next();
// schedule a retry...

// and when it succeeds
generator.Reset();

Networking

Network Utils

include/ola/network/NetworkUtils.h has helper methods for converting between endian formats, and converting IPv4 addresses to strings and visa-versa.

IP Addresses & Socket Addresses

include/ola/network/IPV4Address.h include/ola/network/SocketAddress.h have classes used to represent IP Addresses and Socket Addresses.

SelectServer & Sockets

The SelectServer is the dispatcher at the core of OLA and is defined in include/ola/network/SelectServer.h. It waits for events, and when an action occurs calls the specified method. The SelectServer can also be used to register Timeouts (called every N ms) and Loop functions (shouldn't be used).

Threads and Locking

Threads

ola/include/thread/Thread.h

This works as you'd expect.

class MyThread: public Thread {
 protected:
   void *Run() {
     // write code for the new thread here
   }
};

MyThread thread1;
thread1.Start();  // blocks until the thread is running
// continue, and later
thread1.Join();

Mutexes & Condition Variables

ola/include/thread/Mutex.h

This provides Mutexes (non-recursive) and Condition Variables (Monitors).

Mutex mu;
int counter;

void Foo() {
  mu.Lock();
  counter++;
  mu.Unlock();
}

While this works, it's not good coding practice. Consider what happens if someone later changes this to:

Mutex mu;
int data_to_protect;

bool Foo() {
  mu.Lock();
  counter++;
  if (counter % 10)
    return true;
  mu.Unlock();
  return false
}

Now the Mutex isn't unlocked when counter is a multiple of 10. On the next call the program will deadlock.

To avoid this, we use the MutexLocker class.

void Foo() {
  MutexLocker locker(&mu);
  counter++;
}

It saves a line of code, and unlocks the Mutex automatically when it goes out of scope.

RDM

Write me.

Timecode

include/ola/timecode/TimeCode.h include/ola/timecode/TimeEnums.h

A simple class for handling Timecode data. See OLA Timecode for more info.

JSON

include/ola/web/Json.h

A set of classes for constructing JSON data structures and serializing them.

JsonObject obj;
obj.Add("name", "simon");
obj.Add("Australian", true);
obj.Add("age", 21);

JsonArray *lucky_numbers = obj.AddArray("lucky_numbers");
lucky_numbers->Append(2);
lucky_numbers->Append(5);

// returns the above as JSON.
const string json_output = JsonWriter::AsString(obj);

OLAD

RPC System

The RPC system is what allows clients to communicate with the OLA Server. It uses Protocol Buffers as the data interchange format. You can see the message definitions in the Ola.proto file.

Clients communicate with OLAD over a TCP socket connected to localhost::9010. The RPC port olad listens on can be changed with the --rpc-port command line option but this will require the clients to be updated as well. Because the communication is over localhost we don't need to worry about dropped messages causing the socket buffers to fill up. You can still exceed the socket buffers if the client sends faster than the server can receive, but this usually indicates a poorly written client.

Client Side

The code required on the client side is the similar irrespective of langage. All that's required to write an OLA client is an implementation of Protocol buffers. See the Protocol Buffers, Other Languages page.

Client code needs to:

• Open a TCP connection to localhost:9010
• Provide an API, which constructs protocol buffer objects, serializes them and writes them to the TCP socket.
• Read data from the TCP socket, de-serialize it into protocol buffer objects and deal with the response data.

The only tricky bit is matching the responses to the requests. This task is usually handled by a StreamRpcChannel class, see the Python or C++ code.

Server Side

On the server side, OLAD creates a listening socket and then calls InternalNewConnection each time a client connects. This sets up a OlaClientService object, which is what gets called when a new message is received. The OlaClientService objects call into OlaServerServiceImpl.cpp which is where the message handling takes place.

HTTP Server

olad/OlaHttpServer.h implements the OLA specific behavior and runs as a separate thread. This uses the OlaCallbackClient which has a TCP connection open to the OLAD core, just like any normal client.

Plugin System

Read a walkthrough of the OSC plugin here.

We'll use plugin to refer to the entire module (Plugin, Devices & Ports), and Plugin to refer to the class that inherits from Plugin.

Plugins create and register Devices, which each consist of 0 (obviously not useful) or more Ports. A Plugin generally does a bit of work when it starts to detect devices, then leaves all work to the individual Devices and Ports.

Each plugin implements the classes in blue. Of course, you can choose not to inherit from the BasicPort / Device / Plugin classes and do everything yourself.

PluginAdaptor

The PluginAdaptor is the interface between plugin code and the core OLA objects. Each Plugin object has a pointer to a plugin adaptor in the instance variable m_plugin_adaptor.

Plugins

The AbstractPlugin interface is defined in include/olad/Plugin.h. The Plugin class implements most of this interface, and leaves Id(), PluginPrefix(), StartHook(), StopHook(), SetDefaultPreferences() and Description() to be implemented by the child classes.

The startup sequence for a Plugin object is:

• From within DynamicPluginLoader::LoadPlugins an instance of the plugin is created
• If the ShouldStart() method returns False, nothing else happens, otherwise the Start() method is called.
• The Start() method calls LoadPreferences() which in turn calls SetDefaultPreferences(), this last method gives the Plugin the opportunity to setup the Preferences object.
• if SetDefaultPreferences() doesn't fail, StartHook() is called where new Devices are created. m_plugin_adaptor->RegisterDevice() should be called to add new Devices.

During the shutdown sequence:

• Stop() is called, which in turn calls StopHook()
• StopHook should call m_plugin_adaptor->UnregisterDevice() for any devices registered during the start phase.
• delete is then called on the Plugin object

At any time a the following methods can be called:

• Id()
• Name()
• Description()

Devices

The interface to Devices is defined in include/olad/Device.h as AbstractDevice, again the Device class implements most of this interface, leaving the derived classes to fill in a couple of methods:

• DeviceId() - returns a unique persistent string identifying this device
• StartHook() - this should create the port objects for a device.

Ports

Ports are the objects that actually read/write DmxBuffers. Defined in include/olad/Port.h there is the base interface Port, and then two child interfaces, InputPort and OutputPort. The BasicInputPort and BasicOutputPort provide partial implementations for these two interfaces.

At a minimum , an OutputPort needs to provide the following methods:

• WriteDMX(const DmxBuffer &buffer, uint8_t priority)
• Description()

And an InputPort needs to provide:

• ReadDMX()
• Description()

A call to ReadDMX() is triggered by calling DmxChanged() on a InputPort object. This causes the universe the port is bound to to fetch the new DMX data. Both ReadDMX() and WriteDMX() must be non-blocking, blocking here will delay the main processing loop. To satisfy this, most ports use this sequence of events:

• register a Socket for reading with the SelectServer

// some time later

• receive notification that there is new data on the socket
• read the data and copy it to a buffer
• call DmxChanged() to notify the bound Universe we have new data
• the Universe then calls ReadDMX()

Often more than one port will use the same file descriptor. This means the device is responsible for reading the data and dispatching to the right port.

Here's an example of how dmx data is received from the UsbPro Device.

The UsbProDevice will have been registered using plugin_adaptor->RegisterSocket(). When input becomes available the following sequence happens:

 device->action() // signals the device that new data is available
   widget->recv() // tells the widget to read more data
     widget->do_recv() // reads the data from the fd
       widget->handle_cos() // handles the change-of-state message from the widget
         device->new_dmx() // signal the device that new dmx data has arrived 
          port->DmxChanged() // signal the universe that new dmx data has arrived
            // if this port is bound to a universe, the universe will then call
            port->ReadDMX()
              device->get_dmx()
               widget->get_dmx()

Of course, the plugin authors are free to implement this however they like.

Config Messages

Config messages are handled a little differently for two reasons:

* The configure() method in a plugin has to return a response immediately. We don't want to block because we'll delay all lla processing. The new RPC subsystem removes this limitation.
* Sending a PARAMETER_REQUEST to the widget doesn't generate a response immediately (in fact it may not generate one at all).

To work around this, we send a parameter_request when we start the device, and then anytime we set parameters. In the meantime we store the parameters in the widget object and return those. The sequence looks like:

 device->configure()
   device->config_get_params()
     widget->get_parms()

• What is the interface between the LLA core and LLA plugins?

 See above and the files plugin.h, device.h and port.h. The create() call will be passed a PluginAdaptor object which can then be used to register/unregister file descriptors, loop functions, timeouts and devices.

• What is the interface between the LLA core and other apps/clients to LLA like QLC?

 All clients should use the LlaClient library. This needs better documentation.

• How is functionality split between the usbpro plugin and the example program?

The example program constructs configuration request messages and sends them (using LlaClient) to the Lla Core. The core routes this message to the plugin, which then returns a response message. This response is passed back to the client.

Ideas for easy configuration

For some users, it will be useful to have a "auto-connect" feature. When a attached device is discovered (either when LLA i started or when a new device is attached), the user could be asked if the available ports (input as well as output) should be patched to the lowest available universes.

• Enable auto-connect ( OFF|connect whatever comes first|connect by stored patch layout)
• Save a given combination of devices (just by type or with unique ID's from serial numbers, USB device ID's etc)

Which devices cannot be autodetected?

About device config messages

We need a way to tune settings on a port/device that the LLA Core doesn't know about. To enable this, the LlaClient provides a method dev_config(unsigned int dev, LlaDevConfigMsg *msg)

The LlaDevConfigMsg is an interface which declares one method: pack(uint8_t buffer, unsigned int length). On the device side, we declare a method configure(uint8_t *request, int length)

So to use this:

On the client

 MyObserver::dev_config(unsigned int dev, uin8_t *res, unsigned int length) {
   MyLlaDevConfigMsg msg = parse_message(data, length);
   // do something with the result
 }
 
 int main() {
   // all the setup code
 
   MyObserver observer;
   // the observer gets the dev_config() callback
   lla_client->set_observer(&observer);
 
   MyLlaDevConfigMsg msg;
   // set some fields
   msg.foo = 1
   lla_client->dev_config(device_id, &msg); //calls pack() on the message
 }

In the device:

 MyDevice::configure(data, length) {
   MyLlaDevConfigMsg msg = parse_message(data, length);
   // do something with the message
 
   MyLlaDevConfigMsg *response = new MyLlaDevConfigMsg();
   // response is deleted by the lla core
   return response;
 }

The tool app "lla-usbpro"

The purpose is to set and get the settings that reside in the USB Pro box.

The communication with USB Pro's seems to go via the LLA core, and lla-usbpro registers as a LLA client, and uses some event handlers.

As defined in the device spec. (PDF from Enttec):

label=3 response

• 1. data byte= Firmware version LSB. Valid range is 0 to 255.
• 2. data byte=Firmware version MSB. Valid range is 0 to 255.
• 3. data byte=DMX output break time in 10.67 microsecond units. range=[9-127] (96.03 - 1355.09 micro seconds)
• 4. data byte=DMX output Mark After Break time in 10.67 microsecond units. range=[1-127] (10.67 - 1355.09 micro seconds)
• 5. data byte=DMX output rate in packets per second. range=[1-40]
• x. data byte= some user configuration of the requested size

The serial number is is decoded (from 4 bit Binary Coded Decimal) in lla-usbpro, not the plugin.

Unsupported USB devices

• Peperoni Rodin1 (but this is supported directly by QLC)
• Peperoni USBDMX21
• USB DMX from usbdmx.com
• Sandsys UMX2
• Sandsys UMX4

Peperoni and usbdmx are probably easy to implement. The specs and source code examples are available.