Tracked components

Kismet manages complex objects with arbitrary serialization and logging by implementing “tracked elements”; Tracked elements are introspectable in C++ and can be dynamically exported to other formats via the serialization API, such as JSON which is used heavily in the web interface.

Tracked elements represent a compromise between the efficiency of native C++ types and the necessity of dynamic data handling and serialization.

Unless you are writing a native C++ plugin, you do not need to worry about tracked elements at all; if you ARE, then there are some guidelines to follow.

Anatomy of a tracker_element

Generally speaking a plugin should not define a new tracker_element itself, it should implement a tracked_component (more on these in the next section); however they form the basic building blocks of both tracked_component and of the data held by custom components.

All tracked elements are derived from tracker_element; This super-class contains no actual data storage.

Via subclassing and templates, tracker_element implements basic scalar data types (non-POD basic data types such as std::string, uuid, mac_addr and similar), basic numeric types of the standard sizes (uint8_t through uint64_t, 4-byte float and 8-byte double), and various complex data types such as vector and map. Optimized versions of vectors and maps are provided for common combinations.

tracker_element subtypes are named accordingly, i.e.:

tracker_element_string
tracker_element_uint8
tracker_element_int32
tracker_element_vector
tracker_element_double_map

and can be found in tracked_element.h.

Vectors and Maps

tracker_element represents complex data types in two main formats:

A vector or map (typically implemented as an unordered map or as a robinhood hash map for high speed access) of tracker_element records. This allows constructing complex dictionary-like or list-like records of arbitrary fields, tracked_component complex records, MAC addresses, and so on.
Optimized containers like tracked_element_vector_double and tracked_element_double_map_double which contain, respectively, raw double values and raw double:double key:value pairs. Other, similar containers are also defined.

If your plugin stores purely numerical data, such as a vector of signal values, it is significantly more efficient to use the tracked_element_vector_double (or equivalent DoubleMapDouble) objects, and will consume less than half the RAM using a traditional tracker_element_vector would use.

Each tracker_element derivative also contains:

A trackertype::... type which enforces type-safety on get/set operations
A uint32_t signature which is used to enforce type-safety on complex types
An optional local name
A 16-bit unique ID which is used to map the field name
A reset function used to reset the object when it is recycled

Each derived type contains:

A get/set function for its type
Optional coercive_set functions which allow importing generic types (such as strings to UUID or mac_addr, and various precision numbers to numeric types)
A clone_type function which is responsible for returning a std::shared_ptr instance of a new object of this type, which is integral to the field creation and tracking system.
Optional additional support functions, such as standard STL iterator implementations for vector and map elements

`tracker_element` Lifecycle

tracker_elements are nearly always stored and processed as C++11 std::shared_ptr smart-pointer managed objects. This allows usage counting and automatic deletion of a tracker_element, which may be found hosted in many dynamic representations of data over its lifecycle.

To reduce RAM fragmentation, memory allocation tracking, and generally improve performance, all basic tracker_element types are maintained in dynamic object pools where they are automatically cached and reset.

Tracked elements can be manually created from the pool using the Globalreg::new_from_pool<type>() system.

It is almost never necessary (or wise, or even possible) to create a naked tracked_element or a naked tracked_element * pointer.

`tracker_component`

A tracker_component is derived from a tracked_element_map, which implements a dictionary of fields.

Collections of data maintained and exposed by a plugin should be derived from the tracker_component class; Kismet uses this class to maintain all internal state about devices and other data, and a tracker_component can be directly inserted into an existing device record.

The tracker_component also implements a collection of helper macros which will automatically define getter/setter functions, with optional casting, as well as support for dynamic field creation on first use.

The tracker_component has all of the existing tracked_element features, but also specifies two additional core functions:

register_fields() which is called during construction and is responsible for defining all the fields that this object needs (typically using the register_field function)
reserve_fields(std::shared_ptr<TrackerElement> e) which is called during construction and is responsible for allocating any custom fields which cannot, for some reason, be allocated automatically by the registration system. Typically the only actions which are needed here are re-typing complex fields during loading from stored data.

Tracker components can be registered with the global object pools via Globalreg::enable_pool_type (defined in globalregistry.h). If registered in the object pool, it must define the reset() method.

Deriving from `tracker_component`

Any data you wish to expose as an object must be derived from tracker_component. To do this, there are a number of functions you must implement.

First, the Boilerplate:

For our example, we want to mimic the behavior of the Kismet messagebus so that we can display messages on the web UI.

We need to create a message record which has a timestamp, the message content, and a set of flags.

class web_tracked_message : public tracker_component {
public:
    // tracker_component base constructor
    web_tracked_message() :
    tracker_component() {
        // We always have to call register and reserve
        register_fields();
        reserve_fields(NULL);
    }
    
    // Constructor which takes the ID
    web_tracked_message(int in_id) :
    tracker_component(in_id) {
        // We always have to call register and reserve
        register_fields();
        reserve_fields(NULL);
    }

    // The final default constructor, which takes both and ID and a pointer to
    // existing object data; this is used to craft a custom object from a base
    // which already contains these fields; this can happen when loading stored
    // data or deserializing IPC data
    web_tracked_message(int in_id, std::shared_ptr<TrackerElement> e) :
        tracker_component(in_id) {
        // Again the register and reserve, but this time we pass the
        // pre-existing element to the reserve_fields object
        register_fields();
        reserve_fields(e);
    }
    
    // We need to provide a signature for this type, since all tracker_components
    // are TrackerType::TrackerMap at their base; typically this signature can be
    // a quick checksum of the type name.
    virtual uint32_t get_signature() const override {
        return adler32_checksum("web_tracked_message");
    }

    // We need to provide the clone type function; in this case we can decide if we
    // want to generate the new object from the global pool or from a std::make_shared.
    // In this case we'll use pools.
    virtual std::shared_ptr<tracker_element> clone_type() override {
        using this_t = typename std::remove_pointer<decltype(this)>::type;
        auto r = Globalreg::new_from_pool<this_t>();
        r->set_id(this->get_id());
        return r;
    }

    // Since we're using the object pool, we have to provide a reset function; we'll
    // call reset on our fields (defined next)
    virtual void reset() override {
        timestamp->reset();
        message->reset();
        flags->reset();
    }

}

This creates a tracked object, but there is nothing inside of it. To do that, we need to add some data fields.

TrackerElement Fields

For fields that are always present, we only need to track the field itself. Lets define some basic type fields; notice that we always define them as std::shared_ptr, and that we use the proper TrackerElementXyz sub-type for each.

protected:
	std::shared_ptr<tracker_element_uint64> timestamp;
	std::shared_ptr<tracker_element_string> message;
	std::shared_ptr<tracker_element_int32> flags;

Registering Fields

To actually use the fields, they need to be initialized with a type and name. This is done in the register_fields() class method.

protected:
    virtual void register_fields() override {
        // Call the parent register_fields() function, you MUST do this
        tracker_component::register_fields();

        register_field("webmsg.message.timestamp", "message timestamp", &timestampt);
        register_field("webmsg.message.message", "message content", &message);
        register_field("webmsg.message.flags", "message flags", &flags);
    }

register_field(...) is part of the base tracker_component class, and handles the connection between a tracked data set and the tracking system which assigns fields.

To register a field, you need:

A field name (which must be unique, you can ensure uniqueness by including your plugin type in the name; Kismet uses basic namespacing conventions to keep fields unique)
A human-readable description; this will be shown in the tracked_fields page and is helpful for future developers and consumers of your API
A pointer to the shared pointer where your field resides. This allows the field registration system to automatically create your fields and place them in your variables for you.

register_field will return the internal id of the field which is created - most of the time this is not necessary, however when registering more complex objects or a dynamic object it may be necessary to save this ID.

Accessing the Data: Proxy Functions

Now that we have some data structures, we need to define how to access them.

It’s certainly possible to define your own get/set methods, but there are some macros to help you.

The __Proxy(...) macro allows easy definition of a handful of methods in one line, at the expense of slightly obtuse syntax:

__Proxy(name, tracker type, input type, return type, variable)

This expands to define get and set functions (get_name and set_name) which accept input type variables and return return type, while automatically casting it to the type required by the tracker_element, indicated by tracker type.

What this really allows you to define in a single line cast-conversions between compatible types and define standard get/set mechanisms. For example, for a simple unsigned int element, flags, defined as tracker_uint32, you might use:

public:
    __Proxy(flags, uint32_t, uint32_t, uint32_t, flags);

This would expand to define:

public:
    virtual uint32_t get_flags() const {
        return (uint32_t) get_tracker_value<uint32_t>(flags);
    }
    virtual void set_flags(uint32_t in) {
        flags->set((uint32_t) in);
    }

This looks fairly standard, but allows for more interesting behavior to be defined simply. For instance, we want to hold a standard unix timestamp (time_t) in the timestamp field, however there is no tracker_element primitive for timestamps. However, if we do the following:

public:
    __Proxy(timestamp, uint64_t, time_t, time_t, timestamp);

Now we have a get and set pair of functions which accept time_t and transparently cast it to a uint64_t when saving or reading from the tracker_element variable. The same trick can be used to make automatic get and set functions for any data type which can be cast directly to the internal tracked type.

Additionally, individual get and set functions can be proxied via __ProxyGet(...) and __ProxySet(...) if you wish to only expose the get or set, or if you provide a custom get or set function which is more complex. Numerical values can also define __ProxyIncDec(...) or __ProxyAddSub(...) to generate increment/decrement (++ and –) and addition/subtraction functions automatically. Fields which represent a bitset can use __ProxyBitset(...) to define bitwise set and clear functions.

There are some other tricks for accessing data which is represented by complex data types, we’ll cover them later.

Building from other Data Structures

Often you will want to automatically populate a tracked component from an existing data structure.

Due to how trackercomponents are generated, it is not possible to extend the constructors and import data that way, however creating a function for importing data is trivial, especially once the __Proxy(...) functions are defined.

To continue our Messagebus example, let’s define a method for adapting a message to our tracker_component:

public:
    void set_from_message(string in_msg, int in_flags) {
        // Since we used __Proxy to define get and set functions, we'll
        // call them instead of doing custom get/set and risking doing
        // it wrong.
        
        set_timestamp(globalreg->timestamp.tv_sec);
        set_message(in_msg);
        set_flags(in_flags);
    }

Accessing from outside the object

It may be necessary to allow access from outside callers. This is only required for other code directly accessing your object; for exporting your data via the REST interface and other serialization methods, so long as your data is in tracker_element objects it will be handled automatically.

If you do need to provide access to your data objects, there are several methods you can utilize:

Method one: Provide functions which interface to the complex type

In some instances you may wish to write methods which provide access to the complex type. For example, assuming that Sharedtracker_element example_vec is a TrackerVector, and you wish to add the record e to it, it may make sense to implement access thusly:

public:
    void example_vec_push_back(Sharedtracker_element e) {
        example_vec->push_back(e);
    }

Essentially the same is hiding the internal data structure via your object API: add_foo(...) may internally add to the vector, without ever explicitly exposing the actual data types.

Method two: Proxy the TrackedElement directly

By directly returning a pointer to the TrackedElement you can allow consumers to wrap the complex data themselves. A proxy macro similar to the others is provided:

public:
    __ProxyTrackable(example_vec, TrackedElementVector, example_vec);

__ProxyTrackable(...) takes a name, and the type of shared pointer to return. This can be used to automatically cast custom data objects to the proper type on access, here, we use TrackedElement and don’t change anything. Finally, it takes the variable.

A caller could use this via:

    ...
    TrackedElementVector ev(foo->get_example_vec);
    for (TrackedElementVector::iterator i = foo->begin; 
            i != foo->end(); ++i) {
        ...
    }
    ...

Serialization

Serialization is handled by the tracker_component and tracker_element system automatically. Since the types of the fields are introspectable, serialization systems should be able to export nested data automatically.

The only aspect of serialization that a custom tracker_component class needs to consider is what happens prior to serialization. This is handled by the pre_serialize() method, and is called by any serialization/export class.

This method allows the class to do any updating, averaging, etc before its contents are delivered to a REST endpoing, XML serialization, or other export.

For example, the RRD object uses this method to ensure that the data is synced to the current time:

public:
    virtual void pre_serialize() {
        // Always call the parent in case work needs to be done
        tracker_component::pre_serialize();

        // Call an internal funtion for adding a sample; we add '0' to our
        // current sample and set the time, this fast-forwards the RRD to
        // 'now' and computes history for us in case we didn't see an update
        // in a long time
        add_sample(0, time(0));
    }

Using `tracker_component` objects elsewhere

Sometimes you will want to use a tracker_component in a class that is not, itself, a component: creating data for serialization is a good example.

tracker_component objects can be created and used as normal, with one important exception: The internal object management system expects them to be managed via shared_ptr<>. Failure to pass the proper owning shared_ptr record can result in memory being freed prematurely.

One uncommon condition happens when implementing high-level global constructs which are, themselves, trackable elements: for example, the system_monitor class provides its data by directly serializing itself. This must be accounted for by creating a create_xyz(...) method and assigning the initial shared_ptr<> reference to a GlobalRegistry variable.

A more common condition is when creating temporary records for exporting, for instance, a temporary list of devices. In these cases, the shared_ptr<> reference can be held by the calling function until the task is complete:


int foo::bar() {
    auto c = std::make_shared<some_component>(some_component_id);

    // Do stuff, 'c' is automatically freed when all references expire.
    // This can include passing it to a serializer, embedding it in an object
    // via add_map or similar, etc.

}

It is important to remember that due to the mechanics of shared_ptr<> that two shared pointers created referencing the same object are not identical. There must be one, single, canonical shared_ptr which defines an object, and all other references must be copied from that pointer. Attempting to cast an existing object to a shared_ptr<> or creating a shared_ptr<> from a this pointer will result in multiple usage counts to the same memory and premature freeing of the object.

Dynamic fields

Since tracked objects tend to be used to assign attributes to devices, and there can be tens or hundreds of thousands of devices per session, it can be important to balance the demands on RAM with the demands on CPU. Even a single std::shared_ptr to a field can consume up to 16 bytes of ram per device (two 8-byte pointers).

Kismet offers two, similar, ways to deal with this.

Tracked components

Anatomy of a tracker_element #

Vectors and Maps #

tracker_element Lifecycle #

tracker_component #

Deriving from tracker_component #

First, the Boilerplate: #

TrackerElement Fields #

Registering Fields #

Accessing the Data: Proxy Functions #

Building from other Data Structures #

Accessing from outside the object #

Method one: Provide functions which interface to the complex type #

Method two: Proxy the TrackedElement directly #

Serialization #

Using tracker_component objects elsewhere #

Dynamic fields #