SVS networking

a description of the networking system used in SVS and the core modules supporting it.

introduction
overview of SVS networking system

how SVS clients communicate
outline of the client classes in the svs_core.network package

data packets
formatting data for exchange over the network

notes on Twisted
some information on working with the Twisted network library

client shell commands
setting up simple commands to interact with a client

introduction

All network clients within SVS have their origins in the GenericClient class in the clientuser module of svs_core.network. Two other client classes are provided in clientuser: ScriptableClient and GraphicalClient. ScriptableClient extends GenericClient with the ability to respond to simple script or shell commands, and provides the basis for creating a simple interactive client. GraphicalClient adds functionality for attaching a Graphical User Interface (GUI) and is extended from ScriptableClient. Most applications built with SVS are likely to use ScriptableClient and GraphicalClient rather than extend a new class directly from GenericClient.


fig 1.) inheritance diagram for client classes showing main methods (click to enlarge)

The two modules primarily required for creating clients are svs_core.network.clientuser and svs_core.network.packets. For a detailed understanding of them consult the source code and API notes.

page contents

how SVS clients communicate

The SVS client is built on top of the clients from the 'Perspective Broker' package of the Twisted network library. Twisted uses a deferred networking model. When a method is called that communicates with another client or server, rather than the client waiting for the result before returning, it makes the network call, and then registers a second 'callback' method which will be called whenever the result comes back from the remote client or server (hence it is known as deferred). This way it can carry on with other tasks whilst waiting for the result and without having to create a second thread process. When GenericClient first connects to the server with the connect method, for example, it registers the result_connected method as a callback which will be triggered when the server responds to the client logging on to it:

def connect(self, group, host, port):
    self.clusterGroupName = group
    self.host = host
    self.port = port
    self.logMessage("connecting to server %s:%d ..." % (self.host, self.port))
    factory = pb.PBClientFactory()
    reactor.connectTCP(self.host, self.port, factory)
    d = factory.login(credentials.UsernamePassword (self.profile.name, self.clientPassword), client=self)
    d.addCallback(self.result_connected) # callback registered
    reactor.run()

def result_connected(self, perspective):
    self.logMessage("connected to server.")
    self.avatar = perspective
    self.joinClusterGroup()

example 1.) connection to server and callback.

A naming pattern is used when defining methods to help clarify their function in regard to networking and other apsects of client usage. This is based on the naming pattern used in Twisted. Methods starting with result_, as in result_connected above, are all methods used as callbacks when communicating with the server. Methods starting with remote_ are all local methods within a client which can be called remotely. Other network method prefixes are shown in table 1:

method prefix description
remote_
result_
avatarResult_
groupResult_

table 1.) naming prefixes for network methods.

The remote_ prefix is derived directly from Twisted, the others have been added as part of SVS. The 'avatar' and 'network group' also derive from Twisted's 'Perspective Broker' system. Avatars are server-side proxies for the clients and are used in handling remote procedure calls on clients across the network. Cluster groups are a way of organising clients into smaller, exclusive, groups on the server - it is called the 'clustermanager' in the API for this reason. One server can manage several groups, and in this way multiple SVS networks (called 'clusters' in the API) can operate over the same server connection without conflicting with one another. SVS has been setup to implement these in a default form that should not require any alteration in creating a new application system, only the standard SVS clients need to be altered. The SVS implementations of ther server, avatar and groups are found in the modules clustermanager (server), clientavatar, and clustergroups in the svs_core.network package.

The basic process of network communication is shown in fig. 2 below. A server must be up and running already. A client connects to the server with the connect command. The server checks that the client is on its authentication list. If it is not on the list, the client is refused a connection. If it is, an avatar is created to represent the client on the server-side and the client will received an identifier for this which it wil then use for network communications. Once the client has an avatar it can join a cluster group by calling the joinClusterGroup. If the group exists the client will be added to it, if the group does not exist a new one will be created and the client added to that. As it is the avatar which handles this the callback for it is avatarResult_joinClusterGroup. Once the client has joined a cluster group it can communicate with other clients within the same group.


fig. 2.) network interaction when a client connects to the server).

The haveJoinedClusterGroup method may be the first one that should be overridden by a custom client, as all actions performed prior to this are necessary before any meaningful activity can happen between clients. The haveJoinedClusterGroup method can be extended to retrieve specific data from other clients, or send data to them.

There are various ways in which clients can send data to one another. The use of these is explained in using the basic tools. The methods which you are most likely to override for a custom application, however, are: handleDataPacket, getDataForLabel, notifyListeners, remote_notify, and remote_receiveChatMessage.

page contents

data packets

Information is exchanged between SVS clent in formatted data packets. The packets are a custom object, DataPacket, with specific properties that are used for determining delivery and content handling. Any data type that is compatible with serialisation system of Twisted's Perspective Broker package can be sent in a packet (this includes basic Python data types, arrays, tuples, dictionaries, etc, but see Twisted documentation for custom classes).

The DataPacket class is defined in the module: svs_core.network.packets. Rather than explicitly creating DataPacket instances, clients should use the makeDataPacket utility function in svs_core.network.packets. This ensures that the data packet properties are properly defined for network usage. The makeMessagePacket function also creates a DataPacket object, but one which is specifically formatted for sending messages, such as chat, between clients.

The properties of DataPacket are:

property field description
created
sender
recipient
content
label
timeToLive

The created property is automatically defined when the packet is made and sender is the only obligatory property that must be defined.

The label identifier is used by the handleDataPacket method of the client to determine how it will process the packet on receipt. Normally a client will match the packet label against an internal set of predefined labels, for which corresponding handling methods can be triggered:

def handleDataPacket(self, dataPacket):
    if not dataPacket:return
    dataLabel = dataPacket.label
    if not dataLabel:return
    # time
    if dataLabel == 'time':
        self.handleTime(dataPacket.content)
    # timelog
    if dataLabel == 'time_log':
        self.handleTimeLog(dataPacket.content)
    # client joined network
    elif dataLabel == 'client_joined':
        self.handleClientJoined(dataPacket.content)
    # client departed network
    elif dataLabel == 'client_departed':
        self.handleClientDeparted(dataPacket.content)
    else:
        try:self.logMessage("data packet received: '%s'" % dataPacket.content)
        except:pass

example 2.) example code for handling received data packets.

Although messages are sent as data packets, these are processed by a different method: remote_receiveChatMessage. The default behaviour for this is to print the message out to the client's console, however, some clients may wish to override this method to provide custom handling of messages. In the GameWorldClient from svs_demogame, for example, chat messages from player clients can be used to start and stop a game. The content of the message is treated in a similar way as the label of a data packet, if it matches a predefined string it then triggers a custom handling method:

def remote_receiveChatMessage(self, msgPacket):
    if len(msgPacket.recipient) != 1:return
    if msgPacket.recipient[0] == self.profile.name:
        msg = msgPacket.content.lower()
        if msg == 'start':
            self.startGame()
        elif msg == 'stop':
            self.stopGame()

example 2.) custom handling of chat messages.

There are two other custom packet objects contained in this module: DataRequest and ListenRequest. These are used for retrieveing data from other clients and registering as listeners for notification events on other clients. As with DataPacket there are utility functions, makeDataRequest and makeListenRequest, which should be used when creating these packets.

When a client receives a DataRequest packet, it processes this using the getDataForLabel method, which will trigger specific handling methods in response to the packet label:

def getDataForLabel(self, dataRequest):
    if not dataRequest.label:return GenericClient.getDataForLabel(dataRequest)
    # get time
    if dataRequest.label == 'time':
        return self.getTime(dataRequest.sender)
    # get time log
    if dataRequest.label == 'time_log':
        return self.getTimeLog()
    # pass to parent method
    GenericClient.getDataForLabel(dataRequest)

example 2.) example code for handling received data requests.

ListenRequest packets are handled slightly differently. ListenRequest is used to register clients for regular updates for particular types of data or notification of events. The specific data or event is defined by the label property of the ListenRequest. When another client receives a ListenRequest, it places the requesting client on a listing under that specific label. When data, or events, relating to that label are updated, all clients registered as 'listeners' for that label are notified. For example, clients can register to listen for announcements of when other clients join and leave the network. ListenRequest packets are processed by the handleListenRequest method which is part of the Listenable class defined in: svs_core.utilities.notification. Normally this method should be used 'as is' and need not be redefined by implementing clients. It is built into the standard ClientAvatar class by default (the notification register is held by the server's client proxy rather than on the client itself), so is already available for all clients derived from Generic Client.

To register as a listener, the client simply calls the startListeningTo method (defined in Generic Client), providing the name of the client it wants to listen to and an identifier label. A client can stop listening by calling the stopListeningTo method. Data received as a listener can be processed just as any other data packet using handleDataPacket.

page contents

notes on Twisted

SVS's networking is built on Twisted's 'Perspective Broker' system. It should not be necessary to learn Twisted in order to develop clients for SVS, but if you wish to change low-level aspects of networking behaviour, or create custom classes for data exchnage, you should consult the Twisted documentation.

Two key issues are worth knowing, however. One affects the otherall network architecture of SVS and the other how clients implement networking support.

The architecture utilises a proxying system that enables methods to be called on remote clients as through they were local objects - a form of Remote Method Invocation (RMI). When a client connects to the server, a proxy version of the client is created on the server side using the ClientAvatar class. All communication between clients, or clients and the server goes via these proxies. Whilst most functionality is handled on the remote clients themselves, some is handled within the ClientAvatar. For example, data can be cached on the proxy so that other clients can retrieve it directly from the server rather than the client, thus reducing potential network traffic. The notification system is also implemented on the proxy rather than the main client. This maintains a register of other clients who have registered as 'listeners' to be notified of specific data or events on the remote client. The remote client doesn't need to know who these are, it simply needs to pass on notification data to its proxy who then forwards it to the other clients. This also helps reduce potential network traffic and speeds up the processing of notification events.

For clients, Twisted uses a single-threaded 'deferred' network system, called a 'reactor'. When a client communicates to the network, rather than waiting for the result before continuing, it instead registers a callback function that is triggered when the specific network communication returns. This is described in how SVS clients communicate above. This can have an impact on coding user interfaces for SVS clients, as, conventionally, one thread might be used for the network and another for the interface, but Twisted's reactor can block the interface thread. Twisted have provided some compatibility features for widely used interface toolkits such as Tkinter and wxWidgets, that allow the reactor to integrate with the threading or updating mechanism of these interface systems. Where such compatibility has not been provided, however, the reactor must be used in slightly different fashion from that described in the Twisted documentation.

Interfaces based on toolkits that are integrated with Twisted, register themselves with Twisted, set up the protocols to be used and then call the run method on Twisted's reactor. As the default ClientGUI class of svs_core.gui.clientgui is based on the Tkinter it toolkit, it registers itself as follows:

from Tkinter import *
from twisted.internet import tksupport


class ClientGUI:
    def __init__(self, client):
        self.client = client
        self.viswin = None

    def build(self):
        """
        Creates interface components.
        """
        # make root window
        self.root = Tk()
        self.root.title('SVS client: %s' % self.client.profile.name)
        self.root.geometry("%dx%d%+d%+d" % (400, 200, 0, 0))
        self.root.bind('<Destroy>', self.destroy)
        # register with twisted
        tksupport.install(self.root)
        # add standard components
        self.buildStandardComponents()

And then the client class initiates the run loop, as in the connect method of GenericClient:

def connect(self, group, host, port):
    self.clusterGroupName = group
    self.host = host
    self.port = port
    self.logMessage("connecting to server %s:%d ..." % (self.host, self.port))
    factory = pb.PBClientFactory()
    reactor.connectTCP(self.host, self.port, factory)
    d = factory.login(credentials.UsernamePassword(self.profile.name, self.clientPassword), client=self)
    d.addCallback(self.result_connected)
    reactor.run()

For toolkits that have not been integrated with Twisted this approach may not work as they often have their own runloop which will conflict with that of the reactor. Clients built with these will need to overide the connect method of GenericClient and provide their own update function that can be called by the toolkits own runtime loop:

def connect(self, group, host, port):
    self.clusterGroupName = group
    self.host = host
    self.port = port
    self.logMessage("PDClient connecting to server %s:%d ..." % (self.host, self.port))
    factory = pb.PBClientFactory()
    reactor.connectTCP(self.host, self.port, factory)
    d = factory.login(credentials.UsernamePassword(self.profile.name, self.clientPassword), client=self)
    d.addCallback(self.result_connected)
    reactor.startRunning() # don't use reactor thread
    self.updateNetwork()


def updateNetwork(self):
    """
    Updates network connection.
    """
    reactor.runUntilCurrent()
    reactor.doIteration(1)

In this example the updateNetwork will need to be periodically called by the interface's own runloop in order to process any network activity. An example of such a client is given in the creating clients tutorial on alternative interfaces.

page contents

client shell commands

Clients derived from ScriptableClient can be controlled from an interactive shell. This supports simple single line commands which allow custom functionality to be made easily accessible to users.

Shell commands can be added into a client by adding methods with one of the command prefixes, similar to the network prefixes described above. The are two shell command prefixes:

method prefix description
cmdprivate_
cmdpublic_

table 3.) naming prefixes for shell command methods.

Any methods defined with these prefixes are automatically made available to the interactive shell. This is made possible through the createCommandList method of the 'CommandHandler' object (self.commandHandler of ScriptableClient), which parses the methods of the client class and creates lists of appropriately named methods. For more details of how shell commands are handled see the CommandHandler in the scriptcommands module of the svs_core.commands package. Private commands are only accessible to the local user, whereas pubic commands can be called by other users over the network.

All shell command methods should return a 'CommandResult' object providing a status result ('OK' or 'ERROR'), an optional response message (mostly for errors) and, if required, result data for the operation performed by the method. This is done through calling the makeCommandResult function (imported from the scriptcommands module):

from svs_core.commands.scriptcommands import *
from svs_core.utilities.constants import svs_const


def cmdprivate_disconnect(self, cmd):
    """
    Disconnect from server.
    """
    self.disconnect()
    return makeCommandResult(cmd, message="disconnected", status=svs_const.OK)


def cmdprivate_get(self, cmd):
    """
    Get data from client-side cache.
    """
    try:label = cmd.args[0]
    except IndexError:label = None
    dataPacket = self.getLocalData(label)
    if dataPacket:
        return makeCommandResult(cmd, message="data: '%s'" % dataPacket.content, result=dataPacket, status=svs_const.OK)
    return makeCommandResult(cmd, message="no data found", status=svs_const.ERROR)

example 2.) examples of shell command methods.

The text returned as the 'message' parameter of makeCommandResult will be displayed in the in the output console of the client. If the 'status' is 'OK' it will be displayed via the client's statusMessage method. If the 'status' is 'ERROR' it will be displayed via the client's errorMessage method. These two methods can be overridden to handle output in different ways (such as writing to a log file).

If a shell command is called with '?' as its first argument, the doc string provided with the matching command method will be returned and the command is not executed. This is used to provide info for users.

page contents