Crowd Programming Engine (CPE)

"A Distributed Agent-Based system with PyClips, RandomForests and B-D-I connected over a shared Tuple-Space powered by google spreadsheets or something better when we find it." - Charles Kosta, Sc.D.

R and Python are joining forces

The most ambitious crossover event of the year—for programmers

Last month, McKinney announced the founding of Ursa Labs, an innovation group intended to improve data-science tools. McKinney will partner with RStudio—Wickham’s employer, which maintains the most popular user interface for R—on the project. The main goals of Ursa Labs are to make it easier for data scientists working in different programming languages to collaborate, and avoid redundant work by developers across languages. In addition to improving R and Python, the group hopes its work will also improve the user experience in other open-source programming languages like Java and Julia.


Requirement for using PyClips and NWS to create a crowd of intelligent Actors

1. Registering with Remote-Assert-Agent (on same node as NWSServer)

2. Each PyClips Instance is a Named Actor and can request Roles

3. PyClips blocks (Fetch) when there are no rules to fire; does FetchTry polling between rule executions.

4. Python Functions available in the Right Hand Side of Rules


4. Python Functions available in the Right Hand Side of Rules

PyCLIPS gives the ability to users to call Python code from within the CLIPS subsystem. Virtually every function defined in Python can be called from CLIPS code using the special CLIPS function python-call. However, since CLIPS has different basic types than Python, in most cases it would be useful for modules that implement function to be called in the CLIPS engine to import the PyCLIPS module themselves, in order to be aware of the structures that CLIPS uses.

Functions have to be registered in PyCLIPS in order to be available to the underlying engine, and the registration process can dynamically occur at any moment.

A simple example follows:

>>> import clips
>>> def py_square(x):
        return x * x
>>> clips.RegisterPythonFunction(py_square)
>>> print clips.Eval("(python-call py_square 7)")
>>> print clips.Eval("(python-call py_square 0.7)")

Getting Started with NetWorkSpaces

We are ultimately trying to use NetWorkSpaces (NWS) as a way for Python, R, and Java agents running on different machines to
communicate and coordinate with one another.

To get started with Python NWS, we'll first have to install the NWS server, and the Python NWS client.

pip install nwsserver (

pip install Twisted (

Next, an NWS server must be started. This can be done from a shell
using the nws command, as follows:

% twistd -ny .local/nws.tac

From another window, start an interactive Python session, and import
the NetWorkSpace class:

% python
>>> from nws.client import NetWorkSpace

Next, create a work space called 'cpe':

>>> ws = NetWorkSpace('cpe')

This is using the NWS server on the local machine. Additional arguments
can be used to specify the hostname and port used by the NWS server if

Once we have a work space, we can write data into a variable using the
store method:

>>>'joe', 17)

The variable 'joe' now has a value of 17 in our work space.

To read that variable we use the find method:

>>> age = ws.find('joe')

which sets 'age' to 17.

Note that the find method will block until the variable 'joe' has a
value. That is important when we're trying to read that variable from a
different machine. If it didn't block, you might have to repeatedly try
to read the variable until it succeeded.

There are times when you don't want to block, but just want to see if
some variable has a value. That is done with the findTry method.


From another window, start an interactive Python session, and import
the NetWorkSpace class:

% python
>>> from nws.client import NetWorkSpace

Next, connect to a work space called 'cpe':

>>> ws = NetWorkSpace('cpe')

This won't create the 'cpe' work space, since it already exists. Now you
can execute a operation such as:

>>> x = ws.fetch('joe')
>>> x

you should see that you got the value 17 from the previous function;
only this time you fetched it permanently. Now try to execute the the fetch again.

>>> x = ws.fetch('joe')

Your program will now be blocked! in this session, watch it block for a minute, and then execute a 'store' in
the other session:

>>>'joe', 18)

You should see the previous session return from the fetch command and x should now be 18.

That's a basic sample of this technology, which of course can become much more complex. Keep in mind
that these programs could have been on different machines; or hosted on Amazon.

More importantly, an R program and a python program
and even a Java program could be working cooperatively.

NetWorkSpaces for Python

NetWorkSpaces: a framework to coordinate distributed programs

NetWorkSpaces (NWS) is one way to write parallel programs. It allows you to take advantage of a multi-core machine; multiple virtual machines, as well as cloud-based clusters, using languages such as Python, R, Java, and Matlab. With NetWorkSpaces for Python, you can execute Python functions and programs in parallel using methods very much like the standard Python map function. In some cases, you may be able to parallelize your program in minutes, rather than months.

For example, here's a simple Python NWS script:

          from math import sqrt
          from nws.sleigh import Sleigh
          s = Sleigh()
          for x in s.imap(sqrt, xrange(10)):
              print x

It looks pretty simple, but you'll need to be familiar with the imap function in the standard itertools module.

PyClips (Available via PIP)

This module embeds a fully functional CLIPS engine into Python, and gives the developer a more Python-compliant interface to CLIPS without cutting down on functionalities. In fact CLIPS is compiled into the module in its entirety, and most API functions are bound to Python methods. However the direct bindings to the CLIPS library (implemented as the _clips submodule) are not described here: each function is described by an appropriate documentation string, and accessible by means of the help() function or through the pydoc tool. Each direct binding maps to an API provided function. For a detailed reference for these functions see Clips Reference Guide Vol. II: Advanced Programming Guide, available for download at the CLIPS website.

PyCLIPS is also capable of generating CLIPS text and binary files: this allows the user to interact with sessions of the CLIPS system itself. An important thing to know, is that PyCLIPS implements CLIPS as a separated engine: in the CLIPS module implementation, CLIPS ``lives'' in its own memory space, allocates its own objects. The module only provides a way to send information and commands to this engine and to retrieve results from it.