--- title: Examples keywords: fastai sidebar: home_sidebar summary: "Examples of the PCT library in use." description: "Examples of the PCT library in use." nb_path: "nbs/00_examples.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
 
{% endraw %} {% raw %}
%load_ext autoreload
%autoreload 2
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
{% endraw %} {% raw %}
#import sys
#sys.path.append("..")
{% endraw %} {% raw %}
import gym 
render=False
runs=1
{% endraw %} {% raw %}
render=True
runs=2000
{% endraw %}

Cartpole

Cartpole is an Open AI gym environment for the inverted pendulum problem. The goal is to keep the pole balanced, by moving the cart left or right.

The environment provides observations (perceptions) for the state of the cart and pole.

0 - Cart Position
1 - Cart Velocity
2 - Pole Angle
3 - Pole Angular Velocity

It takes one value, of 0 or 1, for applying a force to the left or right, respectively.

The PCT solution is a four-level hierarchy for controlling the perceptions at goal values. Only one goal reference is manually set, the highest level which is the pole angle of 0.

This example shows how a perceptual control hierarchy can be implemented with this library.

{% raw %}
import matplotlib.pyplot as plt
import numpy as np
from pct.hierarchy import PCTHierarchy
from pct.putils import FunctionsList
from pct.environments import CartPoleV1
from pct.functions import IndexedParameter
from pct.functions import Integration
from pct.functions import GreaterThan
from pct.functions import PassOn
{% endraw %}

Create a hierarchy of 4 levels each with one node.

{% raw %}
cartpole_hierarchy = PCTHierarchy(levels=4, cols=1, name="cartpoleh", build=False)
namespace=cartpole_hierarchy.namespace
cartpole_hierarchy.get_node(0, 0).name = 'cart_velocity_node'
cartpole_hierarchy.get_node(1, 0).name = 'cart_position_node'
cartpole_hierarchy.get_node(2, 0).name = 'pole_velocity_node'
cartpole_hierarchy.get_node(3, 0).name = 'pole_angle_node'
#FunctionsList.getInstance().report()
#cartpole_hierarchy.summary(build=True)
{% endraw %}

Create the Cartpole gym environment function. This will apply the "action" output from the hierarchy and provide the new observations.

{% raw %}
cartpole = CartPoleV1(name="CartPole-v1", render=render, namespace=namespace)
/home/ruperty/anaconda3/envs/dev37/lib/python3.7/site-packages/gym/core.py:173: DeprecationWarning: WARN: Function `env.seed(seed)` is marked as deprecated and will be removed in the future. Please use `env.reset(seed=seed) instead.
  "Function `env.seed(seed)` is marked as deprecated and will be removed in the future. "
{% endraw %}

Create functions for each of the observation parameters of the Cartpole environment. Insert them into the hierarchy at the desired places.

{% raw %}
cartpole_hierarchy.insert_function(level=0, col=0, collection="perception", function=IndexedParameter(index=1, name="cart_velocity", links=[cartpole], namespace=namespace))
cartpole_hierarchy.insert_function(level=1, col=0, collection="perception", function=IndexedParameter(index=0, name="cart_position", links=[cartpole], namespace=namespace))
cartpole_hierarchy.insert_function(level=2, col=0, collection="perception", function=IndexedParameter(index=3, name="pole_velocity", links=[cartpole], namespace=namespace))
cartpole_hierarchy.insert_function(level=3, col=0, collection="perception", function=IndexedParameter(index=2, name="pole_angle", links=[cartpole], namespace=namespace))
{% endraw %}

Link the references to the outputs of the level up.

{% raw %}
cartpole_hierarchy.insert_function(level=0, col=0, collection="reference", function=PassOn(name="cart_velocity_reference", links=['proportional1'], namespace=namespace))
cartpole_hierarchy.insert_function(level=1, col=0, collection="reference", function=PassOn(name="cart_position_reference", links=['proportional2'], namespace=namespace))
cartpole_hierarchy.insert_function(level=2, col=0, collection="reference", function=PassOn(name="pole_velocity_reference", links=['proportional3'], namespace=namespace))
{% endraw %}

Set the highest level reference.

{% raw %}
top = cartpole_hierarchy.get_function(level=3, col=0, collection="reference")
top.set_name("pole_angle_reference")
top.set_value(0)
{% endraw %}

Link the output of the hierarchy back to the Cartpole environment.

{% raw %}
cartpole_hierarchy.summary(build=True)
cartpoleh PCTHierarchy
**************************
PRE: None
Level 0 Cols 1
cart_velocity_node PCTNode
----------------------------
REF: cart_velocity_reference PassOn | 0 | links  proportional1 
PER: cart_velocity IndexedParameter | index 1 | 0 | links  CartPole-v1 
COM: subtract Subtract | 0 | links  cart_velocity_reference cart_velocity 
OUT: proportional Proportional | gain 1 | 0 | links  subtract 
----------------------------
Level 1 Cols 1
cart_position_node PCTNode
----------------------------
REF: cart_position_reference PassOn | 0 | links  proportional2 
PER: cart_position IndexedParameter | index 0 | 0 | links  CartPole-v1 
COM: subtract1 Subtract | 0 | links  cart_position_reference cart_position 
OUT: proportional1 Proportional | gain 1 | 0 | links  subtract1 
----------------------------
Level 2 Cols 1
pole_velocity_node PCTNode
----------------------------
REF: pole_velocity_reference PassOn | 0 | links  proportional3 
PER: pole_velocity IndexedParameter | index 3 | 0 | links  CartPole-v1 
COM: subtract2 Subtract | 0 | links  pole_velocity_reference pole_velocity 
OUT: proportional2 Proportional | gain 1 | 0 | links  subtract2 
----------------------------
Level 3 Cols 1
pole_angle_node PCTNode
----------------------------
REF: pole_angle_reference Constant | 0 
PER: pole_angle IndexedParameter | index 2 | 0 | links  CartPole-v1 
COM: subtract3 Subtract | 0 | links  pole_angle_reference pole_angle 
OUT: proportional3 Proportional | gain 1 | 0 | links  subtract3 
----------------------------
POST: None
**************************
{% endraw %} {% raw %}
cartpole_hierarchy.insert_function(level=0, col=0, collection="output", function=Integration(gain=-0.05, slow=4, name="force", links='subtract', namespace=namespace))
{% endraw %}

Set the names and gains of the output functions. This also shows another way of getting a function, by name.

{% raw %}
FunctionsList.getInstance().get_function(namespace=namespace, name="proportional3").set_name("pole_angle_output")
FunctionsList.getInstance().get_function(namespace=namespace, name="pole_angle_output").set_property('gain', 3.5)

FunctionsList.getInstance().get_function(namespace=namespace, name="proportional2").set_name("pole_velocity_output")
FunctionsList.getInstance().get_function(namespace=namespace, name="pole_velocity_output").set_property('gain', 0.5)

FunctionsList.getInstance().get_function(namespace=namespace, name="proportional1").set_name("cart_position_output")
FunctionsList.getInstance().get_function(namespace=namespace, name="cart_position_output").set_property('gain', 2)
{% endraw %}

Add a post function to convert the output to 1 or 0 as required by the Cartpole environment.

{% raw %}
greaterthan = GreaterThan(threshold=0, upper=1, lower=0, links='force', namespace=namespace)
cartpole_hierarchy.add_postprocessor(greaterthan)
{% endraw %}

Add the cartpole function as one that is executed before the actual hierarchy.

{% raw %}
cartpole_hierarchy.add_preprocessor(cartpole)
{% endraw %}

Set the output of the hierachy as the action input to the Cartpole environment.

{% raw %}
cartpole.add_link(greaterthan)
{% endraw %}

Sit back and observe the brilliance of your efforts.

{% raw %}
cartpole_hierarchy.set_order("Down")
{% endraw %} {% raw %}
cartpole_hierarchy.summary()
cartpoleh PCTHierarchy
**************************
PRE: CartPole-v1 CartPoleV1 | 0 | links  greaterthan 
Level 3 Cols 1
pole_angle_node PCTNode
----------------------------
REF: pole_angle_reference Constant | 0 
PER: pole_angle IndexedParameter | index 2 | 0 | links  CartPole-v1 
COM: subtract3 Subtract | 0 | links  pole_angle_reference pole_angle 
OUT: pole_angle_output Proportional | gain 3.5 | 0 | links  subtract3 
----------------------------
Level 2 Cols 1
pole_velocity_node PCTNode
----------------------------
REF: pole_velocity_reference PassOn | 0 | links  pole_angle_output 
PER: pole_velocity IndexedParameter | index 3 | 0 | links  CartPole-v1 
COM: subtract2 Subtract | 0 | links  pole_velocity_reference pole_velocity 
OUT: pole_velocity_output Proportional | gain 0.5 | 0 | links  subtract2 
----------------------------
Level 1 Cols 1
cart_position_node PCTNode
----------------------------
REF: cart_position_reference PassOn | 0 | links  pole_velocity_output 
PER: cart_position IndexedParameter | index 0 | 0 | links  CartPole-v1 
COM: subtract1 Subtract | 0 | links  cart_position_reference cart_position 
OUT: cart_position_output Proportional | gain 2 | 0 | links  subtract1 
----------------------------
Level 0 Cols 1
cart_velocity_node PCTNode
----------------------------
REF: cart_velocity_reference PassOn | 0 | links  cart_position_output 
PER: cart_velocity IndexedParameter | index 1 | 0 | links  CartPole-v1 
COM: subtract Subtract | 0 | links  cart_velocity_reference cart_velocity 
OUT: force Integration | gain -0.05 slow 4  | 0 | links  subtract 
----------------------------
POST: greaterthan GreaterThan | threshold 0 upper 1 lower 0  | 0 | links  force 
**************************
{% endraw %} {% raw %}
cartpole_hierarchy.draw(font_size=10, figsize=(8,12), move={'CartPole-v1': [-0.075, 0]}, node_size=1000, node_color='red')
{% endraw %} {% raw %}
cartpole_hierarchy.save("cartpole.json")
/home/ruperty/anaconda3/envs/dev37/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)
{% endraw %} {% raw %}
import networkx as nx
gr = cartpole_hierarchy.graph()
print(nx.info(gr))
print(gr.nodes())
Name: 
Type: DiGraph
Number of nodes: 18
Number of edges: 21
Average in degree:   1.1667
Average out degree:   1.1667
['greaterthan', 'force', 'CartPole-v1', 'cart_velocity_reference', 'cart_position_output', 'subtract', 'cart_velocity', 'cart_position_reference', 'pole_velocity_output', 'subtract1', 'cart_position', 'pole_velocity_reference', 'pole_angle_output', 'subtract2', 'pole_velocity', 'pole_angle_reference', 'subtract3', 'pole_angle']
{% endraw %}

Run the hierarchy for 500 steps.

{% raw %}
cartpole_hierarchy.run(1,verbose=False)
1
{% endraw %} {% raw %}
cartpole_hierarchy.run(runs,verbose=False)
/home/ruperty/anaconda3/envs/dev37/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)
0
{% endraw %} {% raw %}
cartpole.close()
{% endraw %}