Run a session of the SCML world (2020)

The SCML world (Supply Chain Management League) runs on top of NegMAS.

In this tutorial, you will test a run of this world.

Firstly, let’s import everything from the SCML app

from scml.scml2020 import *

There are several ways to create an SCML world. One of the simplest is to use the generate method of the SCML2020World class. This will create a dict that can be passed to the SCML2020World constructor as keyword arguments.

Here we explicitly set construct_graphs to True which slows the simulation yet allows us to see graph representations of what is happening in the world.

agent_types = [DecentralizingAgent, BuyCheapSellExpensiveAgent,
               IndDecentralizingAgent, MovingRangeAgent]
world = SCML2020World(
    **SCML2020World.generate(
        agent_types=agent_types,
        n_steps=50
    ),
    construct_graphs=True,
)

Let’s draw a graph to see what is in this world

_, _ = world.draw()
../_images/01.run_scml2020_6_0.png

If you want to just test your installation (and do not care whether you get an accurate indicator of agent performance), you can set the number of steps to a small value (e.g. n_steps=10).

Now you can run this world simulation by just calling run.

world.run_with_progress() # may take few minutes
100%|██████████| 50/50 [05:11<00:00,  6.24s/it]

Let’s see what happened in this run. Firstly, how many negotiations were conducted over time

plt.plot(world.stats['n_negotiations'])
plt.xlabel('Simulation Step')
plt.ylabel('N. Negotiations')
plt.show()
../_images/01.run_scml2020_10_0.png

It is clear that many negotiations happened at the beginning of the simulation with smaller number later. That is expected as the agents at the first and last production layer receive more exogenous contracts in the beginning.

Several other market statistics are available:

pprint(list(_ for _ in world.stats.keys() if "@" not in _  ))
['n_registered_negotiations_before',
 'n_contracts_dropped',
 'n_contracts_nullified_now',
 'n_bankrupt',
 'trading_price_0',
 'sold_quantity_0',
 'unit_price_0',
 'trading_price_1',
 'sold_quantity_1',
 'unit_price_1',
 'trading_price_2',
 'sold_quantity_2',
 'unit_price_2',
 'trading_price_3',
 'sold_quantity_3',
 'unit_price_3',
 'trading_price_4',
 'sold_quantity_4',
 'unit_price_4',
 'productivity',
 'market_size',
 'production_failures',
 'bankruptcy',
 'n_contracts_executed',
 'n_contracts_erred',
 'n_contracts_nullified',
 'n_contracts_cancelled',
 'n_breaches',
 'breach_level',
 'n_contracts_signed',
 'n_contracts_concluded',
 'n_negotiations',
 'n_negotiation_rounds_successful',
 'n_negotiation_rounds_failed',
 'n_negotiation_successful',
 'n_negotiation_failed',
 'n_registered_negotiations_after',
 'activity_level',
 'step_time',
 'total_time']

Let’s start by seeing how long did each step take (note that stats access the stats as a Dict[str, List] but stats_df access the same data as a pandas dataframe.

plt.bar(range(world.n_steps), world.stats_df['step_time'])
plt.xlabel('Simulation Step')
plt.ylabel('Time (s)')
plt.show()
../_images/01.run_scml2020_14_0.png

There are statistics specific for each agent that all have “_{agent_name}”. Lets check what is available for the winner agent:

winner = world.winners[0]
pprint(list(_ for _ in world.stats.keys() if winner.name in _ ))
['spot_market_quantity_00Dec@0',
 'spot_market_loss_00Dec@0',
 'balance_00Dec@0',
 'inventory_00Dec@0_input',
 'inventory_00Dec@0_output',
 'productivity_00Dec@0',
 'assets_00Dec@0',
 'bankrupt_00Dec@0',
 'score_00Dec@0']

The convension is that agent names has the form {ind}{Type}{process} where ind is a unique index, Type is a shortened version of the agent’s type name, and process is the process the agnet can run. Note that the agent’s input product has the same number as its process and its output product has the next number (i.e. an agent that runs process 1, has input product 1 and output product 2).

We can see that 8 pieces of information are available (for each time-step of the simulation):

  • bankrupt If true, the agent is bankrupt.

  • balance The money the agent has in its wallet (account).

  • inventory (input) The number of units of the agent’s input product available in its inventory (by the end of the simulation step).

  • inventory (output) The number of units of the agent’s output product available in its inventory (by the end of the simulation step).

  • assets The value of the agent’s assets (input and output products in inventory) evaluated at the trading price

  • spot market quantity The quantity bought by this agent from the spot market (of its output product on this step). This can only happen as a result of a product-breach.

  • spot market loss The spot market price for the agent. This value will go up the more the agent buys from the spot market and will be used to calculate the price for this agent at future steps. This way agents that depend on the spot market instead of negotiation get punished.

  • productivity The fraction of the agent’s production lines that were active at a given time-step.

  • score The score of the agent according to the evaluation rule of ANAC SCML 2020

Let’s see how did our agent do

#show the first and last value of each of the agent statistics
pprint({k:(v[0], v[-1]) for k, v in world.stats.items() if winner.name in k })
{'assets_00Dec@0': (162.0, 483.28091511614116),
 'balance_00Dec@0': (18008, 22287),
 'bankrupt_00Dec@0': (False, False),
 'inventory_00Dec@0_input': (0, 0),
 'inventory_00Dec@0_output': (9, 24),
 'productivity_00Dec@0': (0.9, 0.0),
 'score_00Dec@0': (-0.0004972925185103326, 0.24481381686142506),
 'spot_market_loss_00Dec@0': (0.3, 2.352),
 'spot_market_quantity_00Dec@0': (0, 0)}
stats = pd.DataFrame(data=world.stats)
fig, axs = plt.subplots(2, 3)
for ax, key in zip(axs.flatten().tolist(), ["score", "balance", "assets", "productivity",
                         "spot_market_quantity", "spot_market_loss"]):
    ax.plot(stats[f"{key}_{winner}"])
    ax.set(ylabel=key)
fig.show()
../_images/01.run_scml2020_19_0.png

We can for example check the activity level of this world (defined as the total amount of money transferred due to trade)

plt.plot(world.stats['activity_level'])
plt.xlabel('Simulation Step')
plt.ylabel('Activitiy Level ($)\nTotal Money Transferred')
plt.show()
../_images/01.run_scml2020_21_0.png

We can see a picture of contracting in this world as follows:

plt.plot(world.stats['n_contracts_concluded'], label='Concluded Contracts')
plt.plot(world.stats['n_contracts_cancelled'], label='Cancelled Contracts')
plt.plot(world.stats['n_contracts_signed'], label='Signed Contracts')
plt.plot(world.stats['n_contracts_executed'], label='Executed Contracts')
plt.legend()
plt.xlabel('Simulation Step')
plt.ylabel('N. Contracts')
plt.show()
../_images/01.run_scml2020_23_0.png

We can also check the breaches that happened

plt.plot(world.stats['breach_level'])
plt.xlabel('Simulation Step')
plt.ylabel('Total Breach Level')
plt.show()
../_images/01.run_scml2020_25_0.png

Notice that there can be multiple winners

winner_profits = [100 * world.scores()[_.id] for _ in world.winners]
winner_types = [_.short_type_name for _ in world.winners]
print(f"{world.winners} of type {winner_types} won at {winner_profits}%")
[00Dec@0] of type ['decentralizing'] won at [24.481381686142505]%

Let’s check how did the first winner’s inventory changes over time:

# find the keys in stats for the input and output inventory
in_key = [_ for _ in world.stats.keys() if _.startswith(f'inventory_{winner}_input')][0]
out_key = [_ for _ in world.stats.keys() if _.startswith(f'inventory_{winner}_output')][0]

# find input and output product indices
input_product, output_product = winner.awi.my_input_product, winner.awi.my_output_product
# draw
fig, (quantity, value) = plt.subplots(1, 2)
quantity.plot(world.stats[in_key], label=f"Input Product")
quantity.plot(world.stats[out_key], label=f"Output Product")
quantity.set(xlabel='Simulation Step', ylabel='Winner\'s Total Storage (item)')
quantity.legend()
value.plot(np.array(world.stats[in_key]) * np.array(world.stats[f"trading_price_{input_product}"])
              , label=f"Input Product")
value.plot(np.array(world.stats[out_key]) * np.array(world.stats[f"trading_price_{output_product}"])
              , label=f"Output Product")
value.set(xlabel='Simulation Step', ylabel='Winner\'s Inventory Value ($)')
value.legend()
fig.show()
../_images/01.run_scml2020_29_0.png

We can actually check what happens to ALL competitors:

from scml.scml2020.world import is_system_agent
fig, (profit, score) = plt.subplots(1, 2)
snames = sorted(world.non_system_agent_names)
for name in snames:
    profit.plot(100.0 * (np.asarray(world.stats[f'balance_{name}'])/world.stats[f'balance_{name}'][0] - 1.0), label=name)
    score.plot(100 * np.asarray(world.stats[f'score_{name}']), label=name)
profit.set(xlabel='Simulation Step', ylabel='Player Profit Ignoring Inventory (%)')
profit.legend(loc='lower left')
score.set(xlabel='Simulation Step', ylabel='Player Score (%)')
fig.show()
../_images/01.run_scml2020_31_0.png
from scml.scml2020.world import is_system_agent
fig, (profit, score) = plt.subplots(1, 2)
snames = sorted(world.non_system_agent_names)
for name in snames:
    profit.plot((np.asarray(world.stats[f'balance_{name}'])), label=name)
    score.plot(np.asarray(world.stats[f'score_{name}'])*(world.stats[f'balance_{name}'][0]), label=name)
profit.set(xlabel='Simulation Step', ylabel='Player Balance ($)')
profit.legend(loc='lower left')
score.set(xlabel='Simulation Step', ylabel='Player Score Unnormalized ($)')
fig.show()
../_images/01.run_scml2020_32_0.png

or just look at the end of the game

fig, (score, profit) = plt.subplots(1, 2)
final_scores = [100 * world.stats[f"score_{_}"][-1]
                for _ in world.non_system_agent_names]
final_profits = [100 * world.stats[f"balance_{_}"][-1] / world.stats[f"balance_{_}"][0] - 100
                 for _ in world.non_system_agent_names]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Score (%)")
profit.set(ylabel="Final Profit (%)")

fig.show()
../_images/01.run_scml2020_34_0.png
fig, (score, profit) = plt.subplots(1, 2)
final_scores = [world.stats[f"score_{_}"][-1] * (world.stats[f"balance_{_}"][0])
                for _ in world.non_system_agent_names]
final_profits = [world.stats[f"balance_{_}"][-1]
                 for _ in world.non_system_agent_names]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Unnormalized Score ($)")
profit.set(ylabel="Final Balance  ($)")

fig.show()
../_images/01.run_scml2020_35_0.png

You can inspect what happened in the simulation by plotting different output statistics. For example, we can see how did the trading price of different products change over the simulation time.

fig, axs = plt.subplots(2, 2)
for ax, key in zip(axs.flatten().tolist(), ["trading_price", "sold_quantity", "unit_price"]):
    for p in range(world.n_products):
        ax.plot(world.stats[f"{key}_{p}"], marker="x", label=f"Product {p}")
        ax.set_ylabel(key.replace("_", " ").title())
        ax.legend().set_visible(False)
axs[-1, 0].legend(bbox_to_anchor=(1, -.5), ncol=3)
fig.show()
../_images/01.run_scml2020_37_0.png
fig, axs = plt.subplots(1, 2)
for ax, key in zip(axs.flatten().tolist(), ["spot_market_quantity", "spot_market_loss"]):
    for a in world.non_system_agent_names:
        ax.plot(world.stats[f"{key}_{a}"], marker="x", label=f"{a}")
        ax.set_ylabel(key.replace("_", " ").title())
        ax.legend().set_visible(False)
axs[0].legend(bbox_to_anchor=(1, -.2), ncol=4)
fig.show()
../_images/01.run_scml2020_38_0.png

You can dig futher to understand what happened during this siumulation. For example, let’s see some of the contracts that were signed:

# create a view with only signed contracts
contracts = world.contracts_df
signed = contracts.loc[contracts.signed_at>=0, :]
fields = ["seller_name", "buyer_name", "delivery_time", "quantity", "unit_price",
          "signed_at", "executed", "breached", "nullified", "erred"]
signed[fields].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name buyer_name delivery_time quantity unit_price signed_at executed breached nullified erred
703 07Buy@2 09Dec@3 9 66 31 2 False True False False
722 06Buy@2 09Dec@3 10 65 31 2 False True False False
733 06Buy@2 10Dec@3 9 62 31 2 False True False False
700 06Buy@2 10Dec@3 10 61 31 2 False True False False
698 07Buy@2 10Dec@3 8 53 31 2 False True False False
916 08Ind@2 09Dec@3 15 46 31 4 False True False False
709 06Buy@2 09Dec@3 7 45 31 2 False True False False
905 06Buy@2 10Dec@3 22 45 31 4 False False False False
2254 05Dec@1 08Ind@2 36 38 35 26 True False False False
913 06Buy@2 09Dec@3 24 31 31 4 False False False False

Let’s check some of the contracts that were fully executed

signed.loc[signed.executed, fields].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name buyer_name delivery_time quantity unit_price signed_at executed breached nullified erred
2254 05Dec@1 08Ind@2 36 38 35 26 True False False False
1247 05Dec@1 08Ind@2 14 28 35 8 True False False False
2450 08Ind@2 09Dec@3 39 23 45 34 True False False False
828 08Ind@2 09Dec@3 27 22 48 3 True False False False
2399 08Ind@2 10Dec@3 37 20 45 31 True False False False
1725 00Dec@0 03Ind@1 23 20 26 15 True False False False
2222 04Dec@1 08Ind@2 35 19 37 25 True False False False
2225 04Dec@1 07Buy@2 30 19 35 25 True False False False
2449 03Ind@1 07Buy@2 39 19 35 34 True False False False
2288 00Dec@0 05Dec@1 37 19 18 27 True False False False
signed.loc[signed.breached, fields[:-4] + ["breaches"]].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name buyer_name delivery_time quantity unit_price signed_at breaches
703 07Buy@2 09Dec@3 9 66 31 2 07Buy@2:product(1.0)
722 06Buy@2 09Dec@3 10 65 31 2 06Buy@2:product(1.0)
733 06Buy@2 10Dec@3 9 62 31 2 06Buy@2:product(1.0)
700 06Buy@2 10Dec@3 10 61 31 2 06Buy@2:product(1.0)
698 07Buy@2 10Dec@3 8 53 31 2 07Buy@2:product(1.0)
916 08Ind@2 09Dec@3 15 46 31 4 08Ind@2:product(0.5869565217391305)
709 06Buy@2 09Dec@3 7 45 31 2 06Buy@2:product(1.0)
2070 01Ind@0 05Dec@1 30 27 18 22 01Ind@0:product(1.0)
737 01Ind@0 04Dec@1 13 24 18 2 01Ind@0:product(0.5833333333333334)
983 00Dec@0 05Dec@1 14 24 18 5 00Dec@0:product(0.5833333333333334)

We can now see how does the singning day affect delivery day, product and quantity

fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "quantity", "product_index"]):
    ax[i].scatter(signed.signed_at, signed[x])
    ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Signing Day")
fig.show()
../_images/01.run_scml2020_46_0.png
fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "unit_price", "product_index"]):
    ax[i].scatter(signed.quantity, signed[x])
    ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Quantity")
fig.show()
../_images/01.run_scml2020_47_0.png

Did any agents go bankrupt and when?

bankruptcy = {a: np.nonzero(stats[f"bankrupt_{a}"].values)[0]
        for a in world.non_system_agent_names}
pprint({k: "No" if len(v)<1 else f"at: {v[0]}" for k, v in bankruptcy.items()})
{'00Dec@0': 'No',
 '01Ind@0': 'No',
 '02Mov@0': 'No',
 '03Ind@1': 'No',
 '04Dec@1': 'No',
 '05Dec@1': 'No',
 '06Buy@2': 'at: 17',
 '07Buy@2': 'at: 43',
 '08Ind@2': 'No',
 '09Dec@3': 'No',
 '10Dec@3': 'No',
 '11Mov@3': 'No'}

You can see what happened during this simulation by drawing graphs at different steps. The meaning of different edge colors can be drawn as follows:

from negmas import show_edge_colors
show_edge_colors()
../_images/01.run_scml2020_51_0.png

You can see what happened in this world in a series of graphs using the draw method

world.draw(steps=(0, world.n_steps), together=False, ncols=2, figsize=(20, 20))
plt.show()
../_images/01.run_scml2020_53_0.png

You can also run a simple animation to see what happens at every step (you need to download the jupyter notebook and execute it to see the animation) :

# prepare animation
from time import perf_counter
from negmas.helpers import humanize_time
import gif
from pathlib import Path

# define animation parameters
draw_every = 1 # draw every 5 steps (drawing is REALLY slow)
single_graph = False # show a graph for every event type
# what = ["contracts-signed", "contracts-breached", "contracts-executed",]
fps = 5

# define the animation function. Simply draw the world
@gif.frame
def plot_frame(s):
    world.draw(#what=what,
               steps=(s-draw_every, s),
               together=single_graph, ncols=3, figsize=(20, 20))

# create frames
frames = []
for s in tqdm(range(world.n_steps)):
    if s % draw_every != 0:
        continue
    frames.append(plot_frame(s))

path = Path.cwd() / "run.gif"
path.unlink(missing_ok=True)
gif.save(frames, str(path), duration=1000 // fps)
HBox(children=(FloatProgress(value=0.0, max=50.0), HTML(value='')))

then show the animation

from IPython.display import HTML
HTML('<img src="run.gif">')

Note that this graph shows only what happened in the last draw_every steps.

Let’s see some statistics about the simulation.

Running a tournament

Now that you can run simple world simulations, let’s try to run a complete tournament and see its results. Let’s start by running a standard tournament (in which each agent is represented by a single factory). Running a collusion tournament will be exactly the same with the only difference that anac2020_std will be replaced with anac2020_collusion.

Note that in the real competition we use thousands of configurations and longer simulation steps (e.g. 50 \(\le\) n_steps \(\le\) 500).

from scml.scml2020.utils import anac2020_std
tournament_types = agent_types + [RandomAgent]
# may take a long time
results = anac2020_std(
    competitors=tournament_types,
    n_configs=12, # number of different configurations to generate
    n_runs_per_world=1, # number of times to repeat every simulation (with agent assignment)
    n_steps = 10, # number of days (simulation steps) per simulation
    print_exceptions=True,
)

Who was the winner?

results.winners
['decentralizing_agent']

How many simulations were actually run?

len(results.scores.run_id.unique())
60

The total number of simulations \(n_{s}\) will be \(n_t \times n_c \times n_r\) where \(n_t\) is the number of competitor agent types, \(n_c\) is the number of configurations, and \(n_r\) is the number of runs per configuration

We can also see the scores that every agent type got

results.score_stats
agent_type count mean std min 25% 50% 75% max
0 buy_cheap_sell_expensive_agent 60.0 -0.460880 0.399413 -2.202297 -0.512401 -0.321267 -0.249171 -0.044618
1 decentralizing_agent 60.0 0.180795 0.205134 -0.123303 0.034685 0.142906 0.323354 0.657287
2 ind_decentralizing_agent 60.0 0.161280 0.199404 -0.122533 0.023384 0.120836 0.239894 0.832956
3 moving_range_agent 60.0 -0.052376 0.081866 -0.211015 -0.108780 -0.052406 0.000000 0.146121
4 random_agent 60.0 -0.538307 0.375828 -1.905989 -0.540328 -0.391387 -0.352393 -0.162312

You can also do statistical significance testing using ttest or kstest (with multi-comparison correction)

results.kstest
a b t p n_a n_b n_effective
0 ind_decentralizing_agent decentralizing_agent 0.133333 6.647506e-01 60 60 60
1 ind_decentralizing_agent random_agent 1.000000 2.070074e-35 60 60 60
2 ind_decentralizing_agent buy_cheap_sell_expensive_agent 0.983333 2.484089e-33 60 60 60
3 ind_decentralizing_agent moving_range_agent 0.633333 1.306907e-11 60 60 60
4 decentralizing_agent random_agent 1.000000 2.070074e-35 60 60 60
5 decentralizing_agent buy_cheap_sell_expensive_agent 0.983333 2.484089e-33 60 60 60
6 decentralizing_agent moving_range_agent 0.650000 2.904237e-12 60 60 60
7 random_agent buy_cheap_sell_expensive_agent 0.366667 5.569711e-04 60 60 60
8 random_agent moving_range_agent 0.983333 2.484089e-33 60 60 60
9 buy_cheap_sell_expensive_agent moving_range_agent 0.850000 2.164592e-22 60 60 60

see the total score

results.total_scores
agent_type score
0 decentralizing_agent 0.142906
1 ind_decentralizing_agent 0.120836
2 moving_range_agent -0.052406
3 buy_cheap_sell_expensive_agent -0.321267
4 random_agent -0.391387

or the aggregated statistics of the world. For example, let’s draw the activity level for different simulations.

plt.errorbar(range(len(results.agg_stats)),
             results.agg_stats.activity_level_mean,
             np.sqrt(results.agg_stats.activity_level_var)
             )
plt.xlabel("Simulation Number")
plt.ylabel("Activity Level")
plt.show()
../_images/01.run_scml2020_73_0.png

We can even get the scores of every agent belonging to every agent type at every simulation

results.scores.loc[:, ["agent_name", "agent_type", "score"]].head()
agent_name agent_type score
0 00Ind@0 ind_decentralizing_agent 0.626623
1 01Dec@0 decentralizing_agent 0.322342
2 02Ran@1 random_agent -1.372634
3 04Buy@2 buy_cheap_sell_expensive_agent -0.334364
4 05Mov@2 moving_range_agent -0.134590

or inspect any statistic we like

ax = sns.violinplot(data=results.stats, x="step", y="activity_level")
ax.set(ylabel="Activity Level / Business Size ($)", xlabel="Step Number", yscale="log")

plt.gcf().show()
../_images/01.run_scml2020_77_0.png

Let’s see how did the location at the production graph affect the score of each type.

results.scores["level"] = results.scores.agent_name.str.split("@", expand=True).loc[:, 1]
sns.lineplot(data=results.scores[["agent_type", "level", "score"]],
             x="level", y="score", hue="agent_type")
plt.plot([0.0] * len(results.scores["level"].unique()), "b--")
plt.show()
../_images/01.run_scml2020_79_0.png

Now that you can run simulations and complete tournament, let’s see how are we going to develop a new agent for the SCML2020 league \(\rightarrow\)

Download Notebook.