Run a session of the SCML world (2020)¶
The SCML world (Supply Chain Management League) runs on top of
NegMAS
.
In this tutorial, you will test a run of this world.
Firstly, let’s import everything from the SCML app
from scml.scml2020 import *
There are several ways to create an SCML world. One of the simplest is
to use the generate
method of the SCML2020World
class. This will
create a dict that can be passed to the SCML2020World
constructor as
keyword arguments.
Here we explicitly set construct_graphs
to True which slows the
simulation yet allows us to see graph representations of what is
happening in the world.
agent_types = [DecentralizingAgent, BuyCheapSellExpensiveAgent,
IndDecentralizingAgent, MovingRangeAgent]
world = SCML2020World(
**SCML2020World.generate(
agent_types=agent_types,
n_steps=50
),
construct_graphs=True,
)
Let’s draw a graph to see what is in this world
_, _ = world.draw()

If you want to just test your installation (and do not care whether you get an accurate indicator of agent performance), you can set the number of steps to a small value (e.g. n_steps=10).
Now you can run this world simulation by just calling run
.
world.run_with_progress() # may take few minutes
100%|██████████| 50/50 [05:11<00:00, 6.24s/it]
Let’s see what happened in this run. Firstly, how many negotiations were conducted over time
plt.plot(world.stats['n_negotiations'])
plt.xlabel('Simulation Step')
plt.ylabel('N. Negotiations')
plt.show()

It is clear that many negotiations happened at the beginning of the simulation with smaller number later. That is expected as the agents at the first and last production layer receive more exogenous contracts in the beginning.
Several other market statistics are available:
pprint(list(_ for _ in world.stats.keys() if "@" not in _ ))
['n_registered_negotiations_before',
'n_contracts_dropped',
'n_contracts_nullified_now',
'n_bankrupt',
'trading_price_0',
'sold_quantity_0',
'unit_price_0',
'trading_price_1',
'sold_quantity_1',
'unit_price_1',
'trading_price_2',
'sold_quantity_2',
'unit_price_2',
'trading_price_3',
'sold_quantity_3',
'unit_price_3',
'trading_price_4',
'sold_quantity_4',
'unit_price_4',
'productivity',
'market_size',
'production_failures',
'bankruptcy',
'n_contracts_executed',
'n_contracts_erred',
'n_contracts_nullified',
'n_contracts_cancelled',
'n_breaches',
'breach_level',
'n_contracts_signed',
'n_contracts_concluded',
'n_negotiations',
'n_negotiation_rounds_successful',
'n_negotiation_rounds_failed',
'n_negotiation_successful',
'n_negotiation_failed',
'n_registered_negotiations_after',
'activity_level',
'step_time',
'total_time']
Let’s start by seeing how long did each step take (note that stats
access the stats as a Dict[str, List] but stats_df
access the same
data as a pandas dataframe.
plt.bar(range(world.n_steps), world.stats_df['step_time'])
plt.xlabel('Simulation Step')
plt.ylabel('Time (s)')
plt.show()

There are statistics specific for each agent that all have “_{agent_name}”. Lets check what is available for the winner agent:
winner = world.winners[0]
pprint(list(_ for _ in world.stats.keys() if winner.name in _ ))
['spot_market_quantity_00Dec@0',
'spot_market_loss_00Dec@0',
'balance_00Dec@0',
'inventory_00Dec@0_input',
'inventory_00Dec@0_output',
'productivity_00Dec@0',
'assets_00Dec@0',
'bankrupt_00Dec@0',
'score_00Dec@0']
The convension is that agent names has the form {ind}{Type}{process} where ind is a unique index, Type is a shortened version of the agent’s type name, and process is the process the agnet can run. Note that the agent’s input product has the same number as its process and its output product has the next number (i.e. an agent that runs process 1, has input product 1 and output product 2).
We can see that 8 pieces of information are available (for each time-step of the simulation):
bankrupt If true, the agent is bankrupt.
balance The money the agent has in its wallet (account).
inventory (input) The number of units of the agent’s input product available in its inventory (by the end of the simulation step).
inventory (output) The number of units of the agent’s output product available in its inventory (by the end of the simulation step).
assets The value of the agent’s assets (input and output products in inventory) evaluated at the trading price
spot market quantity The quantity bought by this agent from the spot market (of its output product on this step). This can only happen as a result of a product-breach.
spot market loss The spot market price for the agent. This value will go up the more the agent buys from the spot market and will be used to calculate the price for this agent at future steps. This way agents that depend on the spot market instead of negotiation get punished.
productivity The fraction of the agent’s production lines that were active at a given time-step.
score The score of the agent according to the evaluation rule of ANAC SCML 2020
Let’s see how did our agent do
#show the first and last value of each of the agent statistics
pprint({k:(v[0], v[-1]) for k, v in world.stats.items() if winner.name in k })
{'assets_00Dec@0': (162.0, 483.28091511614116),
'balance_00Dec@0': (18008, 22287),
'bankrupt_00Dec@0': (False, False),
'inventory_00Dec@0_input': (0, 0),
'inventory_00Dec@0_output': (9, 24),
'productivity_00Dec@0': (0.9, 0.0),
'score_00Dec@0': (-0.0004972925185103326, 0.24481381686142506),
'spot_market_loss_00Dec@0': (0.3, 2.352),
'spot_market_quantity_00Dec@0': (0, 0)}
stats = pd.DataFrame(data=world.stats)
fig, axs = plt.subplots(2, 3)
for ax, key in zip(axs.flatten().tolist(), ["score", "balance", "assets", "productivity",
"spot_market_quantity", "spot_market_loss"]):
ax.plot(stats[f"{key}_{winner}"])
ax.set(ylabel=key)
fig.show()

We can for example check the activity level of this world (defined as the total amount of money transferred due to trade)
plt.plot(world.stats['activity_level'])
plt.xlabel('Simulation Step')
plt.ylabel('Activitiy Level ($)\nTotal Money Transferred')
plt.show()

We can see a picture of contracting in this world as follows:
plt.plot(world.stats['n_contracts_concluded'], label='Concluded Contracts')
plt.plot(world.stats['n_contracts_cancelled'], label='Cancelled Contracts')
plt.plot(world.stats['n_contracts_signed'], label='Signed Contracts')
plt.plot(world.stats['n_contracts_executed'], label='Executed Contracts')
plt.legend()
plt.xlabel('Simulation Step')
plt.ylabel('N. Contracts')
plt.show()

We can also check the breaches that happened
plt.plot(world.stats['breach_level'])
plt.xlabel('Simulation Step')
plt.ylabel('Total Breach Level')
plt.show()

Notice that there can be multiple winners
winner_profits = [100 * world.scores()[_.id] for _ in world.winners]
winner_types = [_.short_type_name for _ in world.winners]
print(f"{world.winners} of type {winner_types} won at {winner_profits}%")
[00Dec@0] of type ['decentralizing'] won at [24.481381686142505]%
Let’s check how did the first winner’s inventory changes over time:
# find the keys in stats for the input and output inventory
in_key = [_ for _ in world.stats.keys() if _.startswith(f'inventory_{winner}_input')][0]
out_key = [_ for _ in world.stats.keys() if _.startswith(f'inventory_{winner}_output')][0]
# find input and output product indices
input_product, output_product = winner.awi.my_input_product, winner.awi.my_output_product
# draw
fig, (quantity, value) = plt.subplots(1, 2)
quantity.plot(world.stats[in_key], label=f"Input Product")
quantity.plot(world.stats[out_key], label=f"Output Product")
quantity.set(xlabel='Simulation Step', ylabel='Winner\'s Total Storage (item)')
quantity.legend()
value.plot(np.array(world.stats[in_key]) * np.array(world.stats[f"trading_price_{input_product}"])
, label=f"Input Product")
value.plot(np.array(world.stats[out_key]) * np.array(world.stats[f"trading_price_{output_product}"])
, label=f"Output Product")
value.set(xlabel='Simulation Step', ylabel='Winner\'s Inventory Value ($)')
value.legend()
fig.show()

We can actually check what happens to ALL competitors:
from scml.scml2020.world import is_system_agent
fig, (profit, score) = plt.subplots(1, 2)
snames = sorted(world.non_system_agent_names)
for name in snames:
profit.plot(100.0 * (np.asarray(world.stats[f'balance_{name}'])/world.stats[f'balance_{name}'][0] - 1.0), label=name)
score.plot(100 * np.asarray(world.stats[f'score_{name}']), label=name)
profit.set(xlabel='Simulation Step', ylabel='Player Profit Ignoring Inventory (%)')
profit.legend(loc='lower left')
score.set(xlabel='Simulation Step', ylabel='Player Score (%)')
fig.show()

from scml.scml2020.world import is_system_agent
fig, (profit, score) = plt.subplots(1, 2)
snames = sorted(world.non_system_agent_names)
for name in snames:
profit.plot((np.asarray(world.stats[f'balance_{name}'])), label=name)
score.plot(np.asarray(world.stats[f'score_{name}'])*(world.stats[f'balance_{name}'][0]), label=name)
profit.set(xlabel='Simulation Step', ylabel='Player Balance ($)')
profit.legend(loc='lower left')
score.set(xlabel='Simulation Step', ylabel='Player Score Unnormalized ($)')
fig.show()

or just look at the end of the game
fig, (score, profit) = plt.subplots(1, 2)
final_scores = [100 * world.stats[f"score_{_}"][-1]
for _ in world.non_system_agent_names]
final_profits = [100 * world.stats[f"balance_{_}"][-1] / world.stats[f"balance_{_}"][0] - 100
for _ in world.non_system_agent_names]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Score (%)")
profit.set(ylabel="Final Profit (%)")
fig.show()

fig, (score, profit) = plt.subplots(1, 2)
final_scores = [world.stats[f"score_{_}"][-1] * (world.stats[f"balance_{_}"][0])
for _ in world.non_system_agent_names]
final_profits = [world.stats[f"balance_{_}"][-1]
for _ in world.non_system_agent_names]
plt.setp(score.xaxis.get_majorticklabels(), rotation=45)
plt.setp(profit.xaxis.get_majorticklabels(), rotation=45)
score.bar(world.non_system_agent_names, final_scores)
profit.bar(world.non_system_agent_names, final_profits)
score.set(ylabel="Final Unnormalized Score ($)")
profit.set(ylabel="Final Balance ($)")
fig.show()

You can inspect what happened in the simulation by plotting different output statistics. For example, we can see how did the trading price of different products change over the simulation time.
fig, axs = plt.subplots(2, 2)
for ax, key in zip(axs.flatten().tolist(), ["trading_price", "sold_quantity", "unit_price"]):
for p in range(world.n_products):
ax.plot(world.stats[f"{key}_{p}"], marker="x", label=f"Product {p}")
ax.set_ylabel(key.replace("_", " ").title())
ax.legend().set_visible(False)
axs[-1, 0].legend(bbox_to_anchor=(1, -.5), ncol=3)
fig.show()

fig, axs = plt.subplots(1, 2)
for ax, key in zip(axs.flatten().tolist(), ["spot_market_quantity", "spot_market_loss"]):
for a in world.non_system_agent_names:
ax.plot(world.stats[f"{key}_{a}"], marker="x", label=f"{a}")
ax.set_ylabel(key.replace("_", " ").title())
ax.legend().set_visible(False)
axs[0].legend(bbox_to_anchor=(1, -.2), ncol=4)
fig.show()

You can dig futher to understand what happened during this siumulation. For example, let’s see some of the contracts that were signed:
# create a view with only signed contracts
contracts = world.contracts_df
signed = contracts.loc[contracts.signed_at>=0, :]
fields = ["seller_name", "buyer_name", "delivery_time", "quantity", "unit_price",
"signed_at", "executed", "breached", "nullified", "erred"]
signed[fields].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name | buyer_name | delivery_time | quantity | unit_price | signed_at | executed | breached | nullified | erred | |
---|---|---|---|---|---|---|---|---|---|---|
703 | 07Buy@2 | 09Dec@3 | 9 | 66 | 31 | 2 | False | True | False | False |
722 | 06Buy@2 | 09Dec@3 | 10 | 65 | 31 | 2 | False | True | False | False |
733 | 06Buy@2 | 10Dec@3 | 9 | 62 | 31 | 2 | False | True | False | False |
700 | 06Buy@2 | 10Dec@3 | 10 | 61 | 31 | 2 | False | True | False | False |
698 | 07Buy@2 | 10Dec@3 | 8 | 53 | 31 | 2 | False | True | False | False |
916 | 08Ind@2 | 09Dec@3 | 15 | 46 | 31 | 4 | False | True | False | False |
709 | 06Buy@2 | 09Dec@3 | 7 | 45 | 31 | 2 | False | True | False | False |
905 | 06Buy@2 | 10Dec@3 | 22 | 45 | 31 | 4 | False | False | False | False |
2254 | 05Dec@1 | 08Ind@2 | 36 | 38 | 35 | 26 | True | False | False | False |
913 | 06Buy@2 | 09Dec@3 | 24 | 31 | 31 | 4 | False | False | False | False |
Let’s check some of the contracts that were fully executed
signed.loc[signed.executed, fields].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name | buyer_name | delivery_time | quantity | unit_price | signed_at | executed | breached | nullified | erred | |
---|---|---|---|---|---|---|---|---|---|---|
2254 | 05Dec@1 | 08Ind@2 | 36 | 38 | 35 | 26 | True | False | False | False |
1247 | 05Dec@1 | 08Ind@2 | 14 | 28 | 35 | 8 | True | False | False | False |
2450 | 08Ind@2 | 09Dec@3 | 39 | 23 | 45 | 34 | True | False | False | False |
828 | 08Ind@2 | 09Dec@3 | 27 | 22 | 48 | 3 | True | False | False | False |
2399 | 08Ind@2 | 10Dec@3 | 37 | 20 | 45 | 31 | True | False | False | False |
1725 | 00Dec@0 | 03Ind@1 | 23 | 20 | 26 | 15 | True | False | False | False |
2222 | 04Dec@1 | 08Ind@2 | 35 | 19 | 37 | 25 | True | False | False | False |
2225 | 04Dec@1 | 07Buy@2 | 30 | 19 | 35 | 25 | True | False | False | False |
2449 | 03Ind@1 | 07Buy@2 | 39 | 19 | 35 | 34 | True | False | False | False |
2288 | 00Dec@0 | 05Dec@1 | 37 | 19 | 18 | 27 | True | False | False | False |
signed.loc[signed.breached, fields[:-4] + ["breaches"]].sort_values(["quantity", "unit_price"], ascending=False).head(10)
seller_name | buyer_name | delivery_time | quantity | unit_price | signed_at | breaches | |
---|---|---|---|---|---|---|---|
703 | 07Buy@2 | 09Dec@3 | 9 | 66 | 31 | 2 | 07Buy@2:product(1.0) |
722 | 06Buy@2 | 09Dec@3 | 10 | 65 | 31 | 2 | 06Buy@2:product(1.0) |
733 | 06Buy@2 | 10Dec@3 | 9 | 62 | 31 | 2 | 06Buy@2:product(1.0) |
700 | 06Buy@2 | 10Dec@3 | 10 | 61 | 31 | 2 | 06Buy@2:product(1.0) |
698 | 07Buy@2 | 10Dec@3 | 8 | 53 | 31 | 2 | 07Buy@2:product(1.0) |
916 | 08Ind@2 | 09Dec@3 | 15 | 46 | 31 | 4 | 08Ind@2:product(0.5869565217391305) |
709 | 06Buy@2 | 09Dec@3 | 7 | 45 | 31 | 2 | 06Buy@2:product(1.0) |
2070 | 01Ind@0 | 05Dec@1 | 30 | 27 | 18 | 22 | 01Ind@0:product(1.0) |
737 | 01Ind@0 | 04Dec@1 | 13 | 24 | 18 | 2 | 01Ind@0:product(0.5833333333333334) |
983 | 00Dec@0 | 05Dec@1 | 14 | 24 | 18 | 5 | 00Dec@0:product(0.5833333333333334) |
We can now see how does the singning day affect delivery day, product and quantity
fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "quantity", "product_index"]):
ax[i].scatter(signed.signed_at, signed[x])
ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Signing Day")
fig.show()

fig, ax = plt.subplots(1, 3)
for i, x in enumerate(["delivery_time", "unit_price", "product_index"]):
ax[i].scatter(signed.quantity, signed[x])
ax[i].set(ylabel=x.replace("_", " ").title(), xlabel="Quantity")
fig.show()

Did any agents go bankrupt and when?
bankruptcy = {a: np.nonzero(stats[f"bankrupt_{a}"].values)[0]
for a in world.non_system_agent_names}
pprint({k: "No" if len(v)<1 else f"at: {v[0]}" for k, v in bankruptcy.items()})
{'00Dec@0': 'No',
'01Ind@0': 'No',
'02Mov@0': 'No',
'03Ind@1': 'No',
'04Dec@1': 'No',
'05Dec@1': 'No',
'06Buy@2': 'at: 17',
'07Buy@2': 'at: 43',
'08Ind@2': 'No',
'09Dec@3': 'No',
'10Dec@3': 'No',
'11Mov@3': 'No'}
You can see what happened during this simulation by drawing graphs at different steps. The meaning of different edge colors can be drawn as follows:
from negmas import show_edge_colors
show_edge_colors()

You can see what happened in this world in a series of graphs using the
draw
method
world.draw(steps=(0, world.n_steps), together=False, ncols=2, figsize=(20, 20))
plt.show()

You can also run a simple animation to see what happens at every step (you need to download the jupyter notebook and execute it to see the animation) :
# prepare animation
from time import perf_counter
from negmas.helpers import humanize_time
import gif
from pathlib import Path
# define animation parameters
draw_every = 1 # draw every 5 steps (drawing is REALLY slow)
single_graph = False # show a graph for every event type
# what = ["contracts-signed", "contracts-breached", "contracts-executed",]
fps = 5
# define the animation function. Simply draw the world
@gif.frame
def plot_frame(s):
world.draw(#what=what,
steps=(s-draw_every, s),
together=single_graph, ncols=3, figsize=(20, 20))
# create frames
frames = []
for s in tqdm(range(world.n_steps)):
if s % draw_every != 0:
continue
frames.append(plot_frame(s))
path = Path.cwd() / "run.gif"
path.unlink(missing_ok=True)
gif.save(frames, str(path), duration=1000 // fps)
HBox(children=(FloatProgress(value=0.0, max=50.0), HTML(value='')))
then show the animation
from IPython.display import HTML
HTML('<img src="run.gif">')

Note that this graph shows only what happened in the last draw_every
steps.
Let’s see some statistics about the simulation.
Running a tournament¶
Now that you can run simple world simulations, let’s try to run a
complete tournament and see its results. Let’s start by running a
standard tournament (in which each agent is represented by a single
factory). Running a collusion tournament will be exactly the same with
the only difference that anac2020_std
will be replaced with
anac2020_collusion
.
Note that in the real competition we use thousands of configurations and longer simulation steps (e.g. 50 \(\le\) n_steps \(\le\) 500).
from scml.scml2020.utils import anac2020_std
tournament_types = agent_types + [RandomAgent]
# may take a long time
results = anac2020_std(
competitors=tournament_types,
n_configs=12, # number of different configurations to generate
n_runs_per_world=1, # number of times to repeat every simulation (with agent assignment)
n_steps = 10, # number of days (simulation steps) per simulation
print_exceptions=True,
)
Who was the winner?
results.winners
['decentralizing_agent']
How many simulations were actually run?
len(results.scores.run_id.unique())
60
The total number of simulations \(n_{s}\) will be \(n_t \times n_c \times n_r\) where \(n_t\) is the number of competitor agent types, \(n_c\) is the number of configurations, and \(n_r\) is the number of runs per configuration
We can also see the scores that every agent type got
results.score_stats
agent_type | count | mean | std | min | 25% | 50% | 75% | max | |
---|---|---|---|---|---|---|---|---|---|
0 | buy_cheap_sell_expensive_agent | 60.0 | -0.460880 | 0.399413 | -2.202297 | -0.512401 | -0.321267 | -0.249171 | -0.044618 |
1 | decentralizing_agent | 60.0 | 0.180795 | 0.205134 | -0.123303 | 0.034685 | 0.142906 | 0.323354 | 0.657287 |
2 | ind_decentralizing_agent | 60.0 | 0.161280 | 0.199404 | -0.122533 | 0.023384 | 0.120836 | 0.239894 | 0.832956 |
3 | moving_range_agent | 60.0 | -0.052376 | 0.081866 | -0.211015 | -0.108780 | -0.052406 | 0.000000 | 0.146121 |
4 | random_agent | 60.0 | -0.538307 | 0.375828 | -1.905989 | -0.540328 | -0.391387 | -0.352393 | -0.162312 |
You can also do statistical significance testing using ttest or kstest (with multi-comparison correction)
results.kstest
a | b | t | p | n_a | n_b | n_effective | |
---|---|---|---|---|---|---|---|
0 | ind_decentralizing_agent | decentralizing_agent | 0.133333 | 6.647506e-01 | 60 | 60 | 60 |
1 | ind_decentralizing_agent | random_agent | 1.000000 | 2.070074e-35 | 60 | 60 | 60 |
2 | ind_decentralizing_agent | buy_cheap_sell_expensive_agent | 0.983333 | 2.484089e-33 | 60 | 60 | 60 |
3 | ind_decentralizing_agent | moving_range_agent | 0.633333 | 1.306907e-11 | 60 | 60 | 60 |
4 | decentralizing_agent | random_agent | 1.000000 | 2.070074e-35 | 60 | 60 | 60 |
5 | decentralizing_agent | buy_cheap_sell_expensive_agent | 0.983333 | 2.484089e-33 | 60 | 60 | 60 |
6 | decentralizing_agent | moving_range_agent | 0.650000 | 2.904237e-12 | 60 | 60 | 60 |
7 | random_agent | buy_cheap_sell_expensive_agent | 0.366667 | 5.569711e-04 | 60 | 60 | 60 |
8 | random_agent | moving_range_agent | 0.983333 | 2.484089e-33 | 60 | 60 | 60 |
9 | buy_cheap_sell_expensive_agent | moving_range_agent | 0.850000 | 2.164592e-22 | 60 | 60 | 60 |
see the total score
results.total_scores
agent_type | score | |
---|---|---|
0 | decentralizing_agent | 0.142906 |
1 | ind_decentralizing_agent | 0.120836 |
2 | moving_range_agent | -0.052406 |
3 | buy_cheap_sell_expensive_agent | -0.321267 |
4 | random_agent | -0.391387 |
or the aggregated statistics of the world. For example, let’s draw the activity level for different simulations.
plt.errorbar(range(len(results.agg_stats)),
results.agg_stats.activity_level_mean,
np.sqrt(results.agg_stats.activity_level_var)
)
plt.xlabel("Simulation Number")
plt.ylabel("Activity Level")
plt.show()

We can even get the scores of every agent belonging to every agent type at every simulation
results.scores.loc[:, ["agent_name", "agent_type", "score"]].head()
agent_name | agent_type | score | |
---|---|---|---|
0 | 00Ind@0 | ind_decentralizing_agent | 0.626623 |
1 | 01Dec@0 | decentralizing_agent | 0.322342 |
2 | 02Ran@1 | random_agent | -1.372634 |
3 | 04Buy@2 | buy_cheap_sell_expensive_agent | -0.334364 |
4 | 05Mov@2 | moving_range_agent | -0.134590 |
or inspect any statistic we like
ax = sns.violinplot(data=results.stats, x="step", y="activity_level")
ax.set(ylabel="Activity Level / Business Size ($)", xlabel="Step Number", yscale="log")
plt.gcf().show()

Let’s see how did the location at the production graph affect the score of each type.
results.scores["level"] = results.scores.agent_name.str.split("@", expand=True).loc[:, 1]
sns.lineplot(data=results.scores[["agent_type", "level", "score"]],
x="level", y="score", hue="agent_type")
plt.plot([0.0] * len(results.scores["level"].unique()), "b--")
plt.show()

Now that you can run simulations and complete tournament, let’s see how are we going to develop a new agent for the SCML2020 league \(\rightarrow\)
Download Notebook
.