©Copyright JASSS

JASSS logo ----

Oliver Will (2009)

Resolving a Replication That Failed: News on the Macy & Sato Model

Journal of Artificial Societies and Social Simulation 12 (4) 11
<https://www.jasss.org/12/4/11.html>

For information about citing this article, click here

Received: 29-Aug-2009    Accepted: 27-Sep-2009    Published: 31-Oct-2009

PDF version


* Abstract

The paper at hand aimes at identifying the assumptions that lead to the results presented in an article by Michael Macy and Yoshimichi Sato published in PNAS. In answer to a failed replication, the authors provided the source code of their model and here the results of carefully studying that code are presented. The main finding is that the simulation program implements an assumption that is most probably an unwilling, unintended, and unwanted implication of the code. This implied assumption is never mentioned in Macy and Sato's article and if the authors wanted to program what they describe in their article then it is due to a programming error. After introducing the reader to the discussion, data that stem from a new replication based on the assumptions extracted from the source code is compared with the results published in Macy and Sato's original article. The replicated results are sufficiently similar to serve as a strong indicator that this new replication implements the same relevant assumptions as the original model. Afterwards it is shown that a removal of the dubious assumption leads to results that are dramatically different from those published in Macy and Sato's PNAS article.

Keywords:
Replication, Social Dilemma Situations, Trust, Simulation Methodology, Cooperation

* Introduction

1.1
In Will and Hegselmann (2008a) we reported to JASSS on how we failed to replicate the results presented in Macy and Sato (2002). We considered several aspects of the model as causes of our failure. The most plausible explanation we could find, was that we had missed the way in which transaction costs were implemented. Lacking the opportunity to analyse the source code of the model, we were not able to verify this conjecture. Shortly after the publication of our paper, the authors' reply appeared in the same journal (Macy and Sato (2008)). No clear answer on why our replication might have failed was given in this text (Will and Hegselmann 2008b) but Macy and Sato published their source code in answer to our article. In the present text, I am presenting the insights I gained from studying the code carefully, trying to uncover those assumptions of the original model that we got wrong in our replication.

1.2
Macy and Sato's model is quite appealing. It implements a social structure that distinguishes between interactions in (a) more or less small groups of people that know each other comparatively well and (b) usually larger groups of strangers. This is a structural scenario that corresponds well to real-world interactions of people. The model predicts that social mobility, as long as it is not extremely high, has a large positive effect on trust among strangers. In a world with probably increasing social mobility this is an interesting finding. Thus Macy and Sato's work is of some interest to scholars of simulation but it needs to be clarified what assumptions lead to the published results.

1.3
In the following section a condensed overview of the assumptions that I extracted from the source code of the model is provided. The main finding is that there is an exclusion of certain agents from reinforcement learning that is most probably an unmeant, unintended, and unwanted implication of the code. This implied assumption is never mentioned in Macy and Sato (2002) and if the authors wanted to program what they describe in their article then it is due to a programming error. The subsequent section compares data that stem from a new replication based on these assumptions with the results published in Macy and Sato's PNAS article. The replicated results are sufficiently similar to serve as a strong indicator that the replication implements the same relevant assumptions as the original model. Afterwards I provide data from simulations in which the seemingly unmeant exclusion of agents from reinforcement learning mentioned above is repaired. This modified version leads to results that are dramatically different from those published in Macy and Sato (2002).[1]

* Assumptions of the model in Macy and Sato (2002) extracted from the source code

2.1
The following subsections give an overview of the assumptions that I extracted from the source code provided in Macy and Sato (2008). They are ordered according to the sequence of events in a simulation and each time step.

Initializing

2.2

Moving

2.3

Entering the market

2.4

Matching

2.5

Cooperation

2.6

Trust

2.7

Payoffs

2.8

Learning

2.9
There are two types of learning in the model: Social learning from a role model and reinforcement learning based on the agents' own experience. Agents that do not have a partner do not learn at all.
Social Learning
2.10
Reinforcement Learning
2.11
2.12
This last assumptions is never mentioned in Macy and Sato's paper though their central claim crucially depends on it (see section 4.2). There is no obvious justification for it and - assuming that the authors wanted to program what they describe in their article - the most plausible diagnosis is, that it is an unmeant, unwantend and unintended consequence of some lines of code: in short a programming error.
2.13
A look at the code shows us that the authors experimented with agents that have two probabilities to enter the market: one in case of being newcomers and one if not.[7] Both of these probabilities can change via reinforcement learning but not both during the same time step since an agent can be either a newcomer or not but not both and only the respective probability can be changed. To remove the assumption of two probabilities, the probability to enter the market as a newcomer is set to the value of the probability for entering the market as a neighbour after initialising the agents[8] and after each learning event.[9]
2.14
This causes a serious problem: For all agents that are newcomers and whose probability to enter the market in case of being a newcomer was changed by reinforcement learning, this probability will afterwards be overwritten by the (unchanged) probability of entering the market in case of not being a newcomer. Thus for agents that are newcomers reinforcement learning does not have any effect on their probability to enter the market.
2.15
At first sight, I did not expect this problem to have any substantial effect on the results of the model. However, the results I present in section 4 tell a very different story.

* Successful replication

3.1
Using the following figures you can compare plots that stem from the replicated model based on the assumptions described in section 2 with plots from the original paper.[10] Information on the figures is given in the captions of Macy and Sato's figures. Note that each combination of a mobility rate and a neighbourhood size was repeated 20 times.

3.2
When comparing the plots, one can easily see that the results from the replicated model match those from the original model very well. Not only in qualitative but also in quantitative terms they are very much the same. This indicates that it is very likely that the assumptions I extracted from the source code match at least the relevant ones of those that were implemented by Macy and Sato. Anyhow, this is only an argument to the best explanation. It might still be that there are other reasons for the good fit of the results but, in my opinion, this is quite unlikely. Furthermore, the high degree of quantitative similarity indicates that 20 repetitions of each parameter combination are sufficient to control for random effects of the model.

Figure
Figure 1. Plots of data from replicated (left) and original model (right).

Figure
Figure 2. Plots of data from replicated (left) and original model (right).

Figure
Figure 3. Plots of data from replicated (left) and original model (right).

Figure
Figure 4. Plots of data from replicated (left) and original model (right).

Figure
Figure 5. Plots of data from replicated (left) and original model (right).

* Modified assumptions

4.1
Now, that I have good reason to believe that I uncovered the assumptions that lead to the results in Macy and Sato (2002), I want to explore the effects of changing some of them. As the implementation of the reinforcement learning seems to imply an assumption that is implemented unintentionally, the most urgent issue to check is how the model reacts if this problem is corrected. I therefore took the code of the successful replication and changed it such that newcomers can learn by reinforcement learning on whether or not they should enter the market as well. Afterwards exactly the experiments of Macy and Sato described in the previous section were repeated.[11]
4.2
It turns out that after this seemingly small issue concerning the set of agents that might learn from reinforcement learning is changed, the model behaves dramatically different. Figure 6 shows the most interesting plot. We can see that for small levels of mobility the plots that show the indicators of market interaction over time are very similar in both cases. But for rates of mobility above 0.4 the results become more different the larger the rate of mobility gets. This is consistent with the modification of the code since the rate of mobility determines the number of newcomers and what was changed was whether or not newcomers can learn from reinforcement learning on whether they should enter the market.
4.3
The most important point is that trust in strangers does not increase with the rate of mobility. Actually the level of trust in strangers remains below or near 0.2 for all levels of mobility. This is very different from Macy and Sato's finding that for mobility rates between 0.1 and 0.9 the expected level of trust in strangers grows (approximately) linearly from a level of 0.1 to 0.7.[12] Figure 7 shows us that there are actually some runs in which a high level of trust in strangers was reached. However, (ignoring mobility rates 0 and 1) it does not look as if there is a correlation between the probability of high levels of trust and the rate of mobility.

Figure
Figure 6. Plots of data from "corrected" (left) and original model (right).

Figure
Figure 7. Plots of data from "corrected" (left) and original model (right).

* Conclusion

5.1
The previous sections suggest that the results presented in Macy and Sato (2002) stem from a source code that implied an assumption the authors did not intend to implement. If the model were robust against this assumption, everything would be fine. But it turns that the model crucially depends on that assumption.
5.2
The results presented here account for the importance of replication. Computer programs may imply assumptions we did not want to implement and this might even happen to the best in social simulation. Rigourous replication can detect such problems and furthermore uncover the assumptions that lead to published results. Thus replication is an important issue in the field social simulation and it is good to see that there is an increasing number of publications on replicating models.[13]


* Notes

1 Both implementations were written in FORTRAN and can be downloaded from http://pe.uni-bayreuth.de/?coid=21&q=detail&mid=125. The download also contains the Mathematica notebooks that were used to generate the plots presented in this paper.

2 This is a minor difference to our former repplication since in Will and Hegselmann (2008a) our variation of N was more fine-grained. E.g., to get an average neighbourhood size of 30, we conducted 7 runs with 34 neighbourhoods and the remaining 13 with 33 instead of taking 33 neighbourhoods in all 20 repetitions.

3 This is equivalent to the signalling mechanism suggested in Will and Hegselmann (2008a).

4 I use the standard notation here: If one agents defects and the other cooperates, the defector gains a payoff given by T(emptation) and the cooperator one given by S(ucker). In case of mutual cooperation both gain the R(reward payoff) and if both defect they receive the P(unishment) outcome.

5 In Will and Hegselmann (2008a) we assumed that newcomers can be role models.

6 In Will and Hegselmann (2008a) we assumed that all agents that have a partner learn either by social or reinforcement learning and that both options are equally likely.

7 The variables are named "PropMktN" and "PropMktS" in the source code. Line 492 shows their application.

8 Line 348 of the source code.

9 Line 753 of the source code. Here, you also find a note, that the distinction should be eliminated.

10 Here is a description of what the values in the figures mean. The exact definition of each value was extracted from the source code of Macy and Sato's model.

Market size:
Is the share of agents that entered the market (see 2.4).
Trust strangers:
Of all agents that have a partner and entered the market, "Trust strangers" gives the share that trust.
Reading signals:
Is the mean value of all agents' probability to conduct the signal-reading instead of the parochial strategy when deciding on trust or distrust (see 2.7).
Trust newcomers:
Of all agents that have a partner, are newcomers, and interact in the neighbourhood, "Trust newcomer" gives the share that trusts
Trusts neighbours:
Of all agents that have a partner, are not newcomers, and interact in the neighbourhood, "Trust neighbours" gives the share that trusts

11Not only were the parameters the same but I also used the same random seeds to initialise the model.

12Note that this modified model produces results that are not only different from those in Macy and Sato's article but also from those of the very first replication in Will and Hegselmann (2008a). Theses differences are mainly due to further differences concerning the learning algorithm mentioned in footnotes [5] and [6]. The model's high sensitivity to small modifications of the learning algorithm will be adressed in future work.

13See for example van de Rijt, Siegel and Macy (2009).


* References

MACY M and Sato Y (2002) Trust, cooperation and market formation in the US and Japan. Proceedings of the National Academy of Sciences, 99, pp. 7214-7220.

MACY, M and Sato Y (2008). 'Reply to Will and Hegselmann'. Journal of Artificial Societies and Social Simulation 11 (4) 11 https://www.jasss.org/11/4/11.html.

VAN DE RIJT, A, Siegel, D and Macy, M (2009). 'Neighborhood Chance and Neighborhood Change: A Comment on Bruch and Mare'. American Journal of Sociology 114, 4, pp. 1166-80.

WILL, O and Hegselmann, R (2008a). 'A Replication That Failed — on the Computational Model in 'Michael W. Macy and Yoshimichi Sato: Trust, Cooperation and Market Formation in the U.S. and Japan. Proceedings of the National Academy of Sciences, May 2002''. Journal of Artificial Societies and Social Simulation 11 (3) 3 https://www.jasss.org/11/3/3.html.

WILL, O and Hegselmann, R (2008b). 'Remark on a Reply'. Journal of Artificial Societies and Social Simulation 11 (4) 13 https://www.jasss.org/11/4/13.html.

----

ButtonReturn to Contents of this issue