The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism

: TheOverview, DesignconceptsandDetails(ODD)protocolfordescribingIndividual-andAgent-Based Models (ABMs) is now widely accepted and used to document such models in journal articles. As a standardized document for providing a consistent, logical and readable account of the structure and dynamics of ABMs, some research groups also find it useful as a workflow for model design. Even so, there are still limitations to ODD that obstruct its more widespread adoption. Such limitations are discussed and addressed in this paper: the limited availability of guidance on how to use ODD; the length of ODD documents; limitations of ODD for highly complex models; lack of sufficient details of many ODDs to enable reimplementation without access to the model code; and the lack of provision for sections in the document structure covering model design rationale, the model’s underlying narrative, and the means by which the model’s fitness for purpose is evaluated. We document the steps we have taken to provide better guidance on: structuring complex ODDs and an ODD summary for inclusion in a journal article (with full details in supplementary material; Table point readers to relevant sections of the model code; update the document structure to include sections on model rationale and evaluation. We also further advocate the need for standard descriptions of simulation experiments and argue that ODD can in principle be used for any type of simulation model. Thereby ODD would provide a lingua franca for simulation modelling.


Introduction
. Individual-or agent-based models (ABMs in the following) began to be widely applied in ecology in the s and in the social and social-ecological sciences in about (Vincenot ). While mathematical models can be described completely and concisely in the universal language of mathematics, mathematics is neither a complete nor convenient and concise way to capture the important characteristics of simulation models. Consequently, the descriptions of early ABMs were di icult to write and read because no one knew where to place or expect di erent kinds of information about the model and in what detail. ABM descriptions were o en incomplete, so that a complete understanding of the model su icient enough to allow reimplementation was not possible. .
Incomplete descriptions violate the central requirement of science that materials and methods must be specified in su icient detail to allow replication of results. Yet ABMs all have common characteristics, which implies that a common language for them could be both useful and feasible. To take advantage of this situation, the ODD protocol (Overview, Design concepts, Details; Grimm et al. , ) was proposed as a standard format for describing ABMs. The development and adoption of ODD is, in fact, one of several parallel initiatives to overcome the "replication crisis", which has been recognised in many disciplines as preventing scientists from building e iciently on existing data, methods, experimental designs, and  ; Wilkinson et al. ). .
ODD was designed to make it easier to write and read ABM descriptions and facilitate model replication, while not being overly technical. ODD model descriptions (ODDs, herea er) can include equations and short algorithms, but are based on written text and intended to be read by humans. They are independent of the hardware and so ware used to implement the model. ODD consists of seven elements ( Figure ). Conceptually they are divided into the three categories "Overview," "Design concepts," and "Details;" hence the acronym ODD. Each of these categories serves a di erent purpose: giving an overview, explaining how design concepts important for ABMs were used, and explaining all the details of the "machinery" of the model. While these three categories explain the structure of ODD model descriptions, ODD itself is defined by its seven elements.
Figure : Structure of model descriptions following the ODD protocol. The protocol itself consists of seven elements, which should be used as given, including the numbering. The categories O (Overview), D (Design concepts), and D (Details) are meant as comments, but not used in ODD model descriptions. There are specific design concepts; if some of them do not apply, they can be le out; if a model includes an important concept of general interest, which is not yet part of ODD, it can be added at the end of the Section 'Design concepts'.
. In its first element " . Purpose and patterns" the purpose of the model and the patterns that serve as model evaluation criteria are briefly described (the new aspect "patterns" is defined below). The element " . Entities,

.
In the following section, we briefly list the advantages of using a standard format and the extent to which ODD has been used so far. As the developments in ODD's purpose have inevitably raised issues and challenges that were not considered when ODD was first created, we next summarize these challenges. Here we first describe all these issues and challenges (Section ) and then present possible solutions (Section ). Finally, we provide an outlook on the further development and use of ODD (Section ). In particular, we propose that most elements of ODD are applicable to any simulation model, not just ABMs, so that ODD could become a "lingua franca" for simulation modelling in general.

ODD's Benefits and Current Use
. Other standards for reporting on simulation models have been proposed, especially in the area of discrete event simulation (Monks et al. ; Wilkinson et al. ), but they focus on providing checklists of things that need to be communicated. To our knowledge, none of them have been used for ABMs yet, which may be because discrete event simulation is a mature field that has its own terminology and technique while many ABMs are developed by domain experts with little background in simulation science. .
Because it is used by domain experts from many fields, ODD is less technical and has a strong focus on facilitating communication within and across disciplines. Once a specific communication format has been defined, established and learned, both authors and readers know exactly where to put and expect what kind of information. The benefits of standard formats are particularly obvious for scientific publications, where the introduction, materials and methods, results, and discussion sections serve well-understood purposes and are presented in a particular order. The establishment of such a standard structure for the description of ABMs was the main purpose of ODD (Grimm et al. , ).
. The hierarchical structure of ODD allows the reader to get an overview of the entire model before being asked to consider details. This implies some redundancy but makes reading model descriptions more e icient because readers can decide how much detail they want to go into. The "Design concepts" element provides additional information for understanding why the model was designed as it was. It also serves as a checklist for authors and readers: each of these concepts (Figure ) a ects the scope and utility of the model, so it is important that modellers explicitly consider them. For example, the most important concept details which key processes in the model are imposed via empirical parameters and rules and which arise from adaptive decision-making of the agents. The ability to represent dynamics arising from adaptive decision-making is a primary benefit of ABMs and can make ABMs more flexible and predictive (Grimm & Berger ; Railsback ; Stillman et al. ), but usually it is possible to represent only a few processes this way; selecting them is a critical design decision that should be made carefully and justified clearly. .
A er its initial publication (Grimm et al. ), ODD was quickly adopted by ecological modellers, so that a first update of ODD and its description, based on its use in more than publications, could be published in (Grimm et al. ). ODD was also introduced to modellers in the social sciences (Polhill ; Polhill et al. ), presented in several handbook chapters (Grimm ; Grimm et al. ; Grimm & Railsback a; Railsback & Grimm ), and used in a textbook (Railsback & Grimm ). ODD has also been supplemented with elements to describe ABMs involving human decision making and behaviour (ODD+D; Müller et al. or describe how data were used in detail to parameterize the ABM (ODD+ D; Laatabi et al. ). ODD also became a key component of "TRACE" (TRAnsparent and Comprehensive model Evaludation; Grimm et al. , a standard format for documenting all elements of model development, including parameterization, model analysis, and model "evaludation" (evaluation and validation; Augusiak et al. ). .
The bibliometric analysis of Vincenot ( ) shows that ODD has contributed to the integration of agent-based modelling across disciplines, specifically by linking the disciplines that historically used the terms "individualbased" and "agent-based" and therefore remained separate for many years. Integration was indeed a declared purpose of ODD: once a common language for describing ABMs exists, it becomes easier to learn from each other's models because it is easier to compare and understand di erent ABMs, even if they were developed in di erent disciplines. Moreover, ODD was designed to make model descriptions complete and thereby enable replication, so that submodels describing certain processes or behaviours can more easily be reused in new models. .
ODD is most widely used by ecologists, but its use is also increasing in social (Hauke et al. ) and other sciences (Figure ; Vincenot ). The non-technical nature of ODD facilitates this development: modellers can just use ODD instead of having to create their own documentation format. Moreover, with increasing usage of ODD within each scientific community, more readers, including reviewers and editors, can take advantage of the standardization and will come to expect its use. While ODD has quickly established itself as a widely used format and the absolute number of publications using it has been steadily increasing, its spread has been unequal among scientific domains ( Figure ). The proportion of papers using ODD has grown significantly only in the life sciences and, overall, changes have been slight since . Figure : Annual trends in the use of the ODD protocol were produced on the basis of Scopus searches performed on January , . The list of ABM papers was retrieved through the query 'TITLE-ABS-KEY('agentbased model * ' OR 'individual-based model * ') AND TITLE-ABS-KEY('simulation'))'. These results were crossed with the list of ABM papers citing at least one of the two ODD papers, Grimm et al. ( , ). Post-processing in R was used to group publications by year and by research fields and generate annual usage rate of ODD in the four scientific domains (obtained through aggregation of research field IDs provided by Scopus; as per "Categories" in Supplement S ). In Google Scholar, the article was cited times, the one (November , ).
. We do not expect that ODD will ever be used by all modellers and admit that it has limitations for some kinds of models, e.g., discrete-event simulations (Monks et al. ). One reason for not using ODD in disciplines other than ecology is obvious: modellers from, e.g., economics, geography, or physics do not usually read ecology journals and therefore may not yet have encountered ODD, or they may perceive it as specifically designed for ecological ABMs. In addition, some scientists simply appear resistant to standardization. However, there are also reasons why the rationale, design, and current use of ODD itself may have prevented its wider use; we address these reasons next.

Issues and Challenges
. Here we briefly list important issues and challenges of ODD that could limit its use. In Section we then suggest possible solutions.
ODD guidance materials are few and not o en used . Inherent to any scientific writing, a factor that limits the wider acceptance and use of ODD is that following an established format without some initial training does not guarantee that the writing is as clear, concise, wellorganized, and unambiguous as it could be. However, so far, the guidance material for ODD has been limited to a template provided in the supplement of Grimm et al. ( ), which was not open access and has not been widely used. The guidance provided in the template is also relatively brief and (now) outdated with respect to the changes in ODD described in Section . . .
The rationale and structure of ODD are simple, so its use should in principle be simple, but comments from users and our experience from reading many ODDs indicate that this is o en not the case. Most ODD users report that they initially found it di icult to understand key elements of ODD, especially "state variables" and some of the design concepts. Consequently, many early ODDs have not fully used the protocol as intended, reducing its value for standardization and understanding. The weaknesses and lack of availability of the template certainly contributed to these problems. .
However, a more fundamental reason for an incomplete or incoherent use of ODD is that modellers tend to describe the mental representation of the model or the narrative of the model rather than the structure and processes of the program that implements the model. These meta-descriptions of mental models are inevitably incomplete; they are simplified versions of our computer models because it is not possible to mentally represent the entire program code. As a result, ODDs are sometimes sloppy, incomplete, and do not provide enough detail to re-implement the model. .
When we think about an ABM, we o en focus on its emergent behaviour, but for the ODD we have to describe the low-level processes and interactions from which that behaviour arises. A motto for this issue could be: "Describe what the program does, not what you think the model does". The lessons from this experience are that the primary ODD learning materials -the template and checklist -need to be more widely available and used, and need to better encourage complete description of what the final model's computer program does. There certainly is also the need to describe a model's underlying story, or narrative, but ODD is not the place for this (but see Section . on "TRACE" and . on "Summary ODDs").

It is not clear whether the rationale for model design should be explained in ODD .
When ODD was originally conceived, its authors were concerned strictly with the need to describe models accurately and e iciently and did not address the question of whether, and how, an ODD should also describe why each element of a model was designed the way it was. Consequently, some ODDs are strictly descriptive, stating only what the model includes and does, while others are also explanatory, describing the processes and reasoning used to design each element. As ODD has developed into a tool supporting the design of ABMs, the importance of explanation has become obvious. Providing the rationale for model design throughout an ODD can have two major benefits. First, it can greatly increase model credibility by encouraging model developers to think carefully about and test each design decision, and by providing evidence of such careful design to readers and model clients. Second, it can increase the re-use of modelling techniques and theory throughout a discipline by (a) encouraging model developers to look for existing techniques as part of a careful model design process and (b) making it easier for subsequent modellers to know which parts of existing models are suitable for re-use. . There are several ways that the rationale for model design can be included productively in an ODD. One is simply to provide the basis for all parameter values (e.g., taken from which literature, and why; developed from what data, and how; estimated via calibration, how) and submodels representing processes. Another is to fully implement and analyse each major submodel separately to show that it produces reasonable results under all conditions that can occur in the full ABM. And the "pattern-oriented modelling" strategy discussed in Section . can provide the rationale for critical design decisions such as what variables and scales are used and how agent behaviours are represented. .
The lesson learned from this experience is that describing the rationale for an ABM's design in ODD has substantial benefits, although there are likely to be situations in which an ODD should be limited to just model description.

ODDs are o en long and must be summarized .
A third factor that appears to limit use of ODD is that ODDs tend to be long for non-trivial ABMs. This is only partly due to the hierarchical structure of ODD, which implies some redundancy, and to the "Design concepts" section, which adds discussion of the model's underlying rationale that is not absolutely essential for reproducibility. The main reason for this length is that most ABMs are more complex than typical mathematical models, so a model description that contains all the details necessary for replication can easily lead to description that are -or more pages long. It also simply takes more space to describe an algorithm than to provide an equation. Therefore, ODDs o en seem longer than ad hoc model descriptions simply because ODD requires model descriptions to be more complete. ODDs are also less succinct than the specification languages that exist in certain ABM communities (e.g. Z-notation; Wooldridge ), which are cryptic to untrained readers. Because ODDs aim to be understandable directly by everybody, they are written in plain English and thus inherently longer than a specialized formalization language with predefined syntax. .
As result of their length, complete ODDs o en end up as supplements to journal articles and are not necessarily reviewed or read as o en or as carefully. Authors may thus feel that the e ort required to write an ODD is not justified, and instead provide only a summary description of their model in the article and, for example, o er their program code on request. This approach is not satisfactory because a full model description is a conditio sine qua non for the use of ABMs as a scientific tool (Grimm et al. , ). A more satisfactory approach would be a complete ODD accompanied by an e icient, standard way to summarize it for journal articles. Until now, there has been no guidance for e iciently producing a summary ODD that is both concise and precise enough for the main text of a journal article. So far there are only ad hoc solutions for such summary ODDs (e.g., Ayllón et al. ; Phang et al. ; Weiss et al. ), which di er widely in structure and depth.

ODD lacks a hierarchical approach for highly complex ABMs
. ; Gallagher et al. ). For such models, ODDs are longer than -pages and o en have submodels which by themselves require or more pages. The development of each complex submodel could benefit from its own ODD-like protocol to, e.g., document the patterns used as criteria for evaluating the submodel, the submodel's entities and processes, and how the design concepts were implemented. However, ODD does not explicitly include or encourage a hierarchical approach in which certain submodels can be described in a format similar to an entire ODD.

Creating a new ODD when re-using parts of a model is ine icient .
Good models are o en re-used by creating new versions that are applied to new problems; and experienced modellers o en borrow theory or other model components from existing models. In fact, facilitating re-use is a primary motivation for ODD and other standardization e orts. A number of ODD users have therefore encountered the issue of how to prepare an ODD for a new version of an existing model, or for a model that includes parts taken from a previous model. Is it necessary to produce a completely new ODD for each version of a model? Or can we instead prepare a "delta-ODD" that describes only the changes made from the previous version? When we re-use parts of an existing model, can we simply cite its ODD, or should we copy the relevant parts of its ODD, or must we re-create those parts? Re-creating an existing ODD certainly seems ine icient and a potential risk for introducing (but also identifying and correcting) errors.
. While the best way to describe a model modified from one that came with an ODD will depend on the situation, the lesson is that standard guidance on doing so could benefit both authors and journal editors.
ODD still seems ambiguous .
Another reason why some modellers may not use ODD is that they perceive it as too narrative in format to be complete or precise enough to make models reproducible (Amouroux et al. ). In part, this perception is due to the characterization of ODD as a "written" format, which could be misinterpreted as "purely verbal". Of course, all relevant equations and algorithms of a model must be included in an ODD, but even written descriptions with text, pseudo-code, and equations may not contain all the details required for unambiguous replication. Many modellers believe, with good reason, that reading and understanding a model's computer code is the only completely reliable way to understand what it does. Unfortunately, nothing about ODD so far has facilitated use of the computer code as another, and sometimes essential, way of describing models. Until now, ODD has focused entirely on describing models so completely in text that it is unnecessary to read their code. This focus has been intentional, because a complete model description is essential for tasks including so ware testing and because computer code can be at least as di icult to understand as a written description. But links between ODD and code could provide a lifeline for readers determined to fully understand exactly how a model works when the written description is ambiguous. .
Interestingly, there is also a contrary view on narrative model descriptions because di erent disciplines have di erent communication cultures. While ODD can be considered too narrative or verbal in the natural sciences, the opposite can be the case in the social sciences. In social sciences, more than in natural sciences, there is typically a narrative that runs through each model element (the choice of agents, processes, model purpose, parameters, etc.). Social science modellers o en construct their models from this narrative, and through this narrative they communicate their models to the readers. ODD with its separate sections can make it di icult for the reader to see the common thread, i.e., the story that connects them all. This challenge reflects a fundamental di erence between natural sciences and sociology in particular, the former being based on the notion of entities and variables, while the latter is based on the notion of discourse and communication. While ODD originated in the natural sciences style, there are ways that it can better present a model as a narrative, and doing so could make the ODD more e ective and the model more understandable.
. The lesson is: the extent to which ODDs should be narrative vs. mathematical and technical can vary among disciplines, and ideally ODD can accommodate this variation. One obvious way to meet ODD's goal of providing complete description even in more narrative descriptions is to help readers find the computer code implementing each part of the model.

ODD is not clearly linked to pattern-oriented modelling and does not describe criteria for a model being fit for its purpose .
As discussed in Section , ODD has developed into a valuable tool for designing ABMs; but it is not the only such tool. "Pattern-oriented modelling" (POM; Grimm & Railsback b; Grimm et al. ; Railsback & Grimm ) is a strategy for basing the design, evaluation, and parameterization of ABMs on patterns observed in the real system addressed by the model. In brief, POM involves identifying observed patterns that characterize the model system with respect to the model purpose, i.e. are driven by the mechanisms, occur at the scales, etc., believed important for the model purpose. Once the patterns are identified, POM then consists of (a) selecting model entities, state variables, and scales so that the patterns could be reproduced; (b) testing "theory" for agent behaviours by showing whether alternative submodels for behaviour cause the patterns to emerge from the model; and (c) using quantitative patterns to identify useful parameter values (Wiegand et al. ). Railsback & Johnson ( , ) provide a particularly explicit example of POM. .
Presumably because it addresses three of the biggest challenges in agent-based modelling (How do I find the right level of detail? How do I model agent behaviour? How do I estimate parameter values?), POM appears to have become widely used. Vincenot ( ) determined that the primary publication on POM (Grimm et al. ) is, like the papers describing ODD, among six "fusion papers" that contributed most to unification of agentbased modelling across disciplines. Even when POM is not used explicitly, observed patterns are almost always important in model design. Very o en, the purpose of an ABM is to explain patterns observed in reality, so these patterns are important criteria for whether the model is useful. The simplest toy ABMs that do not pretend to represent specific real systems are still based on general patterns observed in reality; very common examples include equilibrium population dynamics, segregation in cities, and flocking in birds. Identifying such patterns explicitly in the ODD helps its authors answer important questions such as: Why was the model designed as it was? What criteria were or could be used to determine whether a model is suitable for the stated purpose? .
However, integrating POM with ODD, such as by describing in an ODD what patterns were used in what ways to design the model, has proven frustrating: there is no specific place in ODD for emergent patterns, and it is not obvious where they should be described. The element "Design concepts/Theoretical and empirical background" of ODD+D (Müller et al. ) includes references to patterns, but we suggest to make this link more prominent by referring to them early on in an ODD. Moreover, because the same observed patterns are used in multiple ways, it seems e icient to describe them early in an ODD so they can simply be cited as needed in later sections. The lesson is that ODD needs a place to describe the patterns used to design and evaluate a model, and that place should be before describing things like entities and state variables that may be chosen using the patterns as a basis.

Improving Clarity, Replication, and Structural Realism with ODD
. This section describes the proposed solutions to the issues and challenges identified in Section . In some cases, solutions have been implemented in revisions to ODD and updates to its supplements, especially a completely revised ODD guidance and checklist.

ODD learning materials and guidance .
To address the issues described in Sections . , we thoroughly updated the ODD learning materials (Supplement S ). For each ODD element the learning material provides an ( ) overview explaining the scope and rationale of this element, ( ) explicit guidance, e.g. "Do not confuse parameters with state variables" and corresponding explanations, and ( ) short examples taken from existing ODDs. These examples are for illustrative purposes only, so they should not be copied but used as guidance. Separate guidance and checklists are provided for each of the design concepts of ODD's element " . Design concepts". As an example for the structure and content of the guidance provided in Supplement , Table presents the overview, guidance, and checklist for the ODD element " . Pattern and purpose".

Inclusion of rationale .
Section . discusses the benefits of including in an ODD information about what model design decisions have been made and why. We can envision circumstances in which users may decide to restrict an ODD to only description and not rationale, e.g., in the user documentation of a widely used model or when describing a model developed by someone else, e.g., those in the NetLogo models library (Wilensky ). However, in our new guidance we now explicitly encourage discussion of the rationale behind each ODD element. Each ODD element includes the optional subsection "Rationale" where information about why a certain model design was chosen can be provided to add credibility to the model's design and help readers to better understand, and possibly re-use, the model design. .
Including the rationale for model design can o en substantially increase the length of an ODD: it can take many pages to explain the literature review, data analysis, analysis of alternative approaches, etc., that lead to a model design decision. The following subsection addresses ways to deal with problems caused by ODD length. An alternative, but not mutually exclusive, way to document the rationale underlying a model is to produce a TRACE document, which includes justifications of all major elements of a model's design (Augusiak et al. ; Grimm et al. ). Two example TRACE documents are provided in Supplement S .

Summarizing ODDs .
Sections . identifies the length of many ODDs as a challenge because o en only a summary can be included in the main text of a publication. Including the rationales for model design as recommended in Section . can aggravate this problem by making ODDs considerably longer. We can provide some guidance, from our experience, for producing summary ODDs suitable for the main text of journal publications. This process introduces more of the narrative style of the social sciences (Section . ) to enhance overall comprehension of a model, with detail intentionally relegated to the full ODD. .
The purpose of a summary ODD is to provide a narrative description of the entire model and at the same time be specific enough that the main results of the model can be understood without necessarily resorting to the full ODD model description. We recommend always writing a full ODD first, whether it can be included in the main text of a document or needs to be included in the supplement.
. The summary should start with the three overview elements ( Figure ) brought into a more narrative, storylike form. Section titles should be omitted and long lists of state variables moved to tables. Entities can, if this improves the narrative, be described together with the processes they execute. However, to help readers find further details in the full ODD model description, keywords from ODD should be used and italicized, in particular: purpose, entities, state variables, scales, processes, schedule, design concept, initialization, and submodel. The only design concepts and submodels that should be presented in some detail are those essential to understanding the main idea of the model and the application addressed by the publication. The ODD elements "initialization" and "input data" should be briefly described if they are essential, e.g., if scenarios with di erent initial conditions are implemented or if input data are essential drivers of model dynamics. If external drivers are key elements of the question addressed with the model, for example when exploring e ects of climate change, it should be said here where in the model and how their e ects are represented. .
If the resulting summary model description still does not fully capture the overall narrative of the model, which might be the case for certain model purposes and within certain areas (social sciences), the overall narrative of the model might be presented first, without reference to the terminology and structure of ODD, but then the summary ODD and a link to the full ODD should follow. In any case, it would be worthwhile considering a graphical representation of the model's ODD; an example of such a "visual ODD" is provided in Figure   . General advice for describing models through narratives is the same as for scientific writing in general, where the context is provided first before presenting something new (Gopen & Swan ). In Supplement S , we provide a template for writing summary ODDs, and examples of summary ODDs.

Hierarchical ODDs of highly complex models .
If an ABM includes submodels which by themselves require or more pages of description, we recommend writing "nested ODDs": the submodel is described largely in the same way a full ABM is described, by its own, slightly reduced, ODD which should include the Section "Purpose and patterns", Section "Process overview and scheduling", and Section "Submodels". The other ODD elements, "Entities, state variables, and scales", "Design concepts", "Initialization", and "Input data" should still refer to the entire model, not to specific submodels. .
Further means for keeping ODDs of highly complex models readable and useful are: ( ) group parameters according to the submodels in which they are used rather than providing a single large table of model parameters, which are otherwise presented at the beginning of the "Submodels" element; and ( ), if numerous equations are used, summarize these in tables and explain the rationale of each equation in the text. This disentangling of equations and text is more concise, provides a better overview, and is easier to read and understand (e.g., ODD of Galic et al. ). In Supplement S , we provide an example of a nested ODD (Gallagher et al. ).
Developing ODDs when re-using parts of existing models .
How can modellers develop an ODD e iciently when parts of the model are re-used from other models or when producing a new version of an existing model? How can existing ODDs be re-used? Our recommendations depend on the exact situation.
. It is certainly e icient to describe a new version of an existing model by preparing a "delta-ODD" that describes only the parts of the model that have been changed. The "delta-ODD" identifies the ODD elements that have changed and provides new description for those elements; examples are given throughout the textbook of Railsback & Grimm ( ) and in Railsback & Harvey ( ). However, this approach is appropriate only when the ODD of the original model is readily available to anyone attempting to use the delta-ODD, for example by being re-published as a supplement or freely available on the Internet with a link provided in the delta-ODD.

.
Our experience has been that the "delta-ODD" approach is o en not appropriate or feasible because: the original ODD is not freely and easily available, or a journal expects model-based publications to include complete, stand-alone model descriptions, or the new model includes only part of the previous model, with many new parts. Therefore, we o en need to prepare a new, complete ODD for a model that is partly, or largely, the same as a previously described model. Making it e icient to prepare such ODDs is important for encouraging standardization and re-use of models and model components. .
The most e icient way to prepare a new, complete ODD based in part on a previous model is simply to copy the relevant parts of the previous model's ODD, giving full credit to the original authors and making it clear which work is theirs. Supplement S includes an ODD of a model that was adopted and modified two times, by slightly di erent teams of authors (Johnston et al. , , ); a further ODD of a modified model is presented in Supplement S .

.
To do so of course requires that copyright issues be clarified, and there are steps that authors can take to make their ODDs easier for others to re-use. Copyright concerns apply to ODDs published in open access repositories as well as to those published as supplements to proprietary journals. For most journals, supplements are not subject to the copyright of the journal as long as the journal has not invested in them via their layout or any other way, so their copyright remains with the author. Therefore, authors of ODDs published as supplements to journal articles can, at the time of publication, give permission for others to later re-use parts of the ODDs, but they should carefully check the journal's Copyright Transfer Agreement before they do so.
. We recommend that ODD authors add a license, e.g. in a footnote that sets out the terms of use of the ODD in the same way that terms of use should be stated for the model's so ware. This license should include a "copyle " statement which requires that future users of an ODD or parts of it do not restrict its use by their own license statements. Standard licenses are widely used in open so ware communities, for example GNU or more generally UNIX developers (one common so ware licence is discussed at: https://www.gnu.org/licenses/ fdl-1.3.en.html). They allow others to build on existing work and reuse ODD text while providing full credit to the original work through references. Some of these licenses also include a disclaimer of liability, (i.e. "no warranty" disclaimer, e.g. clause of the CC-BY-SA), which protects the creator against legal action in case the ODD does not perform as expected. In Supplement S , we provide an example license and links to additional licensing information, but each author must make their own decision on whether and how to allow re-use of ODDs and so ware. For example, you might want to restrict the license to the factual parts of the ODD, but not to the narrative ones. .
When preparing an ODD based on that of an existing model, it is critical to clearly indicate that the ODD includes material from a previous one, give full credit to original authors, clearly distinguish the re-used from original work (e.g., by setting new and deleted text in di erent fonts or colours), and ensure that the new ODD is published under terms that do not violate the original material's license (reciprocity requirement especially).

Reducing ambiguity by linking ODD to code .
To many simulation scientists, a model's computer code is its most authoritative and unambiguous description. Therefore, providing clear links between ODD and the computer code can make a model seem more transparent and its description less ambiguous. If readers can easily find and read the code that implements any part of an ODD, they are more likely to thoroughly understand and even replicate the model. Most programming languages are relatively similar and seem understandable over the few lines typically needed to code ABM algorithms or submodels, so even readers unfamiliar with a language can try to understand an algorithm from its code and check whether it matches the ODD narrative. Here, we recommend ways to provide such links, which first requires making a model's code (appropriately licensed and copyrighted) available with its ODD. Most journals do not require code to be made available but doing so is widely considered good scientific practice in line with current trends towards open science.

.
There are well-known limitations of publishing model code. Teams that have invested years in the development of complex models certainly want to benefit from this investment before others do; however, code for key algorithms and data structures could be provided instead of the full program. Furthermore, a model's code itself is not always su icient for understanding exactly how it executes; information may also be required about the compiler or interpreter, code libraries, any numerical solution methods (Seppelt & Richter ), and even the hardware and operating system. Therefore, we recommend that at least revision numbers of external soware/library, architecture (e.g. x , / bits) and operating system version always be provided. Further, computer code can be misinterpreted, and important so ware details can be specified in places other than the code statements; especially, the popular NetLogo platform provides many extremely useful primitives that must be fully understood (almost always possible from their documentation) to understand a model code. .
One way to link ODD descriptions to computer code is simply through careful naming conventions. Using the same names for variables, parameters, and submodels in both ODD and code makes it easier to find the code implementing specific parts of a model. Similarly, code comments can be used to identify where specific ODD elements, algorithms, or even numbered equations, are programmed.
. Links between ODD and code can also be more comprehensive and specific. Becher et al. ( ) provided both the ODD model description and the computer code for their complex model of honey bee colonies, BEEHAVE, in a single file that includes hyperlinks from the ODD to the corresponding code. Another potential approach is to add notes or a table to the ODD to identify the code locations (file, procedure or function name, or even line number) where each ODD element, submodel, or algorithm is implemented.
Linking ODD to pattern-oriented modelling .
In Section . , we identified the important benefits and popularity of POM as another important tool, alongside ODD, for designing ABMs and documenting their validity and credibility. We also identified the problem that there has not yet been a specific place in ODD where the observed patterns used in POM should be stated. We therefore supplement the ODD element " . Purpose and patterns" to explicitly include the patterns that are used in POM to design and evaluate the ABM or otherwise serve as criteria for whether the model is realistic enough for its intended purpose. The first ODD element was chosen as the location where these patterns are stated for two key reasons. The patterns can be used, via POM, to select the entities, state variables, and scales of a model, so the patterns need to be identified before these components of model structure are described and justified in ODD element . The patterns are identified with the model's purpose because purpose and patterns are tightly linked: explaining the patterns is o en part of the model's purpose, and we use the patterns to determine whether the model is suitable for its purpose. Furthermore, in the new ODD guidance we recommend that the overall purpose of the model be explained in more detail, as di erent purposes imply di erent criteria for the design and evaluation of a model. In addition to the specific purposes of a model, it is also helpful to clearly state the general type(s) of the model's purpose. General types of model purpose can be described in broad categories such as understanding/explanation, prediction, and demonstration of ideas (Roughgarden et al. ), but there are also more refined categories, which strongly a ect how a model is designed, analysed, and to be evaluated (Edmonds et al. ). These more specific categories seem to be most relevant in social sciences where models are o en used to illustrate narratives rather than to represent a specific, empirical system.

.
Examples of this new " . Purpose and patterns" ODD element are provided by Railsback & Grimm ( ); this textbook includes patterns in the first ODD element for seven ABMs. These examples include some very simple models, not specifically related to any real system, for which observed patterns were nonetheless important in establishing the model's purpose. One important caveat with POM is the need to also report patterns observed in reality that were considered essential prior to model development, but which the model consistently failed to reproduce. Reporting, in the first ODD element, only those patterns that the model could capture would resemble "HARKing" (Hypothesizing A er Results are Known). Better practice would be to report on missing patterns, which would indicate that important processes were not yet identified, or understood. Such reporting would prime the reader for topics that will likely be covered in the discussion, promoting readability of the article as a whole.

Outlook
. Here we outline possible future developments and applications of ODD. Each would require careful consideration and testing by the modelling community. We hope that these and other ideas to increase the usefulness of ODD and related approaches will be pursued not only by ourselves, but by the entire modelling community.

ODD for non-agent-based simulation models
.
Only a few parts of the ODD protocol apply specifically to ABMs. Those parts are all in the " . Design concepts" element, especially emergence (of behaviour), adaptation, objectives, learning, prediction, sensing, and collectives. Other ODD elements are relevant to any simulation model. Thus, it seems possible to describe any simulation model with ODD.
. ODD has in fact already been used for models that are not ABMs. ) used ODD to describe an ecological-economic model of pastoral-nomadic range management that is not agent-based and employs di erence equations as submodels. Lamonica et al. ( ) similarly used ODD to describe a model that is fully based on ordinary di erential equations.
. Besides these, ODD has also been used to document hybrid models (Vincenot et al. ) in which ABMs integrate other modelling approaches. For instance, DEB-IBM (Martin et al. , ) is an ABM of water fleas (Daphnia) in which the energy budget of individuals is modelled using Dynamic Energy Budget theory (DEB; Kooijman ), which is formulated via ordinary di erential equations. Similarly, ODDs were produced for several System Dynamics (SD)-Individual-based (IB) hybrid models, which took advantage of SD stock-andflow modelling to render continuous processes in ABMs and visualize feedback loops. This approach was used to explain spatio-temporal patterns in plant communities (Vincenot et al. , ), reconstruct cell-based morphogenesis mechanisms (Hay Mele et al. ), render memory e ects in spatial resource use by foragers (Vincenot et al. ), study the e ect of population structure on epidemic resurgence (Vincenot & Moriya ), and simulate lake restoration scenarios (Martin & Schlüter ). Large hybrid models o en couple existing mathematical models of environmental systems with ABMs or other simulation models of the human component, e.g., combining models from hydrology, vegetation science, and social science to address land use change (Drogoul et al. ; Janssen et al. ). In this context, hybrid models are increasingly used to address the feedbacks among environmental compartments that can no longer be ignored in times of rapid global and regional change (Ayllon et al. ). .
All the foregoing models formulated based on di erential equations or algorithms (e.g. decision trees) or both were relatively seamlessly integrated into ODDs describing complete models. Because simulation modelling in general is much more mature than agent-based modelling, its existing literature on model reporting (e.g., Monks et al. ) should be considered in adapting ODD to non-ABM simulation models. ODD already appears useful in its current form in the particular case of hybrid agent-based models (or more generally modular models coupling interacting self-standing submodels), but experience may identify modifications or supplements to ODD to improve its value for these increasingly important models. We can already recommend adding a subsection in the ODD element "Submodels" to describe under which framework submodels were coupled, how and when they interact, and which the particular variables and processes are through which data exchange and synchronization takes place ("hooks" between the submodels; see for example appendix in Vincenot et al. ).

ODD and description of simulation experiments .
An ODD corresponds to the "Materials" part of the Materials and Methods section of a scientific publication because it describes the virtual laboratory in which we conduct simulation experiments. The "Methods" equivalent then must describe how we used the materials -the model -in simulation experiments. Previous publications on ODD recommended that an ODD be followed by a section entitled "Simulation Experiments" but provided no further guidance.
. We debated expanding ODD to include a protocol for describing simulation experiments but chose not to do so. A protocol for describing simulation experiments could be used by authors as a checklist to ensure that important elements of model calibration and analysis are documented, and readers would know where to expect what kind of information, just as with ODD. However, standardizing the description of simulation experiments could be just as complex as ODD currently is, and deserves separate, in-depth consideration (see, e.g., the recommendations of Waltemath et al. ). TRACE, though, is a format for supplements, not for the main text. What might be needed is a format corresponding to TRACE but which is suitable for main texts. While we cannot provide a full-fledged solution here, in the Supplement S we provide a possible template; its elements are adopted from TRACE, and they are explained in detail in Grimm et al. ( ).

Automated links between ODD and code .
Model development should always be a process that begins with written documentation that is carefully discussed, reviewed and revised; then, at some point, the model design must be "frozen" and implemented in a computer program, with the written description updated as so ware development identifies mistakes and ambiguities. Automated links between written description and so ware are naturally an ideal -some or all of the so ware could be produced from the written description and perhaps the description could be updated automatically when code is modified. We are unaware of active e orts to produce such links between ODD and ABM so ware, but technically, it seems possible to write ODDs in such a way that they can be automatically converted into the backbone of the corresponding computer code. ABM is a direct form of object-oriented formalization, and thus related models are mostly implemented using object-oriented programming (OOP). This property is visible in ODD's descriptive structure, in which for instance "entities", "state variables/attributes" and "processes" are clear equivalents to OOP classes, attributes and methods. As a result, the standardized list of entities, attributes and processes of an ODD could be used to generate their counterpart in OOP code skeletons (e.g. Java classes). Similar automated processes already exist for many OOP languages in the case of UML to code conversion.
. Several cautions are in order in pursuing the goal of linking ODD and code. First, care must be taken that ODD is still written for people, not computers. Only verbal, non-technical descriptions of a model force us to try to understand what a model is, how it works, and why it was designed in a certain way. For example, markup languages can be read by people but their main purpose is to be read by computers, which means they do not su iciently force us to think about the model. Second, our experience with several ABM platforms that attempted to partially automate development, e.g., by providing graphical tools for outlining so ware, were that these were useful mainly for tasks that are relatively easy anyway and did not eliminate the more challenging tasks of coding each model's unique details. Third, the semantic links between classes in object-oriented programming and everyday meanings in natural language are not as trivial or straightforward as may at first appear to the naïve programmer (LaLonde & Pugh ; Polhill ; Polhill & Gotts ).The code template produced from the model's design is thus not necessarily the best (most e icient and/or readable) way to implement the so ware. .
Fourth, the practice of preparing and 'locking o ' design documentation prior to implementation in code echoes the rather out-dated 'waterfall' approach to so ware engineering (Adenowo & Adenowo ). This might work fine in mono-disciplinary or similar contexts where there is a small team of people with similar expertise; it will be less suitable in inter-and especially transdisciplinary contexts, where more iterative and 'agile' approaches to so ware development are generally seen as more e ective (e.g. Étienne ; Moyo et al. ). .
Another option worth considering for ODD is "literate programming" (Knuth ) where "programs are written as an uninterrupted exposition of logic in an ordinary human language, much like the text of an essay, in which macros are included to hide abstractions and traditional source code." Organizing ODD's maintenance and development .
So far, the use of ODD as the standard format for describing ABMs has been promoted by a small but diverse group of experienced modellers. The original ODD publication (Grimm et al. ) asked users to cite it, which allowed us to monitor how coherently and e iciently ODD was used and therefore produce the update and this article. .
One long-term issue for ODD is who will continue to maintain and update it. Although ODD is used in a textbook (Railsback & Grimm ), recommended by the CoMSES/OpenABM network (https://www.comses.net/), and some journals (e.g. JASSS, Ecological Modelling), it is worth considering alternative ways to organize the maintenance and development of ODD and related standards. Existing networks, such as CoMSES, might be appropriate if they are not domain-specific. .
A second issue is monitoring and promoting the quality of ODD applications. One way to improve the coherent use of ODD could be by allowing ODDs to be reviewed and certified by users who have undergone training and produced high-quality ODDs themselves. "O icial" ODD certifiers could be qualified by, e.g., attending one-day courses, submitting their own ODDs, and being "examined" by reviewing test ODDs (for example in a quality check process similar to what is implemented in CRAN; Cran ).

Discussion
. ODD has been used much more widely than anticipated by its original developers, which indicates that a standard format for describing ABMs, and possibly other simulation models, is needed and useful. The initial success of ODD convinces us that it should be used even more, and more coherently, in the future. ODD is a positivefeedback technology: the more it is used, the more valuable it is to its users and scientific communities. ODD has already contributed to unifying agent-based science by connecting the previously separate bodies of literature that used the terms "agent-based" and "individual-based" (Vincenot ). Further benefits include facilitated comparison, linking, and reviewing of models (e.g. Berger et al. ) and the re-use of useful and validated submodels of particular behaviours ("pattern-oriented theory development"; Railsback & Grimm ). .
We have summarized issues and challenges that could prevent wider and more coherent use of ODD and have presented possible solutions. We also provided an outlook on possible future developments. Overall, by promoting and improving ODD we hope to contribute to the maturation of agent-based modelling as a scientific tool (Lorscheid et al. ). The language of mathematics developed over hundreds of years, so we cannot expect a lingua franca for ABMs, or simulation models in general, to emerge within a few years. .
A major, and perhaps most important, part of this article is the supplements, which contain guidance, templates, and examples. We therefore summarized these supplements in Table . We encourage developers and users of ABMs to use this material and thereby contribute to the development of ODD and related standards. Feedback and new ideas are welcome as well, for example via the forum on CoMSES.net. We also ask users of ODD to please continue to cite the ODD publications (Grimm et al. and this article) so that the use of ODD can be monitored.