Improving Execution Speed of Models Implemented in NetLogo

: NetLogo has become a standard platform for agent-based simulation, yet there appears to be wide-spread belief that it is not suitable for large and complex models due to slow execution. Our experience does not support that belief. NetLogo programs often do run very slowly when written to minimize code length and maximize clarity, but relatively simple and easily tested changes can almost always produce major increases in execution speed. We recommend a five-step process for quantifying execution speed, identifying slow parts of code, and writing faster code. Avoiding or improving agent filtering statements can often produce dramatic speed improvements. For models with extensive initialization methods, reorganizing the setup procedure can reduce the initialization effort in simulation experiments. Programming the same behavior in a different way can sometimes provide order-of-magnitude speed increases. For models in which most agents do nothing on most time steps, discrete event simulation – facilitated by the time extension to NetLogo – can dramatically increase speed. NetLogo’s BehaviorSpace tool makes it very easy to conduct multiple-model-run experiments in parallel on either desktop or high performance cluster computers, so even quite slow models can be executed thousands of times. NetLogo also is supported by efficient analysis tools, such as BehaviorSearch and RNetLogo, that can reduce the number of model runs and the effort to set them up for (e.g.) parameterization and sensitivity analysis.


Introduction
. Agent-based models (ABMs) have become essential tools in social (and other) sciences, and NetLogo (Wilensky making it easier to test both the so ware and model design. Therefore, NetLogo appears to have a reputation as especially suited for relatively simple ABMs intended mainly to communicate ideas. But for computationally intensive "serious" ABMs, there is a belief that NetLogo's execution speed is such a constraint that models will need to be re-implemented in lower-level languages (e.g., Sklar ; Bouquet et al. ; Lammoglia et al. ). .
This belief that NetLogo is inherently unsuited for large models is not well supported (Tisue & Wilensky ). The belief likely originated in part with the understanding that NetLogo and the Java language it is based on are interpreted instead of compiled. This understanding is no longer as meaningful as it once was: many parts of NetLogo are in fact compiled (Sondahl et al. ); and modern versions of languages that run on the Java virtual machine are no longer considered seriously slower than other languages (Wikipedia ). Railsback et al. ( ) compared version . of NetLogo to other ABM platforms and found its execution slower, but not dramatically slower, than the fastest (RePast and MASON). That comparison was made before NetLogo was converted from an interpreted to mainly compiled language (Sondahl et al. ). .
In fact, people deciding which platform to use for large ABMs need to be aware that NetLogo's compiler now includes a number of optimizations and clever designs (Sondahl et al. ; CCL (Center for Connected Learning, Northwestern University) ) that could o en make NetLogo faster than platforms lacking such attention to execution speed. Especially, NetLogo's optimizations are likely to make it faster than simple code written in a standard programming language by an inexpert or time-constrained programmer. As an illustration, Lytinen & Railsback ( ) found version . of NetLogo about times faster than the "ReLogo" element of RePast (Ozik et al. ) for several example models of modest complexity.
. Even though NetLogo does not appear inherently too slow for serious modeling, our experience using and teaching agent-based modeling indicates that many NetLogo programs can in fact be very slow until their timeconsuming "bottlenecks" are found and remedied. We encourage students to write the first versions of their so ware in NetLogo's natural style, using its primitives and characteristic code structures to make program statements as simple and understandable as possible. This approach allows models to be programmed and tested rapidly, but it is not uncommon for the initial so ware to be prohibitively slow. However, we have also found that such prohibitively slow programs can almost always be sped up, o en by orders of magnitude, with a few simple changes. A few common NetLogo statements are o en the bottleneck, not because the NetLogo primitives are poorly designed but because they require extensive computations that would be time-consuming in any programming language. Using the strategy and techniques we provide here, speeding up a NetLogo program usually takes, in our experience, less than a day and the changes can easily be tested to find any mistakes that were introduced. However, one technique for producing computationally e icient models, discrete event simulation, a ects the entire model design and needs to be considered from the start.

.
Our first objective here is to describe a process for finding execution speed bottlenecks in NetLogo code and techniques that o en speed up execution. We present a five-step strategy that includes ways to quantify execution speed and a set of o en-e ective techniques. We provide NetLogo programs illustrating some of the techniques online.
. As a second objective, we also present ways of making large simulation experiments on NetLogo models feasible. Once a model has been developed, tested, and made computationally e icient, it is then analyzed via simulation experiments to understand both the model and the system it represents. We address two tools for implementing simulation experiments: NetLogo's "BehaviorSpace" experiment manager and the popular "RNetLogo" package that can run NetLogo models from the R statistical package (Thiele et al. , ).

.
We assume that readers are familiar with the basics of NetLogo and therefore make reference to standard parts of NetLogo (e.g., "patches" as square grid cells; "turtles" as mobile agents; "primitives" as the programming language's built-in commands) without explaining them. We provide example code statements to illustrate the issues and solutions we describe; text in Courier font is NetLogo commands (primitives) or code. The tests we used to find and evaluate potential speed increases were all conducted in version . of NetLogo.
. Example NetLogo models illustrating techniques we describe are available for download at http://www.railsbackgrimm-abm-book.com/JASSS-models.html (archived at: "http://www.webcitation.org/ nFYwsz C). References below to an "example model" refer to NetLogo files available at this site.

A Strategy for Producing E icient NetLogo Code
. We advocate the following five-step strategy for building an e icient NetLogo model. Details on conducting steps -are then provided in following sections. This strategy depends on consistent use of one or several test scenarios, each being a carefully selected and documented model version; set of inputs (input files, parameter values); model settings such as which output files are written, the rule for when the model stops, and display settings such as when the View is updated; and hardware -the computer the tests are executed on. These scenarios might include a short test case, a typical simulation, or an especially demanding extreme case.
. Write the initial so ware in NetLogo's natural style, making program statements as simple and understandable as possible. (However, one technique for making models e icient, discrete event simulation, should be considered from the start.) Test the so ware thoroughly to find the inevitable mistakes before proceeding further, e.g., by using methods described in Chapter of Railsback & Grimm ( ). (Common programming mistakes can themselves slow down execution immensely, so code testing o en produces major speed increases. However, we do not address e ects of programming errors, instead focussing on improving code that has already been tested.) . Measure execution time of the initial so ware under the test scenario(s) discussed above, using NetLogo's timer.
. Try simple speed-up methods that do not require code changes and determine whether they result in acceptable execution times. Even slow execution speeds may be acceptable if computer time is less important than the programming time needed for further speed-up steps.
. If further improvement is needed, use NetLogo's profiler extension to identify the slow procedures.
. Attempt to speed up the slow procedures using several techniques which do require (usually minor) changes to the program. As these methods are used, repeat the execution time measurements to determine how much improvement has been made. A er each technique is implemented, test the code to eliminate any errors that were introduced by showing that it produces the same results as the initial version.

Measurement of Execution Speed and Detection of Bottlenecks
. An essential step to speeding up a model is measuring execution speed so that particularly slow parts can be identified and so that attempts to speed up execution can be evaluated quantitatively. NetLogo provides two tools for doing so.
. The first tool is the timer, used via the primitives reset-timer and timer. The timer primitive simply reports the clock time, in seconds, since reset-timer was last executed. It can be used, for example, to report the time taken for a full model run (as needed for steps and of the above strategy) by modifying the go procedure: The timer primitive can also be included among the reporters used to produce output for a BehaviorSpace experiment, so the execution time is a model output. (The example model "InRadius-vs-DistanceMyself.nlogo" illustrates this method.) When using timer to investigate execution speed, be aware that NetLogo code runs substantially slower the first -times it is executed a er being edited (possibly due to the Java virtual machine re-compiling the revised code and some hardware caching; Wikipedia ( )). Therefore, measurements need to be repeated until they produce consistent results.
The profiler extension packaged with NetLogo is also an essential tool for understanding and quantifying execution speed. It can be used as an alternative to timer for steps -of the above strategy for producing e icient code, and is required for step . The profiler (thoroughly documented in the NetLogo User Manual) reports the time spent in each procedure. In the profiler's output report, look for procedures with high values of "exclusive time" (the time spent executing code within the procedure); these should be the targets of e orts to speed up. (If the "go" procedure has high exclusive time, it can be because much of the execution time is spent updating the display. Re-run the profiler with "view updates" turned o .) Figure illustrates the profiler's value.
When measuring execution time, it is important to avoid misleading results due to hardware issues. Measurements used to compare versions of the code should, to be comparable to each other, be made on the same machine and in the absence of competition for computer resources (CPU availability, memory) with other soware. Use tools like the Windows CPU meter or Task Manager to make sure at least one processor core and ample RAM are available just for NetLogo.

Simple Steps to Increase Speed
. There are several steps that NetLogo users can take as soon as a model's program has been written and tested.
While these steps do not always result in substantial speed increases, they are simple and safe to try.
. First, make sure the NetLogo profiler is deactivated by commenting out the statements that use it. Profilers use up computer resources and by themselves slow a model down.
. Second, check and possibly adjust settings for the NetLogo view (the two-dimensional graphical display). Although these settings generally have no e ect when a model is run in BehaviorSpace with view updates turned o (as models typically are for serious simulation experiments), they can dramatically slow down performance of some models when the view is in use. Setting the view update to "on ticks" (once per time step) can speed up some models. Models that execute each tick very quickly can have their speed limited solely by the "frame rate" setting, which by default limits NetLogo to ticks per second. It is essential to understand the complex e ects of this setting, explained in the "View updates" section of NetLogo's programming guide.
. Third, try a -bit version of NetLogo. Starting with version . , NetLogo is distributed in both -and -bit mode. Some models run substantially faster (e.g., -% faster) in -bit mode (which can also allow models with more agents and larger spaces to run without exceeding memory limits). However, -bit NetLogo does not always help: the model of Ayllón et al. ( ), in which relatively few agents execute many calculations, actually took % longer to execute in -bit compared to -bit versions of NetLogo . .

.
Finally, when a model is ready for simulation experiments executed without view updates, users can comment out code statements that set agent colors and shapes that are strictly for display purposes. (

Techniques for Speeding Up Slow Code
. For many models, the above simple steps will not provide substantial execution speed benefits, so users must then revise their code to make the slow procedures more e icient. In our experience, NetLogo programs that run very slowly usually do so because of how key primitives -especially with -are used. In the following subsections we discuss how to avoid these primitives or use them more e iciently. We also identify some other code revisions that can provide substantial benefits for some models. These techniques were identified from our own experience and from the NetLogo Users Group. .
We remind users that all of these techniques are likely to introduce errors, which can be found by carefully comparing the modelâĂŹs results to those obtained before the revisions. Start by documenting one or several test scenarios (discussed above) and saving the output from them, including detailed output (e.g., file output generated every tick, perhaps via BehaviorSpace). Then make the code changes and, if they succeed in producing worthwhile speed increases, test whether the code still produces the same results that the original test cases did. (Whether the revised code should produce exactly the same results as the original can depend on whether random-seed is used to control the sequence of random numbers and whether the revisions result in subtle ). In (A) NetLogo's view was updated each tick, while it was not in (B) ("view updates" was turned o ). As a consequence, the go procedure used almost seconds ( , milliseconds of exclusive time) in (A) and only . seconds in (B). In (C), file output was turned on; generating over megabytes of output increased the time used by the update-output procedure by only seconds ( milliseconds exclusive time, compared to in A). The cost of file output is small, presumably due to hardware caching. In all runs, shade-patches was among the most time-consuming procedures; it is for display purposes only and could be turned o for simulation experiments, saving seconds per model run.
changes such as a ecting the order in which agents execute actions.) Debug the code changes by resolving the di erences between the new and test output.

Make Agent Filtering Statements E icient -Or Eliminate Them
. NetLogo's primitives for filtering (subsetting) agentsets are extremely useful for writing simple, clear code statements such as ask turtles with [ not happy? ] [ find-new-spot ] (from the NetLogo library's Segregation model). The with primitive examines an agentset (in this example, all turtles) and creates a new agentset containing those members of the first agentset that meet a criterion (here, their variable happy? has a value of false). Other examples of primitives that examine an agentset and create a subset of it are in-radius, max-n-of, max-one-of, and with-min. These primitives are o en combined in particularly powerful, but computationally demanding, statements such as min-n-of 10 (turtles with [size < 5]) [distance myself] (which reports the nearest turtles with size less than five). The ability to write such statements is a key feature of NetLogo, making it easy to identify agents with particular characteristics even as those characteristics change during a simulation.
. These filtering primitives are both commonly used and o en slow. When with or related primitives are used to filter a large agentset -e.g., all the turtles or patches in a large model -they must perform the calculations necessary to evaluate the subsetting criterion for each member of the agentset. When we look for ways to speed up a NetLogo code, we o en start (a er using the profiler to identify the slow procedures) by searching for statements that use with. We have experienced numerous models in which just reprogramming such statements has dramatically decreased execution time, o en by several orders of magnitude. .
The following subsections describe ways to make filtering statements faster or to avoid them.

By using global agentsets
.
When a subset of patches or turtles is used repeatedly in a model, it can be saved as a global variable instead of being re-created many times via with or other filtering primitives. For example, the co ee farm model of Railsback & Johnson ( , ) represents land uses via patch colors: forest is green, shade-grown co ee is grey, sun-grown co ee is yellow, etc. The model o en uses information for just one land use type, for example the number of birds in shade-grown co ee patches. Such information could be obtained via count turtles-on patches with [pcolor = grey]. It is far faster, though, to create global variables that each contain the agentset of patches of each land use type. These variables are initialized in the setup procedure via statements such as: set forest -patches patches with [ pcolor = green ] set shade -coffee -patches patches with [ pcolor = grey ] set sun -coffee -patches patches with [ pcolor = yellow ] .
Then the statement to count birds on shade co ee becomes count turtles-on shade-coffee-patches, which does not require NetLogo to look at all patches and identify the ones with grey color and, therefore, is far faster.
. This global variable technique is especially e icient in the co ee farm model because the land use types (patch colors) do not change during a simulation. However, it can still be very e icient even if the agentsets contained in the global variables do change, which requires updating the global-variable agentsets. If the model represented conversion of forest land to co ee production (some patches have their color changed from green to yellow), then the global variables would need to be updated. Usually this would be done by simply repeating the statements used to create them in the first place. As discussed below, this kind of change can be error-prone and should be tested carefully.
. Our example model "With-vs-global-vars.nlogo" illustrates the use of global variables to avoid filtering primitives. With the NetLogo view updates set to "on ticks", the version using patches with [pcolor = yellow] executed ticks in seconds. The version using a global variable for yellow patches executed in . seconds, a X decrease in execution time. With view updates turned o , the di erence was much more dramatic, a decrease of almost X in execution time.

By using agentsets as patch or turtle variables .
This technique is similar to the use of global variables to hold agentsets, but works when each patch (or turtle) needs to use an agentset that di ers among patches (or turtles) but does not change over time (or changes rarely, compared to how o en the agentset is used). In our example model "With-vs-agent-vars.nlogo", turtles move each tick to a randomly chosen patch that is green and within a radius of patches. The natural way for a turtle to identify the potential movement destinations is via the statement patches in-radius 10 with [pcolor = green]. This statement, however, requires NetLogo to examine many patches to determine if the radius and color criteria are met. (in-radius is cleverly designed to examine only the patches within a square defined by the radius.) Instead, during model setup, patches can be given a variable green-destinations that is initialized via: .
Then turtles on blue patches would use green-destinations as their set of potential destinations. In the example model, this change decreased execution time by . X with the view updated on ticks and by X with view updates o . .
Another example is the wild dog model described in Sect. . . of Railsback & Grimm ( ). In this model, "packs" are a NetLogo "breed"; each represents a group of dogs. A pack could refer to the dogs that belong to it via dogs with [my-pack = myself], where my-pack is a dog variable that identifies its pack; but it is faster for each pack to have a variable pack-members that is an agentset of all the dogs belonging to it. This agentset is updated when new dogs are born or older dogs leave the pack. (The alternative of modeling the dog-pack relation using links is discussed below.) By using local agentsets .
When filtering primitives like with are used more than once within a procedure to obtain the same subset, it is o en much faster to create a local variable that holds the subset. A common example is when we ask a subset of turtles or patches to do something, but that subset can sometimes have no members. In our example model "With-vs-local-vars.nlogo", red patches are rare. When we ask turtles to do this: move -to one -of patches in -radius 20 with [ pcolor = red ] .
we will get a run-time error whenever there are no red patches within a radius of . A solution is to tell the turtles to do something else (move to one of the green patches, which are common) if there are no red patches: .
In the example model, this technique reduced execution time from seconds to seconds for ticks (view updating had little e ect). .
To use this technique, search slow procedures for any filtering statements that are used more than once to produce exactly the same subset of agents. (But be aware that the same statement may not produce the same results later if the agents have changed.) If such statements are found, use let to introduce a new local variable that holds the subset of agents, and then use that local variable instead of repeating the filtering statement.
(But also consider whether a global or turtle/patch variable might be more appropriate.)

By using the table extension
.
In some models, each agent has a unique value of some variable, and we want to find the agent that has a specific value of that variable. For example, the frog model of Railsback et al. ( ) uses a unique patch variable cell-number: each patch in the model space has its own value of cell-number. Input data for such patches are referenced by cell number instead of by patch coordinates: e.g., the input data lists the cell number and ground elevation of each patch. Therefore, when setting up the patches from the input data file, we need to find the patch that has each cell number so we can set its elevation. The natural way to do this in NetLogo is: ask patches with [ cell -number = the -input -cell -number ] [ set elevation the -input -elevation ] where the-input-cell-number and the-input-elevation are local variables with values read from the input file. .
In cases like this when it is necessary to find the patch (or turtle) with a specific value of some patch (or turtle) variable, and that variable is unique (no two agents have the same value of it) and static, it can be much faster to use NetLogo's table extension instead of the with primitive. In our example with cell numbers, the .
It is very important to realize, before implementing this technique, that it does not allow BehaviorSpace experiments to run on multiple processors: each experiment must run on only one processor. Multiple processors can be used simultaneously only by opening multiple copies of the model in separate instances of NetLogo.
. The speedup technique is to separate model initialization into two stages that contain the procedures that do and do not need to be repeated to re-run the model a er the initial setup. Here, we refer to setup as the procedure that is used first to create the world and completely initialize the model, and reset as a procedure that re-initializes only those parts of the model that need to be reset between model runs. The trick is for reset to clean up and re-initialize things like turtles without using the primitive clear-all, which erases everything. The setup procedure can be organized like this: to setup ; a global procedure to fully initialize the model , used once . clear -all set -parameters ; the procedure that sets all global variable values build -world ; the procedure that reads in patch variables and sets up the World reset ; the procedure that re -initializes the model between runs end .
The reset procedure can then be organized like this: to reset ; a global procedure to re -initialize the model between runs ; check to make sure setup has been executed if some -global -parameter = 0 [ error " You ran reset before running setup !" ] ; manually clean up without clearing the World ask turtles [ die ] clear -output clear -all -plots reset -ticks ; now create turtles and do everything else needed to initialize the model ; including re -setting non -static global variables ... end .
With this organization, setup must be executed once at the very beginning, but new model runs can be initialized by only running reset. BehaviorSpace experiments can use reset as the "setup commands", as long as setup is manually executed before the experiment is started and only one processor is used.
. When model initialization is organized this way (but not in the standard way, with all initialization in setup, Be-haviorSpace experiments can safely vary parameters that are not on the Interface. BehaviorSpace changes parameter values before it executes its "setup commands", so with a standard setup organization the value given to a parameter by BehaviorSpace would be overwritten if the same parameter was also set in setup. With the reset organization shown here, global parameter values are initialized during setup, not in the reset procedure that we tell BehaviorSpace to use as its setup command. Therefore, parameter values set by BehaviorSpace are not overwritten.

Use State Variables Instead of Links
. NetLogo's "links" are objects that track a relationship between two turtles, o en used to represent networks: a turtle can use primitives such as my-links to identify and interact with other turtles it is linked to. However, the same link relationship can be modeled using turtle state variables instead of links: a turtle can have a state variable that is simply an agentset of the other turtles it is linked to. Such state variables must be updated by turtle procedures that are typically more complex and error-prone than simply creating or removing a link.
. The wild dog model code in Sect. . . of Railsback & Grimm ( ) uses this state variable approach. For example, each dog has a state variable for the pack it belongs to, and each pack has a variable containing an agentset of all the dogs that belong to it. As dogs leave their original pack and create new packs, the code must carefully update these variables. We have re-implemented this model using links instead of state variables: each pack creates a link between itself and each dog that belongs to it, and dogs that leave a pack destroy their link to it. The code using links is simpler and less error-prone than the original. However, the version using NetLogo links takes over six times longer to execute. (This model contains little other than managing links among dogs and their packs, so the cost of creating and destroying links dominates its execution speed.) .
This experience indicates that models in which many links are created and destroyed could potentially be sped up considerably by using state variables instead of links. The state-variable approach is more error-prone, so we recommend such models first be built using the simpler link approach and then, if necessary, converted to the state-variable approach and carefully tested against the original code.

Try Alternative Statements
. There are o en more than one way to code a particular function in NetLogo, and sometimes one way is substantially faster than another. One common example is an agent (turtle or patch) searching for other agents within some distance of itself. The most natural way to program this search is using the in-radius primitive, e.g., turtles in-radius search-radius. However, the same search can also be coded as turtles with [distance myself <= search-radius]. In our example model "InRadius-vs-DistanceMyself.nlogo", we found in-radius to outperform distance myself when the radius was small and the number of agents being searched large, while distance myself was much faster when the search radius was larger. The trout model of Ayllón et al. ( ), with approximately trout agents searching a few hundred patches, ran approximately four times faster using the distance myself approach, even though this search is only a small part of the trout behavior each tick. (The speed di erence between in-radius and distance myself has been discussed on the NetLogo user forum, with some users finding extremely large di erences in execution speed between the two alternatives.) .
Another example of speed di ering among statements that produce exactly the same results is the order in which boolean conditions appear in logical statements using and and or. For example, the statements how many turtles there are, where they are, and how many are red. The di erences between these statements in execution time are because they change whether NetLogo (a) first excludes turtles beyond a radius of and then checks the color of the remaining ones, or (b) first excludes all the non-red turtles and then checks the distance of the red ones.
. Di erences in execution speed among alternative statements that perform the same function are di icult to predict. Our advice is that if profiling shows a procedure to be particularly slow, then simply experiment with alternative code statements to see if they make a di erence.

E icient Model Design: Discrete Event Simulation as an Alternative to Time Steps
. Typical NetLogo models use time step simulation, in which all model actions are called from the "go" procedure, which is executed once each tick. In some models, many agents do nothing on many ticks, so the code must include conditional statements that determine whether or not each agent should execute some procedure on the current tick. For example, a model of vehicle tra ic must use a very short time step (a few seconds or minutes) to capture the movement of vehicles being driven. But at any time, most vehicles are parked and not moving at all; hence, the NetLogo code must ask each vehicle whether it is currently moving and then decide whether it should do anything. Over many agents and many ticks, just checking which agents should do something can become a major computational burden. .

Discrete event simulation (DES)
is an alternative to the time step method of organizing how events are scheduled in a simulation. DES is widely used (there are many books and specialized so ware platforms for DES), but not supported well by NetLogo. With DES, there is a global schedule that keeps track of simulated time, and events are added to the schedule with a specific time to be executed. Each event consists of a particular agent or set of agents, the procedure they are to execute, and the simulated time at which they are to execute it. The modeler can think in more natural terms about what procedures the agents should execute when (e.g. "have agent X execute procedure Y at time Z", or "have agent X execute procedure Y at Z minutes from now"). The scheduler then executes the appropriate procedures on the appropriate agents at the appropriate time in chronological order. There is no longer a tick-by-tick progression in the model; instead, actions are executed according to the schedule and the "dead" time between successive actions is skipped. (If convenient, some actions can happen at regular ticks while others are scheduled at times on or between ticks.) DES is therefore most useful for models where agents spend a lot of time idle despite it being knowable when they need to act next. With DES, each action is performed only when needed, with no conditional testing and very little overhead. .
A second benefit of DES is that it avoids complications associated with the assumption of time step simulation that multiple events happen simultaneously once per tick. For example, if agents are competing for a resource (e.g., parking places), the order in which they arrive at a location may matter very much. Typically, if multiple agents attempt to take an action or access a resource at the same tick, careful attention must be paid to the ordering to avoid artificial bias (which is why the ask primitive randomizes the order in which agents execute an action). With DES, events can happen at any time, not just at time steps, making the order of events purely chronological and more an outcome of the model than an artifact of the simulation style. In some cases, this leads to more natural logical flow of the model and can avoid di icult-to-diagnose bugs or biases in results.
. The time extension for NetLogo (https://github.com/colinsheppard/time/); available via the "Extensions" link from the NetLogo home page) was designed in part to facilitate DES in NetLogo programs. It includes primitives to link NetLogo's ticks to a specific time interval (i.e., so each tick represents one day, or one week, or . seconds), to schedule actions (as NetLogo "tasks") at specific simulation times or to be repeated whenever a specific amount of time has passed, and to label output in time units.
. If a NetLogo model addresses a problem for which DES seems more natural and e icient than conventional time-step simulation, then it can be designed from the start to use the time extension to schedule actions instead of (or in addition to) the standard "tick" approach. A model of automobile use in Delhi, India, was implemented using DES in NetLogo with the time extension (Sheppard et al. b,a), and was several orders of magnitude faster than a pilot version using standard time steps.

E icient Execution of Simulation Experiments
. The goal of model design, implementation, testing, and the previous speed-up steps is to reach the point where the model is ready to be analyzed and applied to a scientific question. Typical analyses include parameterization (calibration), sensitivity and uncertainty analyses, and analysis of scenarios that address research questions (Railsback & Grimm ). These analyses typically require simulation experiments that use many -often, hundreds or thousands -of model runs. Without e icient ways to implement such experiments, analysis of large and complex models can be impractical. .
One key to making standard kinds of model analysis practical is to use e icient analysis designs, which are widely discussed in the simulation literature. The "BehaviorSearch" tool (Stonedahl & Wilensky ) includes several sophisticated algorithms (e.g., genetic algorithms and simulated annealing) for fitting parameter values in NetLogo models; BehaviorSearch should be an easy and e icient approach to parameterization for many complex NetLogo models. Thiele et al. ( ) make several other well-known techniques for reducing the computational burden of model analysis available via RNetLogo (discussed below). Those techniques include Latin hypercube sampling for uncertainty analysis and the "Morris screening" approach to sensitivity analysis. We strongly recommend that NetLogo users find and use such e icient analysis methods but do not discuss them further here. .
Instead, we discuss two ways of executing simulation experiments and computational techniques for making them more e icient.

Executing BehaviorSpace Experiments on a High-Performance Cluster
. NetLogo's BehaviorSpace tool is designed to automate the setup and execution of simulation experiments. Be-haviorSpace can perform multiple model runs in parallel, with the number of parallel runs up to or even exceeding the number of processor cores in the computer. Therefore, the execution time for a large BehaviorSpace experiment is usually limited by how many cores are available. .
The use of a High-Performance Computing (HPC) cluster, typically a large massively-parallel computing system with many processor cores and a shared storage system connected together via a fast network, allows Behav-iorSpace to execute large numbers of model runs simultaneously and makes extensive simulation experiments feasible even for very slow models. In one example, Ayllón et al. ( ) report parameterization and sensitivity analysis experiments that each required thousands of runs of a model that can take hours to days per run, made possible via HPC and BehaviorSpace. In another example, Sheppard et al. ( b,a) used HPC to run their vehicle model millions of times to optimize electric vehicle charging station locations in Delhi. Many universities and research laboratories have HPC clusters, and some commercial "cloud computing" services o er free trial access to clusters. .
Using NetLogo on a HPC cluster is not necessarily di icult, especially if administrative support is available. Doing so requires installing NetLogo and Java Runtime Environment so they are accessible to all nodes; and a batch file that specifies the path to the NetLogo and Java directories, the path to and name of the NetLogo file, and the BehaviorSpace experiment to run and its options (e.g., output file name and format). (It is important to use the "table" output format because the alternative "spreadsheet" format stores all results in memory and thus can consume all available memory.) In a HPC cluster, BehaviorSpace experiments are run in "headless" mode, that is, without any graphical user interface. Documentation and tools for using NetLogo headless and on an HPC cluster are available in the BehaviorSpace section of the NetLogo User Manual and by searching on-line.

E icient Use of RNetLogo Model Analysis: Design of Experiments
. RNetLogo is a package for running NetLogo models from the popular R statistical so ware (Thiele et al. , ). RNetLogo can, for example, send NetLogo a set of input values and then receive and analyze model results; this allows R programs to set up, execute, and analyze NetLogo simulation experiments. Here, we provide a few tips for making the link between R and NetLogo computationally e icient. .
First, set the Java options in R to values that configure the Java Runtime Environment most e iciently for running NetLogo, e.g., by making adequate memory available. These options are set via a statement such as (the exact statement may depend on the operating system): . Second, keep in mind that R is a vector-oriented language and handles vectors more e iciently than individual values. Therefore, using RNetLogo in a way that uses vector operations to send data to and from R will be much faster and more stable than mass calls with single values.
. When using the RNetLogo commands NLGetAgentSet and NLGetAgentPatches, if possible use the list return option via as.data.frame=FALSE in combination with agents.by.row=TRUE or patches.by.row=TRUE, instead of receiving the data from NetLogo as a data frame.
. Finally, keep in mind that R programs can be executed in parallel by using, for example, the "parallel" package. This capability allows parallel execution of RNetLogo experiments similar to that discussed above for Behav-iorSpace experiments.

Conclusions
. NetLogo is widely recognized as an e icient platform for agent-based simulation, in the sense that it allows modelers, including both beginners and experienced ones, to move rapidly through the design, programming, and testing stages and on to using models for analysis and developing scientific understanding. However, Net-Logo is not widely recognized as an e icient platform in the sense of providing low (or at least reasonable) execution times for working models. Potential users who hear NetLogo's reputation as being too slow or limited for big models, or who perhaps look only at the very simple examples that (understandably) dominate NetLogo's built-in models library, may be discouraged from selecting NetLogo as a platform for large scientific models. Choosing another platform, especially those requiring programming in a base language such as C++ or Java, comes at a high cost: programming will take much longer, mistakes will be harder to find and hence more expensive, the tools necessary for testing and understanding models (graphical and interactive displays, experiment managers like BehaviorSpace) must be developed, and -unless the code is designed as cleverly as NetLogo's primitives appear to be -the result may turn out to actually be slower than NetLogo. .
There is now su icient evidence that NetLogo is neither inherently slow nor incapable of handling large and complex models. Our previous experience comparing ABM platforms (Railsback et al. ; Lytinen & Railsback ), while confirming the general understanding in computer science that it is not simple or straightforward to say which platform or programming languages are faster or slower, indicates that NetLogo is not dramatically slower at executing models than other popular platforms are. Perhaps the best evidence that NetLogo is suitable for large scientific models is that many such models have now been successfully implemented and analyzed extensively in NetLogo (Table ).

Model and citation
Computational challenges  There is actually at least one good reason to prefer NetLogo for large models: the set of tools for running simulation experiments e iciently. The convenience of using BehaviorSpace or RNetLogo to run simulations in parallel, on either a desktop computer or HPC cluster, means that large simulation experiments can be underway and finished in less time than they would be in many other platforms. E icient model analysis packages such as BehaviorSearch (Stonedahl & Wilensky ) and the RNetLogo recipes of Thiele et al. ( ) can let parameterization and sensitivity analysis be completed with fewer model runs and far less e ort than required for platforms lacking such tools.

.
Our overall advice to beginners in implementing ABMs in NetLogo is to not worry about execution speed while writing and testing the program, but to then consider the strategy and techniques provided in this article before running analyses that use possibly thousands of simulations. The initial focus in NetLogo programming should be on writing code that is easy to understand and test. Then, with experience, NetLogo users can quickly learn to avoid ine icient programming. But even experienced NetLogo programmers continue to find new ways of increasing execution speed, making it valuable to check the on-line resources for NetLogo programmers, including NetLogo's User Group, Stack Overflow, and Github sites, for solutions to specific speed issues. The reason for this is that most speed issues are context-dependent: the solutions we describe here are o en, but not necessarily always, e icient; some techniques help in some situations and hurt in others; and unique models are likely to have execution limitations unlike those of other models. Therefore, the most important thing to learn is to use the NetLogo timer and profiler extension to carefully check where a program is slow and whether modifications, o en trial and error, actually make it faster.