Rob J. Nadolski, Bert van den Berg, Adriana J. Berlanga, Hendrik Drachsler, Hans G.K. Hummel, Rob Koper and Peter B. Sloep (2009)

Simulating Light-Weight Personalised Recommender Systems in Learning Networks: a Case for Pedagogy-Oriented and Rating-Based Hybrid Recommendation Strategies

Journal of Artificial Societies and Social Simulation vol. 12, no. 1 4
<https://www.jasss.org/12/1/4.html>

For information about citing this article, click here

Received: 01-Apr-2008    Accepted: 11-Aug-2008    Published: 31-Jan-2009

Abstract

Recommender systems for e-learning demand specific pedagogy-oriented and hybrid recommendation strategies. Current systems are often based on time-consuming, top down information provisioning combined with intensive data-mining collaborative filtering approaches. However, such systems do not seem appropriate for Learning Networks where distributed information can often not be identified beforehand. Providing sound way-finding support for lifelong learners in Learning Networks requires dedicated personalised recommender systems (PRS), that offer the learners customised advise on which learning actions or programs to study next. Such systems should also be practically feasible and be developed with minimized effort. Currently, such so called light-weight PRS systems are scarcely available. This study shows that simulation studies can support the analysis and optimisation of PRS requirements prior to starting the costly process of their development, and practical implementation (including testing and revision) during field experiments in real-life learning situations. This simulation study confirms that providing recommendations leads towards more effective, more satisfied, and faster goal achievement. Furthermore, this study reveals that a light-weight hybrid PRS-system based on ratings is a good alternative for an ontology-based system, in particular for low-level goal achievement. Finally, it is found that rating-based light-weight hybrid PRS-systems enable more effective, more satisfied, and faster goal attainment than peer-based light-weight hybrid PRS-systems (incorporating collaborative techniques without rating).

Keywords:

Recommendation Strategy; Simulation Study; Way-Finding; Collaborative Filtering; Rating

Introduction

1.1

Learning Networks (LN) can facilitate self-organized, learner-centred lifelong learning. LN consist of participants and learning actions that are related to a certain domain (Koper and Sloep 2002). Participants can be learners, teachers, counsellors or providers and can have various roles in different circumstances. Learning actions (LAs) can use any type of learning resource or events (like a course, assignment, discussion, lesson, website, blog) that intends to help learners to acquire a certain competence when participating in LN. For effective competence acquisition, lifelong learners should have a clear overview of what LAs are relevant to them. They need advice in choosing from a large and dynamic collection of LAs those that best fit their current needs and accomplishments. In short, they need support to find their way in a LN.

1.2

Personalised Recommender Systems (PRS) can provide this support, as their aim is to help users prevent information overload by delivering personalised advice (see Drachsler, Hummel & Koper 2008). Most readers will probably have come across well-known (commercial) online systems providing advise to costumers browsing the web looking for a book or movie of their liking. For instance, amazon.com (which started as an online bookstore, but now is also selling other media), would suggest alternatives with a specific book like "others that bought X [this book], also bought Y [other books]", using information about the (buying) behaviour of their other customers. As learners in a LN have a large variety of different learning goals, it is important that they receive personalised recommendations from PRS for the best next LA. Learners in LN can also benefit from information about other (successful) learners, but requirements for recommendations in such a learning context differ from the (commercial) context of buying a book or DVD. PRS for learning use 'pedagogical rules' which consider characteristics of available LAs, the current learner and their peers (i.e., content-based or ontology-based techniques), and the collective learners behaviour (i.e., collaborative techniques).

1.3

PRS are already successfully employed in some formal e-learning domains (Andronico et al., 2003; Farzan and Brusilovsky, 2006; Tang and McCalla, 2004a, 2004b). However, their usefulness for LN is questionable as most of existing PRS heavily rely on rather specific and intensive data provisioning, data maintenance and data-mining, making them 'heavy-weight' systems. On the contrary, PRS in LN ask for a 'light-weight' and more generally applicable approach. Such a light-weight approach can be characterised by minimizing the effort on behalf of the participants, and by taking into account that LN do not have clear boundaries or structures like in more formal learning settings. Regarding the first characteristic, using complex and dedicated ontologies might lead to most effective recommendations, but would at the same time be highly domain-specific and too time consuming to maintain. Regarding the second characteristic, it will not always be possible to identify best next LAs for each learner at any time beforehand, as newly added LAs change the LN and enable other learning paths which were formerly unknown. In these, constantly changing and dynamic LN, it would be also be too time consuming and/or practically infeasible to provide all necessary - standardised - information beforehand in order to cater for personalised recommendations. For the same reasons, recommendations provided by human intervention is not an option either.

1.4

Consequently, PRS for LN should be designed different from already existing PRS for formal e-learning. As feasibility is an important motivator for light weight PRS, in order to provide practically feasible recommendations in LN, we propose to use (a) a limited registration of the behaviour of the current learner and their peers including rating (i.e., collective behaviour) in the LN and (b) a limited set of LA-characteristics and learners' characteristics. Simulations should help us find minimum sets of most critical user behaviour and LA-characteristics (third characteristic of light-weight approach). It is proposed to use minimal hybrid PRS, combining top-down and bottom-up techniques. Mainly bottom-up recommendation strategies (RS) seem to be feasible in lifelong learning with a changing and large number of LAs. This is because such collaborative filtering strategies require nearly no maintenance and improve through the emergent behaviour of the community (Hummel et al. 2007).

1.5

We intend to provide individualised support for each lifelong learner in a LN to increase their goal attainment and satisfaction, and to minimize their study time. We call such support: sound recommendations. In order to arrive at designing them, the compound key question of our research is:

What RS and which limited set of LA-characteristics and learners' characteristics is needed in a light-weight hybrid PRS to enable sound recommendations within LNs, and which behaviour minimally needs to be traced?

1.6

This article focuses on a simulation study that addressed this question and that intends to provide insight into specific RS and PRS key variables affecting learning outcomes. It took real data and findings of preceding studies as a starting point (Drachsler et al. 2008). We present a conceptual simulation model that is based upon a well established approach towards social science simulations (Gilbert and Troitzsch 1999) and is largely in line with Kopers' model (Koper 2005), its implementation, the enabled recommendation strategies, and the results obtained from various simulation runs.

1.7

In this introduction we will now first turn into general PRS and their shortcomings towards e-learning. Thereafter, we compare our approach for PRS in LN with existing PRS in e-learning. At the end of this introduction we will briefly address our previous work.

Shortcomings of general PRS for e-learning

1.8

PRS support their users by preventing information overload through the selection of personalised items, content and services (Adomavicius and Tuzhilin 2005). After collecting information about users and items, different RS and recommendation techniques are used for calculating recommendations. Recommendation techniques can be roughly divided into three categories: content-based techniques, collaborative filtering techniques and hybrid techniques. Content-based techniques recommend items to the current user based on what this user liked before. Collaborative filtering techniques recommend items to the current user based on what users with a similar profile as the current user liked. Hybrid techniques combine these two techniques (for a detailed overview, see: Drachsler, Hummel and Koper 2008; Van Setten 2005; Manouselis and Costopoulou 2007). Recommendation strategies (RS) consist of rules that allow reasoning when applying which (combination of) techniques to calculate recommendations.

1.9

Application of PRS in e-learning is different (McCalla 2004). Whereas, as an example, Movielens recommendations (http://movielens.umn.edu) are entirely based on the interests and tastes of the users in movies, most preferred learning actions might not be pedagogically most adequate. McNee, Riedl, and Konstan (2006) identified the definition of users and purpose as important challenge for designing a PRS. Not only learners' preferences (like the interest in a certain sub domain, preferred learning strategy, preferred presentation style) should be considered, but also their goal, (prior) competence level, and available time. It is beneficial for PRS in LN if pedagogical rules derived from educational psychology research are applied (Koper and Olivier 2004). Since PRS in e-learning are meant to support the learning process, the RS should consist of relevant pedagogical rules describing pedagogy-oriented relations between learners' characteristics and LA-characteristics. As an example, we know that from Vygotsky's "zone of proximal development" (Vygotsky 1978) follows the pedagogical rule 'recommended learning actions with a level just over learners' current competence level'. Other pedagogical rules are: 'go from more simple to more complex', 'learners' effort will increase if they get more satisfied'. Pedagogical rules imply the availability of specific metadata for LAs and of the up to date registration of (a minimal set of) learners' characteristics. Although not specific for PRS in e-learning, also the characteristics of the LN itself (e.g., number of learners, number of LAs, number of sub domains) should be considered as these can effect the impact of PRS on learning outcomes.

1.10

Ideally, PRS in e-learning should assist learners in finding learning actions that perfectly match their profile (competences and preferences), keep them motivated and enable them to complete their LAs in an effective and efficient way. Purely model-based recommendations need fine-grained tracing of learner characteristics and matching these to LA-characteristics. Such ontology-based recommendations are very costly. Each time a new LA is added, detailed and standardized identification will be needed that cannot be automatized. Besides this, every time when learners increase their competence after successfully LA-completion, a new matching process between the updated learner model and learning actions will be needed.

1.11

Collaborative Filtering (CF) addresses both problems. Basically, CF lets peer learners filter out most adequate LAs for the current learner in which the PRS-system does not need to know learners' detailed characteristics. The matching process is not performed from learner model to learning materials, but from one learner model (current learner) to other learners' models. However, purely CF-based PRS systems have severe shortcomings (Van Setten 2005), especially the cold-start problem (new users, new items, and scarcity of past user actions) which can be overcome by using hybrid PRS system, that also use (a limited number of) LA-characteristics and learner-characteristics. Hybrid PRS systems have been shown to outperform purely CF-based PRS systems (Balabanovic and Shoham 1997; Claypool et al. 1999; Good et al. 1999;, Melville, Mooney and Nagarajan 2002; Pazzani 1999; Soboro and Nicholas 2002). Tang and McCalla (Tang 2004b) performed a simulation study claiming that hybrid PRS systems can even be as effective as purely model-based PRS.

1.12

Although purely model-based PRS mostly outperform other PRS-systems, they are not appropriate for LN with a fast changing and potentially huge number of LAs. Furthermore, a hybrid PRS with intensive model-based data maintenance and data-mining CF techniques would also be impractical as they induce enormous network traffic (distributed data) and require huge computing power. What instead is needed is a light-weight hybrid PRS with minimized data provisioning, data maintenance, and data-mining.

Related work

1.13

Pioneering work on the application of recommender systems in e-learning has been done by Tang and McCalla. Learners of a data mining course received personalised paper recommendations using a hybrid technique with an ontology-based learners' model and CF with rating (Tang and McCalla 2004a, 2004b). Learners with similar interests were clustered before using classic CF techniques within each cluster to identify learners with similar interests. As they dealt with a small and limited domain, real user data could be collected guiding the construction of their PRS in subsequent studies (Tang and McCalla 2005). They, for instance asked learners to indicate their preferences when deciding whether or not to read specific papers. In a dynamic set of papers only the fittest papers were included (Tang and McCalla 2004a, 2004b). Although Tang and McCalla's approach is very useful, it was only applied in a limited domain. Furthermore, LAs in LNs will generally not be limited to reading papers, but will include a much broader range of learning activities which hampers LA data provisioning. Finally, updating and keeping track of learners' models and CF might need heavy data-mining and implicate rather heavy-weight PRS.

1.14

Andronico's hybrid PRS ( InLinx) - also recommending papers - combines content analysis, clusters learners and recommends didactical resources matching learners' requirements and interests also taking their strengths and weaknesses into account (Andronico et al. 2003). Ultimately, this results in a personalised learning program. The InLinx paper recommendation tool appears more prototypical as Tang and McCalla's work and suffers from similar drawbacks mentioned before.

1.15

The success of CourseAgent - recommending courses - highly depends on learners' feedback (Farzan and Brusilovsky 2006). The system provides implicit feedback through interactions. Through ratings and evaluation-questions, learners can provide explicit feedback. For giving explicit feedback, an incentive is included, namely: observing progress towards career goal. CourseAgent's more light-weight approach seems promising but is insufficient within LN without clear boundaries. Unfortunately, the effects of ratings were not identified. The need for provision of explicit feedback by learners might appear to be a showstopper. We suggest keeping users' explicit contribution towards PRS success to an absolute minimum, and to make the PRS more light-weight than CourseAgent.

1.16

As Tang and McCalla did, Hsu first clustered users with data mining, and then proceeded with a hybrid technique of both content analysis and CF to advise reading lessons (Hsu 2008). Although Hsu showed PRS's effectiveness, it is unclear if learners increased their reading skills when following advices. Furthermore, it was only applied in a limited and fixed domain and its success might depend too much on considerable learner input.

1.17

RACOFI (Rule-Applying Collaborative Filtering) Composer combines a CF engine, that works with users' ratings for learning resources, with an inference rule engine that is mining association rules between the learning resources and using them for recommendation (Anderson et al. 2003; Lemire et al. 2005a; Lemire 2005). RACOFI studies have not yet assessed recommender systems' pedagogical value, nor did they report user evaluation results. So, it is unclear if their approach is fruitful for our purposes.

1.18

Shen and Shen propose a different approach to learning resources' recommendation (Shen and Shen 2004). Their recommender system is based upon sequencing rules guiding users through concepts of a topics-ontology in a Computer Networks course. Learners' competence gaps are identified and appropriate resources are proposed. The required mapping between the topics-ontology and competences makes this approach time consuming and it has been only applied to a relatively small domain with clear boundaries. Authors claim that learners appreciated this approach, but clear evaluation results from the pilot were absent. As their recommender system did not take learners' preferences into account, it is unclear if this would indicate that learners' preferences are not that important when providing recommendations. We argue that learners' preferences do matter when providing recommendations (McCalla 2004; McNee, Riedl and Konstan 2006).

1.19

To sum up this overview of related work: Although PRS are already successfully employed in some domains of e-learning, their usefulness for LN is questionable as most of existing PRS heavily rely on intensive data provisioning, data maintenance and data-mining whereas PRS in LN ask for a light-weight more general applicable approach, minimizing the effort on behalf of the stakeholders (learners and providers). Furthermore, PRS's critical success factors in their actual deployment need more research. For example, the effects of ratings on learning outcomes are still unclear.

Previous work

1.20

Our research developed and used a prototypical PRS in an introductory Psychology course. We wanted to investigate to what extent the system influenced learners' goal attainment and the time learners needed for goal attainment (Drachsler et al. 2008). A content-based technique was used when only information about the learners was available. Otherwise a CF technique was applied and combined with the former based upon three learner characteristics (sub domain interest, available study time, and study motive). The field experiment showed learners who received personalised recommendations to study more effectively (i.e., more goal attainment) than learners in a control group which did not receive any recommendations. There were no differences in time efficiency between both groups. Although this experiment showed promising results, at the same time various practical constraints (e.g., limited number of learners and LAs) made it difficult to investigate other treatments such as other hybrid RS with or without rating.

1.21

If we want to develop a PRS for learners in a LN we face the problem that limited real data are currently available from user studies. On the other hand, general findings with respect to pedagogical rules, aforementioned studies, our own field experiment, and expert reasoning can assist us in articulating a conceptual simulation model for PRS. Simulations can support defining requirements of a PRS for LN before actually starting the costly process of development, implementation, testing and revision in real education. Field experiments with real learners need careful preparation as they cannot be easily repeated or adjusted within a condensed timeframe. Another advantage of simulations is that they bypass some ethical and practical constraints of field experiments.

1.22

Although simulations are a well established approach in social science (Gilbert and Troitzsch 1999), they have been sparsely used for recommendations in e-learning and LN. Tang and McCalla (2004b) performed a simulation study showing no differences between model-based and hybrid-based paper recommendations included rating. Koper (2005) described a simulation study showing that selecting units of learning (i.e., a structured set of learning actions) informed by indirect social interaction, increased learner retention in a LN when compared to a selection without indirect social interaction. Inclusion of CF-based selection of units-of-learning appeared beneficial for learner retention. Koper asked for solutions to improve the RS (i.e., decrease the matching error). As a possible solution to decrease the matching error we suggest using hybrid techniques with ratings to improve recommendations as they enable a light-weight PRS. Our earlier simulation study with a preliminary conceptual simulation model already identified positive effects of hybrid recommendation techniques as compared to non-hybrid techniques (Berlanga et al. 2007). To the best of our knowledge, no study specifically addresses the effect of rating towards recommendations in LN. This is rather surprising as many previously described hybrid PRS included rating ((Tang and McCalla 2004a, 2004b; RACOFI, CourseAgent).

1.23

We reiterate the compound key question of our research:

What RS and which limited set of LA-characteristics and learners' characteristics is needed in a light-weight hybrid PRS to enable sound recommendations within LNs, and which behaviour minimally needs to be traced?

1.24

Sound recommendations enable more learners to achieve their goal (i.e., graduating) in less time and with more satisfaction. In other words, we strive for more, faster, and more attractive graduation. As we acknowledge that ontology-based recommendations can be assumed to fit this aim most closely, we focus on 'more graduation'.

Method: simulation set-up for recommendations in Learning Networks

2.1

This method section describes hypotheses for our simulation study and its setup. After presenting hypotheses, we (2005). This conceptual simulation model represents the minimized set of LA- and learner characteristics. The model will be elaborated in three subsections: model variables (first subsection), measurement variables (second subsection), and recommendation strategies (third subsection), followed by the setup of the simulation (fourth subsection). The final subsection describes the conditions and treatments as included in our simulation study.

2.2

After this method section, the result section will present the main results of the simulation runs. The final section discusses these results and identifies preferred RS in view of our key question, describes some limitations of this study, and provides some suggestions for future research.

2.3

Our two main hypotheses are:

H1: PRS recommendations yield more, more satisfied, and faster graduation than no recommendations
H2: ontology-based and rating-based recommendations from PRS show no differences for graduation, nor satisfaction, nor time to graduate.

2.4

Both hypotheses consist of three measurements (graduation, satisfaction at graduation, and time to graduate). Confirmation of H1 basically would show the need of using PRS recommendations, and hereby specifically the appropriateness of rating-based PRS recommendations. Confirmation of H2 would show that rating-based recommendations are a practically feasible alternative for ontology-based recommendations.

2.5

In addition to researching these main hypotheses, this study sought to verify the effect of the availability of an algorithm for the cold-start problem (Balabanovic and Shoham 1997; Claypool et al.1999; Good et al. 1999; Melville, Mooney and Nagarajan 2002; Pazzani 1999; Soboro and Nicholas 2002). According to this, non-ontology based recommendations including a cold-start algorithm yield to more, more satisfied and faster graduation. Finally, this study intended to compare various hybrid PRS to identify an ideal, minimal hybrid recommendation strategy.

2.6

The conceptual simulation model describes variables, their initial value distribution, as well as their relationships, which are often represented by formulas. This model is used as input for implementation within a simulation environment, i.e., Netlogo 4.02 (Wilensky 1999). Figure 1 shows an integrated picture of the program flow in the simulation environment (the thick lines and arrows) and the conceptual simulation model. We start with a global description of the conceptual simulation model before clarifying all variables and relationships between them.

Model variables, relations, formulas and their implementation within the simulation

2.7

The model takes into account learner profiles for current learner and their peers as well as the LA-characteristics. The RS produces a personalised recommendation of a best next LA. Individual learner behaviour is modelled in the learner model. It is indirectly influenced by peers if CF is included in the RS during setup, which is dealt with in the run model. The amount of alignment between learner profile and LA-characteristics, as well as the current state of the learners model (for example: effort) influence the chance of successful LA-completion. The RS supports better alignment between learner profiles and LA-characteristics, whereas the learner model represents learners' changing behaviour when trying to achieve goals.

2.8

Table 1 presents an overview of all variables in the conceptual simulation model and their implementation within the simulation. Some variables are related by formulas and are further detailed out in Table 2. Table 2 summarizes relations and formulas, describing pedagogical rules in the conceptual simulation model. These rules are used within the RS. The model is implemented in such a way that weighting values for all variables could be easily adjusted. This is very helpful if more empirical data would become available.

2.9

Each [Learner profile] consists of a [Preference profile], a [Competence profile] and some other learner characteristics (like goal or available study time), all being used within the RS. Preferences b and c in the [Preference profile] could for example be learning strategy, presentation style, or price to enrol.

2.10

The [Competence profile] is restricted to one competence which can include up to three levels for the [Goal]. It is assumed that a learner will only start studying LAs that can contribute to achieving the goal. Successfully completed LAs contribute to their associated level. Each competence level included in the goal can have its own number of successfully completed LAs for its mastery, specified at the simulation setup. For simplification of the simulation, learners start with the same [Learner competence level] and have the same goal. A [Learner competence level] indicates the learner's achievement with respect to the goal and is updated by the outcome of [Success].

Figure 1. Conceptual simulation model and simulation program flow. Details about Recommendation Strategy (RS) are included in Figures 2a, b. Details about Setup are presented in Figure 3.

[Available study time] has the same magnitude as the simulation tick time (1 tick = 1 week).

[Effort] is initially normally distributed amongst learners, but changes during study (see Table 2). Effort is the key variable which determines dropout (Ryan and Deci 2000). If Effort gets below zero, a learner will drop out the LN and will not graduate. Effort depends on previous effort, [Preference GAP], [Competence GAP], [Constraints] and [History of Success/Failures] (abbreviated as: History) on the last three LA examinations.

The [Preference GAP] measures alignment between learners' preferences and corresponding LA-characteristics. The smaller the gap, the better the alignment.

The [Competence GAP] measures alignment between [Learner competence level] and the [LA-competence level] of the LA-chosen. A perfect match occurs if [LA-competence level] is one level above [Learner competence level] (Vygotsky 1978). Please note that [Learner competence level] is a variable with a specific value at a certain point of time, whereas [LA-competence level] is a constant. Mismatches for preferences and/or competences will decrease effort, whereas better matches or preferably perfect matches will increase effort.

[Constraints] (like fatigue, being in the flow, a noisy or quiet or study room, stress) can influence the amount of effort learners want to invest when studying. Constraints are randomized at each studied LA. For calculation purposes, we define constraints as '1' in case of positive effects, '-1' in case of negative effects, and '0' in case of a neutral effect. As constraints are considered to be a multidimensional construct, a more fine grained approach could be used. For the sake of simplicity, we have conceived these constraints to be a one-dimensional construct.

[History] also affects [Effort]. Several successes in a row is expected to increase effort (more motivated), whereas successive failure will be detrimental to learner's effort, and ultimately could result in drop out.

[Obedience] differs between learners but remains constant for each learner. Obedience is similar to predictive utility that measures influences of system predictions upon users' willingness whether or not 'consuming' an item (i.e., obeying the recommendation) (Konstan et al. 1997; Walker et al. 2004). Predictive utility depends upon the domain in which the recommender system is used, and is a function of the value of the predictions, the cost of consuming items, and the ratio of desirable/undesirable items. In an earlier study we identified an obedience level of 60 % (Drachsler et al. 2008) which is similar to other studies (Bolman et al. 2007; Cranen 2007).

[Required study time] is the time a learner invests before doing an LA-examination. If a competence GAP occurs, a learner needs more (in case of knowledge deficiency) or less time (in case of knowledge surplus) than the [estimated study time] of the LA-chosen. Required study time is used in the simulation variable [required time]: the quotient of required study time and simulation tick time.

If [Success] is true, the learner passes LA-examination and achievement of the learner competence level corresponding with the LA competence level and goal will improve. If the studied LA is too far above the learners' current Learner competence level (in other words, there is a very huge competence GAP), it does not matter how much effort the learner invest, it will always lead to failing this LA-examination. However, for LA's that normally would be somewhat beyond learners' scope of possibilities, more effort can lead to their successful completion.

Except for [rating], all LA-characteristics remain unchanged. Learners' rating of a LA ([rate LA-studied]) is influenced by whether or not the learner successfully completes this LA, the [Preference gapmatch], and the [Competence GAP]. [Preference gapmatch] is a coarser variable than [Preference GAP] and has only two values. It is 0 in case of a perfect match if all learner preferences are the same as corresponding LA-characteristics, whereas a value of -1 indicates that one or more learner preferences are different from their corresponding LA-characteristics.

Table 1: Overview variables in conceptual simulation model and their implementation within the simulation
Variable	Description	Implementation in simulation	Input for
		Range (initialization)	Formula
Learner profile	------	-----------	n.a.	RS
- goal	competence level(s)	in set up, same for all, allows 3 levels and needed number of LAs	no
- available study time	------	M = 20 hours/week, SD = 5 (Normally-distributed)	no	required study time
- effort (and scaled effort)	investment to study	M = 10 , SD = 3 (Normally-distributed)	yes	Success/dropout, rst
- obedience	follow up recommendation	M = .6, SD = .15 (Normally-distributed)	no	RS
- constraints	fatigue, flow, stress, a.s.o.	[-1, 0, 1] (Randomized) for each studied LA	no	effort
- preference profile	------	------------	n.a.
- preference a (interest sub domain)	------	number of sub domains, in setup (Randomized)	no	preference GAP
- preference b	------	[b1, b2, b3] (Randomized)	no	preference GAP
- preference c	------	[c1, c2, c3] (Randomized)	no	preference GAP
- competence profile	------	-------------	n.a.
- learner competence level (s) (LCL)	goal accomplishment	0, updated if number of successfully completed LAs matches level	yes	competence GAP
- successfully completed LAs	contribution towards goal	an integer for each applicable level in the goal	yes	competence level
- study state		[studying (in progress), graduated, dropout]		dropout, graduated
LA characteristics - estimated study time - sub domain (flavour a) - preference b (flavour b) - preference c (flavour c) - competence level (CL) - rating	------ ------ ------ ------ ------ ------ perceived usefulness after studying	--------- 100 hours for each LA number of sub domains in setup (Randomized) [b1, b2, b3] (Randomized) [c1, c2, c3] (Randomized) same as level(s) in goal, in setup (Randomized) 'missing', updated after each completion, integer [1, 5]	n.a. no no no no no yes	RS required study time preference GAP preference GAP preference GAP competence GAP RS
- preference GAP - Preference gapmatch	alignment preferences and flavours alignment preferences and flavours	Integer [-3, 0] Integer [-1, 0], more coarse as 'preference GAP', [-3, 0] → [-1,0]	yes	effort rate LA studied
- competence GAP	alignment learner competence level (LCL) and CL of LA	Integer [-2, 2]	yes	effort, rate LA studied, Success, required study time
- required study time (rst)	invested time before LA-examination	[50, 100, 150]	yes	Studying LA
- Success	learner passes or fails LA-examination	Boolean	yes	rate LA studied, LCL

Notes: Formulas (for variables changed during exploitation) are described in Table 2. n.a. = not applicable

Table 2: Formulas and descriptions for all variables with changing values in conceptual simulation model
Variable	Description	Formula in simulation	Input for
Learner profile - effort (and scaled effort)	investment to study and satisfaction during study	Effort = PE + SUM (w1*PG, w2*G(CG), w3*Constraints, w4*History Success/Failures) - PE= Previous Effort; PG = Preference GAP; CG = Competence GAP - w1=w2=w3=w4=1 (all weighting values); PG: -3, -2, -1, 0; CG: -2, -1, 0, 1, 2 - G(CG): G(-2)=-1, G(-1)=1, G(0)=0, G(1)=-1, G(2)=-2,→ output [-2, 1] - Constraints: -1, 0, 1 - History Success/Failures (abbreviated as: History) for each LA: -1, 0, 1 →, output [-3, 3] Effort is scaled in order to be able to deal with different weighting values for its input variables (all being 1 for this study) and is scaled for calculating Success: Scaled Effort (SE). IF 0≤ Effort < 7 → Scaled Effort = 1; IF 7≤ Effort ≤ 13 → Scaled Effort = 2; IF 14 ≤ Effort ≤ 20 → Scaled Effort = 3	Success, dropout, required study time (rst)
Learner profile - competence profile - learner competence level (s) - successfully completed LAs	goal accomplishment contribution towards goal	A specific LA competence level is mastered if the number of successfully completed LAs with a specific LA competence level matches the corresponding definition in the goal (specified at the setup). The number of successfully completed LAs for a specific LA competence level is stored within 'successfully completed LAs'.	competence GAP
LA characteristics - rating (updated each time this LA is studied)	perceived usefulness after studying	Rating = [w3*previous LA-rating + w1*G (individual rating) + w2*H(CF-rating)]/[w1+w2+w3] - w1 = 0.25, w2 = 0.25, w3 = 0.5; w1+w2+w3 = 1 - previous LA-rating = average rating for all learners having studied this LA so far - CF-rating = average rating for ad-hoc group-members to which the current learner belongs when completing this LA (using Slope One Algorithm by Lemire and Maclachlan (2005)). - individual rating = [J(Success) + K(Preference gapmatch) + L(Competence GAP)]/[10] - J(0)=25, J(1)=35 - K(-1)= -7.5, K(0)= 7.5 - L(-2)= -7.5, L(-1)=0, L(0)=L(1)= 7.5, L(2)= -7.5 - individual rating (i.e., Rate LA studied): 1, 2, 3, 4, 5 (Note: 5 for successfully completed, preference GAP is 0, and CG=0 or 1) - H([CF-rating]) → 1, 2, 3, 4, 5 (CF-rating (average) should be round off)	RS
- preference GAP - Preference gapmatch	alignment preferences and flavours	Preference GAP = - SUM ((Learner preference(xi) - (LA preference(xi)), i = 1, 2, 3 Learner preference(xi) - LA preference(xi) = 1 if Learner preference(xi) ≠ LA preference(xi) Learner preference(xi) - LA preference(xi) = 0 if Learner preference(xi) = LA preference(xi) The SUM is 0 if the preference GAP is 0, indicating that there is a perfect match. For calculation purposes (in effort) we use negative values if there is no perfect match.	effort rate LA studied
- competence GAP	alignment competence level of learner and competence level of LA	Competence GAP = (Learner competence level - LA competence level) +1 Learner competence level: 0, 1, 2 LA competence level: 1, 2, 3 For calculation purposes a symmetric distribution of competence GAP with integers ranging from [-2, 2] was preferred. Therefore, a perfect match results in a competence GAP of 0.	effort, rate LA studied, Success, required study time
- required study time (rst)	invested time before LA-examination	rst= FSA* ((1 + H(CG)*) LA Estudy time) - FSA = Factor Scaled Effort: FSA= 0.8 if SE = 3, FSA= 1.0 if SE = 2, FSA= 1.2 if SE = 1 - Estudy time = estimated study time; - H(CG) is a function with competence GAP (CG) as input - H(CG) = 0,5 if CG = -3, -2; H(CG) = 0 if CG = -1, 0 ; H(CG)= -0.5 if CG = 1, 2 A learner needs to invest 50% more time in case of knowledge deficiency (CG=-2) and 50% less time in case of a knowledge surplus (CG= 1 or 2). The required study time is in line with the LA estimated study time if the learner has adequate prior knowledge (CG=-1, or 0).	studying LA
- Success	learner passes or fails LA-examination	Success = Scaled Effort + H(CG) - CG= Competence GAP; - Scaled Effort: 1, 2, 3 - H(CG): H(-2)=-2, H(-1)=0, H(0)=0, H(1)=0, H(2)=0 → output [-2, 0] The learner successfully completes a LA if Succes ≥ 0, otherwise the LA examination fails.	rate LA studied, learner competence level

Measurement variables

2.11

Three measurements were used to test this study's hypotheses.

Graduation. Learners will dropout if their effort falls below zero and when they consequently fail to reach their goal. Reaching their goal equals graduating. Identifying effectiveness of the RS from PRS will be based upon the percentage of learners reaching their goal (Graduates). A higher percentage of graduates indicates more effectiveness.

Satisfaction.Satisfaction is measured when learners achieve their goal. We suppose that a higher proportion with maximum satisfaction at goal completion indicates that they are more satisfied at graduation than a lower proportion with maximum satisfaction at graduation. Learners' willingness to voluntary invest a certain effort to study is here regarded to be similar to satisfaction.

Time to graduate. Learners reaching their goal within the LN graduate. The impact of RS from PRS on time efficiency is determined by identifying learners' total study time for achieving their goal: time to graduate. The less time to graduate, the more time efficient.

Enabled Recommendation Strategy (RS)

2.12

Application of the RS ultimately results in a LA-chosen by the learner (see Figures 2a, 2b). The simulation can deal with seven treatment groups and one control group. The RS can partly be defined in the simulation setup. Recommendations can be based upon: Ontology (O), Peers (P), Ontology and Peers (OP), resulting in three treatment groups. Rating can be switched on or off in the setup thereby enabling four additional treatment groups: Ratings (R), Ontology and Ratings (OR), Peers and ratings (PR), Ontology, Peers and Ratings (OPR). Learners choose the highest rated LA matching their interest. Ontology can be switched on or off in the setup for implementation purposes.

2.13

Only learners belonging to the control group won't get any recommendations. They always choose 'common sense', using 2 out of 4 LA-characteristics perfectly matching corresponding learner-characteristics: (1) interest sub domain (preference a), (2) preference b, (3) preference c, and (4) competence level. Which 2 out of 4 characteristics are used randomly changes for each choice. Although their choice is arbitrary, it is not necessarily worse when compared to choices made by learners in the treatment groups, where sometimes even none, 1, or 2 of 4 LA-characteristics have to be selected, when 3 are not available, to feed the recommendation. Additional runs using a randomly selected 3 out of 4 LA-characteristics for the control group 'common sense' condition have been carried out as well to estimate this effect, comparing for instance results of graduation percentages with level 1 goals and three domains. Additional results for learners graduating show the control group to indeed substantially improve (from 17 to 60%) but to still remain significantly under the worst performing treatment group (being 'P' with 65.8%).

2.14

Learners in treatment groups will always get recommendations and their obedience level determines the chance whether they obey or not. If they don't obey, they will always choose a LA 'common sense' complying with their - not already completed - goal. So, learners not following recommendations act like learners in the control group.

2.15

Only for the first LA-recommendation, the [RS cold-start-algorithm] is used which is identical to the RS ontology-algorithm that is also used for further recommendations in case of treatment groups O, OR, OP and OPR. The [RS ontology-algorithm] uses four criteria: a. learner interest sub domain (preference a) equals LA-sub domain; b. learner competence level is one level below LA competence level; c. learner preference b equals value corresponding LA-characteristic; and d. learner preference c equals the value of the corresponding LA-characteristic. At the cold-start, it is evident that all four criteria can be met and the recommended LA will be randomly chosen from LAs meeting all four criteria. Whether all four criteria will be met, depends on the available set of LA. This can be warranted while setting up a simulation study, but will not always be possible in a real-life situation. The RS ontology-algorithm always uses the same four criteria, but if not all four criteria can be met with remaining LAs, criteria will be dropped. First (d) will be dropped, then (d) and (c), then (d), (c), and (b). Of course, all recommended LAs will always comply with learners' goal and will not yet been successfully completed.

2.16

For all successive LA-recommendations, the RS searches for suitable peers in case of the treatment groups (P, PR, OP, OPR) by inclusion of the [RS peer-algorithm]. The [RS peer-algorithm] uses two criteria : (i) a peer should have the required LA-history, and (ii) a peer should have a similar profile. If the current learner has successfully completed LA 'Y', peers should also have completed LA 'Y' successfully. But peers should also have at least another successfully completed LA which is not yet completed by the current learner: let's say LA 'W'. In that case, LA 'W' is a candidate for recommendation. A similar profile means that peers and the current learner have the same interest (preference a), the same learner competence level, and - preferably two - but at least one equal value on preferences b and c.

2.17

If criterion (i) is not met and the learner is part of group (P, PR), a LA meeting learner interest sub domain (preference a) will be recommended. In case the learner is part of group (OP, OPR), the ontology-algorithm of the RS will be followed. If criterion (i) is met, criterion (ii) will be checked. If criterion (ii) is not met and the learner is part of group (P, PR), a LA meeting learner interest (preference a) will be recommended. In case the learner is part of group (OP, OPR), the ontology-algorithm of the RS will be followed. If criterion (ii) is met, there is at least one peer available. Suppose there are four potential peers p1, p2, p3, p4 which have successfully completed (Y, Z), (Y, X), (Z, Y), (Y, Q, P) whereas the current learner has successfully completed LA 'Y'. The recommended LA is randomly chosen from (Z, X, Z, Q, P).

2.18

Finally, if rating is part of the RS, the recommended LA also needs to be the highest rated from possible candidates and will only be randomly chosen if possible LA-candidates have the same value for rating. In the previous example, this would be the highest rated from (X, Z, P, Q). Ratings are determined using the formula in Table 2, in which the CF-part uses the Slope-one Algorithm by Lemire and Maclachlan (2005). This algorithm is the simplest form of CF based on ratings. CF aims to predict the ratings of one individual based on his past ratings and on a (large) database of ratings contributed by other users. It does not use linear regression (f(x) = ax + b), but a simpler predictor (f(x) = x + b), having half the number of regressors: Slope-one. As can be seen from this, 'Peers' and 'Ratings' are both based upon CF but use different data and algorithms.

2.19

Please note that it is possible in the setup (see 2.4) to specify various levels for the goal (1, 2, 3). As soon as the learner achieves a specific level, LAs with that level will no longer be considered in the RS. The RS copes with obedience by using a 'change-function' P that each time chooses a random value from [0,1]. If P <= [obedience of current learner], the recommendation will be followed. Otherwise, the recommendation will not be followed and - as already mentioned before - the learner will choose a LA based on 'common sense'.

Figure 2a. Recommendation Strategy (RS) for the simulation program. See Figure 2b for more detail

Figure 2b. Recommendation Strategy (RS) for the simulation program in detail

Setup of simulation

2.20

Figure 1 also shows the simulation program flow. During the setup the following characteristics can be considered/defined (see Figure 3):

• the Recommendation Strategy (RS)
• the goal: ranging from low-level goal (level 1) towards high-level goal (includes level 3)
• the learning network characteristics (number of learners, number of LAs, number of sub domains which also determines number of interests at the learner side, and the number of LAs for each competence level in the Goal to be achieved)
• other, general settings of the simulation (e.g., run length)

2.21

Considering these values, the setup procedure initializes the environment as follows:

• the defined number of sub domains is randomly uniformly distributed across LAs
• each LA is initialized with a random competence level (1 to 3), estimated study time, sub domain, preference b, preference c.
• each learner is initialized with an uniformly distributed random: interest sub domain (one of the sub domains), preference b, preference c, available study time
• each learner starts with learner competence level 0 and will have the same goal
• each learner starts with a normally distributed random effort (M = 10, SD = 3)

Figure 3. Screenshot of the simulation in Netlogo. Level-1 goal: 14 LAs level-1, 2 sub domains, 400 LAs, 250 learners for each group, run length 7 years: 2 years and 8 weeks are passed. Treatment 'O' then has 174 graduates, 5 drop outs, and 71 participants still studying (in progress)

Conditions and treatments

2.22

Every condition in the simulation was replicated 12 times (i.e., N = 12 runs) to justify the use of classical statistic techniques on resulting data (Law and Kelton, 2000: p. 496), analyzed with SPSS version 15. The source of the simulation program and simulation outcomes can be found at: http://dspace.ou.nl/handle/1820/1212. Each condition included seven treatment groups (O, P, R, OP, OR, PR, and OPR) and one control group (C). Six conditions were included: goal (2) x sub domain (3). A low-end level 1 goal (similar to bachelor) and a high-end level 3 goal (similar to master) were taken. Either, 2, 3 or 4 sub domains were taken as they were all regarded to be representative for studies in higher distance education. In all six conditions, each group included 250 learners that could choose from 400 LAs. For all runs, only 'graduates' or 'dropouts' were allowed after run length. In other words, no participants were 'still studying' after run length. To check the robustness of the conceptual simulation model other conditions have been simulated as well. However, those will only be sparsely reported upon as these conditions showed similar results to the ones included.

2.23

The six conditions and seven treatments were used to test our two main hypotheses.

Results

3.1

The result section presents findings for the three measurement variables for the 6 conditions relating to the differences found between the 7 treatments. The first subsection addresses test results for H1. The second subsection refers to test results for H2. The third subsection presents more global findings for verifying the cold-start effect. Finally, the fourth subsection reports results on the identification of an ideal minimal hybrid RS.

Test results hypothesis 1

3.2

Hypothesis 1 (PRS recommendations yield more, more satisfied, and faster graduation than no recommendations) was tested by using regular statistical methods, such as analyses of variance on a global level, continuing with Bonferroni's correction when using multiple comparisons at a more detailed level. Here, multiple comparisons were always conducted between the control group and one of the seven treatment groups. Multiple comparisons for other 'pairs' are reported in subsections two and four.

3.3

Graduation. Data means and standard deviations for percentage of Graduates for all conditions are presented in Tables 3 and 4. Analyses of variance showed - for all six conditions a significant difference in the percentage of Graduates between all groups. More detailed analysis (see Table 5) showed that the control group always had a significant smaller percentage of graduates than all other treatment groups. This confirms part 1 of our first hypothesis (part 1) that PRS recommendations yield to more graduation than no recommendations.

3.4

In each condition, all 12 runs showed that the control group always had significant fewer graduates than all other groups. For space limitations, these are not reported in more detail.

Table 3: Graduation, satisfaction, time to graduate for a goal including one level (similar to Bachelor). With cold-start algorithm
# sub domains	Variables	Treatment														No treatment
		Ontology (O)		Peers (P)		Ratings (R)		OP		OR		PR		OPR		Control (C)
		M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
2	Graduates
	- percentage	92.9	2.2	70.5	2.7	94.5	1.3	88.2	2.7	93.5	1.6	79.2	3.3	91.1	2.5	20.2	4.3
	- perc. max. satisf.	47.3	3.5	9.1	1.9	42.9	3.1	23.1	3.1	49.0	3.4	17.9	2.5	32.7	3.1	3.3	3.0
	- time	1658	362	1823	364	1530	258	1665	339	1650	355	1781	366	1641	350	2264	393
3	Graduates
	- percentage	91.1	2.4	65.8	5.1	93.9	1.3	84.5	1.7	91.8	1.9	73.5	3.9	88.7	1.5	17.4	3.9
	- perc. max. satisf	39.2	4.0	6.6	2.0	33.3	3.1	16.7	3.0	37.7	2.4	13.4	3.1	24.5	2.4	2.5	1.3
	- time	1670	373	1843	352	1580	281	1661	341	1669	367	1816	361	1669	356	2286	413
4	Graduates
	- percentage	89.3	3.2	60.8	3.7	89.4	2.4	83.0	4.0	88.9	1.9	72.2	2.7	85.4	2.4	15.3	5.3
	- perc. max. satisf	33.2	4.4	5.7	1.9	27.4	2.9	13.7	3.1	32.7	3.0	11.3	2.3	20.1	2.1	1.7	2.5
	- time	1680	363	1853	353	1626	300	1670	332	1673	350	1820	365	1676	349	2251	400

Notes: number of learners = 250 (for each group), number of LAs = 400, run length = 7 year, 12 runs. Goal: 14 LAs level 1. Perc. max. satisf. = percentage graduates with maximal satisfaction. Time = time to graduate. All treatments include a cold-start algorithm.

Table 4: Graduation, satisfaction, time to graduate for a goal including three levels (similar to Master). With cold-start algorithm
# sub domains	Variables	Treatment														No treatment
		Ontology (O)		Peers (P)		Ratings (R)		OP		OR		PR		OPR		Control (C)
		M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
2	Graduates
	- percentage	92.1	1.7	61.5	4.3	90.6	2.1	85.1	3.4	92.5	2.1	73.6	4.6	87.8	2.2	10.4	2.9
	- perc. max. satisf.	92.5	1.6	62.1	4.4	68.9	2.5	72.8	3.4	91.2	2.1	70.3	2.1	76.3	2.0	47.9	11.8
	- time	5234	443	5504	587	5223	394	5247	490	5204	444	5423	557	5209	460	6057	543
3	Graduates
	- percentage	88.9	2.2	51.4	3.8	83.8	2.6	79.9	2.9	89.9	2.5	65.5	5.0	83.50	3.3	8.1	2.3
	- perc. max. satisf	87.2	2.3	53.7	2.4	61.8	3.0	63.8	4.2	88.3	2.7	64.4	2.6	71.5	3.7	38.0	12.1
	- time	5246	457	5576	587	5326	430	5282	508	5236	459	5465	565	5268	495	6061	536
4	Graduates
	- percentage	87.3	2.5	45.5	4.9	81.2	3.2	76.1	2.8	87.3	3.0	62.3	4.1	81.3	1.5	6.5	2.1
	- perc. max. satisf	75.6	2.8	44.5	3.3	54.1	4.3	59.6	4.2	74.7	5.0	59.6	4.3	68.3	2.2	26.7	10.0
	- time	5266	479	5581	572	5383	455	5301	518	5275	482	5502	569	5258	494	6073	553

Notes: number of learners = 250 (for each group), number of LAs = 400, run length = 15 year, 12 runs. Goal: 14 LAs level 1, 28 LAs level 2, 14 LAs level 3. Perc. max. satisf. = percentage graduates with maximal satisfaction. Time = time to graduate. All treatments include a cold-start algorithm.

Table 5: Outcomes for (a) Analyses of variance and (b) Multiple comparisons with Bonferroni's correction with respect to Graduation (N = 12 runs)
Goal	# sub-	Analyses of variance		p	Multiple comparisons
	domains	F	MSE		(mean difference between two groups, all p <.05*)
Level-1	2	F(7, 23992) = 1678.02	754	<.05*	control group fewer Graduates than any treatment group
	3	F(7, 23992) = 1552.80	783	<.05*	control group fewer Graduates than any treatment group
	4	F(7, 23992) = 1378.75	775	<.05*	control group fewer Graduates than any treatment group
Level-3	2	F(7, 23992) = 1903.68	937	<.05*	control group fewer Graduates than any treatment group
	3	F(7, 23992) = 1584.61	928	<.05*	control group fewer Graduates than any treatment group
	4	F(7, 23992) = 1495.61	935	<.05*	control group fewer Graduates than any treatment group

3.5

Satisfaction. Data means and standard deviations for percentage of learners with maximum satisfaction at graduation for all conditions are presented in Tables 3 and 4. Analyses of variance showed for all six conditions a significant difference in the percentage of maximum satisfaction at graduation between all groups. More detailed analysis (see Table 6), namely multiple comparisons all with Bonferroni's correction, showed that the control group always had a significant smaller percentage of graduates with maximal satisfaction than any treatment group. This confirms our first hypothesis (part 2) that PRS recommendations yield to more satisfied graduation than no recommendations.

Table 6: Outcomes for (a) Analyses of variance and (b) Multiple comparisons with Bonferroni's correction with respect to Satisfaction at graduation (N = 12 runs)
Goal	# sub-	Analyses of variance		p	Multiple comparisons
	domains	F	MSE		(mean difference between two groups, all p < .05*)
Level-1	2	F(7, 18890) = 533.09	11790	< .05*	C fewer maximum satisfaction at graduation than any T
	3	F(7, 18196) = 451.12	10966	< .05*	C fewer maximum satisfaction at graduation than any T
	4	F(7, 17520) = 352.08	9175	< .05*	C fewer maximum satisfaction at graduation than any T
Level-3	2	F(7, 17800) = 185.62	353	< .05*	C fewer maximum satisfaction at graduation than any T
	3	F(7, 16519) = 182.28	616	< .05*	C fewer maximum satisfaction at graduation than any T
	4	F(7, 15815) = 107.23	634	< .05*	C fewer maximum satisfaction at graduation than any T

Notes: C = control group. T = treatment group. Different 'n'-s in the F-statistics as conditions differ in number of graduates. As before, the same results were found at the level of the separate runs.

3.6

Time. Data means and standard deviations for total study time of Graduates for all conditions are presented in Tables 3 and 4. Analyses of variance showed for all six conditions a significant difference in time to graduate between all groups. More detailed analysis, namely multiple comparisons all with Bonferroni's correction, showed that the control group always needed significantly more time to graduate than any treatment group (see Table 7). This confirms our first hypothesis (part 3) that PRS recommendations yield to faster graduation than no recommendations. As before, almost the same results were found at the level of the separate runs.

Table 7: Outcomes for (a) Analyses of variance and (b) Multiple comparisons with Bonferroni's correction with respect to time to graduate (N = 12 runs)
Goal	# sub-	Analyses of variance		p	Multiple comparisons
	domains	F	MSE		(mean difference between two groups, all p < .05*)
Level-1	2	F(7, 18890) = 416.89	49315069	< .05*	C more time to graduate than any T
	3	F(7, 18196) = 350.54	42896187	< .05*	C more time to graduate than any T
	4	F(7, 17520) = 272.81	32631950	< .05*	C more time to graduate than any T
Level-3	2	F(7, 17800) = 230.24	52926168	< .05*	C more time to graduate than any T
	3	F(7, 16519) = 184.26	45151363	< .05*	C more time to graduate than any T
	4	F(7, 15815) = 157.14	40070442	< .05*	C more time to graduate than any T

Notes: C = control group. T = treatment group. Different 'n'-s in the F-statistics as conditions differ in number of graduates.

Test results hypothesis 2

3.7

Hypothesis 2 (ontology-based and rating-based recommendations from PRS show no differences for graduation, nor satisfaction, nor time to graduate) was also tested by using regular statistical methods, namely Bonferroni's correction when using multiple comparisons at a more detailed level, using mean difference between two groups (O-group and R-group), p < .05. these multiple comparisons were always conducted after analyses of variance on the more global level. As before, results for all measurement variables are presented in tables 3 and 4.

3.8

Graduation. For the low-level goal, multiple comparisons, using Bonferroni's correction, revealed no differences in the percentage of Graduates between the O-group and the R-group, using mean difference between two groups, p < .05. For the high-level goal, the same statistical methods revealed that the o-group had significantly more graduates if the number of sub domains were three or four. For two sub domains with the high-level goal, both groups showed no differences in the percentage of graduates. All results were almost always present at the level of separate runs. This partly confirms part 1 of our second hypothesis. Please recall that there was a significant difference in the percentage of graduation between all groups for all six conditions when analysis of variance was used.

3.9

Satisfaction. Multiple comparisons, using Bonferroni's correction, revealed significant differences in percentage of Graduates with maximum satisfaction between the O-group and the R-group, using mean difference between two groups, p < .05. For all 6 conditions, the O-group had significantly more satisfaction at graduation than the R-group. This rejects part 2 of our second hypothesis. Please recall that there was a significant difference in the percentage of Graduates with maximum satisfaction between all groups for all six conditions when analysis of variance was used.

3.10

Time. Multiple comparisons, using Bonferroni's correction, showed some significant differences in graduation time between the O-group and the R-group, using mean difference between two groups, p < .05. The R-group consistently needed less graduation time than the O-group for the low-level goal. However, the R-group needed more graduation time than the O-group for the high-level goal with three or four sub domains. There were no differences between both groups for the high-level goal with two sub domains. This partly confirms part 3 of our second hypothesis. Please recall that there was a significant difference in graduation time between all groups for all six conditions when analysis of variance was used.

Table 8: Graduation, satisfaction, time to graduate for a goal including one level (similar to Bachelor). No cold-start algorithm
# sub domains	Variables	Treatment														No treatment
		Ontology (O)		Peers (P)		Ratings (R)		OP		OR		PR		OPR		Control (C)
		M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
2	Graduates
	- percentage	94.2	1.5	51.9	4.4	76.5	3.8	88.4	1.7	94.4	1.8	58.5	4.9	88.9	2.5	15.6	4.4
	- perc. max. satisf.	48.4	4.5	6.0	2.1	28.1	2.7	23.0	3.2	49.5	4.4	11.7	3.0	28.8	3.6	3.6	3.3
	- time	1649	348	1902	359	1692	283	1649	335	1644	359	1897	358	1654	342	2334	402
3	Graduates
	- percentage	91.1	2.3	44.0	4.2	70.0	2.8	84.1	3.4	90.8	2.1	53.6	6.0	87.4	2.4	10.7	3.5
	- perc. max. satisf	38.3	3.5	3.9	2.2	22.4	3.0	18.3	4.0	38.5	5.1	9.3	3.3	23.0	4.0	1.6	3.4
	- time	1667	364	1939	350	1730	295	1660	338	1669	363	1919	365	1682	350	2340	387
4	Graduates
	- percentage	90.0	1.9	39.7	5.2	67.6	4.2	82.0	2.9	89.1	2.0	49.1	4.2	85.3	2.2	11.4	3.4
	- perc. max. satisf	33.3	2.7	3.5	1.7	20.1	3.1	13.8	2.4	34.1	4.7	6.0	2.7	19.1	3.5	1.8	1.9
	- time	1685	361	1926	350	1756	312	1674	338	1660	358	1915	344	1673	355	2304	392

Notes: number of learners = 250 (for each group), number of LAs = 400, run length = 7 year, 12 runs. Goal: 14 LAs level 1. Perc. max. satisf. = percentage graduates with maximal satisfaction. Time = time to graduate. All treatments are without cold-start algorithm.

Verification Cold-start effect

3.11

Figures in Tables 8 and 3 are meant to roughly compare the RS with or without a cold-start algorithm. The benefit of using a cold-start algorithm within our RS for non-ontology treatments can be easily identified. Non-ontology based recommendations with a cold-start algorithm yield to more, more satisfied and faster graduation than without a cold-start algorithm. Using Analysis of variance provides statistical evidence for this assertion. For example, for 2 sub domains, comparing graduates percentages in the P-groups between cold-start(CS) and no-CS reveals F (1, 5998) = 226.40, MSE = 207, p < .05, with the no-CS condition resulting in significant fewer graduates than the CS-condition. Because of limited space, and as this fully accords with existing literature on the cold-start issue, it was decided to report only sparsely on these findings.

Minimal hybrid RS

3.12

The final aim of this study was to look for a minimal hybrid recommendation strategy which would allow us to set up planned field studies as simply as possible. For this purpose, some other simulation results are mentioned here. All analysis concerned multiple comparisons with Bonferroni's correction, and were performed on all 6 conditions (goal (2) × sub domain (3)). As before, these multiple comparisons were always conducted after analyses of variance on the more global leve. Again, figures in Tables 3 and 4 were included in these analysis.

3.13

Multiple comparisons, using Bonferroni's correction, showed significant differences in Graduates percentages, time to graduate, as well in the percentage of Graduates with maximum satisfaction between the R-group and the P-group, using mean difference between two groups, with p < .05. For every treatment group, according to the sequential logics of the algorithm, first potential LA-candidates were selected as a subset (based on O or P) of available LA, and then the best-rated (based on R) of this subset were recommended, thus combining for instance "good peers" with "good ratings". For all 6 conditions, the R-group consistently had more graduates, needed less time, as well as had significantly more maximum satisfaction at graduation than the P-group. In other words, rating-based recommendations are preferable to peer-based recommendations. They both have a similar mechanism, but whereas rating-based recommendations need 'peers with similar LA-history', peer-based recommendations additionally ask for 'peers with similar profile' making them somewhat more complex (i.e., less light-weight) than peer-based recommendations.

3.14

Other comparisons (O-OR, OP-OPR, P-PR) obviously do not result in a simpler RS because the simplest ones are R, P, and O. However, it is noteworthy to mention that - for the ontology-based combinations - more complex RS hardly improve the results on those three measurements (percentage graduates, percentage with maximum satisfaction for graduates, time to graduate). With respect to the percentage of graduates and time to graduate, there is no effect for adding rating to O or OP, but goal achievers are more satisfied if rating is added to O or OP. For all six conditions there is a positive effect for adding rating to P (i.e. PR) on all three measurements. But, it should be noted that R has better results on percentage of graduates and time to graduate than PR. For low-level goal achievement, R has better results for the percentage graduates with maximum satisfaction at graduation than PR. For high-level goal achievement with the sub domains being three or four, PR has somewhat better results for percentage graduates with maximum satisfaction at graduation than R (Tables 3 and 4).

3.15

Finally, note in Table 3 and 4 that for all RS an increasing number of sub domains reduce the effect of PRS on all three measures. Some additional simulations in which the number of sub domains was further increased showed a similar, but quite smoothly trend.

Discussion and conclusions

4.1

This last section starts with a summary of results and discussion. Thereafter, it will be discussed in as how far significant findings have real practical implications and can be put into practice. The third subsection addresses the limitations of this simulation study. Finally, some suggestions for future research are included.

Important findings and discussion

4.2

This simulation study clearly demonstrates the advantage of using pedagogy-oriented PRS recommendations in e-learning. These yield to more, and more satisfied graduation than without recommendations.

4.3

This simulation study also shows that - in particular for low-goal achievement - rating-based recommendations are a good light-weight alternative for intensive data maintenance approaches needed by ontology-based recommendations. For a low-level goal, rating-based recommendations yield similar graduation figures and even less time to graduate than ontology-based recommendations, although learners are less satisfied. For a high-level goal, ontology-based recommendations mostly result in better graduation figures and less time to graduate than rating-based recommendations, again learners are less satisfied with rating-based recommendations. At a practical level, differences in time to graduate at both levels might probably be ignored as they are within the range of a couple of percent. For similar reasons, differences in graduation figures for the high-level goal might probably be ignored.

4.4

This simulation study confirms the relevance of inclusion of a cold-start algorithm in the RS (Balabanovic and Shoham 1997; Claypool et al. 1999; Good et al. 1999; Melville, Mooney and Nagarajan 2002; Pazzani 1999; Soboro and Nicholas 2002). But more important, it also shows that rating-based recommendations in comparison with peer-based recommendations result in more, more satisfied, and faster goal achievement. Both RS have a similar mechanism, but whereas rating-based recommendations need 'peers with similar LA-history', peer-based recommendations additionally ask for 'peers with similar profile'. Rating-based recommendations are somewhat simpler (i.e., more light-weight) than peer-based recommendations and thus preferred.

4.5

More complex RS hardly improve the results on the three measurements. This simulation study revealed that there is very limited use of adding ratings to an ontology-based RS (OR, OPR). Although adding ratings is beneficial for a peer-based RS (PR), this does not result in any benefit as this is still mostly clearly outperformed by a rating-based RS (R). Furthermore, these conflict with our principle that a RS should be as simple as possible for implementation purposes.

4.6

Finally, for all RS it was found that an increasing number of sub domains reduces the effect of PRS on all three measures. This seems reasonable as our conceptual simulation model only takes one preference sub domain into account. In addition, more LAs than the 400 included in our simulations might be needed for learners to be able to achieve the goal without having to study non-preferred LAs. In other words, the more sub domains the more chance that there are too few LAs within the preferred sub domain. Although an increasing number of sub domains reduces the effect of PRS on all three measures, it is interesting to see that this can be different for various treatments. For example for the level-1 goal, whereas the mean percentage graduates for P drops from 70.5 % (2 sub domains) to 60.8 % (4 sub domains), for R the mean percentage graduates drops from 94.5 % to 89.4 %. We have no explanation for this significant difference. It shows that R is more robust than P with respect to the number of sub domains. But, it also demonstrates that a conceptual simulation model cannot unequivocally predict its outcomes.

Practical implications

4.7

Every condition in the simulation was replicated 12 times (i.e., N = 12 runs) to justify the use of classical statistic techniques on resulting data (see, Law and Kelton 2000: p. 496). A possible drawback of this approach could be that the law of large numbers (n = 3000 learners for each group, i.e., 12 runs with 250 learners for each group) might explain occurrences of significant differences. However, as almost all effects were also prevalent at the level of separate runs, this possible explanation can be ruled out in our simulation study. Results on the level of separate runs also showed that the conceptual simulation model caters for randomness. Although it is expected that some potential learning paths are more successful than others, there should always be enough flexibility for personalisation and divergence from most successful learning tracks (a track represents actual behaviour of users) because some randomness in behaviour is crucial in a LN for facilitating the discovery of new solutions (Koper 2005).

4.8

When this rather generic conceptual model is applied in a specific real-life learning situation, several design decisions still have to be made on for instance the level of recommendation. PRS can recommend specific learning activities, courses or even entire curricula. Basically this is a granularity issue, with the model catering for all possible levels. To test a preliminary version of a PRS with novice psychology students (see Drachsler et al. 2008), we for instance split up an introductory psychology course to simulate navigation in a curriculum. Eighteen independent learning tasks, comprising the course material, were considered to be the courses of the curriculum, that should be studied in an optimal order, with the PRS advising which tasks to study next.

4.9

To conclude, this simulation study shows the prospective benefit of using ratings within a RS of the PRS. Although it is argued that a rating-based PRS results in a relatively light-weight PRS, it could still put considerable demands on the needed technical infrastructure needed as information in LN is highly distributed. So, although in principle, this could work, but actual implementation is 'the proof of the pudding'. Furthermore, Farzan and Brusilovsky (2006) report about the "users-do-not-like-to-rate" phenomenon. This means that actual deployment of such a system should tackle this problem in a way similar to as was suggested by aforementioned authors, namely by providing a clear incentive/benefit to learners for providing such ratings, for example by receiving credits for doing so (Hummel et al. 2005). On the other hand, other studies mention possible drawback of providing clear incentives on willingness to share knowledge (Bock and Kim 2002). Finally, a considerable number of learners and learning actions are needed to make this approach work.

Limitations

4.10

Our simulation study has several limitations (see also Koper, 2005). We will address four limitations.

4.11

The main one comes from the limited availability of real data. This boils down to the question whether learners really behave as is modelled in the conceptual simulation model. The conceptual simulation model is derived from various theories (see alsoKoper 2005), but at a more detailed level, we were sometimes forced to make arbitrary choices. For example: obedience towards recommendations is personalised, but is still supposed to be stable for each learner. As Koper (2005), we expect that obedience will be positively influenced by the usefulness of the previous recommendations (i.e., increasing confidence), but will also depend upon the way and format in which they are presented (see, Farzan and Brusilovsky 2006). Indeed, further usability-studies are needed that make an inventory of what learners expect from recommendation systems as well as their - partly individual - preferences towards information presentation formats for such systems.

4.12

The second limitation stems from the fact that a conceptual model always represents part of the world, with as drawback that features could still be missing, notwithstanding the expert consultation that was carried out to verify the current model. Four examples are included to clarify this. First, it was not explicitly taken into account that learners' choice in performing a recommended LA depends on the enrolment-costs of that LA. Although this could be represented by preference b or c, there might still be the need for other weighting factors than were used in the current model. Second, we did not take into account some 'constraints' that could act as 'negative feedback' in the conceptual simulation model. For example: LAs might have some enrolment constraints (maximum number of learners, limited start-time, a.s.o.). Third, learners do not interact with each other in the LN, only indirect social interaction is modelled. However, in reality there is always a combination of indirect and direct social interaction influencing the choices made. The lack of direct social interaction by no means should have to be problematic, as has been demonstrated by educational institutes who have adopted a blended learning or online learning approach. For instance, most open universities successfully provide distance education by combining a blended approach to correspondence study with little or none direct social interaction. Fourth, learners can have preference for one sub domain only, whereas specific learners could like several sub domains equally or would like to indicate their preferences more gradually.

4.13

The third limitation comes from the simplification towards competences. Our conceptual simulation model does not deal with competence-hierarchies. These are complex to describe and the PRS system should be able to trace how completion of a certain learning action can contribute to the - partial- attainment of more than one competence. The learning path specification which is currently under development could tackle this problem (Janssen et al. 2008).

4.14

The fourth and final limitation of this simulation study is that it shows that somewhat ontology-based recommendation, namely a limited set of LA-characteristics and learner-characteristics, should preferably be part of the hybrid approach. However, this can induce considerable workload that could be minimized when using a limited vocabulary of meta data. But, how to identify such limited vocabulary that can be expected to be partly domain dependent? And, how to maintain this?

4.15

Obviously, more research is needed to address aforementioned limitations.

Future research

4.16

Our research focuses on light-weight system-based personalised recommendations within LN. It is debatable whether this approach is sufficient or should be augmented with human-based recommendations as is suggested by QSIA (Rafaeli et al. 2004; 2005). As community driven information (CF) becomes more important within the recommendation strategy, it should be noted that this has a bias to popular tastes. Learners with an unusual taste may get fewer qualitative recommendations, and others are unlikely to be recommended unpopular items (of high quality). Further research is needed to justify the augmentation with human-based recommendations as these can be very costly and practically almost impossible because of limited availability of experts.

4.17

Results from these simulation studies will have to be further validated in real-life experimentation. Currently we are preparing a pilot study with a PRS in the context of an open-source repository of learning objects and activities (Open Educational Resources project, Opener). We intend to use real-life rating information, and further elaborate the model described with actual user input from tagging of learning actions.

4.18

The most important finding from our simulation study is the prospective benefit of ratings. Future field experiments on LNs with real learners should verify the value of light-weight system-based personalised recommendations. Such experiments can also assist the further articulation of a conceptual simulation model and subsequent simulation studies. Such iterations between simulations, system development, and field experiments will provide valuable insights into the technical infrastructure needed to provide sound personalised recommendations to lifelong learners in Learning Networks.

References

ADOMAVICIUS G and Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. Knowledge and Data Engineering, 17(6), 734-749.

ANDERSON M, Ball M, Boley H, Greene S, Howse N, Lemire D and McGrath S (2003) RACOFI: A Rule-Applying Collaborative Filtering System. In Proceedings IEEE/WIC COLA'03, Halifax, Canada, October 2003

ANDRONICO A, Carbonaro A, Casadei G, Colazzo L, Molinari A, and Ronchetti M (2003) Integrating a multi-agent recommendation system into a mobile learning management system. Retrieved, October 18, 2007, from http://w5.cs.uni-sb.de/~krueger/aims2003/camera-ready/carbonaro-4.pdf

BALABANOVIC M and Shoham Y (1997) Fab: content-based, collaborative recommendation. Communications of the ACM, 40(3), 66-72.

BERLANGA A J, van den Berg E B , Nadolski R J , Drachsler H, Hummel H G K and Koper E J R (2007) Towards a model for navigation support in Learning Networks. ePortfolio Conference, Maastricht, The Netherlands, October 18-20, 2007. Retrieved, October 18, 2007, from http://dspace.ou.nl/handle/1820/1015

BOCK G W and Kim Y (2002) Breaking the myths of rewards: an exploratory study of attitudes about knowledge sharing. Information Resources Management Journal, 15(2), 14-21.

BOLMAN C, Tattersall C, Waterink W, Janssen J, van den Berg E B, van Es R and Koper E J R (2007) Learners' evaluation of a navigation support tool in distance education. Journal of Computer Assisted Learning, 23, 384-392.

CLAYPOOL M, Gokhale A, Miranda T, Murkinov P, Netes D and Sartin M (1999) Combining content-based and collaborative filters in an online newspaper. Paper presented at the conference ACM SIGIR Workshop on Recommender Systems: Algorithms and Evaluation. August, 19, 1999, Berkeley, CA, USA.

CRANEN K (2007) Using digital consults and determinants in following up advices [Het gebruik van het digital consult en de determinanten voor het opvolgen van het advies]. Master thesis. Enschede, The Netherlands: Twente University.

DRACHSLER H, Hummel H G K and Koper E J R (2008) Personal recommender systems for learners in lifelong learning networks: requirements, techniques and model. International Journal of Learning Technology, 3(4), 404-423.

DRACHSLER, H, Hummel, H G K, van den Berg, E B, Waterink, W, Eshuis, J, Nadolski, R J, Berlanga, A J, Boers, N, and Koper, E J R (2008) Effects of the ISIS recommender system for providing personalised advice to lifelong learners: an experimental study in the domain of Psychology. Accepted for publication.

FARZAN R and Brusilovsky P (2006) "AnnotatEd: A Social Navigation and Annotation Service for Web-based Educational Resources". In: Reeves T C and Yamashita S F (Eds.) Proceedings of World Conference on E-Learning, E-Learn 2006, Honolulu, HI, USA, October 13-17, 2006, AACE, pp. 2794-2802.

GILBERT N and Troitzsch K G (1999) Simulation for the social scientist. Buckingham, Philadelphia, USA: Open University Press.

GOOD N, Schafer J B, Konstan J A, Borchers A, Sarwar B, Herlocker J and Riedl J (1999) Combining collaborative filtering with personal agents for better recommendations. Proceedings of AAAI, 99, 439-446.

HSU M H (2008) A personalized English learning recommender system for ESL students. Expert Systems with Applications, 34(1), 683-688.

HUMMEL H G K, van den Berg E B, Berlanga A J, Drachsler H, Janssen J, Nadolski R J and Koper E J R (2007) Combining Social- and Information-based Approaches for Personalised Recommendation on Sequencing Learning Activities. International Journal of Learning Technology, 3(2), 152-168.

HUMMEL H G K, Burgos D, Tattersall C, Brouns F M R, Kurvers H J and Koper E J R (2005). Encouraging Contributions in Learning Networks by using Incentive Mechanisms. Journal of Computer Assisted Learning, 21(5), 355-365.

JANSSEN J , Berlanga A J, Vogten H and Koper E J R (2008) Towards a learning path specification. International Journal of Continuing Engineering Education and Lifelong Learning, 18(1), 77-97.

KONSTAN J, Miller B, Maltz D, Herlocker J, Gordon L, and Riedl J (1997) Grouplens. Communications of the ACM, 40(3), 77- 87.

KOPER E J R (2005) Increasing learner retention in a simulated learning network using indirect social interaction. Journal of Artificial Societies and Social Simulation, 8(2). Retrieved, October 30, 2007, from https://www.jasss.org/8/2/5.html

KOPER E J R and Olivier B (2004) Representing the Learning Design of Units of Learning. Educational Technology & Society, 7(3), 97-111.

KOPER E J R and Sloep P B (2002) Learning Networks: connecting people, organizations, autonomous agents and learning resources to establish the emergence of effective lifelong learning (OTEC RTD Programme Plan 2003-2008). Heerlen: Open University of the Netherlands. Available from: http://hdl.handle.net/1820/65.

LAW A M and Kelton W D (2000) Simulation Modeling and Analysis. Boston: McGrawHill.

LEMIRE D (2005) Scale and Translation Invariant Collaborative Filtering Systems. Information Retrieval, 8(1), 129-150, January 2005.

LEMIRE D, Boley H, McGrath S and Ball M. (2005) Colloborative filtering and inference rules for context-aware learning object recommendation. International Journal of Interactive Technology and Smart Education, 2(3). August 2005.

LEMIRE D and MacLachlan A (2005) "Slope One Predictors for online rating-based collaborative filtering". In: SIAM Data Mining (SDM'05), Newport Beach, California, April 21-23, 2005.

MANOUSELIS N and Costopoulou C (2007) Analysis and Classification of Multi-Criteria Recommender Systems, to appear in World Wide Web: Internet and Web Information Systems, Special Issue on "Multi-channel Adaptive Information Systems on the World Wide Web",

MCCALLA G (2004) The ecological approach to the design of e-Learning environments: Purpose-based capture and use of information about learners. Journal of Interactive Media in Education. Retrieved October 18, 2007, from http://www-jime.open.ac.uk/2004/7

MCNEE S M, Riedl J, and Konstan J A (2006) Making recommendations better: an analytic model for human-recommender interaction. Proceedings of Conference on Human Factors in Computing Systems, 1103-1108, 22-26 april, 2006. Montréal, Québec, Canada.

MELVILLE P, Mooney R J and Nagarajan R (2002) Content-boosted collaborative filtering for improved recommendations. Proceedings of 18th National Conference on Artificial Intelligence, 187-192. July 28 - August 1, 2002, Edmonton, Alberta, Canada.

PAZZANI M J (1999) A framework for collaborative, content-based and demographic filtering. Artificial Intelligence Review, 13(5), 393-408.

RAFAELI S, Barak M, Dan-Gur Y and Toch E (2004). QSIA - a Web-based environment for learning, assessing and knowledge sharing in communities. Computers & Education, 43, 273-289.

RAFAELI S, Dan-Gur Y and Barak M (2005) Social Recommender Systems: Recommendations in Support of E-Learning. Journal of Distance Education Technologies, 3(2), 29-45.

RYAN R M and Deci E L (2000) Self-determination theory and the facilitation of intrinsic motivation, social developments and well being. American Psychologist, 55, 68-78.

SHEN L and Shen R (2004) "Learning Content Recommendation Service based on Simple Sequencing Specification". In Liu W et al. (Eds.) ICWL 2004, LNCS 3143, 363-370.

SOBORO I M and Nicholas C K (2000) Combining content and collaboration in text filtering. Proceedings of the IJCAI Workshop on Machine Learning in Information Filtering, 86-91. August, 1999, Stockholm.

TANG T Y and McCalla G (2004a) Beyond Learners' Interest: Personalized Paper Recommendation Based on Their Pedagogical Features for an E-Learning System. Proceedings of the 8th Pacific Rim International Conference on Artificial Intelligence (PRICAI 2004), Auckland, New Zealand, August 9-13, 2004.

TANG T Y and McCalla G (2004b) Utilizing Artificial Learners to help overcome the cold-start problem in a Pedagogical-Oriented Paper Recommendation System. Proceedings of AH 2004: International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Eindhoven, The Netherlands, August 23-26.

TANG T Y and McCalla G (2005) Smart Recommendation for an Evolving E-Learning System: Architecture and Experiment, International Journal on E-Learning, 4 (1), 105-129.

VAN SETTEN M (2005) Supporting People in finding information: Hybrid Recommender Systems and Goal-Based Structuring. Doctoral thesis. Enschede, The Netherlands: Telematica Instituut.

VYGOTSKY L (1978) Mind in Society: The development of higher psychological functions. Cambridge: Harvard University Press.

WALKER A, Recker M M, Lawless K and Wiley D (2004). Colloborative Information Filtering: a review and an educational application. International Journal of Artificial Intelligence in Education, 14, 1-26.

WILENSKY U (1999) NetLogo. http://ccl.northwestern.edu/netlogo/. Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL.

Return to Contents of this issue

© Copyright Journal of Artificial Societies and Social Simulation, [2009]