Embedding Systems Thinking into EWB Project Planning and Development : Assessing the Utility of a Group Model Building Approach

Service learning organisations intend to provide opportunities to equitably benefit communities through technical projects. By participating in these activities, engineering students gain critical thinking tools and experiences and a greater sense of civic responsibility. However, with the common lack of project sustainability, two problematic outcomes arise: (1) the true impact of this work may exploit communities for the sole benefit of student learning and (2) engineering students do not carry new perspectives into their professional careers. Past research efforts have established that holistic, systems thinking approaches combat these challenges by allowing engineering students to address social, economic, cultural, and political implications of projects. Therefore, systems thinking could create more successful community-based projects and develop more holistic engineers.

Over the past decade, the emergence of service learning (SL) organisations within universities, such as Engineers Without Borders (EWB), reflect a growing movement in sustainable international development.From SL experiences, engineering students gain global perspectives and engage in "real world" problems unlike standard coursework, whilst working within complex and lowresource contexts.In professional practice, humanitarian engineering applications toward corporate social responsibility can increase the social and environmental sustainability of industry and engineering firms.Additionally, working in a global, cross-cultural context has been identified by industries as a valuable strength for engineering students (ASME 2011;Amadei, Sandekian & Thomas 2009;Chan & Fishbein 2009;Berg, Lee & Buchanan 2016).
Unfortunately, despite efforts toward building a more socially responsible profession, the true impact of SL experiences within humanitarian engineering has been questioned by recent literature.In 'A Methodology for Exploring, Documenting, and Improving Humanitarian Service Learning in the University', Berg et al. (2016) argues that amongst increased academic impact for the students, it is unclear whether SL activities truly provide "clear benefit to the communities involved or if those communities are being exploited in the pursuit of better education for the privileged" (p.6).Additionally, the long-term functionality (i.e.sustainability) of EWB development interventions has come into question.In a study conducted by EWB USA (2015), 202 completed projects were assessed to gauge project sustainability using three attributes of each system: high functionality, successfully performed maintenance, and adequate community capacity to sustain the project.Evaluation revealed that 9% of systems were non functional, 23% were not being maintained, and 12% of communities did not have the capacity to maintain the system (EWB-USA 2015).
To meet the need for improvements in EWB project sustainability, this paper evaluates a systems based project planning framework that can be used to strengthen EWB USA project planning and development activities by integrating systems thinking into the existing project planning structures, such as the EWB USA Planning, Monitoring, Evaluating and Learning (PMEL) framework (EWB USA 2014).The impetus for this approach follows the growing notion that international development projects are inherently complex, adaptive systems that require a holistic understanding of programmatic pathways toward positive community change (Amadei 2016;United Nations 2015;Neely 2015;Ramalingam & Jones 2008).The planning framework described in this paper, developed by Walters et al. (2017) uses group model building with system dynamics to foster systems thinking through factor diagramming and analysis.This study aims to build on Walters & Litchfield (2015) and Walters et al. (2017) by vetting their proposed approach with EWB practitioners.In the sections that follow, this framework, and the processes it contains, is briefly summarised and then is applied within a workshop conducted with a group of EWB USA student and professional team leaders.With the findings from this workshop, the utility of the framework is evaluated through critical observation of participant learning from the authors, as well as insights and perceptions from the participants collected in a post workshop survey.Finally, conclusions regarding the overall benefit of the approach and implications for future application within EWB USA and EWB-International are presented.

METHoDoLoGy
The planning and evaluation framework highlighted here is based on system dynamics modelling (SD), which was first established by Jay Forrester from the Michigan Institute of Technology (MIT) in 1961.Since then, the SD methodology has been applied to study many organisational, social, and technical interactions, most notably sustainable development (Meadows et al. 1972;Bossel 2007;Pruyt 2010;Hjorth & Bagheri 2006).This methodology utilises the development of qualitative and quantitative frameworks and models that evaluate the systematic interactions between and adaptations of technical, social, economic, and environmental factors that influence a particular outcome (Bossel 2007;Bagheri et al. 2010;Walters & Javernick-Will 2015) as well as the feedback mechanisms that emerge (Richardson 2011).Walters & Javernick-Will (2015) and Neely & Walters (2016) applied qualitative system dynamics modelling to assess the sustainability of rural water projects and demonstrated that using this approach can improve systems-based learning within the water for development sector.
A useful way to apply SD modelling techniques is within group model building (GMB) workshops, in which model building teams are guided through a series of steps to extract and shape group knowledge into informative qualitative and quantitative SD models (Vennix 1996;Luna-Reyes et al. 2006;Richardson 2013;Wolstenholme 1999).Within SD models, feedback loops emerge when circular causality takes place between factors (e.g., increase in global warming → melting icecaps → decreased solar reflectance from icecaps → increase in global warming, and so on).Through the process of identifying, discussing, and even simulating these cyclical influences, it is possible to hypothesise the root causes of a particular outcome or system behaviour (Richardson 2011).Walters & Litchfield (2015)  planning with an EWB-USA team working on a rural water project in Peru.This study was later expanded upon by Walters et al. (2017), wherein they formally propose the workshop framework incorporated within the present study.These past works establish an overarching premise that through group modelling, factors and feedback loops can be utilised by EWB-USA team members and project stakeholders to gain insight into the interdependent interactions and pathways between technical and nontechnical factors that influence project sustainability.

research context
To further assess the utility of integrating a GMB workshop into the EWB project process, this paper conducts and evaluates a shortened version of the workshop structure proposed in Walters et al. (2017) with a group of over 30 EWB practitioners and leaders at the EWB-USA West Coast and Mountain Regional Conference in Las Vegas, Nevada on October 15, 2016.This conference is one of EWB USA's largest annual, multi-chapter conferences, gathering EWB USA staff, members, and professionals who are invested in the future direction of EWB projects.Such a broad assortment of conference participants -each with their own unique experiences, perspectives, and insights into EWB project delivery -presented a potent and trustworthy arena to vet the value added of a GMB approach.During the hour-long session, participants were first introduced to the GMB workshop agenda and given a brief synopsis of system dynamics modelling terminology and iconography.Then, participants were guided through a process of workshop activities (Figure 1), which included factor brainstorming, drawing interconnections between and polarity of these factors, and identifying and characterising emerging feedback loops within the final group model known as a Causal Loop Diagram (CLD).Following the session, workshop participants were invited to provide additional feedback by being involved in post-workshop activities (Figure 1).These activities included further analysing the group model and discussing their general perceptions regarding the utility of the GMB approach to EWB project planning.Figure 1 illustrates the chronological flow between research activities in the research plan.

Workshop Activities
The workshop began by asking participants to brainstorm community-level factors that enable or inhibit the sustainability of a theoretical project.To increase individual involvement, the authors asked participants to first form smaller groups and then converged all participants to propose and discuss the proposed factors.
Next, the entire group decided on broad, concrete definitions for each factor.The authors explained that when performed with a single EWB project team, the participants would determine factors specific to the partnering community and project.Once a factor list was developed, the team wrote each factor name on a board and spent the remaining time systematically identifying the relationship between factors using arrows, building a CLD.Each relationship between two factors entailed the existence, polarity (+/-), and strength (scale of 1, weak to 3, strong) of the interaction, as detailed in Walters et al. (2017).Assigning influence polarity allows for the subsequent characterisation of feedback loops, where positive polarity (+) indicates a direct relationship in which an increase in one factor causes an increase in another (or vice versa), and negative polarity (-) indicates a direct relationship in which an increase in one factor causes a decrease in the other (or vice versa).Assigning influence strength allows for the prioritisation of the most impactful factors and feedback loops through structural analysis of the final group model, as described below.Overall, the final deliverable from these modelling activities is a customised systems-web of technical and non-technical factors that help participants understand factors influencing project sustainability.

Post-workshop Activities
After the workshop, the aforementioned structural analyses of the final group model were performed by the authors following previously established methodologies to first elucidate important factors (Walters andLitchfield 2015, Walters et al. 2017) and then identify dominant feedback mechanisms (Walters & Javernick-Will 2015).Centrality analysis was used to quantitatively evaluate the relative importance of each factor based on their interconnection within the model.Per Walters et al.  (2017), degree (direct factor to factor relationships) and betweenness centrality (indirect bridges between factors) were used to score and then rank factor importance based on their interactions.To mathematically perform these centrality analyses the authors used the free network analysis software Gephi version 0.91 (gephi.org).For loop dominance, the final group CLD was redrawn in the free SD software VENSIM PLE (vensim.com) to systematically identify emergent feedback loops.Then, the averaged sum of influence scores contained within each feedback loop was evaluated to reveal which feedback mechanisms were dominant (most influential) based on their average score.
For example, a four factor feedback loop with associated influence scores of 3 (strong), 2 (moderate), 2 (moderate), and 3 (strong), would have an average loop score of 2.5.Further details about these structural analysis processes are outlined in the referenced work.
The application and value of this approach was evaluated by EWB practitioners through a post workshop survey, which collected analysis, insights, and perceptions from the participants.The survey was administered in an anonymous, online format, and all workshop participants were invited to be involved.The post-workshop survey presented participants with a digitised version of the final group CLD, along with tables displaying the ranked factors for each centrality measure, and a table with the top five ranked feedback mechanisms.Participants were then asked the following questions to facilitate assessment of participant learning based on their interpretation of, and reaction to, the model results: 1. Is there anything that pops out when you look at this diagram?In particular, do you see any influences or feedback loops that appear to make sense (or not make sense) given the content of the project the group discussed?
2. Do any of the top five ranked loops in the table above seem to make sense (or not make sense) given the project context?If so, which one(s) pop out and why?What might you be able to conclude about the most important drivers on project success?
3. Given the nature of interpretation from each of the three centrality types outlined, do these results make sense or not?Are there factors that you think should have been (but were not) highly ranked, or vice versa, for any of these centrality measures?
Participants were asked to analyse these outputs and evaluate their ability to reveal project complexity, which were included for vetting and verification of previous work by Walters and Litchfield (2015).Building on this, participants were then asked to give their perceptions on the benefit and applicability of the workshop to EWB project planning and assessment by asking the following questions: 1. What do you think were the most useful aspects of the group model building process?
2. How did the GMB workshop enable insight (if any) into project complexity?
3. Were there any aspects of the modelling process that could be improved?If so, how might you recommend improving them?
4. Where do you think GMB workshops could be used (if at all) within the traditional EWB project planning and assessment phases (assessment, planning, implementation, monitoring and evaluation)?
5. On a scale from 1 (not useful at all) to 10 (very useful), how would you rank the utility of GMB workshops as a tool EWB student teams can use to plan and assess their projects?
The survey was designed to generate feedback that would reveal new perceptions about the approach from EWB professionals and practitioners and provide guidance and evidence for applications within EWB projects.

rESULTS & DIScUSSIoN
Execution of the aforementioned workshop and postworkshop activities yielded a final group CLD that was discussed by the group, structurally analysed by the authors, and then presented to workshop participants for evaluation and analysis within an anonymous survey format.In this section, the final group model is presented, along with the structural analyses used to interpret factor importance and loop dominance.Workshop participant reactions to the implications of these structural analyses, along with their perceptions of workshop utility, are then presented alongside author insights and recommendations for future application of GMB workshops within the EWB project planning and assessment process.

Workshop Session results
The group identified seven community-level interactions and definitions (Table 1) that were modelled into a CLD based on their influences (Figure 2).During the identification and definition of factors, over twenty factors were identified by the group, which they then combined and distilled down to seven overarching factors.Aggregation of factors stimulated rich conversation on factor meaning and interpretation.For example, in the case of the factor "Government", workshop participants discussed how government stability encompassed both internal and external governments.However, when focusing on the social structures within a specific community, participants agreed that the same broad factor could be broken down into the most important "Embedding Systems Thinking into EWB Project ..." -Pugel & Walters decision-making bodies, such as internal governments at the family, class, and occupational level, and external governments at the local, state, and national level.
Structural analysis of the final group CLD (Figure 2) by the authors resulted in ranked scores for each factor's centrality, including degree-out (influence on other factors), degree-in (dependence on other factors), and betweenness (indirect influence), as presented in Table 2.
Analysis also resulted in feedback loop dominance scores, including characterisation and ranking, displayed in Table 3.As mentioned previously, Tables 2 and 3 were then presented to the participants to aid in the structural interpretation of the final group model.2: A digitised version of the final group CLD, with influence strengths designated as 3-strong (red, thick), 2-moderate (blue, medium), and 1-weak (grey, thin).

Survey results
Twelve of the EWB workshop participants provided extensive analysis of and feedback on the GMB workshop through an anonymous survey regarding structural interpretations of the final group CLD and on the utility and application of the approach.Salient participant responses for each question, along with associated insights into the merit and applicability of the proposed framework, are shown in Tables 4 and 5.
Observations noted by the participants showed that substantial insight was gained regarding the influence and dependence between factors as well as the paramount importance of non-technical, community-based aspects that influence the implemented technology.Visual and structural analysis of the final CLD, enabled participants to critically evaluate traditional assumptions made within EWB project planning and discover potentially problematic power relationships between the community and the EWB team.For example, one participant pointed out the connection between community involvement and the community's financial contribution of 5% of overall project costs required by EWB-USA.Realising that introducing additional economic structures could disrupt traditional power dynamics, the participant noted that this could indirectly "hurt a community relationship" and therefore impact project sustainability.Table 4 summarises the type of descriptive language used by participants within the post workshop survey and a summary of the insights gained by the visual and structural analysis of the final group model.
Participant responses regarding the utility of the GMB workshop yielded positive and constructive feedback regarding the modelling process and the complexities that it revealed, along with ways to improve upon future workshops.Participants emphasised the importance of applying the GMB workshop to the project "planning and assessment phases", but that overall it would be helpful "throughout… it should really be a working model that can continuously be improved" (GMB workshop participant).When asked to rank the utility of the methodology as a tool that EWB student teams can use to plan their projects (survey question 8), 80% ranked it as a 7 out of 10 or greater.Table 5 expands further on the added value of GMB workshops, as well as ways in which they might be improved upon and used within EWB project planning and development activities.The diagram itself is a tool that can be used to visualise influences, and reveals the complexities of project sustainability."Getting people to clearly establish a line of logic and questioning if everyone was on the same page."

Top five ranked loops
"Seeing the visual interconnectedness" "Being able to rank which factors and loops are the most important can help with prioritising for projects" "The activity highlights problematic assumptions" The GMB workshop and final CLD can be used to facilitate discussion amongst teams and to reveal interconnectedness, priorities, and assumptions.

How GMB enabled insight into project complexity
"It clearly displayed how complex projects are.With all the arrows pointing all over the place and crossing everywhere, it is very clear how everything relates to each other and the importance of each factor." "It really helped to visualise the complexity, which I think could be really useful for new EWB members." "The workshop helped me see the big picture when it comes to putting together a project." "It was really cool to see the transfer of qualitative rankings to quantitative rankings so influences could be better analysed." Both the process of creating the model and the final CL diagram help to reveal project complexity.Through modelling, project complexity and ultimately sustainability can be clearer through both visualisation and quantification.

Improvements to framework
"This in only an hour (instead of three) the process was pretty flawed in my view.However, if I tried the three-hour process I might have better feedback on how to improve the process." "Difficult to discuss with such a large group (harder to have a shared understanding of things)." "Different trains of logic a person can follow and if the standards and definitions aren't clearly set then it's easy to get confused." Whilst systems thinking perspectives can be gained through this hour-long workshop, the proposed framework would provide longer sessions with a group that shares understanding of a single project with more specific factors.

Applicability to other aspects of EWB project lifecycle
"Throughout!Most importantly in planning and learning -but it should really be a working model that can continuously be improved upon." "It can be applied in every step, especially planning, implementing and evaluating.The more specific the circumstances, the better the model will work (I believe)" "I think these workshops could be useful during the assessment phase to determine a path to success, and again during monitoring to observe how closely the path was followed." The GMB process would supplement both the planning and assessment phases of EWB project planning and assessment by allowing for greater understanding of a project and its impact."Embedding Systems Thinking into EWB Project ..." -Pugel & Walters

Limitations and Future research
In combination with the inherent limitations of GMB and SD modelling, possible limitations exist regarding how this study assessed GMB workshop participant learning and evaluated the utility and merit of GMB for sustainable EWB projects.First, the nature of the study did not afford the evaluation of participant learning based on their prior knowledge.As such, it was not possible to truly ascertain before and after-based improvements in participant understanding.
Second, the open-ended survey questions provided only anecdotal evidence of workshop efficacy, where definitive evidence of GMB utility for EWB project sustainability would conceivably require investigating actual impacts of GMB activities on project outcomes.Finally, CLDs describe highly complex and often unquantifiable relationships in terms of semiquantifiable positive or negative relationships.Whilst this can be a useful way to gain insight into the complexities of a project, it might have provided an incomplete representation of the theoretical project used as an example in the workshop.
In support of the study's findings, however, Walters and Litchfield (2015) and Walters et al. (2017) definitively showed marked shifts in EWB team knowledge post GMB towards a more complete understanding of the systemic factors that influence project success.Similar improvements would be anticipated for future GMB workshops, including the one presented here.Moreover, Table 5 shows a connection between the exercise and participants' insights gained by modelling and interpreting important factors as a group.For example, participants gained a clear vision of the interdependent complexity of factors, developed a "big picture" understanding of the problem, and were able to transfer anecdotal understanding of the project to quantitative insights into factor importance, interaction, and dynamics.Often the true utility of modelling complex systems within a group setting are less contingent on creating an accurate model, and more in gaining knowledge on how system structure influences system behaviour (Bossel 2007, Vennix 1996, Box and Draper 1987;Walters et al. 2016).Moving forward, full verification of the merit and utility of GMB for EWB projects will require studies that draw connections between GMB-informed intervention strategies and practice with the sustainability of these interventions.Additionally, fruitful investigation could focus on the joint application of GMB workshops with community-level stakeholders as a way to validate and strengthen models developed and acted upon by project teams.

coNcLUSIoN AND rEcoMMENDATIoNS
As the engineering field works to become more socially informed and sustainable, it must broaden and adapt to incorporate socio-cultural, economic, and political considerations into applications of science and technology.Therefore, SL experiences and organisations can strengthen their impact in low-resource areas by adapting to incorporate systems thinking into their projects.Unfortunately, many SL projects, such as those implemented by EWB, struggle to fully realise this impact due to a lack of sustained functionality of the built technology.Systems thinking applied to a technical project can be a tool both to highlight the factors that influence sustained functionality as well as to develop a holistic project approach that more effectively promotes social justice.Here we highlight a planning and evaluation framework developed by Walters & Litchfield (2015) and Walters et al. (2017), which uses GMB workshops to engender systems thinking within EWB student team planning and learning.EWB practitioner perspectives are then evaluated to determine how and where the approach could be incorporated into EWB projects to increase project sustainability.
Presented in a workshop format to EWB-USA members at the 2016 West Coast and Mountain Regional Conference, the proposed approach reveals project complexity to GMB participants through the process of brainstorming and defining of factors, the discussion of factor influence, and the interpretation of the final group model (CLD).Participants learn that overall project sustainability is directly and indirectly facilitated by technical factors (that they are familiar with) as well as non technical factors (that they did not immediately consider), enforcing the findings of Walters & Litchfield (2015) and Walters et al. (2017).This work builds on these findings by evaluating the perceived value that this approach adds to traditional planning and evaluation activities based on EWB practitioner opinion and identifying key areas of EWB projects where this approach can be applied.By both visually and quantitatively identifying influential connections, the final group CLD can supplement an EWB project team's discussion of priorities and assumptions that affect project sustainability.Thus, supported by the enthusiastic and constructive feedback given by the GMB workshop participants, it is recommended that the GMB process be integrated into EWB project planning and development activities.As identified by EWB-USA members here, this process could be embedded as a key exercise within the long-term EWB-USA PMEL framework, and could greatly supplement the planning and project assessment phases.
Lastly, as evidenced through this work, the integration of GMB workshops into EWB project planning, assessment and learning can range from a short one-hour modelling workshop to a time-intensive reoccurring set of multi-day workshops held over a project's duration.One compelling example of this fact was mentioned by a workshop participant, who wrote: "Inspired by this workshop, we did a mini-version of this during our first project meeting to help newcomers "Embedding Systems Thinking into EWB Project ..." -Pugel & Walters get an overall picture of the project and understand its complexity.The more we discussed, the more people were able to think about the project, ask questions about it, and become more invested in it.We didn't go through the whole process of looking for feedback loops (due to time constraints), but it was definitely helpful just to have a way to facilitate project discussion." Indeed, it appears the added value of such systems thinking exercises, regardless of the intensity of application, can have a powerful impact on the individual and group's understanding of project complexity.

""
The capacity of the community and community involvement drive the technical aspects of the project (or at least should).When I look at the feedback loops I see community involvement and capacity of the community.""The two quantitative factors of cost and design which rely on technical knowledge are entirely dependent on qualitative assessments of the communityMaintenance has been such a critical issue in many of the water system failures in developing communities that it should be more complex and interconnected to variables than is represented here.I think the same factors that influence community involvement would influence in turn maintenance as well.""I would have thought maintenance would be higher ranked, but it makes sense that design and community involvement would be a bigger influence on the cost and sustainability of a project than the other way around."Whilst technical factors such as maintenance are a focus of project planning, participants realised that project failures such as a lack of maintenance may have been directly or indirectly caused by non technical factors.Thus, a more effective project would consider non technical factors.

Table 3 :
Loop characterisation and ranking

Table 5 :
Participant response on GMB workshop utility and application