O6 Evaluation of the effectiveness of Digital Skills for Culture Course, Quality Assurance and Sustainability
This output is concerned with the evaluation, quality assurance and sustainability of the project and its outcomes.
The Evaluation strategy of Digital Skills for Culture is part of the socio-cognitive engineering methodology (Sharples, 2002) implemented throughout this project and it is a continuing process. The evaluation will consist of different stages and will implemented at every output: evaluation of the concept and guidelines (O1), testing VLH against its own requirements for usability (O2); evaluating the Course Curricula and Content against the needs of the target group (O3), evaluating the e-assessment tools for digital skills against their specific requirements (O4) and evaluating the effectiveness of the pilots in improving digital skills of the adults involved (O5) .
Effectiveness is defined as meeting user goals. For online learning, there are many tools and techniques that are used to evaluate effectiveness (Andone, 2012; Mosakhani, 2010).
Evaluation of the effectiveness of the course and pilots will be conducted continuously and focus on dimensions directly affecting the effectiveness of MOOC: Interactivity, Collaboration, Pedagogy, Content, Technology, Usability, Motivation, Assessment and Support for Learners. A mix of qualitative and quantitative methods will be used to evaluate the effectiveness in order to better understand participants’ perspectives on the effectiveness of MOOCs. These will include an online questionnaire as well as qualitative insights into their experience during the course through testimonials, interviews and focus groups. They will be corroborated with the results from the learning analytics report (O2) in order to create a full picture for the final evaluation report.
The work in this output is divided into four activities – the Quality Assurance Framework (6.1), the overall Evaluation Framework (6.2), the Effectiveness evaluation of the course and pilot (6.3) and the Sustainability plan (6.4). These activities are critical for ensuring the success of the project and will guide work not only within the partnership but also in relation to the wider cultural and heritage community, and to stakeholders from other domains interested in policy formation or implementation in adult education through sustainability of project results.
This output applies concepts from innovation theory, such as the ‘community of innovation’ approach for defining the quality assurance framework and the ‘diffusion of innovation’ model for development of the Sustainability Plan (Rogers, 1995; von Hippel, 1985).
Output Type: Methodologies / guidelines – Evaluation method and tool
O6.1 Quality Assurance Framework
This activity will defines a quality assurance framework for the implementation of the entire project and its components. Quality assurance is a crucial issue in the design of any learning intervention, but particularly so in the case of OERs, MOOCs and any form or open or distance education where students will be accessing materials without the guidance of a teacher.
The methodology will include
– the design of quality assurance standards for single components of the project (especially O1.2, O2.2, O4.1 and O5.3)
– the design of quality assurance process for linking and integrating quality standards to all stages of development of intellectual outputs, e. g. by applying effective design and development methodology such as quality-driven agile development, peer-reviews of iterations and final outcomes as well as quality implementation process and possible corrective actions
– bi-monthly evaluation of quality of work in the project and reports on findings to further the implementation of the project.
Milestones:
O6.1.1.Design of quality assurance standards
O6.1.2. Design of the quality assurance process
O6.1.3. Design of evaluation instruments and periodical progress reports to guide project implementation
Leading: JMEA
Participating: all
O6.2 Evaluation Framework
Unlike QA which is ongoing, project evaluation is episodic, and will need to be conducted both at the end of the project and at key stages during the project (especially O2.5, O3.4, O5.3 and O5.4). It will use both internally collected data and compare it against information from other resources.
Evaluation will be conducted at the end of each intervention involving outside parties from the target groups that the project is designed to help (O5.2, O5.3 and O5.4).
Evaluation will be conducted using face to face interviews and focus groups during project meetings and other interventions, and by means of online tools (surveys) for larger samples.
Final evaluation at the end of the project will include if the project cost effective, if it can be scaled up and replicated elsewhere, and if it is self-sustaining.
For the final evaluation, the project may employ a consultant or outside organisation to provide a fresh perspective on its achievements.
Milestones:
O6.2.1 Identification of evaluation episodes
O6.2.2 Design of quantitative evaluation tools
O6.2.3 Design of qualitative evaluation tools
O6.2.4 Design and implementation of final evaluation
Leading: JMEA
Participating: all
O6.3 Effectiveness of the Digital Skills for Culture Course
To evaluate the course and pilot based on the socio-engineering method, we will also use a mixture of qualitative and quantitative methods in order to both evaluate the course and gather participants’ perspective on the effectiveness of MOOCs more generally (O3.4 and O3.5). This will make use of an online questionnaire with a variety of question types, together with the use of qualitative tools to gather views on experience during the course, including testimonials, interviews and focus group. The evaluation will be run on all partners’ courses, and in all languages for both fully online and blended learning implementations. The final target is to evaluate and validate which of the teaching methods used (fully online or blended-learning) is more appropriate to deliver adult education for the improvement of digital skills.
Milestones:
O.6.3.1. Design of evaluation plan for the Digital Skills course
O6..3.2 Implementation of Quantitative Evaluation
O.6.3.3. Implementation of Qualitative Evaluation
O.6.3.4. The Evaluation Report
Leading: JMEA
Participating: all
O6.4 Sustainability Plan
This activity focuses on the development of a sustainability plan for the project results beyond the project lifetime including the integration into national and EU policies. The methodology used is based on relevant concepts from innovation theory, including the diffusion of innovation and the adoption of innovation models.
Validation of the strategy will be achieved through various methods such as surveys, workshops and public meetings. The final sustainability plan will focus on leadership, objectives, strategies, outreach and activities. The sustainability plan will be developed in close connection with the quality assurance plan and with the involvement of key policy organisations inside and outside the partnership such as TM2021, EDEN, Rijeka2020, NoviSad2021, Kaunas2022, Italian Fondazione Idis-Città della Scienza, Foundation InterArt Triade, Foundation ArtEncounters, European Architects Association, Network of European Museum Organisations (NEMO), etc.
Milestones:
6.4.1 Design of Sustainability Plan
6.4.2 Dissemination of Plan within the partnership
6.4.3 Dissemination of Plan with outside organisations
Leading: JMEA
Participating: all
