A portal of educational resources: providing evidence for matching pedagogy with technology

Nicoletta Di Blas, Alessandro Fiore, Luca Mainetti, Roberto Vergallo, Paolo Paolini
A portal of educational resources: providing evidence for matching pedagogy with technology

ORIGINAL RESEARCH ARTICLE

A portal of educational resources: providing evidence for matching pedagogy with technology

Nicoletta Di Blasa*, Alessandro Fioreb, Luca Mainettib, Roberto Vergallob and Paolo Paolinia

aDepartment of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, Italy; bDepartment of Innovation Engineering, University of Salento, Lecce, Italy

(Received 25 September 2013; final version received 19 February 2014; Published: 6 May 2014)

Abstract

The TPACK (Technology, Pedagogy and Content Knowledge) model presents the three types of knowledge that are necessary to implement a successful technology-based educational activity. It highlights how the intersections between TPK (Technological Pedagogical Knowledge), PCK (Pedagogical Content Knowledge) and TCK (Technological Content Knowledge) are not a sheer sum up of their components but new types of knowledge. This paper focuses on TPK, the intersection between technology knowledge and pedagogy knowledge – a crucial field of investigation. Actually, technology in education is not just an add-on but is literally reshaping teaching/learning paradigms. Technology modifies pedagogy and pedagogy dictates requirements to technology. In order to pursue this research, an empirical approach was taken, building a repository (back-end) and a portal (front-end) of about 300 real-life educational experiences run at school. Educational portals are not new, but they generally emphasise content. Instead, in our portal, technology and pedagogy take centre stage. Experiences are classified according to more than 30 categories (‘facets’) and more than 200 facet values, all revolving around the pedagogical implementation and the technology used. The portal (an innovative piece of technology) supports sophisticated ‘exploratory’ sessions of use, targeted at researchers (investigating the TPK intersection), teachers (looking for inspiration in their daily jobs) and decision makers (making decisions about the introduction of technology into schools).

Keywords: educational portal; educational repository; open educational resources; educational technology; the TPACK model

*Corresponding author. Email: nicoletta.diblas@polimi.it

Research in Learning Technology 2014. © 2014 N. Di Blas et al. Research in Learning Technology is the journal of the Association for Learning Technology (ALT), a UK-based professional and scholarly society and membership organisation. ALT is registered charity number 1063519. http://www.alt.ac.uk/. This is an Open Access article distributed under the terms of the Creative Commons CC-BY 4.0 License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: Research in Learning Technology 2014, 22: 22906 - http://dx.doi.org/10.3402/rlt.v22.22906

Introduction

In the past decade, a large number of repositories of educational resources have flourished on the Internet. Some (or most) of them are open, while others require a fee. Some of them are meant for teachers, others for students and families. They cover all school grades, from pre-school up to higher education, and all possible subjects. Examples are MERLOT, CAREO, Wins-Online, Orange Grove Repository (OER, 2013), Florida’s digital repository, the (Khan Academy, 2014) and (BrainPOP, 2014).

A common feature of these repositories is what may be defined as a ‘content-centric’ approach. Emphasis is first and foremost on what the educational resource is about (e.g. ‘volcanoes’), next on what subject it is related to (e.g. ‘science’) and finally on what school grade it could be applied to (e.g. ‘3rd grade’). This organisation is meant to answer practical needs such as ‘how can I introduce volcanoes to my 3rd graders?’ (by a teacher) or ‘how can I learn more on volcanoes?’ (by a student). But, a number of crucial needs such as ‘how can I improve collaboration within my class?’ or ‘how can I use digital storytelling to foster media literacy?’ (by teachers) or ‘what impact are technologies having on education?’ (by a district manager) are not addressed at all.

To answer questions like the ones above, this paper proposes a novel approach in which technology and pedagogy take centre stage. A portal, backed up by a repository, gathering almost 300 educational experiences involving the use of technology has been built. Among other features, technology factors involve types of hardware, types of software, ways of using HW (Hardware) and SW (Software) and types of activities. Pedagogy factors involve organisation (heterogeneous group-work, homogenous group-work, role-playing, individual assignments, etc.), inclusion problems and solutions (hospitalised children, disabled students, immigrants, disaffected students, etc.) and educational benefits (cognitive benefits, enhanced motivation, improved relationships, etc.). The core idea is to encourage users to look at the experiences not because they cover a specific subject (as in the content-centric approach) but because they face specific pedagogical issues and/or make use of specific technologies. Figure 1(A and B) shows how the portal looks like.

The new paradigm has been designed and implemented in a portal, backed up by a repository that gathers 274 educational experiences from pre-school to high-school (see Figure 1. Each educational experience is tagged through a complex taxonomy encompassing more than 30 parameters (‘facets’) and almost 200 ‘facet values’ (Figure 1B). The repository and the portal were designed and implemented in the frame of L4ALL (Learning For All), an Italian national research project set up to investigate in depth the impact of technology on teaching and learning practices (Ferrari et al. 2012).

Fig 1
Figure 1.  The L4ALL portal of educational experiences; each tile stands for an educational experience (A). (B) shows some of the facets according to which the experiences are classified.

The description of an educational experience includes a detailed analysis based on TPK (Technological Pedagogical Knowledge) factors and various kinds of documents, including audio files of interviews with the teachers, transcripts, materials produced during the educational activities, multimedia presentations and videos. Three types of users are envisioned: teachers, researchers and decision makers in the education field. Preliminary evaluations have been carried out via focus groups with teachers and via experimental research with scholars, investigating seven different issues related to TPK. District managers (at a regional level in Italy) have shown great interest in the portal. The data show high levels of interest and satisfaction. The main request by users is ‘for more’, in terms of the number of experiences available.

The paper is organised as follows. First, the background of the TPACK (Technology, Pedagogy and Content Knowledge) model and an overview of existing repositories of educational resources is provided. Next, the repository and the portal are described, starting with requirements and ending with implementation. Then, the evaluation is introduced, leading to a final section on conclusions and future steps.

Background and the state of the art

TPACK model

The TPACK model/framework (otherwise known as the TPCK model)1 stands for the ‘technology, pedagogy and content knowledge’ model. The model was introduced by Mishra and Koehler (2006) following the insights of Shulman (1986a, 1986b), who in the 1980s had invented the PCK (Pedagogical Content Knowledge) model. The PCK model advocated the need to consider the intersection between pedagogy and content knowledge and train teachers accordingly. The TPACK model follows in the footsteps of the PCK model and advocates the need to consider the intersections between TPACK in order to effectively train 21st century teachers (Koehler and Mishra 2005, 2008; Koehler, Mishra, and Yahya 2007; Mishra and Koehler 2005).

The TPACK model has witnessed enormous success. It would be beyond the scope of this paper to account for all the various implementations, interpretations and developments it has fostered in only a decade. As of June 2013, Mendele (2013) listed 466 academic papers on the TPACK model. A good literature review can be found in Voogt et al. (2013).

Among the various trends of research that stem from the TPACK model, the one of specific interest to this paper discusses the interrelation between the TPACK components, arguing in favour, or against, the relative independence of each ingredient. As Voogt et al. (2012) point out, three different understandings of the TPACK model as a concept have been developed: TPACK as an extension of PCK (Cox and Graham 2009; Niess 2005), TPACK as a unique body of knowledge (Angeli and Valanides 2009; Archambault and Barnett 2010) and TPACK as the interplay of three different knowledge domains (Graham 2011; Koehler and Mishra’ 2005, 2008, 2009, Koehler, Mishra and Yahya 2007). In the first case, TPACK is seen as the integration of subject matter knowledge with technology, teaching and learning knowledge (Niess 2005, p. 510). In other words, TPACK is considered to be an evolution of PCK (Cox and Graham 2009). The integration of these three domains is interpreted as a way to teach using technology (Voogt et al. 2012). In the second understanding, what may be defined as a transformative view of the TPACK model is adopted. For example, Angeli and Valanides (2009) argue in favour of TPACK as a distinct body of knowledge that can be developed and evaluated on its own. In the third understanding, Mishra and Koehler put forth a different interpretation. They see TPACK not as an enhancement of PCK nor as a distinct body of knowledge but rather as a progressive understanding of TK (Technological Knowledge), PK (Pedagogical Knowledge) and CK (Content Knowledge) and all their intersections, leading to seven distinct bodies of knowledge (Koehler and Mishra 2008). Cox and Graham (2009) and Niess (2011) underpin the difficulties that such a conceptualisation, with its seven intersections, brings about. The model is further complicated by the fact that context is added by Koehler and Mishra as a fundamental part of the framework, as an element requiring deep understanding on the teacher’s side. Koehler and Mishra’s position can be defined as an integrative view of TPACK.

With respect to the integrative view of TPACK, it can be observed that the description of an educational experience can address different aspects, including how it was devised, how it was implemented, what was important when it was conceived and what surfaced as important after completion. In addition, the interplay between the different components of the TPACK model can have several interpretations. A teacher, for example, may start with the idea of using tool ‘t’ and then decide that for her goal the best pedagogical approach is approach ‘p’. Once the educational experience is made available in a public portal (such as the one presented in this paper), it may happen that another teacher takes a different path. Investigating experiences with approach ‘p’, he/she may be led to take tool ‘t’ into consideration. In other words, a good educational portal should be able to support several different conceptual pathways, possibly ones not even corresponding to the original goal of the teacher who created the experience in the first place.

Educational repositories

In the mid-1990s, relatively simple learning objects were made available informally as instructors shared syllabi, lesson plans and learning activities. Later, more complex and/or topic-specific repositories came into existence as museums, journals and magazines, educational television productions and other actors placed content on the Web and encouraged it to be used for educational purposes. Many institutions developed learning activities around the objects (thus creating larger, ‘second-order’ learning objects) and made them available from their websites.

Afterwards, websites became populated by learning objects from all disciplines but, instead of housing the objects on their own servers, they often linked them to the original sites where the resources were made available. Large repositories of learning objects are now available on sites such as MERLOT (Schell and Burns 2011), a leading archive for higher education that provides tools for user comments and peer reviews; CAREO (Neven and Duval 2012), a collection of Web-based multidisciplinary teaching materials; and Wisc-Online, which houses the online resources supporting Wisconsin’s technical colleges.

Examples of repositories of learning objects for use in specialised training include those from the aviation industry for which the Aviation Industry Computer-based Training Committee helped in setting standards. Another repository is the (NMC, 2007) Learning Object Initiative, thanks to which many industry and military issues have been resolved. Other early examples in this area include the Educational Object Economy (Roschelle et al. 1999) and Ariadne (Forte et al. 1997). More recently, spurred by the development of the ‘learning object metadata’ standard of the IEEE Learning Technology Standards Committee (LTSC), numerous initiatives have been launched in academic and corporate contexts (Neven and Duval 2012; Primo, Vicari, and Bernardi 2012). Several standards have also been developed in the field of e-learning systems (ELS) as reported by Cos Garcia (2011).

Learning object repositories can be difficult to navigate, and the educational material difficult to integrate into online courses. Schoonenboom, Sligte, and Kliphuis (2009) observe that the literature on the reuse of learning materials has largely focused on the development of materials. The authors developed guidelines that support staff and/or management in cases of (un)successful reuse of existing digital materials and provided methods for teachers in higher education in such cases. They collected existing strategies from 11 literature studies and developed new guidelines from 19 (mainly Dutch) case studies. Lukasiak et al. (2005) present the Smart Learning Design Framework that supports sharing, reuse and adaptation of learning material via a metadata-driven philosophy enabling the technicalities of the system to be imperceptible to the author and consumer. The system proposes the use of pedagogically focused metadata to support and guide the author and to adapt and deliver the content to the targeted consumer. Even though many educational technologies and services have been provided to enhance educational pedagogy, in contrast with what was expected, a drop in participants and subscribers to these educational technology resources among students and instructors was recorded in many different studies (Cuthrell and Lyon 2007; Jones 2009; Willems 2007). These studies have also identified that the reason for such a drop was that a ‘one-size-fits-all’ approach was used instead of an approach that provided educational courseware for students using all available educational technologies (Cuthrell and Lyon 2007; Matar, Khwaldeh, and Hunaiti 2007). Many adaptive ELS were created to satisfy students’ needs and learning styles using different approaches, but their focus and intent was not on unified learning object repositories (Aroyo, Mizoguchi, and Tzolov 2003; Dolog et al. 2004). Andrew, Farrow, and Cooper (2012) argue that online resources for education not only offer opportunities for those with disabilities but also raise challenges regarding how to best adjust resources to accommodate accessibility. The authors carried out a study within the EU4ALL (European Unified Approach for Accessible Lifelong Learning) EU-funded accessibility project at the Open University, United Kingdom, in parallel with studies at three other European universities, developing systems and services to give personalised access to online materials that include alternatives.

To overcome these limits, unified E-learning repositories (UER) were used for simplifying the tasks of sharing learning contents and resources between different universities through a unified access point. Such systems are capable of providing management information regarding contents and users; in addition, they give instructors and course authors the ability to reuse learning contents and to assemble them into courses (Hatala et al. 2007). Different initiatives have been implemented to support the idea of UER among organisations and educational institutions. The first one is called the centralised LCMS (Learning Content Management System) approach, where a central LMS (learning management system) is used to provide the shared courses as ready-made packages that can be used by instructors (Matar, Khwaldeh, and Hunaiti 2007). Such an approach can be performed using any available LMS such as Moodle, ATutor, Blackboard and WebCT. The second one is called the networked approach (NA). The focus of NA is on the learning objects level rather than on ready-made courses, so that each instructor can use learning objects and structure them into courses (Pouyioutas and Poveda 2005). An example of such a system is the learning object network with CAPA. (LON-CAPA, 2013) provides unified e-learning through a dedicated network; in order for universities to use the system, they should structure their servers to be a part of the learning network. LIONSHARE is another UER-oriented platform that uses NA for sharing resources between learners. SPLASH is an NA-oriented system that is actually used as a distributed learning objects repository/metadata tagging tool. SPLASH was developed as a part of the Portal for Online Objects in Learning (POOL) project, which is a consortium of several educational private and public sector organisations aimed at developing an infrastructure for learning object repositories (Perkins 1995). POOL addresses the issues of building such architectures including metadata, software and hardware, and bootstrapping the system with initial content. It also makes tools available for download to help set up similar infrastructures elsewhere and to connect them to POOL.

The authors observe that the tendency of current repositories is to retain content in the form of a broad mix of text documents, videos, audio files and graphics (EDRENE 2009). It also emerges that a few repositories include non-digital materials (e.g. text books). A little less than a third of repositories surveyed have a mix between free and commercial material. What is relatively clear is that educational repositories are mainly created to share learning objects, often characterised by metadata or ready-made courses, intended as an organised set of learning resources related to a specific discipline. However, they largely fail to provide a whole, fully described and reproducible learning experience that can clarify when, where and how materials, digital or not, were used; how the learning process was organised; what educational goals were planned; which educational benefits were generated and what the role of the technology was.

In order to schematise and understand the circumstances and the outcomes around an educational experience, studies have been conducted and conceptual and meta-learning frameworks have been proposed. The study in So (2012) endeavours to understand how teachers make use of online resources to design learning environments that are motivating and cognitively engaging to their learners. The final aim is to propose a resource-based e-learning environment (RBeLEs) framework, based on five teachers’ authentic experiences in planning, designing, implementing and reflecting on such learning environments. In-depth interviews were conducted to collect more information. In-depth interviews have the primary advantage of providing much more detailed information than other data collection methods such as questionnaires, and they are particularly useful when detailed information about a person’s thoughts and behaviours is wanted (Boyce and Neale 2006). The five participating teachers were individually interviewed face-to-face for about 30 minutes before and after lesson implementation. All interviews were semi-structured, audio-recorded, and transcribed. In the pre-lesson interviews, the teachers were encouraged to describe how they had created the teaching designs and how they had planned to implement them in the classroom. The post-lesson interviews, in contrast, focused on understanding how the teaching designs were actually implemented, and what the pupils’ motivations and cognitive engagements were in the lessons.

Frameworks for supporting knowledge capture at source (Verhaart 2002) and collaborative knowledge management systems (Maher and Kourik 2008; Nunamaker 2001) have been proposed in different fields, for example in e-Government (Zhou 2008). The need to assess quality levels in engineering educational programmes led ABET (Accreditation Board for Engineering and Technology) to issue the Engineering Criteria 2000 (EC2000) standard (Lattuca et al. 2006). EC2000 shifted the basis for accreditation from inputs, such as what is taught, to outputs, what is learned. The new criteria specify 11 learning outcomes and require programs to assess and demonstrate their students’ achievement in each of those areas. In Besterfield-Sacre et al. (2000), a framework based on Bloom’s taxonomy (Bloom et al. 1956) has been developed for better specifying the EC2000 outcomes. Bloom’s taxonomy is based on six levels of the cognitive domain: knowledge, comprehension, application, analysis, synthesis, and evaluation. To meet EC2000 assessment criteria, information from all university constituents needs to be routinely collected and tracked longitudinally. The management of EC2000-related information through a data warehousing approach is the main focus of Ingham (2000); the data warehouse approach allowed the management of 100,000 student records covering a 10-year span, as well as 30,000 course records over a similar period of time. This has also led to the advantage of supporting the assessment process.

The OER is Florida’s digital repository for instructional resources. It provides an environment for educators to search for, use, remix, share and contribute educational resources. The repository can also import resources directly from existing LMSs (e.g., Blackboard, Desire 2 Learn, Canvas). The diverse collections available for searching contain a wide range of resources, including open textbooks. Resources are free for educational use but may be protected through various copyright statements associated with each resource. The main audience of the repository is Florida public higher education faculty and staff, but Florida K-12 educators, educators outside of Florida, and guests can also contribute resources.

A growing number of repositories explicitly meant for students and families are becoming available for all levels of schooling. The Khan Academy (KHAN) is a not-for-profit organisation with the goal of bringing innovation to education by providing online materials and resources available free of charge for anyone, anywhere. Students can make use of an extensive video library, interactive challenges and assessments from any computer with access to the Web. Coaches, parents and teachers can have different levels of visibility regarding what their students are learning and doing with the Khan Academy resources. The BrainPOP repository (BrainPOP) encompasses more than 650 curricular topics designed to actively engage students from grades 3 and up. It creates animated, curriculum-based content that supports educators and students in school, at home, and on mobile devices. Resources include movies, quizzes, games, mobile apps, experiments, activity pages and other kinds of learning content covering hundreds of topics within math, science, social studies, English, technology, arts and music and health. All content is aligned to and searchable by US state standards, including Common Core State Standards. PoliLab Kids (PoliLab, 2013) is an Italian initiative that released an online repository of educational videos (lasting 2–3 minutes each) related to various areas (science, math, technology, etc.). Each video is accompanied by a rich set of educational materials including quizzes and a teacher and parents’ guide.

With the aim of driving and supporting the implementation of innovative technological experience at school, a number of technological and conceptual frameworks have been proposed (Bucciero et al. 2011; Di Blas et al. 2012; Vergallo 2012). However, they do not include a sharing platform like a repository, so they lack structuring of the material that is aimed to make it easier to share and reuse the experience. Also, it is evident that adopting well-designed collaboration process models (Barchetti et al. 2011) for supporting empirical research about the role of Information and Communication Technologies (ICT) and its influence on pedagogy-related concepts (various kinds of implementations, inclusion problems and solutions, envisioned benefits, etc.) could ease information-intensive research activities and improve results. Starting from these considerations, we defined the main requirements of the repository’s collaboration and data model as will be described in the next section.

The repository and the portal

The repository and the portal described in this paper were designed and implemented in the framework of the research project, L4ALL, and aimed at detecting how ICT is affecting learning processes at school. The project, taking an evidence-based approach, investigated a large number of educational experiences where ICT had played a prominent role.

Each experience generated a fair amount of material (interviews, analysis, documents, lesson plans, etc.). This sizeable body of documentation was collected with several aims in mind:

  • to foster research by scholars both within the project itself (L4ALL) and the community of researchers at large,
  • to inspire teachers through the evidence of works and experience by other teachers, and
  • to support decision makers in the field of education.

In order to make the reader better understand what kinds of user experiences we mean to support, let us introduce two hypothetical scenarios of use.

Scenario 1

A scholar is investigating the relation between different kinds of group work in digital storytelling activities and the educational benefits achieved by the participants. She accesses a portal where hundreds of educational experiences are gathered. Each educational experience is accompanied by a rich set of materials (documents, presentations, videos, etc.) and is classified according to a set of 30 facets (technology used, benefits, organisation, inclusion issues, etc.) and a high number of facet values (for the facet ‘benefits’, cognitive benefits, relational benefits, etc.). First of all, she selects ‘digital storytelling’ as a technology-based activity. Then, she selects all the kinds of group work (there are four: homogenous, heterogeneous, same-skill and different skill group work) within the facet about organisation. She thus creates S1, a subset including all experiences that combine storytelling with all kinds of group work. S1 shows that digital storytelling and group work generate a number of benefits, among which relational benefits stand out as prominent. She now wonders whether this activity (digital storytelling implemented through group work) may have an impact on inclusion issues, but she wants to leave aside disability since she is not an expert in that field. She just wants to focus on disaffected students, immigrants and students with a poor background. She thus gets to the facet ‘inclusion issues’ and excludes ‘disabilities’. She gets S2, another useful subset that she saves for further investigation.

Scenario 2

A high-school teacher in humanities wants to improve her use of ICT at school, and therefore accesses a portal like the one described above. First, the teacher browses the initial set (S0). She takes a look at the facet about technology, getting an idea of what technologies are most popular. She also takes a look at the ‘organisation’ facet, to see what kinds of pedagogical implementations (heterogeneous group-work, role-play, etc.) are used. Then she selects interactive whiteboards and tablets within the facet about technology (creating subset S1). She has an interactive whiteboard in her class but she seldom uses it. Moreover, she has been told that the school principal will buy a tablet for each student, so she needs to get prepared for this change. She further refines her exploration by selecting ‘motivational benefits’ within the facet about benefits and ‘group-work’ within the facet about organisation (creating subset S2). There are many disaffected students in her class, so it may be a good idea to start trying to involve them by getting the help of the performing students. Now, she notices with dismay that in S2, technical disciplines are overrepresented. Therefore, she goes back to S1 and adds the criterion that humanities be the overall discipline. She also excludes ‘immigrants’ as a problem within the facet about inclusion issues (in the profile of S1 ‘immigrants’ is a major issue, but there are none in her class). She gets S3, a small, interesting set. She saves the contents to investigate them at a later time.

In order to support the aims of the project exemplified by the above scenarios, two developments were needed:

  • A repository: to collect (in an organised fashion) all the materials available for each experience, along with managerial information (dates, who the author is, etc.). Users are the staff of the project.
  • A portal: to provide effective access to the educational experiences. Users are scholars (both within and outside the project’s team), teachers and decision makers.

Although they share a fair amount of information, the repository and the portal (given the different aims and the different types of users) required different design concepts and different implementation strategies that are described in what follows.

The repository

The repository was co-designed and implemented by the GSA-Lab (Graphics and Software Architectures Laboratory) of the University of Salento and the HOC-LAB (Hypermedia Open Center Laboratory) of Polytechnic of Milan. Its main aim was to store educational experiences and ease the collaboration among staff. The repository was the ‘backstage’. In other words, external users (researchers, teachers, decision makers) did not access it nor did they have any perception of its existence.

The design of the repository was driven by two inputs, the ‘static model’ of the information to be stored and the dynamic aspects of the workflow (for experience collection and analysis). The latter was quite complex and distributed among various actors. If the requirements for the ‘static model’ were relatively easy to elicit, the analysis of the workflow was a far more difficult task. The staff of the project were altering their methods as the project proceeded; therefore, the workflow was continuously redefined. At the end of the project, the following were the main roles that surfaced:

  • Interviewer: she carried out the interviews and produced all the materials concerning an experience (as described in the next section).
  • Editor: she reviewed the materials produced by the interviewer, improving the overall quality. In addition, she ‘tagged’ the experience using the proper facet values.
  • Coordinator: she orchestrated and supervised the work of interviewers and editors. Furthermore, she approved publication of the educational experiences in the public portal.

Experience data model

The objects stored in the repository were not simple files or learning contents but instead were educational experiences. From a technical perspective, an experience is a collection of resources (files and descriptions) related to the implementation of a technology-based activity at school. The experience data model defines the type, number and structure of such resources within an experience (see Figure 2).

Fig 2
Figure 2.  Data model of an educational experience within the L4ALL project.

A large variety of materials can be associated with an experience:

  • Interview(s): audio file(s) of interviews of the teacher (before and after the experience).
  • Interview transcripts: the interviews are transcribed and filtered, that is, irrelevant parts are removed and more colloquial sentences are summarised in order to make the transcript more readable.
  • Abridged interview (optional): if the interview includes relevant or interesting sections, the interviewer can decide to extract such parts in a specific document.
  • Contents: any other useful resource, such as materials produced in the course of the experience (e.g. multimedia presentations).
  • FEE (FEatures Extraction) forms: FEE forms are the schemas that structure the relevant information hidden in a set of resources about the experience (characteristics of the whole class, the social/cultural/economic context of the school, the rationale behind the experience, etc.). In L4ALL, there were three different forms: one about expectations, one about results and another about a comparison between the two (performed by a scholar).

Each interviewer had to manage the lifecycle of the experiences assigned to him or her within the repository. A minimal set of metadata (for each of the objects) was added both for fast identification and for managing purposes. So, it was easy to find items such as what was done on a given day, how many interviews the repository held, which FEE forms were missing and what the job done by a specific interviewer was.

Experience collection process

The collection and sharing of educational experiences consisted of four sub-processes: interview, review, tagging and publishing (see Figure 3). There were two types of interviews, one about expectations and one about results. The ‘expectation interview’ was taken before the educational experience began to investigate what the teacher was looking for. The ‘results interview’ was taken after the educational experience was completed. The audio of the interviews was recorded, then a transcript was made and the writing was tightened to eliminate what was not relevant. After that, the three FEE forms were filled in: expectation, results and a comparison between the two. Next, the editor reviewed all the materials and tagged the experience, interpreting the main findings of the FEE forms.

Fig 3
Figure 3.  The collection and sharing of educational experiences.

Implementation

Flexibility in implementation was a key requirement, since the repository actually underwent a continuous cycle of design–implementation–evaluation–redesign as the project moved on and new needs surfaced. Alfresco Enterprise Content Management (ECM) software (Alfresco software 2013) was selected. The key technical features driving this choice were the following:

  • Alfresco is a lead ECM platform, widely adopted by both private enterprises and public corporations. Since the first release in 2005, it has evolved adding several features: Web content management, versioning, records management, workflow management, and so on.
  • The Alfresco Community Edition is free and open source. Alfresco’s source is freely downloadable and customisable. Moreover, the Alfresco community is very active and lots of add-ons are available.
  • In Alfresco, the repository’s interfaces are highly decoupled from the core document management. It is distributed as composed by two main Web applications, Alfresco Explorer, which provides main administration capabilities, and Alfresco Share, which provides a rich customisable collaboration environment.
  • Alfresco provides built-in user profiling management and configurable workflow management. Five versions of the repository were released without a significant re-working effort.
  • Alfresco widely supports open standards that allow a fast content transfer with other ECM environments.

Figure 4 shows the high-level architecture for the repository. The experiences request broker is an architectural element that can arrange the stored experiences and dispatch them according to three different perspectives (interfaces). In our architecture, the user views were decoupled from the experience management (insertion, updating).

Fig 4
Figure 4.  High-level architecture of the repository and portal space.

Figure 5 shows the detailed architecture of the repository. It includes the internal interface, the experience’s data entry built on top of the Alfresco ECM and the search interface. The internal interface was used by the interviewers and was deployed in Alfresco Share as a custom extension. This was structured according to three layers:

  • The persistence layer used the file system to store the experiences materials, while the database was needed for storing user information, user groups, content categorisation, and so on.
  • The business logic layer allowed the development of custom extensions to manage documents, users, groups, categories, permissions, and so on.
  • The integration layer let user interfaces interact with the business logic.

Fig 5
Figure 5.  The detailed repository and the portal architecture.

The search interface mainly intertwined tag-based, full Boolean, weighted and visual search strategies, to provide a rich experience search engine. The tagging experts used both the internal and the search interface. They downloaded the experiences to be tagged from the Alfresco repository, used QSR NVivo 9 (QSR International 2013), a third-party tagging software installed on their desktop environments, then exported the tags in Microsoft Excel format and uploaded xls files to the search interface through an ad-hoc view.

It was crucial to let the repository and the portal remain absolutely independent, to allow both the interfaces to evolve separately and to keep one of the subsystems online while the other was in maintenance/update mode. Developing an Alfresco extension, we defined the experiences data model as a common language adopted by the two interfaces. Moreover, we configured the interaction between the two interfaces by minimising their mutual relationship.

The Alfresco extension we developed included an experience export mechanism for transferring the data from the internal (the repository) to the external interface (the portal). The coordinator partner performed the export task from time to time, for example, when a massive data update was done. It allowed transferring of the resources’ links, and of the content metadata as well, for each completed experience.

The portal

The portal was designed and implemented by HOC-LAB of Polytechnic of Milan, according to the following requirements:

  • to support an innovative way of accessing rich information, emphasising exploration and serendipitous discovery over search,
  • to accommodate ‘niches’ of users (scholars, experienced teachers, novice teachers, managers, experienced researchers, novice researchers, etc.),
  • to emphasise pedagogy and technology (and their intersection) rather than the specific content (subject matter) to which they had been applied,
  • to combine very synthetic information with a large variety of additional content (audios, videos, documents, etc.) without any specific constraint,
  • to combine at-a-glance understanding with the possibility of focusing upon details, and
  • to allow different scenarios of use: supervision, decision making, focused research investigation, broad research investigation, generic inspiration, focused inspiration, and so on.

The portal was implemented using standard Database Technology and an ‘experience engine’ with a novel interface. The portal was made public in December 2012. It currently hosts about 274 educational experiences, carried on in Italian schools, where the use of technology has been crucial.

In order to provide an effective exploration, a rich taxonomy of more than 30 facets and almost 200 facet values was developed. The taxonomy was revised several times since it was crucial to combine powerful exploration with understandability and usability. Thanks to an innovative HCI (Human Computer Interaction) approach (and novel supporting technology), the taxonomy is at the same time a tool for navigation and exploration as well as a tool for feedback. Once a value is selected within a facet, all the other values in all other facets ‘react’ to the choice. For example, if primary school is selected as school level, all the other facets (e.g. the technologies used) modify their values accordingly (so, for example, a scholar can see what technologies are used at primary school level).

Table 1 shows the list of the main facets. They range from factual elements (e.g. which tools are used) to interpretative ones (e.g. the impact generated by the adoption of a technology).


Table 1.  The main facets to each facet, a set of possible values (from 4 minimum to 20 maximum) is associated.
Facet Description
School year Year in which the educational project took place. Rationale: technology rapidly changes. Knowing the year of deployment helps the user to better contextualise what s(he) will read.
School level School level at which the educational experience took place, for example, pre-school, primary school. Rationale: the school level has a crucial impact on the most relevant factors in education–implementation, teacher’s attitude towards inclusion, envisioned benefits, etc.
Monodisciplinary versus Multidisciplinary Whether the project is related to one or more than one subject.
Subject Areas of study and subjects involved. Main distinction between humanities and STEM (Science, Technology, Engineering, and Mathematics) subjects. Further values list all possible subjects (literature, art, history, chemistry, etc.).
Format A format is a pre-defined technology-based educational activity that implies, though they are not strictly mandatory, specific technologies, specific implementation procedures, a workflow of activities, etc. For example, ‘PoliCultura’ (i.e. a specific format for multimedia storytelling deployed by Politecnico di Milano).
Initiative An initiative is a broad ‘umbrella’ (a ministry-funded project, a course, etc.) under which different formats can be found, for example, ‘Classes 2.0’, a ministry-funded initiative to introduce interactive whiteboards into some selected Italian schools.
Geographic location Where the experience took place (region, county, etc.). Rationale: these data are crucial for statistics about technology use in education, for example, for comparing southern with northern regions.
Social context The socio-economic level of the students’ families–‘low’, ‘average’, etc. Rationale: the learning attitudes and performances of the students are oftentimes influenced by the context.
Cultural context The cultural and educational level of the students’ families. The same rationale as for the social context.
Class performances Level of the students’ performances during normal school activities. Rationale: when results of a special activity are analysed, they should be compared to the typical performances of the students during normal school activities.
Class homogeneity Level of homogeneity of the students in terms of performances. For example, they can be ‘highly homogeneous’ but in the sense that they are almost all poorly proficient. Rationale: it is very important for a researcher/teacher to know how the class is performing in order to understand or take inspiration from a specific experience.
Hardware The most important pieces of equipment used during the experience–. PC, iPad, video camera, etc. Rationale: The hardware components have an influence over the experience’s implementation in all its aspects. For example, they may determine roles in the class according to the students’ expertise.
Software The most important pieces of SW used during the experience, for example, tools for image processing such as Photoshop and Gimp. Rationale: the SW influences the experience’s implementation in all its aspects, just like the HW does: it may determine students’ roles.
Digital Content The most important pieces of digital content used during the experience–videos, presentations, apps, etc.
‘Where’ The physical environment where the activities took place–school, lab, home, social space, etc. Rationale: the location is a crucial aspect of the implementation.
‘When’ The hours during which the activities were carried out–during school hours, at home, etc. Rationale: it is important to understand whether the regular school hours were enough.
Curriculum Whether the experience stood within the curriculum or not, for example, an experience could be curricular or extracurricular. Rationale: often technology-based activities are not considered part of the curriculum.
Programme The inclusion of the activities in the official ‘programme’ of the school. Rationale: activities that are officially part of the programme are often felt as more important.
Design How well defined the design was at the start. For example, an experience could be ‘poorly sketched’ or ‘sketched in detail’. Rationale: it is often the case that innovative educational activities cannot be fully designed beforehand.
Human resources Who was directly involved in the experience’s implementation–more than one teacher, the school’s technician, etc. Rationale: to allow evaluation of the effort needed to run the experience.
Other subjects Additional subjects involved in the activities with different roles–the principal, the class’ council, families, etc. Rationale: to understand the complexity of the experience.
Activities Main activities characterising the experience–‘brainstorming’, ‘creation of digital media’, etc. Rationale: it is only by knowing what activities the experience requires that the experience itself can be fully understood from a pedagogical point of view.
Organisation How the class was organised–‘heterogeneous group work’, ‘homogenous group work’, ‘role-playing’, etc. Rationale: same as per activities; more specifically, an organisation is likely to shed light on ‘inclusion’ issues because it shows whether all students were really involved, to what extent and how.
Key features The aspects of the implementation deemed to be particularly important for its success. For example, ‘method’, ‘organisation’, or ‘visibility’ (of the final results).
Inclusion problems List of possible inclusion issues. For example, ‘children with severe learning difficulties’.
Inclusion strategies Possible strategies adopted to overcome inclusion issues. For example, ‘personalised learning’ or ‘peer to peer education’.
Cognitive benefits Benefits at the cognitive level, better understanding of the subject dealt with, capacity to make complex connections, etc.
Motivational benefits Benefits related to enhanced motivation, increased self-esteem, increased motivation towards a specific subject, increased motivation towards school in general, etc.
Relational benefits Benefits related to the sociological sphere–improved relationships within the class, improved relationships with the teacher, negotiation skills, etc.
Communication benefits Benefits related to the students’ communication skills (both with and without technology). For example, increased media literacy, ability to choose and organise digital contents.
Technology-related benefits Benefits related to the students’ technological skills–‘Basic level Techno-literacy’ (technology used to acquire information), etc.

Table 2 shows an example of facet values for the facet ‘activities’. The user’s at-a-glance understanding of a dataset is facilitated by numerous visualisation strategies. For example, all the elements of a set can be visualised as a ‘mosaic’ where shapes and colours are meaningful, as elements over a geographical map emphasising the geographical location of the experiences (see Figure 6) or as thumbnails (to support visually driven selection). Also, within the facets, different visualisations are possible, including absolute numbers, percentages and word clouds. Boolean logic, such as conjunction ‘and’, disjunction ‘or’, ‘exclusive or’, and negation ‘not’, support the use of the portal (see Figures 1 and 7).

Fig 6
Figure 6.  Example of visualisation strategy: the geographical map.

Fig 7
Figure 7.  Example of Boolean-logic supported selections within a facet.


Table 2.  Example of facet values for the facet ‘activities’.
Facet ‘activities’
Value Example
Lecture Traditional lesson
T-Discussions Traditional discussion or brainstorming in class
D-Discussions Digitally supported discussion or brainstorming
T-Individual Traditional individual study or work
D-Individual Digitally supported individual study or work
T-Self-assessment Traditional self-assessment
D-Self-assessment Digitally supported self-assessment
T-Collaboration Collaborative activities, in a traditional way
D-Collaboration Collaborative activities, in a digitally supported way
T-Creation Creation of traditional content or artefacts
D-Creation Creation of digital content or artefacts
Content collection Activities for searching and collecting content
Content collection Activities for searching and collecting content
Problem solving Activities oriented to problem solving
Gaming Game-like activities for learning
Simulations Imitation of a real-world system or activity
Other  

A ‘shopping cart’ allowed the user to save the results of an exploration. Through proper commands, investigations could be saved, stored and shared, or ‘exported’ as PDF files.

If a specific experience was of interest, the user was provided first with an abstract and then with all the available materials, including documents, audio files and presentations.

Evaluation

The evaluation of the L4ALL Portal is a difficult task for several reasons, including the variety of user communities, variety of aims, variety of tasks and variety of scenarios. In the following sections, we describe two specific qualitative evaluations: focus groups with teachers and monitored experimentation with researchers. Other larger quantitative evaluations are planned with teachers for 2014.

Focus groups

Using a portal such as the one provided by L4ALL is a novelty for teachers. Therefore, focus groups were chosen (over surveys) as a way to investigate issues not fully known in advance. We wanted teachers to face the novelty together, share their first impressions and, through discussions and task performances, reach a shared opinion about the tool they were presented with. Three focus groups were held in autumn 2012, in cooperation with three different research groups: HOC-LAB (at Polytechnic of Milan Computer Science School); a team at the University of Perugia (Department of Education) and another at the University of Bologna (Department of Education). On the whole, 32 teachers were involved. They were all working as teachers across all school grades (from pre-school to high-school). The average age was 42. The three focus groups adopted a similar procedure:

  • First, the moderator gave a general introduction to the portal and the goal of the study: what it was about, what kinds of materials could be found and what kinds of scenarios could be supported (allotted time: 10 minutes).
  • Then, the helper showed some of the portal’s affordances: the taxonomy, the different views and the possibility of saving queries and materials (allotted time: 10 minutes).
  • The participants were divided into small groups of three teachers each.
  • Each group was given the following tasks to perform (allotted time: 30 minutes):
    • Take a look at the portal and browse around freely, trying to understand the basics (what the portal is about, what facets and values are used to classify the educational experiences, etc.).
    • Select three facets in the taxonomy and see how the features of the educational experiences vary (e.g. understanding how different technologies affected cognitive benefits).
    • Select some experiences and explore to some extent the content (abstract, synopsis, document, audio files, etc.).
  • During task performance, the focus group moderator and the helper would monitor what was going on within the groups but without interfering.
  • The helper took notes about the teachers’ actions, comments and attitudes.
  • All the participants re-grouped again for the final discussion, prompted by the moderator’s questions (allotted time: 20 minutes):
    • What was your first reaction to the portal?
    • Did it take long to make up your mind about how it works?
    • Do you think that the portal is too difficult from an interactional point of view?
    • Do you think that the portal is too difficult from the point of view of content (i.e. the taxonomy, the materials provided, etc.)?
    • Would you prefer to have a simplified version of the portal?
    • Do you think the portal is useful?
    • Would you use the portal?
  • The helper recorded the discussion and a transcript was made that was later boiled down to the essential comments only.

The outcome of the focus groups was highly encouraging:

  • HCI features: after the initial difficulty of the apparent complexity of the interface, teachers became comfortable with it. Overall, they said that after a few minutes of usage, the interface became quite usable.
  • Experience description: teachers did like how the educational experiences were described. They said that it was not usual for them to describe experiences in that way, but they found a systematic analysis useful for deep understanding.
  • Browsing and serendipity: teachers quickly got very familiar with the browsing method and found the lack of focus on ‘content’ interesting.
  • TPK: teachers said that the complex taxonomy helped them to start realising the relationships between technology and pedagogical issues. Whether to simplify the taxonomy for the teachers leaving the more complex one for researchers only was one of the project’s concerns, but the majority of teachers declared that after a few minutes they became ‘intrigued’ by all the facets and did not want a simpler version.
  • Size of the portal: many teachers pointed out that a larger number of experiences would add value to the portal.

Researchers

In order to evaluate the quality of the portal as a support for research activity, five research groups with different backgrounds (technology and pedagogy) were asked to use the portal for their research activities. Seven different investigations were carried out:

  1. ‘The Unexpected Learning: How Unexpected Benefits Can Be Generated Through ICT-Based Experiences’, by Polytechnic University of Marche
  2. ‘PoliCultura & Moodle: A Blended Learning Environment’, by the University of Perugia
  3. ‘Educational Technology at Primary School Level: A Survey’, by the University of Perugia
  4. ‘Inclusion and Group Work: What Benefits?’ by Polytechnic of Milan
  5. ‘Educational Technology at Junior High-School Level: A Survey’, by University of Bologna
  6. ‘Technology@School: Analysis of an ICT-Based Format’, by University of Bologna.
  7. ‘Investigation on the Relationship Between Technology and Educational Benefits’, by University of Salento.

Some of the reports have already been turned into scientific papers (e.g. Di Blas and Paolini 2013).

The researchers were interviewed extensively and they outlined several advantages introduced by the portal:

  • Content: researchers showed appreciation of the in-depth understanding of the experiences afforded by the portal’s wide range of materials.
  • HCI: as was expected given the type of user, the mechanics of the interface were quickly understood.
  • Browsing and serendipity: the two most important methods of usage were formulating research hypothesis and verifying them and browsing around trying to ‘stumble’ into interesting data. Researchers declared that both methods were effective, and that the latter was a pleasant novelty.
  • TPK: interesting investigations were easily and effectively carried out. Facets were not just used for selection but also for feedback. For example, without even looking at the description of the specific experiences, it was possible to detect that group work (quite typical at primary school level) generates different benefits with respect to the experiences where individual assignments are used. Discovering ‘properties of sets’ (without looking at details) emerged as a major affordance of the portal and it has become a base for future enhancements (as will be discussed in the next section).

Discussion

The work described in this paper presents several new aspects:

  • TPACK and Educational Portals: our contribution lies in the emphasis placed at the TPK intersection. While most educational portals are built around a school grade-content paradigm, we strive to communicate how technology and pedagogy influence each other. This is done in two ways:
    • The description of each educational experience is mainly about the TPK intersection.
    • The taxonomy used to ‘explore’ the set of experiences is based upon TPK facets.
  • Sharing educational ‘evidence’: in the field of education, the most common pattern is that each research group creates its own experimental data (about a phenomenon of interest), collects it and interprets it. This common pattern can be criticised in several respects. First of all, it is an expensive way of proceeding. Young researchers can have difficulty with this and students cannot easily access empirical data. In addition, ‘theoretical’ researchers do not have data to work with. Third, the different ‘interpretations’ cannot be easily (and reliably) compared, since they are based on different data sets.

In September 2012, in a workshop organised in Como, Italy, several experienced and world-renowned researchers agreed that sharing empirical data in education would be valuable for the community. Therefore, we advocate the advantages of making publically available empirical data about innovative learning experiences. The L4ALL portal is open in two ways. It can be expanded with new experiences fed by other research groups and any research group can use it for research and investigations.

  • Teachers are users: teachers can find in the L4ALL portal an unusual source of inspiration and, most important, an occasion for professional growth, while paying attention to TPK’s intersection. They can realise that the relation between pedagogy and technology is multifaceted and complex (e.g. tablets will not save education nor they will destroy it), and that ‘details’ (that they can find in the portal) make the difference. This is what we think we have accomplished, to some extent. Therefore, we advocate the need for using educational portals not just as a ‘hub’ where information can be found but also as a way to encourage teachers to broaden their horizon and their way of planning their activities.
  • Researchers are users: we have indicated that evidence-based research can be carried on well by exploiting all the information gathered in the portal. The researchers found some expected phenomena in the portal, but they also serendipitously discovered new aspects and were stimulated to formulate new research directions. Therefore, we advocate that the empirical evidence should be explored in view of detecting new, unexpected, patterns and not just to prove a hypothesis that was formulated beforehand.
  • Decision makers are users: the Italian ministry of Education and several regional district managers showed great interest. In an open meeting in Rome, they declared that our effort was worthwhile and that we should continue expanding the portal.

Therefore, we advocate the need for basing education policies (e.g. how to use technology at school) not just on anecdotal evidence, nor only on largely dry statistical data, but also on a fine-grained understanding of the issues and of the various implications. Most important is the fact that data about TK and PK, if considered separately, do not provide an adequate insight.

  • Exploratory portals and technology: we have created a novel paradigm for exploring ‘rich data sets’, that is, complex information organised via a complex taxonomy. Its development involved HCI design, user experience design and a strong technological development. Our approach is the development of a new generation of portals, supporting better investigation, exploration and serendipitous discovery. The approach will be based on a general repository-portal model, such as the one described in this paper.

Future work

Several directions for future work are open to add to the research described in this paper:

  • TPACK and educational portals: we are in the process of refining the understanding of how TK and PK influence each other. This will lead to a redefinition (expansion and refinement) of the taxonomy upon which the portal is based. In a forthcoming paper (in cooperation with one of the authors of the TPACK model), we are also investigating how TK and PK are ‘distributed’ among various actors and how they evolve within the same educational experience (Di Blas et al. 2014).
  • Sharing educational ‘evidence’: we are pursuing contacts with other research groups in Italy and with district authorities in order to expand the size and the coverage of the portal, sharing its use. The major problem is cost. Going from 300 experiences to 10,000, as it seems it would be necessary, is a cost that must be covered with a variety of financial resources.
  • Teachers are users: we were delighted by the positive reactions from the teachers. At the moment, we are actively using the portal as an overall resource for training teachers on the job.
  • Researchers are users: five different groups of researchers are pursuing their activities with the portal. As it is in Italian, it is not easily ‘exportable’. We are working in the direction of making it more international.
  • Decision makers are users: the initial reactions of decision makers were quite enthusiastic. The main problem, as mentioned before, is the size of the portal. An Italian regional district would prefer to have between 500 and 1,000 experiences related to its territory in order to have a reliable foundation for a decision. We are signing agreements with some of these districts, in order to make this possible.
  • Exploratory portals and technology: we are working on expanding and improving the paradigm and the user experience. This new vision (that we call EXPLORE) will provide significant enhancement to the ways we access today rich data sets. We are also moving towards applying the same technology to other application areas (such as e-tourism and e-food).

Acknowledgements

We warmly thank all the people who took part in the Learning For All research project for their invaluable contributions. We warmly thank professors Tom Reeves (University of Georgia), Barry Harper (University of Wollongong, Australia) and David Hung (National Institute of Education, Singapore) for the insights on our repository and portal they provided during the seminar Sharing Empirical Evidence: The Impact of Technology on Learning at School (Como, Italy, September 2012).

Note

1. The initial acronym was TPCK. In 2007, it was changed to the easier TPACK name. The change in terminology has not been universally adopted, so both acronyms can be found in the literature.

References

Alfresco software. (2013) Open Source Enterprise Content Management System (CMS)|Alfresco, [online] Available at: http://www.alfresco.com

Andrew, P., Farrow, R. & Cooper, M. (2012) ‘Adapting online learning resources for all: planning for professionalism in accessibility’, Research in Learning Technology, vol. 20, no. 4, pp. 345–361.

Angeli, C. & Valanides, N. (2009) ‘Epistemological and methodological issues for the conceptualization, development, and assessment of ICT-TPCK: advances in technological pedagogical content knowledge (TPCK)’, Computers & Education, vol. 52, no. 1, pp. 154–168. PubMed Abstract | PubMed Central Full Text | Publisher Full Text

Archambault, L. M. & Barnett, J. H. (2010) ‘Revisiting technological pedagogical content knowledge: exploring the TPACK framework’, Computers & Education, vol. 55, pp. 1656–1662. PubMed Abstract | PubMed Central Full Text | Publisher Full Text

Aroyo, L., Mizoguchi, R. & Tzolov, C. (2003) ‘OntoAIMS: ontological approach to courseware authoring’, International Conference on Computers in Education (ICCE2003), Hong Kong, pp. 1011–1014.

Barchetti, U., et al. (2011) ‘Modelling collaboration processes through design patterns’, Computing and Informatics, vol. 30, no. 1, pp. 113–135.

Besterfield-Sacre, M. et al. (2000) ‘Defining the outcomes: a framework for EC-2000’, IEEE Transactions on Education, vol. 43, no. 2, pp. 100–110. Publisher Full Text

Bloom, B. S., et al. (1956) Taxonomy of Educational Objectives: Handbook 1: Cognitive Domain, Longman, New York.

Boyce, C. & Neale, P. (2006) ‘Conducting in-depth interviews: a guide for designing and conducting in-depth interviews for evaluation input’, Carolina Population Center, [online] Available at: http://www.cpc.unc.edu/measure/training/materials/data-quality-portuguese/m_e_tool_series_indepth_interviews.pdf

BrainPOP (2014). BrainPOP - Animated Educational Site for Kids - Science, Social Studies, English, Math, Arts. [online] Available at: http://www.brainpop.com

Bucciero, A., Guido, A. L. & Mainetti, L. (2011) ‘Conceptual design of collaborative virtual environments for education using a theatre based metaphor’, ICST Transactions on e-Education and e-Learning, vol. 11, no. 7–9, pp. 1–16.

Cos Garcia, J. A., et al. (2011) ‘E-learning internationalization standards overview and guidelines’, IEEE Learning Technology Newsletter, vol. 13, no. 3, pp. 24–26.

Cox, S. & Graham, C. R. (2009) ‘Diagramming TPACK in practice: using and elaborated model of the TPACK framework to analyze and depict teacher knowledge’, TechTrends, vol. 53, no. 5, pp. 60–69. Publisher Full Text

Cuthrell, K. & Lyon, A. (2007) ‘Instructional strategies what do online students prefer? Journal of Online Learning and Teaching, vol. 3, no. 4, pp. 153–163.

Di Blas, N., et al. (2012) ‘Multi-user virtual environments for learning: experience and technology design’, IEEE Transactions on Learning Technologies, vol. 5, no. 4, pp. 349–365. Publisher Full Text

Di Blas, N. & Paolini, P. (2013) ‘Technology and group work: inclusion or diversification of talents?’ In Parmigiani, D., Pennazio, V., & Traverso, A. (Eds.). Learning & Teaching with Media & Technology. ATEE-SIREM Winter Conference Proceedings, 7–9 March 2013, Genoa (Italy). Brussels: ATEE aisbl, pp. 218–231.

Di Blas, N., Paolini, P., Sawaya, S. & Mishra, P. (2014) ‘Distributed TPACK: going beyond knowledge in the head’, In M. Searson & M. Ochoa (Eds.). Proceedings of Society for Information Technology & Teacher Education International Conference 2014, (pp. 2464–2472). Chesapeake, VA: AACE.

Dolog, P., et al. (2004) ‘The personal reader: personalizing and enriching learning resources using semantic web technologies’, Third International Adaptive Hypermedia and Adaptive Web-based Systems Conference (AH2004) (pp. 85–94), Eindhoven, The Netherlands.

EDRENE. (2009) – D2.6 State of the art report – II, [online] Available at: http://edrene.org/results/deliverables/EdReNe D 2.6 SoA - II.pdf

Ferrari, L., et al. (2012) ‘“Learning for All”: is everyone learning?’, World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012, Chesapeake, pp. 1782–1792.

Forte, E., Wentland-Forte, M. & Duval, E. (1997) ‘The Ariadne project (part 1): knowledge pools for computer-based and telematics-supported classical, open, and distance education’, European Journal of Engineering Education, vol. 22, no. 1, pp. 61–74. Publisher Full Text

Graham, C. R. (2011) ‘Theoretical considerations for understanding technological pedagogical content knowledge (TPACK)’, Computers & Education, vol. 57, pp. 1953–1960. PubMed Abstract | PubMed Central Full Text | Publisher Full Text

Hatala, M., et al. (2007) ‘Secure communication layer for scalable networks of learning objects repositories’, E-Learning Networked Environments and Architectures, pp. 276–305.

Ingham, J. (2000) ‘Data warehousing: a tool for the outcomes assessment process’, IEEE Transactions on Education, vol. 43, no. 2, pp. 132–136. Publisher Full Text

Jones, A. (2009) ‘Redisciplining generic attributes: the disciplinary context in focus’, Studies in Higher Education, vol. 34, no. 1, pp. 85–100. Publisher Full Text

Khan Academy (2014). A free world-class education for anyone anywhere. [online] Available at: https://www.khanacademy.org/

Koehler, M. & Mishra, P. (2005) ‘What happens when teachers design educational technology? The development of technological pedagogical content knowledge’, Journal of Educational Computing Research, vol. 32, no. 2, pp. 131–152. Publisher Full Text

Koehler, M. & Mishra, P. (2008) ‘Introducing TPCK’, in Handbook of technological pedagogical content knowledge (TPCK) (pp. 3–31), ed AACTE Committee on Innovation and Technology, Routledge, New York.

Koehler, M. J. & Mishra, P. (2009) ‘What is technological pedagogical content knowledge? Contemporary Issues in Technology and Teacher Education, vol. 9, no. 1, pp. 60–70.

Koehler, M., Mishra, P. & Yahya, K. (2007) ‘Tracing the development of teacher knowledge in a design seminar: integrating content, pedagogy and technology’, Computers & Education, vol. 49, no. 3, pp. 740–762. PubMed Abstract | PubMed Central Full Text | Publisher Full Text

Lattuca, R. L., Terenzini, P. T. & VolkWein, J. F. (2006) ENGINEERING CHANGE – A Study of the Impact of EC2000, ABET, Baltimore, MD.

LON-CAPA (2013). The Learning Online Network with CAPA. Open-Source Free CMS/LCMS. [online] Available at: www.lon-capa.org.

Lukasiak, J., et al. (2005) ‘Learning objects and learning designs: an integrated system for reusable, adaptive and shareable learning content’, Research in Learning Technology, vol. 13, no. 2, pp. 151–169. Publisher Full Text

Maher, P. E. & Kourik, J. L. (2008) ‘A knowledge management system for disseminating semi-structured information in a worldwide university’, International Conference on Management of Engineering & Technology (PICMET 2008), pp. 1936–1942.

Matar, N., Khwaldeh, S. & Hunaiti, Z. (2007) ‘Adaptive unified e-learning system for supporting better e-learning approach’, Annual Postgraduate Symposium on the Convergence of Telecommunications Networking and Broadcasting (PGNet 2007), Liverpool John Moores University, Liverpool, UK, pp. 28–29.

Mendele. (2013) Free Reference Manager and PDF Organizer, [online] Available at: http://www.mendeley.com

Mishra, P. & Koehler, M. (2006) ‘Technological pedagogical content knowledge: a framework for integrating technology in teacher knowledge’, Teachers College Record, vol. 108, no. 6, pp. 1017–1054. Publisher Full Text

Neven, F. & Duval, E. (2012) ‘Reusable learning objects: a survey of LOM-based repositories’, In Proceedings of ACM Multimedia (pp. 291–294), ACM Press.

Niess, M. L. (2005) ‘Preparing teachers to teach science and mathematics with technology: developing a technology pedagogical content knowledge’, Teaching and Teacher Education, vol. 21, pp. 509–523. Publisher Full Text

Niess, M. L. (2011) ‘Investigating TPACK: knowledge growth in teaching with technology’, Educational Computing Research, vol. 44, no. 3, pp. 299–317. Publisher Full Text

NMC project (2007). Sparking Innovative Learning and Creativity. [online] Available at: http://archive2.nmc.org/index.php .

Nunamaker, J. F., Jr., Romano, N. C. & Briggs, R. O. (2001) ‘A framework for collaboration and knowledge management’, 34th Annual Hawaii International Conference on System Sciences, pp. 1–12, Hawaii.

OER (2013). The Orange Grove repository, is Florida’s digital repository for instructional resources. [online] Available at: http://florida.theorangegrove.org/og/access/home.do .

Perkins, D. (1995) ‘Outsmarting I. Q The Emerging Science of Learnable Intelligence’, ed. The Free Press, New York.

Polilab Kids (2013). A collection of educational animations, [online] Available at: http://www.polilabkids.it/

Pouyioutas, P. & Poveda, M. (2005) ‘Designing a learning object repository-the views of higher education faculty’, Lecture Notes in Computer Science 3583/2005, pp. 111–121.

Primo, T. T., Vicari, R. M. & Bernardi, K. S. (2012) ‘User profiles and learning objects as ontology individuals to allow reasoning and interoperability in recommender systems’ Global Engineering Education Conference (EDUCON), pp. 1–9, 17–20, Marrakech.

QSR International. (2013) NVivo Research Software for Analysis and Insight, [online] Available at: http://www.qsrinternational.com/products_nvivo.aspx .

Roschelle, J., et al. (1999) ‘Developing educational software components’, IEEE Computer, vol. 32, no. 9, pp. 50–58. Publisher Full Text

Schell, G. P. & Burns, M. (2011) ‘Merlot: a repository of e-learning objects for higher education’, e-Service Journal, vol. 1, no. 2, pp. 53–64.

Schoonenboom, J., Sligte, H., & Kliphuis, E. (2009) ‘Guidelines for supporting re-use of existing digital learning materials and methods in higher education’, Research in Learning Technology, vol. 17, no. 2, pp. 131–141. Publisher Full Text

Shulman, L. (1986a) ‘Paradigms and research programs in the study of teaching: a contemporary perspective’, in Handbook of Research on Teaching, 3rd., ed. M. C. Wittrock, MacMillan, New York, pp. 3–36.

Shulman, L. (1986b) ‘Those who understand: knowledge growth in teaching’, Educational Researcher, vol. 15, no. 2, pp. 4–14. Publisher Full Text

So, W. W. M. (2012) ‘Creating a framework of a resource-based e-learning environment for science learning in primary classrooms’, Technology, Pedagogy and Education, vol. 21, no. 3, pp. 317–335. Publisher Full Text

Vergallo, R. (2012) ‘Classroom 3.0: the real world meets the virtuality through ambient sensing in education’, 12th IEEE International Conference on Advanced Learning Technologies (ICALT), pp. 722–723, Rome.

Verhaart, M. (2002) ‘Knowledge capture at source. Developing collaborative shared resources’, International Conference on Computers in Education, vol. 2, pp. 1484–1485.

Voogt, J., et al. (2012) ‘Technological pedagogical content knowledge – a review of the literature’, Journal of Computer Assisted Learning, vol. 29, no. 2, pp. 109–121. Publisher Full Text

Willems, J. (2007) ‘Does style matter? Considering learning styles in e-learning’, In R. Atkinson, C. McBeath, S.K.A. Soong, & C. Cheers. (Eds.), ICT: Providing choices for learners and learning. Proceedings ASCILITE Singapore 2007 (pp. 1068–1069). Singapore: Nanyang Technological University. http://www.ascilite.org.au/conferences/singapore07/procs/willems-poster.pdf

Zhou, P. (2008) ‘An adaptive framework for managing knowledge in e-government’ International Symposium of Knowledge Acquisition and Modeling (KAM), pp. 69–73, Wuhan.

About The Authors

Nicoletta Di Blas
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, Italy
Italy

Alessandro Fiore
Department of Innovation Engineering, University of Salento, Lecce, Italy
Italy

Luca Mainetti
Department of Innovation Engineering, University of Salento, Lecce, Italy
Italy

Roberto Vergallo
Department of Innovation Engineering, University of Salento, Lecce, Italy
Italy

Paolo Paolini
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, Italy
Italy