3.5 Monitoring and evaluation
Monitoring and evaluation are critical components for school connectivity projects. An effective plan should include methods to (1) evaluate the technical results of Internet connectivity, (2) measure progress towards school connectivity and (3) analyse the impact of broadband access on learning.
Countries with the intent and resources to provide Internet access in schools will want to carry out pilot projects to assess the technical solutions. This is also relevant for countries where schools have narrowband access but now desire to upgrade to broadband. Even in countries with widespread school connectivity, there may be a desire to evaluate new access technologies because of cheaper costs or better performance, e.g. leveraging widespread mobile phone use for m-education.
The evaluation should examine if the type of Internet access used (e.g., ADSL, fixed wireless, third generation mobile, VSAT, etc.) works as expected, whether it can support the number of access points with sufficient bandwidth and what costs are expected.173 The evaluation also provides information about the overall costs of connectivity, such as monthly access tariffs, the costs of retrofitting schools with cabling and redistributing access throughout a school and training and support expenses. These results can then be used to refine the technical solution prior to implementation on a wider scale.
Monitoring is also essential for tracking school connectivity deployments to assess whether they are advancing according to plan. This should cover both new deployments and upgrades from narrowband to broadband connectivity.
Metrics for measuring deployment are fairly straightforward. The Partnership on Measuring ICT for Development, a group of international agencies, has recommended eight core indicators that countries should collect regarding ICTs in schools.174
The basic statistic is to track the number of primary and secondary schools with Internet access (broken down by narrowband and broadband access and public and private schools) and compare it to targets set within a plan. The statistic can also be used to derive indicators such as the percentage of schools with Internet access. Additional statistics can be compiled, and indicators derived, depending on the desired level of analysis. This would probably include a breakdown by primary and secondary schools with additional disaggregation to gauge the impact of specific populations such as the poor, females, persons with disabilities, minorities, ethnic groups, rural inhabitants, etc. This requires extrapolating the number of children affected by the school connectivity.
Table 3-7: Basic Indicators for Monitoring the Status of School Connectivity Deployment
Total number of schools
Total number of students
Number schools with Internet access
Number of schools with broadband Internet access
Number of students covered by Internet access
Number of students covered by broadband Internet access
In addition to comparing the actual status to deployment plans at specific milestones, the monitoring of school Internet connectivity is useful for other applications. For example governments may want to benchmark their school connectivity progress to other countries at similar socio-economic levels.
Despite the fundamental importance of monitoring, it is surprising how few countries compile and publicly disseminate clear and comparable statistics on the status of school connectivity. Collection of the school connectivity statistics should not impose an extra burden on educational systems. Nearly all education ministries publish statistics on the numbers of schools, students, teachers, completion rates, enrollment rates, etc. The number of schools with Internet access should be collected as part of the regular data-gathering processes when schools are asked about those other educational statistics.
Once broadband connections are in place, other indicators that could be employed to gauge the effectiveness of investments in broadband connectivity include:
- Number of teachers trained in online (Internet-enabled) and m-enabled curricula and tools;
- Level of use of online (Internet-enabled) and m-enabled tools or curriculum in classrooms;
- Measurement/testing of students’ ICT abilities;
- Measurement/testing of student performance in subject areas expected to benefit from broadband-enabled curricula and tools;
- Amount of time that school-based "telecentres" are available to the community;
- Number of users of school-based "telecentres"; and
- Percentage or absolute amount of school budget committed to online educational materials.
An important goal is the impact of school connectivity on the learning process as well as on the socio-economic development of the community. Medium-term and long-term objectives can also be assessed, including:
- Changes in classroom management practices after the introduction of new technologies,
- Improvements in perceived educational trajectories of beneficiary students,
- Higher values placed on education by beneficiary families and lower drop-out rate of beneficiary students, and
- Improved performance on standardized tests.
In order to obtain a complete assessment, quantitative and qualitative measurements are important, including test scores or grades, surveys, observation and interviews. The use of an internationally recognized testing instrument can provide not only legitimacy to test results, but a means to compare educational performance against international benchmarks.
A related approach would be for policymakers to identify the benefits delivered by a school connectivity project across a range of development outcomes. For example, policymakers could identify the effects of a connectivity project in terms of progress made toward the Millennium Development Goals, 175 or toward a specific goal, such as poverty reduction. By leveraging school connectivity projects, national policymakers could spur progress on goals such as child health, maternal health, combating HIV/AIDS, environmental sustainability, and global partnership – as well as universal education and gender equality, as mentioned in Section 2.1.1 – through such by-products of school connectivity as access to information, improved educational opportunities, and expanded capability to communicate and collaborate.
Another example would be to follow the lead of the U.S. Trade and Development Agency (USTDA), which looks at benefits across five areas to gauge the impact of its assistance projects:176
- Infrastructure : Identifying ICT/telecommunication facilities to be constructed, the expected connectivity level enabled by such infrastructure, and the number of anticipated users.
- Market-Oriented Reform : A description of any regulation, laws, or institutional changes that are recommended, and the effect they would have if implemented.
- Human Capacity Building : The number and type of positions that would be needed to construct and operate the proposed project, as well as the number of people who would receive training.
- Technology Transfer and Productivity Enhancement : A description of advanced technologies that will be implemented as a result of the project, and description of efficiencies gained.
- Other : Any other developmental benefits to the project.
The examples above provide ideas for how to measure progress, gauge effectiveness and determine development impacts of school connectivity. Policymakers will need to determine the metrics and milestones that will best allow them to determine the effects of their investment in educational broadband, while keeping in mind that the metrics will likely require recalibration periodically.177
Intel Corporation, “Deploying 1:1 e-Learning Environments for the 21st Century,” 2007
174 Statistical Commission Background document. Fortieth session, 24 - 27 February 2009. Items for information: Information and communication technology statistics - REVISIONS AND ADDITIONS TO THE CORE LIST OF ICT INDICATORS, Prepared by the Partnership on Measuring ICT for Development – see http://www.itu.int/ITU-D/ict/partnership/material/CoreICTIndicators_e_rev2.pdf . In addition, Canada, the European Union and the United States have produced statistical reports tracking connectivity in their countries. They provide good examples of the kinds of metrics used. See:
Wells, J., and Lewis, L. (2006). Internet Access in U.S. Public Schools and Classrooms: 1994–2005 (NCES 2007-020). U.S. Department of Education. Washington, DC: National Center for Education Statistics. http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007020
Johanne Plante and David Beattie. (2004). Connectivity and ICT integration in Canadian elementary and secondary schools: First results from the Information and Communications Technologies in Schools Survey, 2003-2004 . Statistics Canada.
empirica. (2006). Benchmarking Access and Use of ICT in European Schools 2006. European Commission.
175 The Millennium Development Goals, available at: http://www.un.org/millenniumgoals/bkgd.shtml.
176 United States Trade and Development Agency, “Proposal and Budget Model Format,” Annex VI, available at http://www.ustda.gov/program/ModelProposalFormatUSFirms2010.pdf .
177 Wagner, Daniel A., Bob Day, Tina James, Robert B. Kozma, Jonathan Miller and Tim Unwin. 2005.Monitoring and Evaluation of ICT in Education Projects: A Handbook for Developing Countries. Washington, DC: infoDev / World Bank. Available at: http://www.infodev.org/en/Publication.9.html