Trina Hurdman

Reading up on PISA 2015

Every three years since 2000 a sample of 15 year olds in Alberta have participated in an international test, the Programme for International Student Assessment (PISA), developed by the Organisation for Economic Co-operation and Development (OECD). The results for the most recent 2015 test were released a couple of months ago in December 2016.

The OECD has created a brief video to provide an overview of how the test is developed, administered and analyzed.

While most reporting seems to focus on ranking the education systems of the participating countries, the key objective is to “inform and support education policy decision making within countries.” To this end, the data collected forms the basis of many publications that examine commonalities between high (or low) performing countries in order to determine effective practices that can be shared internationally. While many of these publications are lengthy, the OECD also creates brief policy notes focused on specific topics that are quite fascinating.

It is important to note that the test is not about academic knowledge, but about how well students can apply a variety of skills in order to solve real-life problems. Also important to remember is that because this is a test of a sample of students, countries or jurisdictions cannot be ranked exactly and confidence intervals must be taken into account. So, while Alberta may have been ranked #2 out of 72 jurisdictions in the world in Science, statistically speaking, Alberta could be anywhere from #2 to #8. Conversely, Alberta was ranked #14 in mathematics, but statistically, could be anywhere from #8 to #26. In reading, Alberta was ranked #3, but again, could be anywhere from #1 to #13. So, while rankings make good headlines, the information that they provide is limited. To examine the data in more detail, the OECD provides a website that produces reports based on what you’re interested in looking at and which countries you’re interested in comparing. If you love spreadsheets, you will love this website.

For those who prefer more of a written analysis, the Council of Ministers of Education, Canada (CMEC) uses the PISA data to create a companion report that discusses the results across Canada in more depth. As many Canadian provinces do well on PISA, it is helpful to examine the practices of other provinces when looking for ideas for improvement.

The following are a few examples of information that can be found in the companion report.

When an overall average is used for rankings, a country can perform well by having well performing students mask the low performance of others. However, the purpose of a good education system is to ensure that all students achieve at least a basic level of competency needed to succeed in life. Thus, it is important to know what proportion of students are achieving at each level (with 1 being the lowest and 6 being the highest). From the chart below, found on page 18 of the CMEC report, we can see that 9% of Alberta students did not achieve the basic level of proficiency in Science, despite Alberta being ranked #2 in the world. While this may be much better than other provinces or the total of the participating OECD countries, there is still work to do when almost one in ten students are not succeeding.

The following table is from page 79 of the CMEC report. Again, due to sampling, you cannot determine whether a jurisdiction has improved or is doing worse based solely on their score. Based on the scores in the math table below, it looks like Quebec has improved over time, but statistically, there is no difference from 2003 to 2015. However, Alberta’s math results have statistically decreased since 2003. (Another table on the same page shows that Alberta has not declined statistically since 2012.)

Another interesting table on page 77 of the CMEC report compares the difference in scores between males and females. While in science, there was no statistical difference, in reading, females outperform male students in every province. In math, males slightly outperform females in Alberta, but this is not true for every province.

Data can be analyzed and interpreted in a myriad of ways. I can easily spend entire days on the PISA website reading about various educational issues and policies espoused by education systems around the world, and I hope that I’ve tweaked your interest in doing some of reading of your own!

Leave a Reply

Your email address will not be published. Required fields are marked *