Indigenous Recognition

Welcome & Introductions

Art Stewart, PPX President & Tom Wileman, Chair, Symposium Committee

Opening Keynote

What Gets Measured Gets Managed: Ministries’ Performance Measurement, Program Evaluation and Annual Reporting

Bonnie Lysyk, MBA, FCPA, FCA, LPA, Auditor General of Ontario

Description

Bonnie Lysyk will speak about the 2023 Ontario Auditor General’s report “What Gets Measured Gets Managed: Ministries’ Performance Measurement, Program Evaluation, and Annual Reporting”. She will note the need for significant improvement in these areas, to better inform government and ministry decision-making, as well as to enhance operational performance: performance measurement, program evaluation, and public reporting.

What Gets Measured Gets Managed: Ministries’ Performance Measurement, Program Evaluation and Annual Reporting, June 20, 2023

The State of the Environment in Ontario, May 16, 2023

Networking Break 

Sponsored by Performance Management Network Inc

Plenary Panel – in room 106H

Future Strategies for Public Audit

This panel will be offered in French from 10:30-11:30 and repeated in English from 11:30-12:30

Description:

This panel examines the development of strategies for legislative and internal public audit in Canada. Paul Forgues, President of the Canadian Audit and Accountability Foundation will speak about the mandate of the Foundation and its impact on strategic thinking in member audit organizations. Professor Luc Juillet, from the University of Ottawa will present research findings regarding strategic thinking in both forms of public audit.

Moderator:
Sharon Clark
Tom Wileman

Panelists:
Paul Forgues, President of the Canadian Audit & Accountability Foundation
Luc Juillet, Graduate School of Public and International Affairs, University of Ottawa

Concurrent 1

Concurrent 2

Path to Performance Measurement Improvement at Employment and Social
Development Canada (ESDC)

John Mitchell, Senior Policy Analyst, Employment and Social Development Canada (ESDC)

A presentation detailing how ESDC’s Performance Measurement Division has been investigating the use and utility of performance information in the department. John will look to support the “How will we know we are getting there?” theme by outlining some of the organization’s experiences and findings from research conducted over the last year. In addition, he will provide an overview of the division’s plans to raise the department’s maturity-level with performance measurement and capacity for results-based management over the coming years.

Concurrent 3

The Government of Yukon Performance Measurement & Evaluation Framework: Centering Integration and Concision in Performance System Design

Stephanie Hawkins, Director Evaluation, Government of Yukon.

Implementation of the Government of Yukon’s new Performance Measurement & Evaluation Framework (the Framework) came into effect on January 1, 2023. The Framework applies to Yukon government programs, policies, and strategies; its scope encompasses evaluation, performance measurement, and the integration of budget, strategy, and results. The Framework is, in many regards, reflective of similar policies such as the Government of Canada’s Policy on Results. The design and implementation of the Performance Measurement & Evaluation Framework, however, makes notable departures from convention in two distinct realms:

1. An intensive “integration-first” approach to the capture, interconnection, and navigation of priority and performance data. By design, priority lenses such as reconciliation, gender, climate, and quality of life are incorporated into the Framework’s design and resulting products, as are systematic data integrations at the whole-of-government scale and beyond; and
2. Added concision in Framework information terminology, concepts, and products; this applies to an adaptation of Outcome nomenclature as well as stricter conventions around common performance measurement concepts such as Outputs, Targets, and KPIs. The first part of this session will provide an overview of the key features of Yukon government’s Performance Measurement & Evaluation Framework: what’s required, when, how, by whom, and to what end for what audience(s). The balance of the session will focus on selected adaptations and innovations developed in support of Framework clarity, simplicity, insight, and utility. Acknowledging the centrality of organizational culture and capacity to Framework success, a summary of strategies and supports deployed in support of Framework implementation and adoption.

Concurrent 4

(NOTE: THIS SESSION WILL BE PRESENTED IN FRENCH)

Developing a Framework for Measuring and Monitoring the Performance of Public-Private Partnerships (P3s) in Infrastructure Projects in Canada

Cheick Alassane Traore, University of Ottawa 

The purpose of this study is to highlight the elements in the literature to report on the performance of P3s and to develop a systematic framework for measuring and monitoring this performance from a public sector perspective and the project life cycle. This framework must be operational and accessible to professionals to evaluate the performance of P3s at different stages of the project life cycle and to overcome the ideological debates surrounding P3s.

This study makes it possible to establish an original performance measurement framework that combines different aspects of performance and the complete life cycle of P3s. It surpasses the current analysis of the performance of P3s, which is limited to construction, costs and infrastructure quality, and explores issues of governance, competition, sustainability of public funding, etc. Using information from nearly 200 P3s projects in Canada, this study allows for the analysis of a wide range of P3s. It fills a knowledge gap on the performance of P3s and enriches the literature on performance.

Concurrent 5

The Current State of GBA+ Analysis in Federal Government Evaluation: How Close Are We to Support the Result-based Management Practice With a GBA+ Lens?  

Deen Taposh, Manager Strategic Policy, Employment and Social Development Canada
Sara Hayati, Evaluator, Employment and Social Development Canada
Nicole Mitri, Evaluator, Employment and Social Development Canada

The use of GBA+ analysis in federal government evaluations has increased over the years (Gauthier et al., 2022). The disproportionate impact of the pandemic has also reinstated the importance of GBA+ analysis in the public sector (Cameron and Lindsay, 2023). The question, however, remains to what extent federal government evaluations are using GBA+ analysis. Learning from an environmental scan of a representative sample of recently published and publicly available federal government evaluation reports, the presentation proposes to examine the following questions:

• ‘to what extent’ federal departments are applying a GBA+ lens in their evaluation reports, including but not limited to, a scan of ‘the aspect’ (program design/engagement/ operation/impact) and ‘reported methodologies’ of the application; and,
• ‘to what extent’ the findings from GBA+ analysis is translating into evaluation ‘recommendations’ in the report.

The findings from the environmental scan will be in an aggregate form without stating the underlying causes. Given the content may explore published findings from several federal departments, the views should be solely attributed to the authors.  

Lunch

Sponsored by Carleton University Public Policy and Program Evaluation

Plenary Panel

Strategic Policy (Spending) Review 

Professor Evert Lindquist, University of Victoria
Professor Rob Shepherd, Carleton University

Don Wilson, Executive Director, Treasury Board of Canada Secretariat
Doaa Saddek, Director, Employment and Social Development Canada
Philippe Thompson, Chief Finances Results and Delivery Officer, Indigenous Services Canada
Susan Lehmann, Canada Free Agent

Case Study Presentations

Gita Zareikar, Carleton University
Keita Hashimoto, Carleton University
Irene Huse, University of Victoria
Hayat Askar, M&E Professional

What do spending and strategic reviews mean for departments, ministries and agencies? Are the days of free flowing money over? What does it mean to be going into austerity?

The pandemic placed many countries, including Canada, into major debt and deficit positions with net debt to GDP levels rising well above 75 percent. In Canada’s case, this figure is closer to 48 percent, which is still significant, with few signs that debt levels will return to pre-pandemic levels anytime soon. The federal government announced that it will be looking to departments and agencies to find $15B in savings over the next five years, which seems insignificant, but is this the real story? It seems fairly certain that budget increases will not be forthcoming, hiring will slow down dramatically, and there will be less reliance placed on outside consultants to support program delivery.

This session will divide the time into two main parts. The first will feature a panel of senior public servants, who will discuss the state of readiness in their departments for austerity measures, systems and capability gaps they are observing, and advice on ways to address these gaps in a constructive way. The second will feature some research undertaken that compares the Canadian experience with spending and strategic reviews with other countries that are performing well post-pandemic, including The Netherlands, Ireland, Nordic countries and Japan. The aim is to highlight that there are several models for implementing reviews and how these contribute to not only expenditure management but to facilitating a broader policy agenda of public service reform.

Concurrent 6

Who Am I to Evaluate That?  How We Aspired to Apply the Values of Diversity and Inclusion in Our Evaluation.

Mary Peters, Senior Evaluator, Treasury Board of Canada Secretariat

How do you know if your best was good enough? Considering two Treasury Board of Canada Secretariat evaluations, this presentation will discuss our approach and the learning curves inherent in inclusive, diverse, equitable and accessible evaluations. We are all more than we appear – how can we bring this to table?

Concurrent 7

Relevance Reconsidered: Exploring Evaluation Criteria for Public Programs and a Forward-Looking Approach in Canada’s Ministry of Natural Resources

(NOTE: THIS SESSION WILL BE PRESENTED IN FRENCH)

Raoul Tamekou Tsowa, Ph.D, Policy Analyst, Natural Resources Canada

The traditional role of evaluators has largely been to analyze past actions or events and assess their outcomes (Rossi, Lipsey and Freeman, 2003). This retrospective analysis provides a vital understanding of what has worked, what has not, and why. Evaluators typically consider a range of criteria, such as effectiveness, efficiency, impact, and sustainability. However, some critics posit that a focus on retrospective analysis might not be enough. Decision-makers often seek guidance for future initiatives, implying that evaluation needs to be more future-oriented. They want to know what actions they should take based on the evaluation findings, and how to apply these learnings to future decisions (Wicker, 2023a). In a few communications, Ted Wicker suggests one way of addressing this concern: by giving more importance to the criterion of ‘relevance’ in evaluations. The concept of relevance in evaluation usually refers to the extent to which the objectives of a project or program are consistent with beneficiaries’ requirements, country needs, global priorities, and partners’ and donors’ policies. By emphasizing relevance, evaluators can help ensure that evaluation findings are not only descriptive of the past but are also prescriptive for future initiatives, reflecting the priorities and needs of the stakeholders involved (2023b).

The purpose of this paper is to reflect on evaluation criteria in the practice of evaluating public programs in Canada. More specifically, we question the place of the criterion of relevance, its meaning, and its evolution in program evaluations carried out by the Ministry of Natural Resources Canada since the 2000s. Also, the article emphasizes that while the criterion of relevance proves heuristic from a forward-looking perspective, it is insufficient. Other solutions are considered which aim to more effectively position the practice of evaluation in the future.

Concurrent 10

Logic Model and Integrated Reporting Model for Arts Funding

Gabriel Zamfir-Enache, Director of Research, Measurement and Data Analytics, Canada Council for the Arts

As a public funding agency for the arts, the Council has a mandate to foster and promote the study and enjoyment of, and production of works in, the arts. Many of its programs, services and activities align with the theme of this year’s symposium, as well as its three streams.

Some of its key activities and areas of focus that parallel the theme of the symposium include, but are not limited to: 

1. Performance measurement with outcomes-based programs, including impacts that can be easily communicated to Canadians. 
2. Innovation in results reporting with an increased capacity in data mining for evidence-based decisions based on quantitative measurement of arts funding related to outputs (e.g. number of grants, recipients, type of support, etc.) as well as an increased capacity in measuring qualitative program outcomes (e.g. direct impact on artistic career of recipients, public engagement, etc.). 
3. Highlighting quantitative and qualitative data related to outputs, outcomes and impacts to support the Council in enabling it to tell a rich story about how its investments strengthen the arts and benefit the Canadian public and promote Canadian creativity on the international stage. 
4. Exploration and integration of digital and new technologies (AI, machine learning) in the arts sector. 
5. Ongoing commitment to Truth and Reconciliation and decolonization, fostering relationships between Indigenous artists and both Indigenous and non-Indigenous audiences, as well as surpassing a goal to triple support for Indigenous creation to $18.9 million by 2020-21. 
6. Exploration of integration of environmentally sustainable approaches and practices in program guidelines, reporting and evaluation, with a focus in reducing the carbon footprint of the arts sector. 

Wayne MacDonald

Who Are You?
The promo for this year’s Symposium suggests an exploration of multiple themes: What are we doing? Why do we need change? How will we know when we get there? The common denominator is “we”. Who are “we”? Recalling the 1978 music classic by The Who, “Who Are You”, the focus is in this presentation is on “who are we” as a PPX community of interest. The presentation will examine several topics: Who joins PPX? What is the collective professional expertise of PPX members who do join? What are PPX membership interests? This descriptive analysis looks at characteristics of PPX membership. The analysis is based on most recent 100+ member registrations, those who joined the PPX over the past 12 months. The aim is to gain a better-appreciation of “who we are” as a community of interest, and this should be a precursor to addressing some of the key themes posed by the 2023 Symposium.

Welcome Reception

Welcome Day 2

Professor Eric Champagne, University of Ottawa and PPX VP

Opening Keynote

Accountability to Canadians in a Decentralized Federation 

Jeannine Ritchot, Assistant Deputy Minister
Multilateral Relations – Intergovernmental Affairs
Privy Council Office

Jeannine Ritchot joined Intergovernmental Affairs, Privy Council Office as the Assistant Deputy Minister of Multilateral Relations in April 2022. In this role, she provides advice and thought leadership on matters affecting the evolution of the federation, including on the federal-provincial-territorial dynamics of key social and economic policy files.

Prior to this, Jeannine was the Assistant Deputy Minister of the Communications and Portfolio Sector at Natural Resources Canada. She has also held executive positions at Treasury Board Secretariat, Health Canada, and the Public Health Agency of Canada, where she acquired extensive experience developing regulation and programs in collaboration with provinces and territories.

In 2020, for her efforts in promoting international and interprovincial regulatory cooperation, she became an inaugural recipient of the Apolitical and World Economic Forum “Agile 50” Award, which recognizes the 50 Most Influential People in the World Revolutionizing Governance. In 2019, Jeannine became the first woman to serve as Chair of the OECD Regulatory Policy Committee. She was also a peer reviewer of governance structures for international regulatory cooperation in the U.K. and Mexico and negotiated the Canada-United States-Mexico Agreement Good Regulatory Practices Chapter. 

Through her community work, Jeannine is a fierce advocate for the advancement of women’s involvement in sport. She mentors University of Ottawa women athletes and works with organizations in Ottawa to ensure that girls have equal access to sports infrastructure. 

Jeannine holds a Bachelor of Arts degree in History from the University of Ottawa and a Master of Arts degree in Conflict Studies from St. Paul University / University of Ottawa.

Networking break

Plenary Panel

Digital Transformation

Description:

What does it mean to have public sector organizations ready to respond to the challenges of today and tomorrow, when service delivery increasingly means digital delivery by default?

From government legislative assemblies to the editorials of the day, much talk revolves around the challenges organizations face in transforming service delivery of key government services to citizens. Why can some jurisdictions make it easy to renew drivers’ licenses where others struggle to support passport renewal?

The prevailing view is that Canada is failing and falling further behind in meeting its citizens’ needs. Indeed, Canada has plummeted from being ranked 3^rd in the world in its digital service delivery to 32^nd in little over 10 years.

So what’s going on? How do we know what is and is not working well on the digital transformation file? What roles have key functions (e.g., results performance, audit, evaluation, and risk management) played to date in digital transformation. How have they helped – or possibly even hindered – transformation? How can these functions become key partners in public service renewal?

Join us for a panel discussion with leaders in digital transformation in Canadian public sectors to hear their thoughts and to ask your questions on these pressing issues.

Moderator:
Jacky Tweedie, Principal, Delta Vega Consulting

Panelists:
Honey Dacanay, Director-General, Digital Policy, Treasury Board Secretariat of Canada
Ryan Androsoff, CEO and Founder of Think Digital
Yves Giroux, Parliamentary Budget Officer

Keynote Panel

Review of the TBS Policy on Results

Ken Stephenson, Director, Policy on Results and Engagement, Treasury Board Secretariat of Canada

Description:

PPX welcomes Kenneth Stephenson’s off er, on behalf of the Secretariat, to provide an update of its ongoing activities in the Policy Review, including the consultations with deputy heads, assistant deputies, and heads of performance measurement and evaluation.

Lunch

Sponsored by Goss Gilroy Inc.

Plenary Panel

Truth & Reconciliation with Indigenous Peoples

Chair:
Professor Veldon Coburn, McGill University
Professor Sarah Wiebe, University of Victoria
Professor Janique Dubois, University of Ottawa

Concurrent 9

Breaking Data Barriers and Silos: How Natural Resources Canada is Advancing Disaggregated Data Collection for Gender-Based Analysis Plus (GBA Plus)

Allain Barnett, Policy Analyst, Gender Based Analysis Centre of Expertise, Strategic Policy, and Innovation
Erica Kaneza, Manager, Gender Based Analysis Centre of Expertise, Strategic Policy, and Innovation

Through sharing data collection experiences in GBA Plus, departments can enhance their capacities to collect and analyze disaggregated data to understand diverse experiences, and to ensure no one is left behind. In this presentation, we share Natural Resources Canada’s practices in GBA Plus data collection. As reflected through data collection information released in Departmental Results Reports (DRR), our GBA Plus Centre of Expertise works to assist NRCan officials build their capacity to collect disaggregated data and measure the impacts of their programs on diverse groups of people.

The GBA Plus Centre of Expertise has consulted with NRCan sectors to establish a baseline on NRCan’s capacity to collect disaggregated data, common challenges experienced, and approaches to address these challenges. Overall results indicated that many NRCan’s programs collected disaggregated data through funded project reporting, workforce data, and engagement/participation. Of the programs collecting disaggregated data, most collected data on gender and Indigenous Peoples and/or communities. Data gaps remain for other identity factors, such as racialized individuals, regional diversity, age, and people with disabilities. Several programs have developed data collection approaches to meet these challenges, including:

• tracking GBA Plus information through multiple sources, 
• developing protocols to maintain confidentiality, 
• supplementing and leveraging data gaps with secondary data sources, and
• developing working groups and networks to share experiences. 

In this session, you will learn about some of the data challenges, experiences with GBA Plus at NRCan and ways you can fulfill reporting requirements related to disaggregated data collection. This session is your first step in understanding how to take to track your organizations’ policies and programs’ impacts on diverse groups of people. This is also a space for practitioners to exchange on their experiences with designing and implementing GBA Plus research and data collection plans.

Concurrent 11

Designing and Delivering Horizontal Programs and Services in the Federal Government 

Cristina Drob, Analyst, Treasury Board of Canada Secretariat
Brian Cho, Analyst, Treasury Board of Canada Secretariat
Robert Macleod, Advisor, Treasury Board of Canada Secretariat

Horizontal Initiatives in the Government of Canada play a crucial role in addressing complex priority issues by bringing together multiple federal departments and agencies. By utilizing a “whole of government approach,” these initiatives ensure that departments work together to achieve common objectives. During the 2022-23 fiscal year there were 30 active Horizontal Initiatives with $134 billion in total spending. 

The progress and outcomes of these Horizontal Initiatives are regularly reported through Departmental Plans and Departmental Results Reports. These reports not only ensure that Canadians stay informed about the government’s work but also provide a mechanism for tracking the government’s performance and making necessary adjustments to better serve Canadians.  

The presentation will provide insights into the strengths and challenges departments face in managing and reporting on Horizontal Initiatives. By sharing best practices and lessons learned with the performance and planning community, the Results Division at TBS aims to support federal departments to better plan and implement Horizontal Initiatives. 

Practical Ways to improve the efficiency and effectiveness of your Procurement activities

Carolyn Montague, Partner, Performance Management Network Inc and President, Canadian Institute for Procurement and Materiel Management (Volunteer)
Craig Szelestowski, Founder, Lean Agility Inc., Director, Lean Programs, Telfer Business School Centre for Executive Leadership, University of Ottawa

In the current climate of cost cutting pressure will be on all public servants to justify the outcome of their procurement activities. Craig Szelestowski of Lean Agility and Carolyn Montague of the Canadian Institute for Procurement and Materiel Management will share with delegates a number of practical ways Project Authorities can improve the efficiency and effectiveness of their procurement activities.

Closing Remarks

Art Stewart, PPX President