Disclaimer:
Please be aware that the content herein has not been peer reviewed. It consists of personal reflections, insights, and learnings of the contributor(s). It may not be exhaustive, nor does it aim to be authoritative knowledge.
Overview
Prepared by (Name of the experimenter)
Cristhian Parra
On date (Day/Month/Year)
26/8/2024
Current status of experimental activity
Completed
What portfolio does this activity correspond to? If any
Formalization and SMEs promotion
What is the frontier challenge does this activity responds to?
Formalization and SMEs promotion
What is the learning question(from your action learning plan) is this activity related to?
To what extent can the direct interacitons of waste pickerst with household improve trust between them?
Please categorize the type that best identifies this experimental activity:
Quasi Experimental (Analytical, observations, etc)
Which sector are you partnering with for this activity? Please select all that apply
Public Sector, Private Sector, Civil Society/ NGOs
Please list the names of partners mentioned in the previous question:
Coca Cola Company (private), Fundación Moisés Bertoni (Civil Society), Soluciones Ecológicas (Private), MADES (public)
Design
What is the specific learning intent of the activity?
To evaluate the impact of a pilot to implement a differentiated waste collection service that connects an association of informal waste pickers' with actual homes in 5 neighborhoods of Asunción, the capital of Paraguay
What is your hypothesis? IF... THEN....
IF we establish and facilitate interaction between waste pickers and households THEN trust relationships will develop among the parties, which in turn will contribute to increasing the level of engagement in activities that make part of the collective dilemma of recycling ( e.g., waste sorting at the point of origin). This hypothesis is fully in display on the causal chain our Recycling as a collective action dilemma blogpost.
Does the activity use a control group for comparison?
Yes, a different group entirely
How is the intervention assigned to different groups in your experiment?
Non-random assignment
What is the timeline of the experimental activity? (Months/Days)
6 months
What is the unit of analysis of this experimental activity?
Households and Waste Pickers. While random assignment was not possible, we did use random selection of the sample to include in the evaluation (on the households' side).
Please describe the data collection technique proposed
This was a mixed-methods experiment that had two different ways of collecting data and evaluating impact. Impact evaluation on households: on the households’ side, two surveys were implemented. First, a characterization survey on the whole population of households that signed up for the pilot. Second, an impact evaluation survey collected from a random sample of households that were included in the pilot (the treatment group) and a random sample of households that signed up but were excluded because they fell outside the neighborhoods selected for the pilot (the control group). Only the universe of households that signed up was considered to mitigate the auto-selection bias that was unavoidable in this initiative (i.e., both groups are groups already interested in taking part of this type of services). Furthermore, the random selection strategy for the impact survey ensured balance in both groups across variables of income, employment, and gender, collected in the first survey. Econometric analysis including standard t-tests to compare means and the use of multivariate linear regression to explore interaction among variables were used to estimate impact on recycling knowledge, recycling behavior, trust, among other things. Impact evaluation on waste pickers: in-depth interviews, a cultural probe and a participatory evaluation workshop were implemented to understand how the new service impacted the lives of the 5 different waste pickers who were in charge of providing the service.
What is the estimated sample size?
50-99
What is the total estimated monetary resources needed for this experiment?
Between 1,000 and 9,999 USD
Quality Check
This activity is relevant to a CPD outcome, The hypothesis is clearly stated, This activity offers strong collaboration oportunities, This activity offers a high potential for scaling
Please upload any supporting links
What are the estimated non- monetary resources required for this experiment? (time allocation from team, external resources, etc) If any.
A very rough approximation would be to say that 20% of the time of our Head of Experimentation, plus 35% of the time of our Lead Data Analyst, went to design, manage, analyze, report and follow-up of the results, throughout the duration of the pilot.
Results
Was the original hypothesis (If.. then) proven or disproven?
On the household’s side: our hypothesis about trust was proven. Impact of the pilot on the levels of trust between households and waste pickers increased by approximately 37% (p<0.01). Meaningful interactions were also higher, by roughly 35% (p<0.05), in the treatment group than in control. No other variables were impacted on the households’ side: knowledge about recycling was higher in control (but both were already too high, leaving little room for improvement), interactions between households also showed no differences, sustainable management of waste was also approximately the same in both treatment and control. On the waste pickers’ side: qualitative evaluation identified two key positive impacts, (1) the perception that households started treating them better, and (2) gaining pride on their daily work, particularly on the eyes of their families and children. Other results of note: while knowledge about recycling was found to be already high in the population we engaged, this did not translate into practice as qualitative observations recorded high levels of frustration from waste pickers on the quality of the recyclable waste they received, which often included materials that were not valuable. Moreover, logistics and organization were noted as an element to improve, waste pickers faced higher mobility costs and the return was not necessarily better, households noted that more frequencies and flexibility was needed. Overall, the service is still not perceived to be financially sustainable. Our impact evaluation, however, opened the door to second phase were policy implications and specific and concrete ideas we propose, inspired by the results, are being considered to inform the implementation and scaling up of the pilot in the 2023.
Do you have observations about the methodology chosen for the experiment? What would you change?
We feel confident about our methodology, and particularly, about the fact that we were able to incorporate this in an action that was not fully under our control to design. It serves as a showcase of how to integrate learning in action, compromising just enough. Other than evaluating the impact, the mixed methods we applied enabled us to engage allies in the process of learning and helped them buying into our proposals. The cultural probe we tried, on the other hand, did not work as well. It generated few in-context responses, and they were mostly too short to provide enough richness. To work better, more time would be needed for this type of methods to work in this type of studies. We only had 2 weeks of reported experiences. In-depth interviews were added to complement this and provided great amount of insight. A participatory evaluation workshop, bringing together waste pickers and institutional allies that were part of the organization, was key to identify consensus points regarding the next stage of the pilot, which will begin in 2023. This workshop in particular was highly praised and well received by everyone, while at the same time generated rich data. The workshop started with waste pickers and organizations working separately, evaluating the initiative and proposing ways to improve it. Both groups then switch places: waste pickers discussed and identified potential issues with proposals made by organizations, and viceversa. A final deliberation brought everyone into the same room to discuss dissensus proposals and gain awareness about consensus proposals.
From design to results, how long did this activity take? (Time in months)
9
What were the actual monetary resources invested in this activity? (Amount in USD)
For evaluation purposes only: 1. Impact evaluation on households (through digital surveys) ~ $3500 2. Impact evaluation on waste pickers (through qualitative methods) ~ $2500 3. Pilot implementation: undisclosed amount by The Coca Cola Company. 4. Other resources: time invested by AccLab Members (average or 2h-4h/week over the full period of the pilot), half time engagement of a social worker involved in strengthening the association of waste pickers, hired by UNDP and supported by AccLab team.
Does this activity have a follow up or a next stage? Please explain
The pilot started with 5 waste pickers, 5 neighboorhods and 1 waste pickers association. It nows cover 19 neighboorhoods, 17 waste pickers, and over 16 associations.
Is this experiment planned to scale? How? With whom?
Yes, with the same allies.
Please add any supporting links that describe the planning, implementation, results of learning of this activity? For example a tweet, a blog, or a report.
Considering the outcomes of this experimental activity, which of the following best describe what happened after? (Please select all that apply)
Solutions tested in this experiment were scaled in numbers
Learning
What do you know now about the action plan learning question that you did not know before? What were your main learnings during this experiment?
We know that trust building is in the pathway towards increasing recycling practices in the city. We know that strengthening cooperative practices and associative organization among waste pickers is a needed investment. We know that knowledge does not translate into practice, but also that households consider recycling a collective action dilemma, opening the way to integrating community participation practices to improve the linkage between households and waste picker associations.
What were the main obstacles and challenges you encountered during this activity?
Logistics and organization of the weekly tours. Issues of breakdown and repair of motorcars that had little support from the social enterprise tasked with coordinating implementation. Financial sustainability for waste pickers: the service is not yet profitable for them, representing often a high opportunity cost (e.g., to attend participants demands, they have to resign other valuables they can collect along the way)
Who at UNDP might benefit from the results of this experimental activity? Why?
Asuncion Green City Project, a large project implemented by the Environment Program of our CO, whose goal is to impact
Who outside UNDP might benefit from the results of this experiment? and why?
The Association of Waste Pickers in San Francisco Neighborhood might benefit if a better coordination and cooperation leads to a higher volume of recyclable recovered and sold by them. They also benefit by strengthening their organization. The Coca Cola Company can benefit by significantly contributing to the recovery of the plastic waste they introduce into the system. The city of Asunción is benefitting by having its first differentiated waste collection service.
Did this experiment require iterations? If so, how many and what did you change/adjust along the way? and why?
Before the final form, our original plan was to implement a survey to measure the impact of a natural experiment already taking place in the city around the installation of the so-called “Eco-points”: recyclable waste containers installed in companies or other places, managed by a Social Enterprise and attended to by a selected waste picker. This was meant to be stage one, later to be followed up by stage 2, experimenting directly with the prototype of a differentiated waste collection service. In between stages, we were continuing our work with the Association of Waste Pickers, investing in activities to strengthen them as an organization. As we designed stage 1, The Coca Cola Company learned of our efforts in mapping routes and experiences of waste pickers, and approach UNDP Asuncion Green City Project to propose a collaboration around this initiative, which was very close to our definition of the stage 2, so we decided to join the effort and skip stage 1.
What advice would you give someone wanting to replicate this experimental activity?
In this activity, we could not ensure that participating household were randomly assigned to treatment or control, because the decision was outside our reach. However, we were able to influence that decision by using our exploration results: the routes we mapped. So, if you are looking to replicate this and have a similar situation, make sure you have a way to incorporate on your models at least a proxy that represents the criteria for assigning a house to treatment or control. In our case, that proxy is essentially the distance of houses to the closest route we mapped. Moreover, considering our universe of ~200 households that signed up, a good decision we made was to run a characterization survey to have data on the whole population that we took into account when randomizing the selection of households to evaluate. Make sure you do the same if you can and your sample is small. It will allow you to choose the best random selection strategy later on. Include in that characterization variables that do not correlate with your intended impact to assess the balance of your random selection, but include also at least one or two variables that do correlate or represent your dependent variable. It will help you run power estimates, among other things. Finally, if external allies come to you with a proposal that is similar or related to what you found as a hypothesis worth exploring, do not hesitate and collaborate, even if it is not on your terms. Finding ways to integrate learning into action, even when you do not have full control of the action, does have a place and generates valuable insights.
Can this experiment be replicated in another thematic area or other SDGs? If yes, what would need to be considered, if no, why not?
Some of the concepts and methods we used can be replicated. Any collective action dilemma where a group of people needs to connect to another group, to achieve a common goal that benefit both parties, can use some elements of this design.
How much the "sense" and "explore" phases of the learning cycle influenced/shaped this experiment? In hindsight, what would you have done differently with your fellow Solution Mapper and Explorer?
Both sensing and exploration activities influenced greatly the design of the experiment, the hypotheses we decided to test, the places and people we engaged, and the things we needed to observe as the experiment unfolded.
What surprised you?
That we were able to integrate very fast an impact evaluation strategy, fully designed by us, into the action that was planned and coordinated by a private sector multinational company. And that they actually paid attention to our results, considering them in the design of the next stage of their agenda.
8Decent work and economic growth
11Sustainable cities and communities
17Partnerships for the goals
Comments
Log in to add a comment or reply.