Disclaimer:
Please be aware that the content herein has not been peer reviewed. It consists of personal reflections, insights, and learnings of the contributor(s). It may not be exhaustive, nor does it aim to be authoritative knowledge.
Overview
Prepared by (Name of the experimenter)
Cristhian Parra
On date (Day/Month/Year)
26/8/2024
Current status of experimental activity
Completed
What portfolio does this activity correspond to? If any
Building innovation capacities
What is the frontier challenge does this activity responds to?
Public and Social Innovation
What is the learning question(from your action learning plan) is this activity related to?
To what extent can we improve the access to better food in vulnerable populations? To what extend increasing local access will result in improvements in the diets of vulnerable families? How can we apply agile R&D methods to develop and test programs with the public sector?
Please categorize the type that best identifies this experimental activity:
Fully Randomised (RCTs, etc.)
Which sector are you partnering with for this activity? Please select all that apply
Public Sector
Please list the names of partners mentioned in the previous question:
Presidential Delivery Unit (National Strategy of Innovation), Municipality of San Juan Nepomuceno
Design
What is your hypothesis? IF... THEN....
If → a strategy for the provision of food coupons is established so that families in urban and peri-urban areas of San Juan Nepomuceno can exchange them for fruits and vegetables at local agricultural fairs (…) then → these families will improve their economic and physical access to these fresh foods with high nutritional value, and they will improve their indices of dietary diversity.
Does the activity use a control group for comparison?
Yes, a different group entirely
How is the intervention assigned to different groups in your experiment?
Random assignment
Describe which actions will you take to test your hypothesis:
The intervention consists of providing coupons to acquire fruits and vegetables from local fairs to beneficiary and non-beneficiary households of social programs. Each coupon has a value of G. 50,000 (roughly USD 9) and will be provided for 4 weeks to each selected household. In the framework of the intervention, there will be two groups of households that will be selected through the strategy of randomization. The random sample will be made up of a total of 280 households grouped as follows: (1) Treatment group: 140 households that for 4 weeks will receive a coupon per week to acquire fruits and vegetables worth G. 50,000 at local agricultural fairs. (2) Control group: 140 households that will not receive coupons for fruit and vegetables.
What is the unit of analysis of this experimental activity?
The study unit are households in urban and peri-urban areas of San Juan Nepomuceno, Department of Caazapá. To account for how vulnerability plays a role on this experiment, we divided the whole population in two blocks: (A) families that receive government money transfer aid, (B) families that do not government receive money transfers. Receiving money transfers from the government is used as a proxy for higher vulnerability in this study. In total, 280 households were randomly selected from the urban and peri-urban census map of San Juan Nepomuceno for the cluster A, and from the Government Database of the Tekopora Program for the block(B). Each block was later randomly assigned to either treatment (receive the food stamps for 4 weeks) or control (do not receive the food stamps). All 280 took part of the experiment.
Please describe the data collection technique proposed
First, we articulated efforts and cooperation with the Ministry of Social Development, to get access to their administrative records of the national direct money transfers program Tekopora. This provided us with a database of all the beneficiaries of the program. On this database, we filtered out all the households that were outside the urban and peri-urban area areas of the municipality of San Juan Nepomuceno, the territory we selected for the experiment. Second, we organized a mapping collaboration session with social workers that work with families in the Tekopora program. Together with them, we mapped each of the households in the database into a physical map that was later digitalized. Third, we used census data to complete our georeferenced household’s database with households that are not from the Tekopora program but are within the urban and peri-urban areas of the same municipality. With this database, we then proceeded to randomly select our sample of participant households using QGis software algorithms to randomize city blocks first, and households within city blocks second, both to select participants, and then randomly assign treatment in a second stage. For the group of selected households without members participating in social programs, we conducted a socioeconomic characterization survey at the beginning of the experiment to have data to generate a balanced treatment group and the control group. Once the intervention period ended, an impact evaluation endline survey was carried out to the households that are part of the treatment group and the control group. This survey collected data using the Household Dietary Diversity Survey (HDDS) questionnaire, adapted from Swindale and Bilinsky (2006) (see link below).
What is the timeline of the experimental activity? (Months/Days)
6 months
What is the estimated sample size?
100-999
What is the total estimated monetary resources needed for this experiment?
Between 10,000- and 20,000 USD
Quality Check
This activity is relevant to a CPD outcome, The hypothesis is clearly stated, This activity offers strong collaboration oportunities, This activity offers a high potential for scaling, This activity has a low risk
Please upload any supporting links
What are the estimated non- monetary resources required for this experiment? (time allocation from team, external resources, etc) If any.
A very rough approximation would be to say that 20% of the time of all the Heads the Lab, plus 20% of the time of our Lead Data Analyst, went toward designing, managing, analyzing, reporting and following-up on the results, throughout the duration of the experiment and the sensing and exploration activities that led to it.
Results
Was the original hypothesis (If.. then) proven or disproven?
Yes, it was. We measured the impact of our intervention by estimating the Average Treatment Effect (ATE). It is a variable that compares the difference in average outcomes between the treatment and control group, when there is random assignment of the treatment. For the calculation, a cross-sectional regression model is estimated, where the dependent variable is the logarithm of the household's dietary diversity index. The ATE turns out to be the parameter that accompanies the treatment dummy variable. The model shows that, on average, treatment group had +0.64 scored on the HDDS index, and this result is statistically significant. In simpler terms, families that had access to the coupons added, on average, a little more than half of a whole new food group to their diets. Looking into the details, the new groups that were added include: green leaves, vegetables rich in vitamin A, pulse, nuts and seeds.
Do you have observations about the methodology chosen for the experiment? What would you change?
We feel confident about our methodology to evaluate impact of the intervention in the diversity of diets. There are many other variables we could have measured and analyzed, and the data we collected will surely allow for other forms of analysis. Particularly, we were able to articulate the cooperation of more than 3 different public institutions to create both the conditions and later facilitate the implementation of the experiment: the social development ministry contributed data on its social programs, the municipality of San Juan Nepomuceno gave the space to gather and map households to categorize census geo spatial data. It also provided the venue and facilitated spaces to discuss options of the intervention and later on, its results with the community. Moreover, the presidential delivery unit helped in all the activities, articulating contacts and opening doors for us to work with allies. As for limitations, we need to explore in the future the offer side. The local fair only offers its own production, which limited a little the spectrum of products available. Seasonal dynamics and the integration of other local fairs or community gardens into the control variables are things to explore. Census data is outdated (last census was in 2012, and was riddled with implementation issues as it happened in the context of political turmoil).
From design to results, how long did this activity take? (Time in months)
6
What were the actual monetary resources invested in this activity? (Amount in USD)
Approximated values are below, for each activity of the experiment. $2.200 = Characterization survey and baseline (data collection, in presence survey) $2.000 = Implementation, logistics and monitoring. $2.700 = Impact evaluation survey (data collection, in presence survey) $4.000 = Food stamps value $300 = Printing of food stamps and other materials $11.200 = Total cost of implementing and collecting data for the experiment $7.500 = Consultant that work throughout this learning cycle as our associate researcher in the field, designing, conducting, and articulating local actors for the implementation of the whole field research. Other resources: time invested by AccLab Members (average or 2h week over the full period of the learning cycle, with an increase to 8h-10h/week during the experiment), half time engagement of a the Social Innovation Coordinator from the Presidential Delivery Unit.
Does this activity have a follow up or a next stage? Please explain
The intervention itself was socialized in a series of co-creation workshops with public institutions, academics, and civil society, to identify specific opportunities to scale either the intervention itself or its methodological contributions to logistics of social assistance public programs and services.
The intervention did not scale, but the methodologies we used allowed us to partner with the National Strategy of Innovation in launching the Public Innovators Proram.
Is this experiment planned to scale? How? With whom?
Possibly. The National University of Asunción, through its Research Direction, has committed already some resources for a version 2 of the pilot, with a focus on the offer side of local fairs and community gardens, and possibly connected to a new Innovation Center for Family Agriculture that the lab is supporting in its co-creation stage with community. Other institutions are interested as well. We will push some of these ideas forward within the context of the new Program to Develop Innovation Capacities in the Public Sector, the new cooperation we will have throughout 2022 with the National Strategy of Innovation. This Program will form a cohort of 50 public officials, selected by contest, in R&D+innovation methodologies and tools. We will support this effort both in its human capital development stage and more importantly, in the design and implementation of 2 pilots, one for public policy innovation and one for public service innovation. If teams from the institutions interested in following up upon this intervention are able to come up with a strong follow up, they will be able to obtain up to $15k for its piloting from our CO TRAC funds.
Moreover, the experiment led to the National Strategy of Innovation finally realizing the value of this type of applied R&D for public innovation, which resulted in the launch of the Public Innovators Program, which is entering its 3rd year of implementation ever since this experiment was completed.
Please add any supporting links that describe the planning, implementation, results of learning of this activity? For example a tweet, a blog, or a report.
Considering the outcomes of this experimental activity, which of the following best describe what happened after? (Please select all that apply)
This experiment influenced public policy at a national or local level, This experiment led to adoption of new ways of working by our partners
Learning
What do you know now about the action plan learning question that you did not know before? What were your main learnings during this experiment?
We know that increasing economic access to fresh vegetables and fruits produced locally has an impact on the dietary diversity. We have also learned how to design and implement the logistics, monitoring, and execution evaluation processes that are needed to deliver programs like the one we piloted, and after a series of co-creation workshops where we deliberated on the results, learned how rich and transferrable these processes are to other public services and programs for food security assistance. We learned how important it is for the lab members to be on the field before, during and after the experiment, in order to increase take up and achieve buy in. And finally, we could watch up close how a strong community organization such as the one that organizes the fair, can be a key partner in food security programs, creating opportunities to increase their productivity by establishing connection with these programs that can sustain a local demand for their products in both crisis times but also throughout the year when programs are of continuous delivery (e.g., the school lunch programs, for example).
What were the main obstacles and challenges you encountered during this activity?
Time. We had little time to both design and implement the experiment. The bureaucratic demands for delivery before the end of the year stressed us thin and limited the room for a good implementation.
Who at UNDP might benefit from the results of this experimental activity? Why?
Both our CO´s environment and inclusive development portfolios can benefit from both the lessons we gathered and the partnerships we were able to mobilize.
Who outside UNDP might benefit from the results of this experiment? and why?
All the partners involved in the experiment: (1) the Ministry of Social Development has expressed interest in transferring lessons from this experiment to their community gardens program that complements the direct monetary transfers they manage, (2) the Presidential Social Cabinet Delivery Unit has interest in learning from the articulations we mobilized to coordinate the multiple institutions they have to engage, (3) the Presidential Delivery Unit has developed and strengthened their R&D+innovation skills in the context of a public policy and social program issue, which will be key in their upcoming Public Innovation Program, and (4) the Municipality of San Juan gained insights into how to localize social program interventions, (5) the local fair increased their demand and was able to do so without renouncing to their principle of only offering their own products (6) FAO Latin America is prioritizing new approaches to food security, and the experience and evidence generated by this learning loop is unique not only in Paraguay, but in the region in general and could serve to orient UNDP-FAO collaboration.
Did this experiment require iterations? If so, how many and what did you change/adjust along the way? and why?
Not so many iterations were needed (or possible), mainly due to the limited time available. As soon as the exploration phase signaled the economic barrier problem as one of the most important ones to solve in our context, our team started to develop the idea of a food stamp intervention, basically following upon the idea of developing public innovation, in this case, innovation for food security assistance programs that would be feasible to take up by the Social Cabinet institutions that were working to provide such assistance during the pandemic. The first modification to this idea incorporated the connection to the local fair, to also incorporate a local economic development approach into the mix. And the second iteration limited the exchangeable products to only those available at the fair, after the local fair uphold their principle of never engaging in reselling of product: they only sell what they can produce.
What advice would you give someone wanting to replicate this experimental activity?
Be proactive in identifying who the key institutional actors are, and who are the people who is working in the field. Engage them and bring them to the table early on, they will be key to both adapt the intervention to the context and make its implementation possible. Secondly, here the target is really on two fronts: public innovation capacities and food security. For our portfolio, the priority is to develop innovation capacities in the public sector, and food security came as one challenge through which develop those capacities, but it is not then endgame of the portfolio. If the priority would be a portfolio in food security in and on itself, then expand the experimentation to include the season dynamics, the offer side, engaging other local fairs, and what other food security interventions are happening in parallel. A food systems approach might be better in this latter case, where you look into the full value chain of the system you are intervening.
Can this experiment be replicated in another thematic area or other SDGs? If yes, what would need to be considered, if no, why not?
The hypothesis itself is replicable: increasing economic access certain goods as means to achieve an impact on improving some aspects of the phenomena that are associated to those goods. The intervention itself, may need adaptation to other thematic areas, but there many examples of “coupons” or “vouchers” approaches and examples. Think for instance on the SMEs Vouchers scheme (https://www.surrey.ac.uk/innovation/funding/sme-innovation-voucher-scheme) as one iteration of the same concept, but taken to the area of business innovation.
How much the "sense" and "explore" phases of the learning cycle influenced/shaped this experiment? In hindsight, what would you have done differently with your fellow Solution Mapper and Explorer?
Everything in the experiment came out of a cycle of three months of sensing and exploring the topic. Before that, we had also spent a year articulating efforts with the National Strategy of Innovation in mapping food security initiatives during the pandemic, both through collaborative map Wendá (map.wenda.org.py) and also through our community social innovation project Moiru. Learning from those initiatives also fed into our sensing and exploration phases for this learning cycle.
What surprised you?
We were particularly surprised by the strength of the community organization behind the local fair Ka’avo: a group of women farmers who were actively engaged in their community organization efforts, and who would establish their own rules regarding how to participate of the intervention. Our original plan involved giving them funding to buy and sell some of the products that were not produced by them, and they were clear with us that getting to the point where they were took them several years, and they wouldn’t break something that was their principle (to sell only what they produce) just to have higher revenues. We all admired this and did what we had to adapt the intervention to those rules, putting their principles first.
1No poverty
2Zero hunger
10Reduced innequalities
