Disclaimer:
Please be aware that the content herein has not been peer reviewed. It consists of personal reflections, insights, and learnings of the contributor(s). It may not be exhaustive, nor does it aim to be authoritative knowledge.
Overview
Prepared by (Name of the experimenter)
Lorena Moscovich
On date (Day/Month/Year)
2023
Current status of experimental activity
Completed
What portfolio does this activity correspond to? If any
It corresponds to our Artificial Intelligence and Democracy portfolio.
What is the frontier challenge does this activity responds to?
This activity addresses an increasingly prevalent challenge in the current context: political polarization, a multi-causal phenomenon whose deepening is a global trend. One of these causes is the use of algorithms aimed at promoting user interaction and participation on social networks. Algorithms tend to display content that aligns with users' beliefs, intensifying their negative feelings to encourage consumption.
Furthermore, specifically in Argentine society, the sociopolitical context favors polarization between supporters of different political parties. In the context of the 40th anniversary of Argentine democracy, we use artificial intelligence tools, specifically natural language processing (NLP) through Chat GPT, to promote consensus and social cohesion. We emphasize what unites Argentine society beyond its political differences. To achieve this, we analyzed presidential inauguration and legislative session opening speeches from the last 40 years. We carry out an exercise that consists of three parts: discourse analysis, exploration of the topics addressed, and generation of a unified discourse from the corpus of analyzed texts. This tool can be generalized to
What is the learning question(from your action learning plan) is this activity related to?
The learning question is: Can AI, through NLP tools, contribute to depolarization?
When analyzing presidential speeches, to answer the learning question we can ask ourselves: Are there points of agreement and consensus in presidential speeches over the years that transcend partisan affiliations? Does the variation or continuity of themes converge over time?
Please categorize the type that best identifies this experimental activity:
Quasi experimental (Analytical, observations, etc)
Which sector are you partnering with for this activity? Please select all that apply
Civil Society/ NGOs, Academia
Please list the names of partners mentioned in the previous question:
Within academia, we work with various departments of Universidad de San Andres, including the Master’s program in Public Administration, the Center for Human Development Studies (CEDH, by its Spanish acronym), the Center for Technology and Society Studies (CETyS, by its Spanish acronym), the Department of Engineering, and the Data Science Lab within the Department of Mathematics and Sciences.
Among the Civil Society partners is Data Género, an organization centered in approaching data and AI from a gender perspective. Fundación Sadosky and Fundar are other partners, both are research institutions and are active in promoting technology for the development of Argentina. We also work with HumaAI, a learning platform focused on technology and computing skills.
Design
What is the specific learning intent of the activity?
The specific learning intent of the activity is to develop a natural language processing (NLP) model to promote consensus and social cohesion through the analysis of political speeches. This model helps to create a common space for understanding and dialogue with the goal of addressing political polarization. In the context of the 40th anniversary of Argentina's democratic recovery, we use this tool to analyse and systematize presidential speeches from 1983 to the present. This activity encourages dialogue and underscores the importance of unity in a polarized context.
What is your hypothesis? IF... THEN....
This exercise consists of three sets of hypotheses:
H1: The continuity of certain topics is positively related to the passage of time.
H2: The continuity or variation of topics is unrelated to the president's party affiliation.
H3: The continuity or variation of topics positively related to the context.
Does the activity use a control group for comparison?
No, it does not use a control group
How is the intervention assigned to different groups in your experiment?
Other
Describe which actions will you take to test your hypothesis:
To assess our hypotheses, we employed Chat GPT to analyze themes in all presidential inaugural and legislative session opening speeches from the past 40 years, from the return of democracy to the present. The initial thematic analysis involved defining 30 themes using an associated word dictionary. Then, we queried Chat GPT to determine if each speech addressed these pre-defined themes.
It's worth noting that due to AI character limitations, we had to segment and summarize each speech, creating an overall summary that consolidated these segments. ChatGPT can only operate with 4000 characters at a time, input (the question) and output (the answer) combined. Since most speeches exceed this number, they were divided into fragments of 2250 characters for analysis. Each of these fragments is then sent to ChatGPT with a prompt asking if that fragment addresses a particular topic (along with the definition of the topic) and, if so, ChatGPT is asked to copy exactly the specific part in which it appears. This action is repeated for each chunk and each of the topics. In this way, we obtain the topics addressed in all the fragments of each speech. From the identified themes in each speech, we examined the frequency of each theme per president over time, along with their associations with other themes and the specific words used in addressing them, among other aspects.
What is the unit of analysis of this experimental activity?
The unit of analysis is presidential speeches. The dataset consists of 52 inaugural and legislative session opening speeches delivered by 9 presidents over the course of 40 years of democracy.
Please describe the data collection technique proposed
To test our hypotheses, we needed text data. We obtained this information from presidential speeches, downloaded it from the national government's official website, and then processed this data through GPT Chat.
What is the timeline of the experimental activity? (Months/Days)
From May to November 2023, the project was carried out in several stages. In the first stage, from May to June, the model was developed, and data were collected from presidential speeches. In the second stage, from June to July, the information was processed. In the third stage, from July to December, these insights were used to write a report and publish an infographic on the topics of the speeches.
What is the estimated sample size?
50-99
What is the total estimated monetary resources needed for this experiment?
Between 10,000- and 20,000 USD
Quality Check
This activity is relevant to a CPD outcome, The hypothesis is clearly stated, This activity offers strong collaboration oportunities, This activity offers a high potential for scaling, This activity has a low risk
Please upload any supporting images or visuals for this experiment.
Please upload any supporting links
Report: https://www.undp.org/es/argentina/publicaciones/report-ai-argentine-intelligence Infographic of topics: https://inteligenciaargentina.org/en/ Infographic of speeches: https://inteligenciaargentina.org/discurso/en/
What are the estimated non- monetary resources required for this experiment? (time allocation from team, external resources, etc) If any.
Results
Was the original hypothesis (If.. then) proven or disproven?
The hypotheses were supported by observational, no experimental, data. We were able to test:
H1 The continuity of certain topics is positively related to the passage of time: The hypothesis was partially supported. Continuity is observed over time in certain themes, such as "macroeconomics," which was mentioned by all presidents. Other themes were predominantly addressed in different periods; for example, "democracy" was mentioned primarily in the post-military dictatorship period, and "technology" has been more prevalent in recent years. Additionally, some themes, though mentioned over time, have changed their connotation over the years. This is the case with "democracy," which initially referred to the political system and, over time, was used to mention the "democratization" of certain rights. In conclusion, there is evidence of similarity in the themes addressed despite variations in their focus and conception over time.
H2 The continuity or variation of topics is unrelated to the president's party affiliation: The hypothesis was confirmed; the variation or continuity of topics seems to depend more on the sociopolitical context than on the president's party affiliation. In other words, presidents from different parties but in similar contexts address similar topics.
H3 The continuity or variation of topics positively related to the context: The hypothesis was confirmed; the mention of certain topics seems to depend, to a greater extent, on a temporal factor than on a partisan one. In other words, presidents from the same period address similar themes.
Do you have observations about the methodology chosen for the experiment? What would you change?
Regarding the methodology chosen to conduct the action, which involves analyzing speeches from various presidents over the last 40 years, it is crucial to address potential biases arising from the frequency with which different presidents spoke. For this reason, we opted to standardize the sample by selecting inauguration speeches and opening sessions of legislative sessions, as these must be delivered by all presidents.
It is also important to consider biases inherent to the AI itself, particularly regarding text generation, as ChatGPT tends to converge to commonplaces and faces the potential presence of biases in the results due to the nature of the data it is trained on.
From design to results, how long did this activity take? (Time in months)
8 months.
What were the actual monetary resources invested in this activity? (Amount in USD)
For this activity, a total of $16,800 was invested. Of that amount, $4,400 was allocated to hiring the consultant who developed the NLP model, and $12,400 went to hiring the agency responsible for the project's identity design, including the website, report, and images for social media, among others.
Does this activity have a follow up or a next stage? Please explain
This activity is part of the 'AI: Argentina Intelligence' project, which aims to explore and spread the uses, regulations, risks, and opportunities of artificial intelligence for development. Specifically, this activity focused on the use of AI for depolarization. Although this activity has concluded, we plan to continue exploring the opportunities that artificial intelligence can offer for development in future stages of the project.
Is this experiment planned to scale? How? With whom?
No
Please add any supporting links that describe the planning, implementation, results of learning of this activity? For example a tweet, a blog, or a report.
Link micrositio: https://www.undp.org/es/argentina/proyectos/ia-inteligencia-argentina Link infographic of topics ( English ): https://inteligenciaargentina.org/en/ Unified discourse infographic link ( English ) : https://inteligenciaargentina.org/discurso/en/ AI report link ( English ): https://www.undp.org/es/argentina/publicaciones/report-ai-argentine-intelligence
Considering the outcomes of this experimental activity, which of the following best describe what happened after? (Please select all that apply)
Learning
What do you know now about the action plan learning question that you did not know before? What were your main learnings during this experiment?
The main learning during this experiment is related to creation of the unified speech. The creation of the unified speech is more than the union of fragments from different texts. It requires identifying common themes, conjugating them, and creating a voice that maintains the style of the presidents. AI tools like ChatGPT add great value to this task, as they not only enable quantitative discourse analysis but can generate text within a context.
What were the main obstacles and challenges you encountered during this activity?
Who at UNDP might benefit from the results of this experimental activity? Why?
Our results highlight how AI tools can be utilized for the analysis and generation of political discourses, as well as for combating political polarization. This has the potential to benefit other areas within UNDP, both in terms of topic and methodology. The democratic governance area could benefit from the results due to the relevance of the topic, while other areas such as gender or environment could leverage the text analysis tool without necessarily being limited to governance topics or depolarization objectives
Who outside UNDP might benefit from the results of this experiment? and why?
The results of this experiment could be beneficial for various sectors beyond UNDP. Firstly, both the Argentine government and public policy designers could directly benefit by obtaining key information to adapt governmental communication effectively. They could use the results as a valuable tool in seeking to reduce political polarization among different parties and exploring new avenues to achieve a more cohesive and unified society.
Furthermore, it may favor other researchers and academics who adopt this method for analyzing discourses and trends over time. Additionally, political analysts could benefit from understanding how certain themes have evolved and converged in different presidential periods, regardless of party affiliation. This could provide valuable insights for the assessment and prediction of political dynamics.
Did this experiment require iterations? If so, how many and what did you change/adjust along the way? and why?
No
What advice would you give someone wanting to replicate this experimental activity?
The replicability of this exercise fundamentally depends on the availability of data. At this point, it is important to clarify that the texts to be analyzed must be faithful to the speeches, especially if they were delivered orally, as the choice of words and the way ideas are expressed are highly relevant for the analysis and generation of text.
Can this experiment be replicated in another thematic area or other SDGs? If yes, what would need to be considered, if no, why not?
This exercise has the potential to be replicated in various activities involving text or speech analysis, as well as text generation. For instance, it could be applied to other thematic areas such as Sustainable Development Goal (SDG) 3, "Good Health and Well-being," or SDG 5, "Gender Equality." Specifically, the experiment could be extended to analyze discourse surrounding gender equality policies and initiatives over the past decades. This would involve exploring how different stakeholders, including government officials, activists, and policymakers, have addressed issues such as gender-based violence and women's empowerment. The aim would be to study the progress made in these areas in recent years and, simultaneously, identify persistent challenges and emerging priorities in the pursuit of gender equality.
How much the "sense" and "explore" phases of the learning cycle influenced/shaped this experiment? In hindsight, what would you have done differently with your fellow Solution Mapper and Explorer?
This action was carried out in collaboration with the exploration area, as it is part of a larger project focused on exploring cutting-edge technology for both the Lab and the country, specifically artificial intelligence.
What surprised you?
16Peace, justice and strong institutions
17Partnerships for the goals
Comments
Log in to add a comment or reply.