Generating Code Explanations using Large Language Models

S. MacNeil A. Tran D. Mogil S. Bernstein E. Ross F. Huang
ACM ICER 2022, ACM SIGCSE 2023

---

Good explanations are essential to efficiently learning introductory programming concepts. To provide high-quality explanations at scale, numerous systems automate the process by tracing the execution of code, defining terms, giving hints, and providing error-specific feedback. However, these approaches often require manual effort to configure and only explain a single aspect of a given code segment. Large language models (LLMs) are also changing how students interact with code. For example, Github’s Copilot can generate code for programmers, leading researchers to raise concerns about cheating. Instead, our work focuses on LLMs’ potential to support learning by explaining numerous aspects of a given code snippet. This poster features a systematic analysis of the diverse natural language explanations that GPT-3 can generate automatically for a given code snippet. We present a subset of three use cases from our evolving design space of AI Explanations of Code.

Curri: Visualizating the Curriculum

S. MacNeil M. Dorodchi E. Al-Hossami A. Benedict MJ Mahzoon D. Desai
ASEE 2020, IEEE FIE 2018

---

Curriculum design can be more of an art than science. Decisions are seldom made based on data. Instead, they are often made based on the expertise of academic advisors and curriculum designers. Curri combines student data with information from the course catalog to visualize when, on average, students take specific courses. This allowed curriculum designers to identify bottlenecks (areas where many courses are taken at the same time) and inversions (areas where a class is taken before the pre-requisites) which can delay graduation. Curriculum designers can adjust to ensure that the courses are more evenly spread out temporally. Curri was designed for curriculum designers and advisors, but might also provide students with data about their own trajectories.

• Keywords: machine learning, visualization

ProbMap: Automatically constructed design galleries

S. MacNeil Z. Ding K. Quan Z. Huang K. Chen S. Dow
ACM UIST 2021

---

Making sense of large unstructured problem spaces is cognitively demanding. Structure can help, but adding structure to a problem space also takes significant effort. ProbMap is a novel application for automatically constructing a design gallery from unstructured text input. Given a list of problem statements, ProbMap extracts and semantically groups the stakeholders to construct hierarchical search facets which enables designers to more efficiently navigate the problem statements. We contribute a novel feature extraction algorithm using natural language processing and a technique for automatically constructing a design gallery. These stakeholders are grouped semantically by clustering stakeholders with higher pairwise similarity together. Preliminary trials show that these techniques, which mirror traditional design activities like stakeholder identification and affinity mapping, provide an initial structure to a large unstructured problem space. This resulted in similar features that would be extracted by humans and sensible clusters.

ProbLib: Helping Novices to Frame Better Problems

S. MacNeil Z. Ding K. Quan T. Parashos Y. Sun S. Dow
ACM C&C 2021

---

Problem framing---the process of defining a problem---has been described by many researchers and designers as the crux of the design process. However, novice designers struggle with problem framing. To better understand this process and the potential for scaffolding, we conducted two studies. In the first study, we analyzed 41 problem statements from an introductory design course and found that novices often omit key information, like the primary stakeholder or the obstacles they face. To get novices to reflect on and include necessary design information, we created a tool called ProbLib that cues novices to explicitly reflect on aspects of the problem such as the stakeholders. To evaluate this approach, we conducted a between-subjects study (N=73) to compare ProbLib with an unstructured open text form. We found that participants using ProbLib wrote higher quality statements, included more information, and were more confident about specifying design needs. We observed creative behaviors such as brainstorming and analogical reasoning.

CoNotate: Adaptive Query Suggestions Support Learning

S. Palani Z. Ding A. Nguyen A. Chuang S. MacNeil S. Dow
ACM CHI 2021

---

When exploring a new domain through web search, people often struggle to articulate queries because they lack domain-specific language and well-defined informational goals. Perhaps search tools rely too much on the query to understand what a searcher wants. Towards expanding this contextual understanding of a user during exploratory search, we introduce a novel system, CoNotate, which offers query suggestions based on analyzing the searcher’s notes and previous searches for patterns and gaps in information. To evaluate this approach, we conducted a within-subjects study where participants (n=38) conducted exploratory searches using a baseline system (standard web search) and the CoNotate system. The CoNotate approach helped searchers issue significantly more queries, and discover more terminology than standard web search. This work demonstrates how search can leverage user-generated content to help people get started when exploring complex, multi-faceted information spaces.

The Active Search Hypothesis

S. Palani Z. Ding S. MacNeil S. Dow
ACM CHIIR 2021

---

While research shows that web search plays a role throughout the creative process, less is known about about how people use web search to learn and frame their thinking about an open problem. People need web search to gather information about a problem area, but this can also influence the rest of the creative process. To understand how web search affects early-stage design, we collected and analyzed search log and self-report data from 34 students in a project-based design class. Participants reported struggling with scoping broad, ill-defined information goals into queries, learning domain-specific language, and assessing the usefulness of information. Analysis found that more active and diverse search behavior (i.e. issuing more frequent and diverse queries, and opening more webpages) related to more progress in early-stage design (i.e. gathering more facts, articulating more insights, and developing better problem frames). Based on these findings, we discuss implications for designing search tools to support peoples’ creative processes.

End-User Design in Canvas

S. MacNeil Z. Ding A. Boone A. Grubbs S. Dow
ACM CSCW 2021

---

Many organizations have adopted design processes that integrate community voices to discover the real problems that communities face through online discussion forums. However, we lack understanding about what information community members share, how that information is structured, and how social interactions affect the design processes at scale. This paper presents a mixed-methods analysis of Canvas, a learning management system, which enables users to contribute to the design of the platform by sharing and deliberating on problems and solutions in a discussion forum. We collected and analyzed 1412 ideas and 18,335 associated comments shared on the Canvas discussion forum. We found that 56.8% of posted ideas articulated a clear problem and only 40.4% included a comprehensive set of information (problem, solution, and contextual details) sufficient to make design decisions. Many of the ideas were duplicates (at least 11.6%); some members posted duplicates to garner attention for their ideas. Finally, we observed that 5% of posters contributed 19.9% of ideas. The distributed nature of design information, the presence of duplicate ideas, and gaming behaviors made it difficult for the community to get oriented to the discussion

D4SD: Design for San Diego

S. MacNeil B. McInnis J. Taylor S. Dow E. Hekler M. Morris
In Progress

---

Design for San Diego (D4SD) is a regional design challenge where San Diegans to come together to discuss and address San Diego's hardest problems through collaborative design activities. This project has been an opportunity to study how students, community members, and domain experts can work together to develop that leverage each of their unique skills and experiences.

IneqDetect: Visualizing Student Conversations

S. MacNeil K. Kiefer D. Takle B. Thompson C. Latulipe
ACM CompEd '19

---

IneqDetect records students’ group conversations that occur during peer learning activities and then visualizes these conversations temporally. In addition to showing the detected speech by group member, IneqDetect shows summaries of the total talk time per speaker and an overall measure of conversational equity. This work builds on prior research that has identified sociocultural inequities between students during peer learning activities. I hypothesized that presenting data to students about their social interactions would help students to communicate more equitably and lead to better team performance and cohesion.

• Keywords: machine learning, hardware, visualization

Educational Design Patterns

N. Dehbozorgi S. MacNeil ML Maher M. Dorodchi
FIE '18, Book Chapter '20

---

Design patterns, which originate in the field of architecture, provide a standardized format in which related problem and solution pairs can be represented. We propose object-based design patterns as a way to capture features that are domain specific and to reduce the amount of explanation necessary in the problem and solution text. We also contribute the idea of using concept maps as a way to represent related patterns.

• Keywords: csed, education, learning, design, design patterns

Nature Net: Designing with and for the Crowd

S. MacNeil S. Abdellahi ML Maher J. Kim MJ Mahzoon K. Grace
DCC '18

---

NatureNet is a citizen science project that, in addition to collecting biodiversity data, invites end-users to contribute design ideas to guide the its future design and development. This paper presents the NatureNet model of crowdsourcing design, then compares an analysis of the design process to published analyses of traditional face to face design processes. The protocol analysis approach is used to segment and code the design ideas submitted to NatureNet. We use the Function-Behavior-Structure ontology as a basis for comparison across the crowdsourced design data and design data collected in face-face sessions. The primary finding of this paper is that crowdsourced design results in a different distribution of cognitive effort when compared to traditional design processes.

• Keywords: Participatory design, Design Cognition

Dimensional Reasoning

S. MacNeil J. Okerlund C. Latulipe
ACM C&C '17, ACC ICCC '17

---

Design spaces are common in both research and design. They help us to understand the bounds of a problem and allow us to explore conceptual spaces. Despite their widespread use and importance, it is not yet understood how people create such spaces. Can people reason construct these spaces easily or is it difficult. Should creating these spaces start with examples or with dimensions? In this work we explore these questions and provide insights about dimensional reasoning and design spaces.
Work presented at CCW
Work presented at C&C in Singapore

• Keywords: dimensional reasoning, design space, sensemaking

Reflection Analytics

FIE '18, FIE '19

---

I've been studying how students reflect on their learning. I've been using a mix of qualitative and quantitative methods to understand how students reflect on their learning and whether it improves agency and awareness. I've been using natural language processing to make sense of the reflections at scale. Using the sentiment vectors from these reflections improved our ability to classify student's risk of obtaining a D, F, or W.

• Keywords: affect, reflection, machine learning, SVM

Swish Sleeve

---

For our senior design course, we created a sleeve that could track arm motion. We were expected to implement each of the components from scratch rather than using off-the-shelf components such as arduino. The main challenge we faced was avoiding gimbal lock as the joints rotated. We used Kalman filtering but as shown in the video there is some delay associated with the filtering and with the XBee connection.

• Keywords: Hardware, OpenGL, PCB, Microcontrollers, XBee, Kalman Filter

Nature Net

H. Killen T. Clegg J. Kim S. MacNeil J. Preece ML. Maher T. Yeh
In Progress

---

NatureNet is a citizen science project where its design is shared between a design team and end-users. Over a four-year period, we have worked with a variety of stakeholders and communities to develop a citizen science app developed by and for the people.

I am the lead developer, part of the design team, and an active member of the research team. I am especially interested in the relationship that exists between these end users and the design team and in understanding how this relationship affects the design process.

• Keywords: Citizen Science, Social Computing, Participatory design

Distributed Lightweight Teams

S. MacNeil C. Latulipe A. Yadav
ACM ICER '15, ACM SIGCSE '16

---

Active learning is important in computer science education, where students often don't have enough opportunities for social learning and development of soft skills. Flipped classrooms can provide social interaction through approaches such as lightweight teams [23], where students collaborate during class in low-stakes peer learning. These teams scaffold positive interdependence [17] by removing high-stakes assignments that heavily impact student's grades.
Given the proliferation of online courses, and MOOCs in particular, it is important to consider whether successful face-to-face pedagogical strategies can be reappropriated for distributed, online contexts. Specifically, we are interested in whether a low-stakes model could provide similar learning benefits when team members collaborate remotely.
This paper presents results from a study that analyzed the efficacy of low-stakes distributed teams. We examined whether low-stakes teams that communicate through Google Hangouts can provide educational benefits, in terms of both engagement and learning outcomes, compared to students who are learning via video in a co-located setting or individually. Results suggest that co-located teams have the highest learning gains, but there are no significant differences between distributed teams and individual work. We discuss implications of these results for practice and future research.

• Keywords: team-based learning, education, cmc modalities

BRIDGES

completed

---

NSF grant to remodel the data-structures course at UNCC. Deliver real-time socially-relevant data to students for use in their java-based assignments. The resulting data-structure is visualized in a web app.

• Keywords: visualization, computer science education, learning

Machine Learning Tutorials

---

Various algorithms posted on github and npm so that other developers could use them. Docco comments in the code provide instant documentation which available from the tutorials website.

• Keywords: javascript, docco, jekyll, machine learning

Surveyr

---

Surveyr is a platform which allows users to provide quizzes in json format. The quizzes can be timed, can include text, videos, and questions. The questions can be asked in 4 different formats (multiple choice, single choice, open-ended, and true/false). Multiple choice questions are shown in the teaser image above. Quizzes can have multiple sections. The server and client communicate to ensure that answers aren't lost when the page is refreshed and users can't gain additional time by refreshing. This reduces the chances that someone could cheat on the quiz. This work was used to support some of my human subjects research.

• Keywords: javascript, angular, koa

Genetic Algorithms

---

Genetic algoriths are algorithms which allow solutions to evolve throughout multiple generations. Candidate solutions are bred with other candidate solutions and probablistically mutated. After a number of generations the population can converge on the best fitting solution. This project provided an opportunity to use clojure and functional programming to solve real problems. This project uses Clojure's libraries hiccup and compojure for the server.

• Keywords: clojure, d3

UIST Website

---

As an opportunity to give back to the academic community, I decided to volunteer my time to serve as the Web and Social Media Chair at UIST 2015. The servers and the broad range of potential users provided me with an opportunity to avoid using my typical web stack and explore creating static mobile-friendly webpages using jade and the php slim framework..

• Keywords: jade, php slim

Social Fitness

completed

---

Social media fitness app where users can upload and share their runs with their friends. Run data could be from nike+ or polar. The idea was to use this website as a way to aggregate information about runners and to perform machine learning to compare performance between and across runner populations.

• Keywords: javascript, d3, wordpress, social media

Visualization Mosaics

S. MacNeil N. Elmqvist
Computer Graphics Forum '13

---

We present a new model for creating composite visualizations of multidimensional data sets using simple visual representations such as point charts, scatterplots and parallel coordinates as components. Each visual representation is contained in a tile, and the tiles are arranged in a mosaic of views using a space-filling slice-and-dice layout. Tiles can be created, resized, split or merged using a versatile set of interaction techniques, and the visual representation of individual tiles can also be dynamically changed to another representation. Because each tile is self-contained and independent, it can be implemented in any programming language, on any platform and using any visual representation. We also propose a formalism for expressing visualization mosaics. A Web-based implementation called MosaicJS supporting multidimensional visual exploration showcases the versatility of the concept and illustrates how it can be used to integrate visualization components provided by different toolkits.

• Keywords: javascript, google charts, d3, visualization

#Talk @Plants

---

This was a quick weekend project to find a way to water my plants during Thanksgiving and Christmas breaks. I have a pretty large garden in my apartment and on my balcony. This was a quick and dirty attempt to keep those gardens alive. It ended up being a blast.

• Keywords: RaspberryPi, Gardening, Nature