Selected Publications (see more)

Generating Code Explanations using Large Language Models

Good explanations are essential to efficiently learning introductory programming concepts. To provide high-quality explanations at scale, numerous systems automate the process by tracing the execution of code, defining terms, giving hints, and providing error-specific feedback. However, these approaches often require manual effort to configure and only explain a single aspect of a given code segment. Large language models (LLMs) are also changing how students interact with code. For example, Github’s Copilot can generate code for programmers, leading researchers to raise concerns about cheating. Instead, our work focuses on LLMs’ potential to support learning by explaining numerous aspects of a given code snippet. This poster features a systematic analysis of the diverse natural language explanations that GPT-3 can generate automatically for a given code snippet. We present a subset of three use cases from our evolving design space of AI Explanations of Code.

Curri: Visualizating the Curriculum

Curriculum design can be more of an art than science. Decisions are seldom made based on data. Instead, they are often made based on the expertise of academic advisors and curriculum designers. Curri combines student data with information from the course catalog to visualize when, on average, students take specific courses. This allowed curriculum designers to identify bottlenecks (areas where many courses are taken at the same time) and inversions (areas where a class is taken before the pre-requisites) which can delay graduation. Curriculum designers can adjust to ensure that the courses are more evenly spread out temporally. Curri was designed for curriculum designers and advisors, but might also provide students with data about their own trajectories.

• Keywords: machine learning, visualization

ProbMap: Automatically constructed design galleries

Making sense of large unstructured problem spaces is cognitively demanding. Structure can help, but adding structure to a problem space also takes significant effort. ProbMap is a novel application for automatically constructing a design gallery from unstructured text input. Given a list of problem statements, ProbMap extracts and semantically groups the stakeholders to construct hierarchical search facets which enables designers to more efficiently navigate the problem statements. We contribute a novel feature extraction algorithm using natural language processing and a technique for automatically constructing a design gallery. These stakeholders are grouped semantically by clustering stakeholders with higher pairwise similarity together. Preliminary trials show that these techniques, which mirror traditional design activities like stakeholder identification and affinity mapping, provide an initial structure to a large unstructured problem space. This resulted in similar features that would be extracted by humans and sensible clusters.

ProbLib: Helping Novices to Frame Better Problems

Problem framing---the process of defining a problem---has been described by many researchers and designers as the crux of the design process. However, novice designers struggle with problem framing. To better understand this process and the potential for scaffolding, we conducted two studies. In the first study, we analyzed 41 problem statements from an introductory design course and found that novices often omit key information, like the primary stakeholder or the obstacles they face. To get novices to reflect on and include necessary design information, we created a tool called ProbLib that cues novices to explicitly reflect on aspects of the problem such as the stakeholders. To evaluate this approach, we conducted a between-subjects study (N=73) to compare ProbLib with an unstructured open text form. We found that participants using ProbLib wrote higher quality statements, included more information, and were more confident about specifying design needs. We observed creative behaviors such as brainstorming and analogical reasoning.

CoNotate: Adaptive Query Suggestions Support Learning

When exploring a new domain through web search, people often struggle to articulate queries because they lack domain-specific language and well-defined informational goals. Perhaps search tools rely too much on the query to understand what a searcher wants. Towards expanding this contextual understanding of a user during exploratory search, we introduce a novel system, CoNotate, which offers query suggestions based on analyzing the searcher’s notes and previous searches for patterns and gaps in information. To evaluate this approach, we conducted a within-subjects study where participants (n=38) conducted exploratory searches using a baseline system (standard web search) and the CoNotate system. The CoNotate approach helped searchers issue significantly more queries, and discover more terminology than standard web search. This work demonstrates how search can leverage user-generated content to help people get started when exploring complex, multi-faceted information spaces.

End-User Design in Canvas

Many organizations have adopted design processes that integrate community voices to discover the real problems that communities face through online discussion forums. However, we lack understanding about what information community members share, how that information is structured, and how social interactions affect the design processes at scale. This paper presents a mixed-methods analysis of Canvas, a learning management system, which enables users to contribute to the design of the platform by sharing and deliberating on problems and solutions in a discussion forum. We collected and analyzed 1412 ideas and 18,335 associated comments shared on the Canvas discussion forum. We found that 56.8% of posted ideas articulated a clear problem and only 40.4% included a comprehensive set of information (problem, solution, and contextual details) sufficient to make design decisions. Many of the ideas were duplicates (at least 11.6%); some members posted duplicates to garner attention for their ideas. Finally, we observed that 5% of posters contributed 19.9% of ideas. The distributed nature of design information, the presence of duplicate ideas, and gaming behaviors made it difficult for the community to get oriented to the discussion

IneqDetect: Visualizing Student Conversations

IneqDetect records students’ group conversations that occur during peer learning activities and then visualizes these conversations temporally. In addition to showing the detected speech by group member, IneqDetect shows summaries of the total talk time per speaker and an overall measure of conversational equity. This work builds on prior research that has identified sociocultural inequities between students during peer learning activities. I hypothesized that presenting data to students about their social interactions would help students to communicate more equitably and lead to better team performance and cohesion.

• Keywords: machine learning, hardware, visualization

Educational Design Patterns

Design patterns, which originate in the field of architecture, provide a standardized format in which related problem and solution pairs can be represented. We propose object-based design patterns as a way to capture features that are domain specific and to reduce the amount of explanation necessary in the problem and solution text. We also contribute the idea of using concept maps as a way to represent related patterns.

• Keywords: csed, education, learning, design, design patterns

Dimensional Reasoning

Design spaces are common in both research and design. They help us to understand the bounds of a problem and allow us to explore conceptual spaces. Despite their widespread use and importance, it is not yet understood how people create such spaces. Can people reason construct these spaces easily or is it difficult. Should creating these spaces start with examples or with dimensions? In this work we explore these questions and provide insights about dimensional reasoning and design spaces.
Work presented at CCW
Work presented at C&C in Singapore

• Keywords: dimensional reasoning, design space, sensemaking