Critique Me: Implications for Facilitating Online Feedback Exchange
A mixed-method study of how creators publicly request feedback in an online critique community
Published & presented at ACM CSCW 2020 · October 17–21, 2020
Many creative workers post their work in online communities, such as Reddit and Dribbble, where members respond with feedback. Prior research focused on prompting feedback providers and helping creators make sense of the feedback they receive. Our study instead aimed to understand and support feedback seekers in effectively requesting feedback to begin with — empirically studying how creators publicly request feedback in an online critique community and how their strategies affect the responses they get.
Background
The UC San Diego ProtoLab, under the UCSD Design Lab, investigates the foundations of collective intelligence, creativity, feedback exchange, and decision making using human-centered design, data science, qualitative methods, and system prototyping. During the summer of 2019, I spent 4 months in ProtoLab under Professor Steven Dow, conducting research on feedback seekers and co-producing a research report on the findings.
Research timeline
Research questions
We focused on the r/design_critiques subreddit, an active community dedicated to feedback exchange across a range of design domains, chosen because it lets us observe a range of feedback interactions across experience levels and design genres, without being membership-based or professional.
Study 1: User interviews
We recruited 12 active users of r/design_critiques through purposeful sampling — all had both provided and requested feedback before. Our protocol asked participants to reflect on their own request strategies, recall experiences providing feedback, and critique recent community posts.
A sample interview transcript
Building affinity diagrams
Using the transcribed interviews, we collaboratively and iteratively built affinity diagrams, surfacing common themes among feedback seekers and providers.
This surfaced four key tensions seekers struggle with: how to present design context, whether to include personal details, whether to request specific or general feedback, and whether to explicitly request expert input.
Study 2: Qualitative coding & computational analysis
We selected 900 post requests from a corpus of 24,867 posts (150 randomly sampled per year, across six years), removing posts unrelated to design feedback or with invalid links — leaving 879 posts and 3,632 corresponding feedback comments as our dataset.
We iteratively developed a 7-strategy coding scheme, reaching Cohen's Kappa ≥ 85%
Regression analysis & computational methods
We built multiple regression models to investigate how features of feedback requests influence the resulting feedback, then calculated two text-based measures — actionability and justification — using natural language processing and semantic analysis.
Results
A majority of requests (89.0%) present design context, but rarely include reasoning about the design process (13.4%). While 87.7% explicitly prompted for feedback, more than half used only general prompts without specific scaffolds.
Bolded = independent variables that led to statistical significance and at least a 10% change in the dependent variables
Despite their effectiveness, these strategies were used by only a small portion of the community (6.1%, 21.8%, and 11.2% respectively).
Design implications
- Support how seekers compose requests — hints and prompts, instead of a free text box, offering key principles as people compose.
- Support how seekers reflect on their designs — an assistant that provides step-wise reflective instruction and examples.
- Support how seekers generate and present variations — an interface to upload, organize, and explain multiple versions, or AI-powered variant generation.
- Initiate private exchanges between novices and experts — a private channel with a "matching" system based on motivational profiles.
Personal takeaways
This project let me experience the full, iterative process of turning research questions into a published paper — not just the user research process, but how to write in the form of storytelling while still conveying data and impact.
Interviewing people of different ages, paths, and careers, all pursuing something within design, taught me there isn't really a "set timeline" or "correct path" for anything. Being introduced to so many perspectives ignited my love for understanding the 'user' in user experience.
I'm so grateful for the experience — from working with an inspirational team, to experiencing academic research firsthand, to living in San Diego for a summer, to attending my first HCI conference and co-authoring my first paper.