Bug Report Automation

AI integration reduces duplicate bug reports at Google
Role
Product Designer
Research Lead
Skills
AI Design
Mixed-Method Research
Duration
Jan - Jun 2024
(6 months)
Collaboration
Product Designers
Software Engineers
Tools Used
Figma
Miro + Google Sheets
React.js

Overview

The Problem

Google's Android Security Vulnerability Rewards Program (VRP) allows people to report bugs (bug reporters) via a form for a monetary reward. However, with 18K bug reporters, Android Security Analysts struggle to identify and remediate novel, high-impact vulnerabilities as over 90% of submissions are duplicates and lack security significance. The sheer volume of low-value reports threatens to let dangerous vulnerabilities slip through the cracks, putting user safety and the reputation of Google’s Android ecosystem at serious risk.

Leadership set a clear project objective: leverage AI to reduce duplicate reports by 30%, establishing this as a key success benchmark.

The Solution

The VRP form provides AI-driven content similarity metrics to bug reporters in real-time. Analyzing user input and existing bug reports, the form provides field similarity breakdowns, side-by-side report comparisons, and links to similar raw reports, allowing bug reporters to quickly refine or withdraw their report. This integration saves bug reporters and Google Security Analysts time and effort to focus on novel threats.
‍
Click here to view our landing page and demo.

Awards & Media

Impact

I led AI deduplication efficacy testing against our accuracy, speed, and reliability benchmarks. Our solution exceeded Google's expectations, delivering a 40% reduction in duplicate reports with significantly faster processing times than manual review, and achieving fair agreement with security analysts.
+40%
ACCURACY
Deduplication Accuracy Goal: 30%
+200%
SPEED
Percent Difference Goal: 50%
.34
AGREEMENT
Inter-Rater Reliability Goal: 0.20
I created a demo video showcasing our solution, which was presented to Google engineers and researchers. They were highly impressed and decided to continue the project after handoff.
My team also won the Innovation Award and was runner-up for the Research Award out of 175 teams alongside being featured in a University of Washington article.
OverviewContextHighlightResearchIdeationDesignEvaluationFinal DesignReflection

Context

Project Framing

AI Security Integration

I collaborated with Google’s Android Security team through the University of Washington Information School, receiving guidance on project direction, technical resources, and stakeholder feedback.

Leadership positioned our project as highly exploratory, focusing on how AI could integrate into their security workflows, and our team was tasked with delivering an end-to-end solution demonstrating this vision.
The intersection of AI and cybersecurity is both highly specialized and emergent, carrying significant stakes given the scale of the Android product ecosystem—key factors to consider as the project moved forward.

Highlight

Final Design

Before

The original VRP form guided bug reporters to provide all necessary technical details. However, with static input fields and no informative progress feedback, reporters received no indication if their submission might be a duplicate—even as they meticulously completed the form.
By introducing real-time similarity notifications and metrics, access to exportable similar reports, and highlighted field-level report comparisons, the system helps reporters review, refine, or withdraw their reports with confidence.

See Final Design for more details.

Research

Literature Review

I supported a 25-document literature review on AI applications in cybersecurity, analyzing peer-reviewed papers, conference proceedings, and research publications to understand the problem space further. Several of my key findings are outlined below:

01 - System requires large volumes of high-quality, context-specific data.  
02 - Data must be actively updated & capable of capturing nuanced patterns.
03 - Meaningful, explainable, & transparent AI integration into security workflows.

We knew our AI solution must be trained on high-quality Android bug reports and have a open, collaborative dynamic with users.

Market Research

I conducted a competitive analysis and matrixing of 25 current AI-driven cybersecurity tools for vulnerability detection and triage, identifying product similarities, human-AI interaction types, and market gaps. Several of my key findings are outlined below:

‍01 - Most tools feature automated detection, integrations, and data organization.
02 - Complex UIs and steep learning curves limit usability.
03 - Poor balance between human and AI roles leads to less effective workflows.

These insights show the need for intuitive automations that facilitate collaborative human-AI interaction.

Stakeholder Discussions

I led four stakeholder discussions with the Android Security team to deepen our understanding of their workflows, pain points, and priorities. Building on our initial research, these conversations revealed greater clarity into the complexity of their review ecosystem.

I synthesized the findings using affinity diagramming, surfacing key themes:

01 - Reliance on specialized manual bug report review.
02 - Fragmented knowledge across a multi-step evaluation process.
03 - Report quality is inconsistent and thus slips through current automations.

Ideation

Idea Prioritization

Idea Scoping

Ideation Synthesis

Divergent Ideation

Leveraging our research insights, I facilitated an ideation workshop that began with individual brainstorming to encourage diverse thinking before converging as a team.

My concepts focused on enhancing the existing VRP form through clearer guidance and progress tracking, supported by AI-driven insights that help reporters understand the novelty and quality of their submissions.
Several of my ideas are outlined below:
Content Flagging
Flag unacceptable form content
 Novelty
QUALITY
AI Gap Filling
AI recommends missing content
 Novelty
QUALTIY
Collaborative Reporting
Hunters build reports together
 Novelty
QUALITY
Report Assistant
Conversational focus AI assistant
 Novelty
QUALITY
Form Walkthrough
Structured report onboarding
QUALITY
Dynamic Incentive
Adjust reward based on quality
QUALITY
Dynamic Templates
Report specific templates
QUALITY
Progress Tracking
Report progress & AI use metrics
QUALITY
Playbook Website
Provide reporting options & outcomes
 Novelty
Quality
Attribute Summary
Summarize missing report attributes
Quality
Automated Deduplication
Detect duplicate content in real time
 Novelty
Incentivized Structuring
Content based reward progression
 Novelty
Quality
Report Chatbot
Answers report content questions
 Novelty
Quality
Report Alerts
Alert if content violates guidelines
 Novelty
Quality
I led a convergent ideation workshop with my team to provide focus to our 34 ideas. I guided the team to identify unique solutions, combine adjacent pairs, and unify similar ideas together. Parent and children ideas were visualized accordingly.

I noticed two possible solution mediums emerge: form and portal. Form features built off the existing report form structure. A portal could offer a larger range of reporting support inclusive of form specific features.

All these ideas and potential mediums raised many design, technical, and impact questions. We needed to find focus and prioritize to hone our project vision.
I created an impact-feasibility matrix to visually prioritize what ideas the team could pursue. I led a co-working session with our engineers to scope technical feasibility while I provided research and design insights.

This process allowed the team to dynamically prioritize ideas, identify constraints, and ensure our next steps provided high impact to Bug Reporters and Android Security stakeholders.
🥇Must Have
Automated Duplicate Check
Automatically detect content similarity and notify bug reporters as they fill out their report
NOVELTY
Automated Flagging & Guidance
AI provides real-time guidance to structure input fields while flagging low-quality content
Quality
Automated Flagging & Guidance
AI provides real-time guidance to structure input fields while flagging low-quality content
Quality
Automated Flagging & Guidance
AI provides real-time guidance to structure input fields while flagging low-quality content
Quality
🥈Should Have
Explicit Form Instructions
Provide formatting guidance for each input filed with specific content instructions
Quality
Form Walkthrough & Examples
Provide a guided walkthrough of each report field with validated high-quality examples
Quality
Placeholder Templates & Content
Include bug-specific report templates with placeholder content for a robust launch point
Quality
Automated Flagging & Guidance
AI provides real-time guidance to structure input fields while flagging low-quality content
Quality
🥉Nice To Have
Collaborative Bug Reporting
Connect reporters together to build a bug report in unison and share potential rewards
NOVELTY
Quality
Report Progress Tracking
Track report progress and AI involvement metrics to validate report efficacy
NOVELTY
Quality
Automated Report Querying
Automatically request missing content from reporters via a secondary submission
Quality
Confirm Experience Level
Gauge reporters experience level to boost signal on potential quality variance
Quality
Through iterative discussions with Google, I determined a strategic focus on automating duplicate bug report checks for reporters based on time constraints and business priority.

This direction addresses critical pain points, including the lack of visibility into duplicate status and the resulting surge in redundant bug report submissions. The remaining features were documented for future investigation.

Design

Lo-Fi Wireframing

Mid-Fi Wireframing

I worked with another designer to build out a low-fidelity flow of the current reporting form, adding in brainstormed elements to aid in duplicate review prior to submission.

I used various shades of grey shapes to represent UI elements to ensure focus remained on the conceptual flow and architecture. Using Figma notes and several coworking sessions, we clarified our ideas and established our design path, allowing us to advance our wireframes to a higher fidelity build.
With a more define set of design requirements now in line, my design teammate and I were primed to create a more flushed out iteration of the UI.

However, as I began this process I realized we were beocming prematurly focused on visual perfectionsim with spacing, color, type, etc. that we began to lose site of our user-focus and design specifications.

After a dicussion with my teammate, we advocated for focusing on mid-fidelity wireframing, allowing us to achieve a balance between detail and conceptual flow.

Evaluation

Expert Review

Usability Testing

AI Benchmarking

I led speed, accuracy, and reliability benchmark assessments for our AI model, harnessing quantitative analysis methods applied to model data derived from AI review of previously triaged reports.

I delivered three key benchmarks, all of which surpassed our projects defined success indicators.
My team and I presented our design to Google engineers, researchers, and other cybersecurity professionals to gain feedback on technical feasibility, process alignment, business applicability, and user experience.

A key takeaway from Google was the need to define focus on what the AI model utilized to determine bug report similarity. Based on our discussion with the Android Security stakeholders and bug report lifecycle documentation review, the team and I identified three critical attributes to hone our model's focus.

I added a color-coded key and respective content highlights to aid visual clarity and support focused attribute comparison.
I facilitated usability test sessions with four cybersecurity professionals, including a founding engineer for Amazon's bug reporting portal. Sessions with Bug Reporters were out of scope due to recruitment feasibility.
‍
My design teammate and I created a higher-fidelity, interactive prototype based on this evaluative data, particularly creating additional flexibility to export reports in select file types for independent analysis.
‍
Landing Page
Phase Selection
Assessment Introduction
ECD Language Guide
Phase One: See The System
See The System Question One
See The System Question Three
See The System Question Four
See The System Question Two
Assessment Results
Sidebar Metrics
Report and field content similarity are displayed as percentages based on AI analysis within the sidebar. Reporters receive a toast notification if their report’s similarity exceeds predefined thresholds. This feature provides visible, focused metrics on a report's likelihood of being a duplicate.
Support Documentation
A one-page support document accessed via the form sidebar provides user's with an overview of key similarity metrics, the four similarity thresholds, important form sections, and recommended next steps if a report is likely a duplicate to ensure a confident, informed experience.
Similar Reports Table
Users can toggle a table of similar historical reports using the 'View Reports' link in the form sidebar. The table displays an overall similarity metric between the current report and past reports, as well as field-level similarity scores within each report row. Every entry includes important metadata, helping users quickly assess potential duplicates and relevant details.
Report Comparison
Selecting a report row reveals a detailed field-to-field similarity breakdown. Each field displays a similarity score and highlights matching content across three key threat attributes. Reporters can view and export similar reports for independent analysis. This feature enables guided yet flexible investigation into potential duplicate reports.

Final Design

With evaluation complete, my design teammate and I arrived at a final high-fidelity prototype ready for implementation. I supported our engineers with front-end development so they could flush out back-end functionality. This design was built and securely handed off to Google.

Reflection

Project Handoff

🎉 Project handoff was a success — by establishing early documentation supported by intentional presentations and discussions, the impact and value of this work was clearly articulated and is now serving as a springboard for security tooling development at Google.

Recruitment Blockers

Due to recruitment and time constraints, I was unable to conduct usability testing with bug reporters as originally planned. Instead, I pivoted to running feedback sessions with security and AI experts, ensuring valuable insights while adapting to project limitations.

💡What I’d do differently: Allocate more time for participant recruitment and explore partnerships with relevant teams earlier in the process to enable direct testing with bug reporters.

Bug Report Red Tape

Facing access restrictions to Android bug reports to train our model, we switched to open-source Chromium reports to maintain project momentum. We documented this adjustment and a highly tranferrable process to Android bug reports, ensuring continued progress despite external blockers.

💡What I’d do differently: Proactively engage with stakeholders to anticipate access issues and seek alternative data sources sooner, minimizing disruption and maintaining a smoother workflow.

Collaborative Exploration

Building the foundation for AI collaboration in bug reporting pushed me to establish new design approaches while branching off of core UX patterns. Tight teamwork, open communication, and genuine team bonding fueled our ability to solve complex challenges—making the process not only innovative, but genuinely energizing and fun.
Landing Page
Phase Selection
Assessment Introduction
ECD Language Guide
Phase One: See The System
See The System Question One
See The System Question Three
See The System Question Four
See The System Question Two
Assessment Results
Landing Page
Phase Selection
Assessment Introduction
ECD Language Guide
Phase One: See The System
See The System Question One
See The System Question Three
See The System Question Four
See The System Question Two
Assessment Results
Wave
EquityUp

Thanks for stopping by, feel free to explore my other projects!

2025 © SAMI FOELL
LINKEDINEMAIL