Shopify Overhauls Bug Bounty Payouts and Processes

Shopify has published its annual review of its Bug Bounty program, covering both 2019 results and a series of changes rolling out immediately in 2020. The company spent last year running private experiments and automating parts of its reporting workflow, and is now applying those lessons to its public program.

Bounties Paid Faster Than Ever

Three program changes take effect as of the publication of the review. First, Shopify will now pay bounties in full within seven days of a report being triaged. The company previously experimented with this approach in its private Shopify-Experiments program and saw enough success to bring it to the public program.

Second, the maximum payout is increasing to $50,000. Bounty ranges are being doubled for three vulnerability classes:

  • Arbitrary Code Execution: now $20,000–$50,000
  • SQL Injection: now $20,000–$40,000
  • Privilege Escalation to Shop Owner: now $10,000–$30,000

Third, researchers who file duplicate reports will now be added to the original report when it exists within HackerOne. Shopify says it is also exploring further ways to share information about internally known issues with hackers.

Learning From Peer Programs

Late in 2018, Shopify reached out to other bug bounty programs to exchange experiences. Those conversations pointed to two concrete improvements: better analytics and a private program for controlled experiments.

The HackerOne platform only reveals when reports are submitted, not who is testing what and how often. Other programs had solved this by leveraging provisioned accounts to track their funnel from invitation to testing. Shopify had long required hackers to register with a specific identifier (currently a @wearehackerone.com email address), a requirement originally meant for investigating suspicious activity, but realized the same data could show how frequently its applications were being tested. Combined with HackerOne API improvements and regular report data exports, this now supports activity reports and program trend tracking.

In mid-2019, Shopify launched Shopify-Experiments, a private program for high-signal, high-impact hackers with a proven track record on HackerOne or in Shopify's own program. The program ran controlled experiments around:

  • expanding scope to understand workload implications
  • paying full bounties after validation and triage
  • mandatory report disclosure and adding hackers to duplicate reports
  • allowing self-closing of valid false positives
  • increasing collaboration with Shopify third-party developers on app testing

One experiment directly shaped the public program. After measuring the workload of an expanded scope privately, Shopify added nearly all of its Shopify-developed apps to the public program's scope on September 11, 2019. That expansion has already produced notable reports, including one from researcher Vulnh0lic that found an OAuth misconfiguration in the Shopify Stocky app. Hackers can earn an invitation to the program with three resolved Shopify reports and an overall signal rating of 3.0 or higher.

Automating Report Handling

Shopify manages incoming reports through a dedicated Slack channel, and in January 2019 implemented HackerOne API calls to change report states, assign reports, post public and private comments, and suggest bounty amounts. The initial syntax proved hard to remember—changing a report state required hackerone change_state <report_id> <state>, while auto-responding used hackerone auto_respond <report_id> <state> <response_id>. The team introduced shorthands and emoji responses, so a state change can now be done with h1 change_state 123456 na, and common invalid reports get closed with a single emoji reaction that posts the appropriate response.

2019 Response Metrics Improve

Response times continued to drop across the board. The average time to first response went from 1 day and 9 hours in 2018 to 16 hours in 2019, with weekends included. Shopify attributes much of that gain to being able to close invalid reports on weekends via Slack. The average time to triage fell from 3 days and 6 hours to 2 days and 13 hours.

The average time to bounty from submission dropped from 14 days in 2018 to 7 days and 1 hour in 2019, and average resolution time from triage dropped from 48 days and 15 hours to 20 days and 3 hours. The company thanked 88 hackers in 2019, up slightly from 86 the prior year.

Disclosures rose substantially. Shopify disclosed 74 resolved bugs in 2019, up from 37 in 2018, continuing its practice of building a resource library for ethical hackers.

Bounty Payouts Down, Report Volume Steady

Despite faster payments, total bounty spending fell. Shopify paid $126,100 in 2019 compared with $296,400 in 2018, and the average bounty dropped from $2,052 to $1,139. The company attributes the decline largely to H1-514, the live hacking event it hosted in October 2018, which paid out over $130,000. The merger of the Shopify Scripts bounty program, which had a $100 minimum bounty, into the core program also pulled the average down. Bounties went to 107 reports in 2019 versus 182 in 2018.

Report volume was roughly steady: 1,379 reports in 2019 compared with 1,306 in 2018. Shopify continues to encourage hackers to participate through hackerone.com/shopify.