1# Security Sheriff 2 3[TOC] 4 5## Important Links 6 7[Chrome Open Security Bugs dashboard, 8go/chrome-security-bugs](http://go/chrome-security-bugs). 9 10[Vulnerability Severity Guidelines](severity-guidelines.md). 11 12[Security Labels](security-labels.md). 13 14[Current Sheriffs](http://go/whos-the-sheriff). 15 16[Sheriff Handoff Log](http://go/chrome-security-sheriff-handoff). 17 18You might also like the [HOWTO: Be A Security Sheriff 19deck](https://docs.google.com/presentation/d/1eISJXxyv7dUCGUKk_rvUI9t9s2xb98QY4d_-dZSa7Wg/edit#slide=id.p). 20 21## What Is A Security Sheriff Or Marshal? 22 23A security sheriff (as well as a security marshal) is a member of a rotation 24that occurs in 1-week time slots, starting on Tuesdays and ending the following 25Monday. All sheriffs and marshals are Googlers and so some links on this page 26might not be externally accessible (or indeed locked down to just Chrome 27Security Googlers). 28 29[Here is the rotation 30schedule](https://docs.google.com/spreadsheets/d/10sLYZbi6QfLcXrhO-j5eSc82uc7NKnBz_o1pR9y8h7U/edit#gid=0). 31 32Sheriffs and marshals ensure that all incoming security issues are triaged 33quickly and correctly. We aim to have every bug triaged and assigned **within 34two business days** (preferably one). This does not include weekends, but please 35ensure you leave a clear queue before the weekend (i.e. on Friday, unless there 36is a holiday) and check first thing after the weekend (i.e. on Monday morning, 37unless there is a holiday). 38 39 40## When Am I The Security Sheriff Or Marshal? 41 42You should get a calendar invite. Please accept it to acknowledge. If you need 43to swap shifts, ask around for a volunteer and then just update the 44[rotation sheet](https://docs.google.com/spreadsheets/d/10sLYZbi6QfLcXrhO-j5eSc82uc7NKnBz_o1pR9y8h7U/edit#gid=0) 45and wait 10 minutes for the calendar invites to be updated. 46 47## I'm The Security Sheriff Or Marshal. What Do I Do? 48 49Each week has a sheriff and marshal, and during their rotation both have 50various important responsibilities: 51 52### Sheriff 53 54* Look at every incoming security bug report on the 55 [dashboard](http://go/chrome-security-bugs). Ensure each is accurately 56 triaged, and actively progressing towards getting fixed. 57* Don't forget to fully triage the low severity bugs. Once a bug is labeled with 58 `Security_Severity-Low `, it disappears from the first sheet and may slip 59 under your radar. 60* Keep the [Sheriff Handoff Log](http://go/chrome-security-sheriff-handoff) up 61 to date. 62* Shout for help if the incoming bug rate is too high ([suggested vocal 63 exercises](https://youtu.be/5y_SbnPx_cE?t=37s)). The first person to ask is 64 the marshal. 65* Make sure all **new bug reports** are triaged completely. That means no red 66 cells on the top of the dashboard. Double-check that OS flags are set 67 properly. For most of the bugs, typically more than one OS is affected, but 68 the dashboard will not highlight it in red. 69* Stay sharp, keep in shape ([hand-stand 70 pushups](https://www.youtube.com/watch?v=jZ1ZDlLImF8#t=50) are standard for 71 the sheriff), and remember you may be [called upon during 72 emergencies](https://www.youtube.com/watch?v=buHaKYL9Jhg). 73 74### Marshal 75 76* Ensure that all incoming queries to the 77 [security@chromium.org](https://groups.google.com/a/chromium.org/forum/#!forum/security) 78 and 79 [chrome-security@google.com](https://groups.google.com/a/google.com/forum/#!forum/chrome-security) 80 lists get a reply (by someone; not necessarily the marshal themselves). See 81 [go/chrome-security-emails](https://goto.google.com/chrome-security-emails) 82 for a dashboard. 83 * Note: external emails will always come in on security@chromium.org as 84 chrome-security@google.com is a Google-only list, but both need to be 85 triaged. 86 * When triaging an email to be handled off of the list, make sure to bcc: the 87 list that it arrived on, so that other people including future marshals can 88 see that it has been handled. 89* Change bugs status to **Fixed** for those that the developer forgets to close. 90 Make sure to read bug comments where developer might point out that it needs 91 more CLs, et c. Wait 24 hours before closing ClusterFuzz bugs, to give 92 ClusterFuzz a chance to close it automatically. 93 * [Starting point](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=Type%3D%22Bug-Security%22+%22Change-Id:%22) 94* Look at the open security bug reports and check that progress is occurring. 95 This does not apply to the **new bug reports** (these are handled by the 96 sheriff), but does apply to the issues on the *Low Severity Bugs* page of the 97 dashboard. The rule of thumb is *if there is any red cell on the dashboard, it 98 needs your attention*. 99* Stay sharp, keep in shape ([finger 100 exercises](https://youtu.be/20elMaVZ9lg?t=47s) are standard for the marshal), 101 and remember you may be called upon during emergencies. 102 103## Life Of A Security Bug 104 105Do as much as you can for the week to triage, shepherd, and wrap up open 106security bugs. What follows are the details of what that entails, but it 107practically means turning all the red cells in the dashboard to green. **If 108you're ever stuck or in doubt, ask for help on #chrome-security! or the 109[Chrome Security Chat](http://go/chrome-security-chat).** 110 111![alt text](apf-right-a-wrong.png "felt: a lot of Chrome vuln reports come from 112well-meaning people who clearly went out of their way to try to right a wrong. 113i like that.") 114 115[link](https://twitter.com/__apf__/status/728776130564526080) 116 117### Diagnose The Issue 118 119![alt text](life-of-a-security-issue.png "Life of a security issue.") 120 121* **If the report is invalid**, remove the **Restrict-View-SecurityTeam** label 122 and mark it **WontFix**. 123* **If the report is a duplicate**, mark it **Duplicate**. If the issue this is 124 a duplicate of is public, remove the **Restrict-View-SecurityTeam** label. 125* **If the report is primarily a privacy issue**, send it to the privacy team: 126 * Add the **Privacy** component so that it enters their triage queue. 127 * CC any security team members, including yourself, who may be interested in 128 the privacy issue. 129 * Change the **Restrict-View-SecurityTeam** label to 130 **Restrict-View-ChromePrivacy**. 131 * Note that security team members don't automatically have privacy bug 132 access, so this will probably make the issue inaccessible to you. 133* **If the report is asking about why something is or is not on the Safe 134 Browsing list:** 135 * Assign it to zbutler@, who will triage it for the Safe Browsing team. 136 * Remove the **Restrict-View-SecurityTeam** label and add the 137 **Restrict-View-Google** label. 138 * Change **Type-Bug-Security** label to **Type-Bug**. 139 * Add the **Security** component. 140 * See below for reporting URLs to SafeBrowsing. 141* **If the report is a potentially valid bug but is not a security 142 vulnerability:** 143 * remove the **Restrict-View-SecurityTeam** label. If necessary, add one of 144 the other **Restrict-View-?** labels: 145 * **Restrict-View-Google** if this is a crash report. 146 * **Restrict-View-EditIssue** if the bug can be abused (e.g. denial of 147 service) 148 * Change **Type-Bug-Security** label to **Type-Bug** (or whatever **Type-?** 149 is appropriate). 150 * Add appropriate component or CCs to ensure it does get triaged. 151 * Add the **Security** component or the **Team-Security-UX** label if the 152 security team should still track the issue (e.g. security features). 153* **If the report doesn't have enough information**, ask the reporter for more 154 information, add the **Needs-Feedback** label and wait for 24 hours for a 155 response. 156* The [security bug template](https://bugs.chromium.org/p/chromium/issues/entry?template=Security+Bug) 157 asks reporters to **attach files directly**, not in zip or other archives, and 158 not hosted at an external resource (e.g. Google Cloud Storage). If the report 159 mentions an online demo hosted somewhere, make sure the reporters attach the 160 source code for the demo as well. 161* **If the bug is a security bug, but is only applicable to Chrome OS**: 162 * The Chrome OS Security team now has their own sheriffing rotation. To get 163 bugs into their triage queue, just set OS to the single value of "Chrome". 164 No other steps or labels are needed. 165 * If you need to ping or ask about Chrome OS bug, [ask their current 166 sheriff](go/whos-the-chromeos-sheriff). 167* **If the report smells like a vulnerability, keep going.** 168 169### Verify And Label The Bug 170 171#### Step 1. Reproduce legitimate-sounding issues. 172 173Ideally, sheriffs should reproduce each bug before triaging, but being efficient 174is also important. It's fine to delegate reproducing bugs in the following 175cases: 176* A bug comes from an automated infrastructure (such as ClusterFuzz or Vomit). 177* A bug comes from a reporter with a solid track record of vulnerabilities (e.g. 178 prolific external researchers or Google Project Zero team). 179* A bug requires a particular device that you don't have available, or any other 180 environment which you don't have ready but a potential code owner would have. 181 182Mention explicitly in your comment that you didn't reproduce a bug before 183assigning it to someone else. 184 185A few components have their own triage processes or points of contact who can 186help. 187 188* V8 bugs can be assigned to the [V8 ClusterFuzz 189 Sheriff](https://rotation.googleplex.com/status?id=5714662985302016) for 190 triage. Note that V8 CHECK failure crashes can have security implications, so 191 don't triage it yourself and instead assign it to V8 ClusterFuzz Sheriff. They 192 can make an informed decision on whether it is a security vulnerability or not 193 and whether it is safe to strip the security tags (**Type=Bug-Security**, 194 **Restrict-View-SecurityTeam**). 195* Skia bugs can be assigned to hcm@chromium.org. Be careful while triaging 196 these! The place where we're crashing isn't necessarily the place where the 197 bug was introduced, so blame may be misleading. Skia fuzzing bugs can be 198 assigned to kjlubick@chromium.org, as Skia is heavily fuzzed on OSS-Fuzz and 199 some issues reported in Chromium are already known or even fixed upstream. 200* URL spoofing issues, especially related to RTL or IDNs? See 201 [go/url-spoofs](http://go/url-spoofs) for a guide to triaging these. 202 203 204Tips for reproducing bugs: 205 206* [https://clusterfuzz.com/upload-testcase](https://clusterfuzz.com/upload-testcase) 207 allows you to upload files to reproduce crashes on various platforms and will 208 identify revision ranges when the regression was introduced. If a test case 209 requires multiple files, they can be uploaded together in a zip or tar 210 archive. Useful fuzzers include:- 211 * repro.html [linux_asan_chrome_mp](https://clusterfuzz.com/upload-testcase?upload=true&job=linux_asan_chrome_mp) 212 or [windows_asan_chrome](https://clusterfuzz.com/upload-testcase?upload=true&job=windows_asan_chrome) 213 * repro.js [linux_asan_d8](https://clusterfuzz.com/upload-testcase?upload=true&job=linux_asan_d8) 214 * repro.pdf [libfuzzer_pdfium_asan / pdfium_fuzzer](https://clusterfuzz.com/upload-testcase?upload=true&job=libfuzzer_pdfium_asan&target=pdfium_fuzzer) 215 or [libfuzzer_pdfium_asan / pdfium_xfa_fuzzer](https://clusterfuzz.com/upload-testcase?upload=true&job=libfuzzer_pdfium_asan&target=pdfium_xfa_fuzzer) 216* When you can't just build from a specific branch locally, check out 217 [https://dev.chromium.org/getting-involved/dev-channel](https://dev.chromium.org/getting-involved/dev-channel) 218 or 219 [https://commondatastorage.googleapis.com/chromium-browser-asan/index.html](https://commondatastorage.googleapis.com/chromium-browser-asan/index.html) 220 for the latest release of a specific version. 221* There are many tools available to help you reproduce various memory issues 222 reliably. If you aren't already familiar with them, check out 223 [AddressSanitizer](https://www.chromium.org/developers/testing/addresssanitizer), 224 [MemorySanitizer](https://www.chromium.org/developers/testing/memorysanitizer), 225 [ThreadSanitizer](https://www.chromium.org/developers/testing/threadsanitizer-tsan-v2), 226 and 227 [UndefinedBehaviorSanitizer](https://www.chromium.org/developers/testing/undefinedbehaviorsanitizer). 228* If you run into issues with a reproducible ClusterFuzz test case (like 229 missing symbols, or if anything else seems off), try uploading the test case 230 again using a different job type with a more mature tool (e.g. ASan on Linux). 231 It may give more complete information. 232 233#### Step 2. Assess the severity. 234 235[See the severity guidelines](severity-guidelines.md). If it's a critical 236vulnerability, act quick! We aim to get users patched in < 30 days. Remember 237that if something requires an unusual configuration or complicated user 238interaction, the severity rating should be lowered. 239 240Bug chains are typically composed of several individual security bugs and 241should be split into a new bug for each potential fix required, so this allows 242each team to work on fixing their part of the chain. In cases like this, leave 243the main bug as the severity/priority of the full chain, and mark child bugs as 244being blockers of the parent bug each with their own separate severity. Each 245child bug can have its own priority. Examples of this in action are [issue 246352369](https://crbug.com/352369) and [issue 453937](https://crbug.com/453937). 247 248Even after initial triage, re-assess the severity while you're looking at a 249security bug update: does it have new information in the bug that could change 250the assessment? Be especially on the lookout for Highs that are really 251Criticals, and Lows that are really Mediums (make sure to account for process 252types and sandbox boundaries). 253 254For V8 issues, it can be hard to identify the correct security severity. If 255you're not sure, please take your best guess, and add the 256`Security_Needs_Attention-Severity` label alongside the regular 257`Security_Severity-*` label. If you do this, the V8 team will check the 258severity later and change it if necessary. 259 260#### Step 3. [Label, label, label](security-labels.md). 261 262Much of Chrome's development and release process depends on bugs having the 263right labels and components. Labels and components are vitally important for 264our metrics, the visibility of bugs, and tracking our progress over time. 265 266Labels to **double-check** (that should already be there if the bug was filed 267using the Security template): 268 269* **Restrict-View-SecurityTeam** 270* **Type-Bug-Security** 271* **If the reporter wants to remain anonymous or if the bug description or 272 comments contain PII**, add **Restrict-View-SecurityEmbargo**. 273 274Generally, see [the Security Labels document](security-labels.md). 275 276**Ensure the comment adequately explains any status changes.** Severity, 277 milestone, and priority assignment generally require explanatory text. 278 279* Report suspected malicious URLs to SafeBrowsing: 280 * Public URL: 281 [https://support.google.com/websearch/contact/safe_browsing](https://support.google.com/websearch/contact/safe_browsing). 282 * Googlers: see instructions at [go/safebrowsing-escalation](https://goto.google.com/safebrowsing-escalation) 283 * Report suspected malicious file attachments to SafeBrowsing and VirusTotal. 284* Make sure the report is properly forwarded when the vulnerability is in an 285 upstream project, the OS, or some other dependency. 286* For vulnerabilities in services Chrome uses (e.g. Omaha, Chrome Web Store, 287 SafeBrowsing), make sure the affected team is informed and has access to the 288 necessary bugs. 289 290##### Labeling For Chrome On iOS 291 292* Reproduce using iOS device or desktop Safari. 293* Assign severity, impact, milestone, and component labels. 294* Label **ExternalDependency**. 295* Label **Hotlist-WebKit**. This label is monitored by Apple friends. 296* File a security bug at [bugs.webkit.org](https://bugs.webkit.org), and CC 297 chrome-ios-security-bugs@google.com. This alias is monitored by the iOS Chrome 298 team so they can be notified when the WebKit bug is fixed. 299* Note the WebKit bug ID in the crbug report. 300 301### Find An Owner To Fix The Bug 302 303That owner can be you! Otherwise, this is one of the more grey areas of 304sheriffing. With experience, you'll figure out good goto people for certain 305areas. Until then, here are some tips. 306 307**Determine the correct component before continuing.** It's not enough on its 308own, but it's a good starting point. Many components will automatically apply 309some CCs who may be able to help you out. If it's a crash bug, see if 310ClusterFuzz is able to provide one (will appear in the same card as the culprit 311CL). You can also use `git hyper-blame` and check OWNERS files to see who might 312own the relevant code. 313 314**For crashes, check to see if ClusterFuzz provides a culprit CL.** Before you 315assign a bug based on this, do a quick sanity check to ensure the CL could have 316caused the bug. If the result seems wrong, apply the Test-Predator-Wrong label 317to the bug and keep going. 318 319If you're able to narrow this to a specific regression range, usually from 320ClusterFuzz for crash bugs, do a quick pass over the git log to see if any CLs 321stand out. If you aren't sure, don't be afraid to add CCs to the bug and ask! 322 323At this point, you'll probably need to dive in and attempt to root cause the 324bug, which is another complicated grey area that you'll figure out with 325experience. Try not to spend too much time on this for any given bug, as some 326cases will simply be too difficult without a deep understanding of certain 327portions of the codebase. 328 329* If you can narrow the bug to a specific file or block of code, or if something 330 stands out as suspicious, try to assign an owner based on `git hyper-blame` or 331 add some CCs based on OWNERS files. 332* If not, consider searching in the issue tracker for people that fixed similar 333 bugs or bugs in similar areas of the code base, such as issues with the same 334 components, recently. For example, let's say you were trying to figure out a 335 good person to assign a Content>Fonts issue. Look for `status=fixed,verified` 336 and query by when the issues were closed after (i.e. w/ in the last 30 days == 337 `closed>today-30`). 338 339Got stuck? Ask #chrome-security or someone from 340[go/chrome-security-sheriff-mentors](https://goto.google.com/chrome-security-sheriff-mentors) 341for help! That's why we're here. Don't be afraid to do this! 342 343Make sure that the person you assign to handle a bug is not OOO. And, generally, 344explicitly CC more than one person on the bug, if possible, and preferably 345people from more than one geographic region. (See the OWNERS file(s) that 346affect(s) the relevant area of code.) 347 348**Sometimes, finding an owner isn't enough to ensure that a bug will get 349fixed.** Check the stale bug list on the security dashboard and try resolve 350some of the problems that might be blocking these issues. If you get in touch 351with a bug owner off of the issue tracker, be sure to have them update the bug 352so that future sheriffs are aware of the status. 353 354> Q: Why isn’t setting the component alone good enough? 355> 356> A: CCs are critical because just assigning to a component is ineffective 357> because the component’s team cannot see the issues unless they have the 358> Security View permissions. 359 360### Using The Permission API Kill Switch 361 362If you find a vulnerability in a Permission API and need to use the Global 363Permissions Kill Switch, then follow [the 364instructions](https://docs.google.com/document/d/17JeYt3c1GgghYoxy4NKJnlxrteAX8F4x-MAzTeXqP4U) 365 366### Wrap Up The Fixed Issue 367 3681. For any **Security_Severity-**{**Critical**, **High**, **Medium**} bugs that 369 **Security_Impact-**{**Beta**, **Stable**}, add **Merge-Requested** so that 370 the fix gets merged into the next release. Exercise discretion according to 371 security severity and risk associated with the bug fix; you can ask the patch 372 author whether any risky code paths are affected. The actual merging and 373 drafting of release notes is taken care of by the [security release management 374 role](https://www.chromium.org/Home/chromium-security/security-release-management). 3751. Chrome's [Vulnerability Rewards 376 Program](https://www.google.com/about/appsecurity/chrome-rewards/index.html) 377 TPM adds the **reward-topanel** label by mass modification, but **do** label 378 any bugs reported by a @chromium.org email that should be rewarded (e.g. "I'm 379 filing this on behalf of" or the like). 380 381## End Of Rotation 382 383Update the [Sheriff Handoff Log](http://go/chrome-security-sheriff-handoff). 384