1# sass-spec 2 3### A cross-implementation [Sass][] test suite 4 5* [Running Specs](#running-specs) 6 * [Dart Sass](#dart-sass) 7 * [LibSass](#libsass) 8* [Spec Structure](#spec-structure) 9 * [HRX](#hrx) 10 * [Specifying Warnings](#specifying-warnings) 11 * [Implementation-Specific Expectations](#implementation-specific-expectations) 12 * [Options](#options) 13 * [`:todo`](#todo) 14 * [`:warning_todo`](#warning_todo) 15 * [`:ignore_for`](#ignore_for) 16* [Spec Style](#spec-style) 17* [Interactive Mode](#interactive-mode) 18 19[Sass]: https://sass-lang.com 20 21[](https://travis-ci.org/sass/sass-spec) 22 23`sass-spec` is the official Sass test suite. It's used by all major Sass 24implementations to ensure that they correctly implement the language. 25 26## Running Specs 27 28Before running specs, you'll need to install [Node.js] 14.14 or newer. Then, from the root of 29this repo, run `npm install`. 30 31[Node.j]: https://nodejs.org/en/download/ 32 33From there, it depends which implementation you're testing: 34 35### Dart Sass 36 37To run specs against [Dart Sass][], the reference implementation of Sass that's 38used for [the `sass` package][] on npm, you'll first need to install [Dart][]. 39Then run: 40 41[Dart Sass]: https://sass-lang.com/dart-sass 42[the `sass` package]: https://npmjs.com/package/sass 43[Dart]: https://www.dartlang.org/ 44 45```sh 46# If you already have a clone of the Dart Sass repo, you can use that instead. 47git clone https://github.com/sass/dart-sass 48(cd dart-sass; pub get) 49export DART_SASS_PATH=`pwd`/dart-sass 50 51npm run sass-spec -- --dart $DART_SASS_PATH 52``` 53 54### LibSass 55 56To run specs against [LibSass][], the C++ Sass implementation that's used for 57[Node Sass][] and other languages' Sass wrappers, you'll need to be able to 58[build LibSass][]. Once you have all the build dependencies: 59 60[LibSass]: https://sass-lang.com/libsass 61[Node Sass]: https://npmjs.com/package/node-sass 62[build LibSass]: https://github.com/sass/libsass/blob/master/docs/build.md 63 64```sh 65# If you already have a clone of the LibSass repo, you can use that instead. 66git clone https://github.com/sass/libsass 67(cd libsass; ./script/bootstrap; make sassc) 68export SASSC_PATH=`pwd`/libsass/sassc/bin/sassc 69 70npm run sass-spec -- --impl libsass -c $SASSC_PATH 71``` 72 73## Spec Structure 74 75Each spec is defined by a directory with an `input.scss` or `input.sass` file 76and either: 77 78* An `output.css` file, in which case the spec asserts that the Sass 79 implementation compiles the input to the output. These specs are known as 80 "success specs". 81* An `error` file, in which case the spec asserts that the Sass implementation 82 prints the error message to standard error and exits with a non-zero status 83 code when it compiles the input. These specs are known as "error specs". 84 85These files may also have [variants that are specific to individual 86implementations][]. 87 88[variants that are specific to individual implementations]: #implementation-specific-expectations 89 90The path to the spec serves as the spec's name, which should tersely describe 91what it's testing. Additional explanation, if necessary, is included in a silent 92comment in the input file. Specs may also contain additional files that are used 93by the input file, as well as various other features which are detailed below. 94 95### HRX 96 97Most specs are stored in [HRX files][], which are human-readable plain-text 98archives that define a virtual filesystem. This format makes it easy for code 99reviewers to see the context of specs they're reviewing. The spec runner treats 100each HRX file as a directory with the same name as the file, minus `.hrx`. For 101example: 102 103[HRX files]: https://github.com/google/hrx#human-readable-archive-hrx 104 105```hrx 106<===> input.scss 107ul { 108 margin-left: 1em; 109 li { 110 list-style-type: none; 111 } 112} 113 114<===> output.css 115ul { 116 margin-left: 1em; 117} 118ul li { 119 list-style-type: none; 120} 121``` 122 123HRX archives can also contain directories. This allows us to write multiple 124specs for the same feature in a single file rather than spreading them out 125across hundreds of separate tiny files. By convention, we include an HRX comment 126with 80 `=` characters between each spec to help keep them visually separate. 127For example: 128 129```hrx 130<===> unbracketed/input.scss 131a {b: is-bracketed(foo bar)} 132 133<===> unbracketed/output.scss 134a {b: false} 135 136<===> 137================================================================================ 138<===> bracketed/input.scss 139a {b: is-bracketed([foo bar])} 140 141<===> bracketed/output.scss 142a {b: true} 143``` 144 145Each HRX archive shouldn't be much longer than 500 lines. Once one gets too 146long, its subdirectories should be split out into separate archives beneath a 147physical directory. Conversely, if a given directory contains many small HRX 148archives, they should be merged together into one larger file. This helps ensure 149that the repo remains easy to navigate. 150 151The only specs that *aren't* written in HRX format are those that include 152invalid UTF-8 byte sequences. The HRX format is itself written in UTF-8, so it's 153unable to represent the files in these specs. 154 155### Specifying Warnings 156 157By default, Sass implementations are expected to emit nothing on standard error 158when executing a success spec. However, if a `warning` file is added to the spec 159directory, the spec will assert that the Sass implementation prints that warning 160message to standard error as well as compiling the output. This is used to test 161the behavior of the `@debug` and `@warn` rules, as well as various warnings 162(particularly deprecation warnings) emitted by the Sass implementation itself. 163 164Warnings can't be specified for error specs, since everything an implementation 165emits on standard error is considered part of the error message that's validated 166against `error`. 167 168### Implementation-Specific Expectations 169 170Sometimes different Sass implementations produce different but equally-valid CSS 171outputs or error messages for the same input. To accommodate this, 172implementation-specific output, error, and warning files may be created by 173adding `-dart-sass` or `-libsass` after the file's name (but before its 174extension, in the case of `output.css`). 175 176When a spec is running for an implementation with an implementations-specific 177expectation, the normal expectation is ignored completely in favor of the 178implementation-specific one. It's even possible (although rare) for one 179implementation to expect an input file to produce an error while another expects 180it to compile successfully. 181 182### Options 183 184Metadata for a spec and options for how it's run can be written in an 185`options.yml` file in the spec's directory. This file applies recursively to all 186specs within its directory, so it can be used to configure many specs at once. 187All options must begin with `:`. 188 189All options that are supported for new specs are listed below. A few additional 190legacy options exist that are no longer considered good style and will 191eventually be removed. 192 193#### `:todo` 194 195```yaml 196--- 197:todo: 198- sass/libsass#2827 199``` 200 201This option indicates implementations that should add support for a spec, but 202haven't done so yet. When running specs for a given implementation, all specs 203marked as `:todo` for that implementation are skipped by default. This ensures that 204the build remains green while clearly marking which specs are expected to pass 205eventually. 206 207Implementations can be (and should be) specified as shorthand GitHub issue 208references rather than plain names. This makes it easy to track whether the 209implementation has fixed the issue, and to see which specs correspond to which 210issue. When marking an issue as `:todo` for an implementation, please either 211find an existing issue to reference or file a new one. 212 213If the `--run-todo` flag is passed to `sass-spec.rb`, specs marked as `:todo` 214for the current implementation will be run, and their failures will be reported. 215 216If the `--probe-todo` flag is passed to `sass-spec.rb`, specs marked as `:todo` 217for the current implementation will be run, but a failure will be reported *only 218if those specs pass*. This is used to determine which specs need to have `:todo` 219removed once a feature has been implemented. This can be used in combination 220with [`--interactive`](#interactive-mode) to automatically remove `:todo`s for 221these specs. 222 223#### `:warning_todo` 224 225```yaml 226--- 227:warning_todo: 228- sass/libsass#2834 229``` 230 231This option works like [`:todo`](#todo), except instead of skipping the entire 232test for listed implementations it only skips validating [that spec's 233warnings](#specifying-warnings). The rest of the spec is run and verified as 234normal. This should not be used for error specs. 235 236#### `:ignore_for` 237 238```yaml 239--- 240:ignore_for: 241- libsass 242``` 243 244This option indicates implementations that are never expected to be compatible 245with a given spec. It's used for specs for old features that some but not all 246implementations have dropped support for. 247 248[which is deprecated]: http://sass.logdown.com/posts/7081811 249 250## Spec Style 251 252The specs in this repo accumulated haphazardly over the years from contributions 253from many different people, so there's not currently much by way of unified 254style or organization. However, all new specs should follow the [style 255guide](STYLE_GUIDE.md), and old specs should be migrated to be style-guide 256compliant whenever possible. 257 258## Interactive Mode 259 260If you pass `--interactive` to `npm run sass-spec`, it will run in interactive 261mode. In this mode, whenever a spec would fail, the spec runner stops and 262provides the user with a prompt that allows them to inspect the failure and 263determine how to handle it. This makes it easy to add [implementation-specific 264expectations][] or mark specs as [`:todo`](#todo). For example: 265 266``` 267In test case: spec/core_functions/color/hsla/four_args/alpha_percent 268Output does not match expectation. 269i. Show me the input. 270d. show diff. 271O. Update expected output and pass test. 272I. Migrate copy of test to pass on libsass. 273T. Mark spec as todo for libsass. 274G. Ignore test for libsass FOREVER. 275f. Mark as failed. 276X. Exit testing. 277``` 278 279[implementation-specific expectations]: #implementation-specific-expectations 280 281Any option can also be applied to all future occurences of that type of failure 282by adding `!` after it. For example, if you want to mark *all* failing specs as 283`:todo` for the current implementation you'd type `I!`. 284 285## Tests 286 287The unit tests for the spec runner are located in the `test/` directory. To run 288these unit tests, run: 289 290```sh 291npm run test 292``` 293
){kind=link}