1--- 2title: "Overview of cloud storage systems" 3description: "Overview of cloud storage systems" 4type: page 5--- 6 7# Overview of cloud storage systems # 8 9Each cloud storage system is slightly different. Rclone attempts to 10provide a unified interface to them, but some underlying differences 11show through. 12 13## Features ## 14 15Here is an overview of the major features of each cloud storage system. 16 17| Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | 18| ---------------------------- |:-----------:|:-------:|:----------------:|:---------------:|:---------:| 19| 1Fichier | Whirlpool | No | No | Yes | R | 20| Amazon Drive | MD5 | No | Yes | No | R | 21| Amazon S3 (or S3 compatible) | MD5 | Yes | No | No | R/W | 22| Backblaze B2 | SHA1 | Yes | No | No | R/W | 23| Box | SHA1 | Yes | Yes | No | - | 24| Citrix ShareFile | MD5 | Yes | Yes | No | - | 25| Dropbox | DBHASH ¹ | Yes | Yes | No | - | 26| Enterprise File Fabric | - | Yes | Yes | No | R/W | 27| FTP | - | No | No | No | - | 28| Google Cloud Storage | MD5 | Yes | No | No | R/W | 29| Google Drive | MD5 | Yes | No | Yes | R/W | 30| Google Photos | - | No | No | Yes | R | 31| HDFS | - | Yes | No | No | - | 32| HTTP | - | No | No | No | R | 33| Hubic | MD5 | Yes | No | No | R/W | 34| Jottacloud | MD5 | Yes | Yes | No | R | 35| Koofr | MD5 | No | Yes | No | - | 36| Mail.ru Cloud | Mailru ⁶ | Yes | Yes | No | - | 37| Mega | - | No | No | Yes | - | 38| Memory | MD5 | Yes | No | No | - | 39| Microsoft Azure Blob Storage | MD5 | Yes | No | No | R/W | 40| Microsoft OneDrive | SHA1 ⁵ | Yes | Yes | No | R | 41| OpenDrive | MD5 | Yes | Yes | Partial ⁸ | - | 42| OpenStack Swift | MD5 | Yes | No | No | R/W | 43| pCloud | MD5, SHA1 ⁷ | Yes | No | No | W | 44| premiumize.me | - | No | Yes | No | R | 45| put.io | CRC-32 | Yes | No | Yes | R | 46| QingStor | MD5 | No | No | No | R/W | 47| Seafile | - | No | No | No | - | 48| SFTP | MD5, SHA1 ² | Yes | Depends | No | - | 49| Sia | - | No | No | No | - | 50| SugarSync | - | No | No | No | - | 51| Tardigrade | - | Yes | No | No | - | 52| Uptobox | - | No | No | Yes | - | 53| WebDAV | MD5, SHA1 ³ | Yes ⁴ | Depends | No | - | 54| Yandex Disk | MD5 | Yes | No | No | R | 55| Zoho WorkDrive | - | No | No | No | - | 56| The local filesystem | All | Yes | Depends | No | - | 57 58### Notes 59 60¹ Dropbox supports [its own custom 61hash](https://www.dropbox.com/developers/reference/content-hash). 62This is an SHA256 sum of all the 4 MiB block SHA256s. 63 64² SFTP supports checksums if the same login has shell access and 65`md5sum` or `sha1sum` as well as `echo` are in the remote's PATH. 66 67³ WebDAV supports hashes when used with Owncloud and Nextcloud only. 68 69⁴ WebDAV supports modtimes when used with Owncloud and Nextcloud only. 70 71⁵ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive 72for business and SharePoint server support Microsoft's own 73[QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash). 74 75⁶ Mail.ru uses its own modified SHA1 hash 76 77⁷ pCloud only supports SHA1 (not MD5) in its EU region 78 79⁸ Opendrive does not support creation of duplicate files using 80their web client interface or other stock clients, but the underlying 81storage platform has been determined to allow duplicate files, and it 82is possible to create them with `rclone`. It may be that this is a 83mistake or an unsupported feature. 84 85### Hash ### 86 87The cloud storage system supports various hash types of the objects. 88The hashes are used when transferring data as an integrity check and 89can be specifically used with the `--checksum` flag in syncs and in 90the `check` command. 91 92To use the verify checksums when transferring between cloud storage 93systems they must support a common hash type. 94 95### ModTime ### 96 97The cloud storage system supports setting modification times on 98objects. If it does then this enables a using the modification times 99as part of the sync. If not then only the size will be checked by 100default, though the MD5SUM can be checked with the `--checksum` flag. 101 102All cloud storage systems support some kind of date on the object and 103these will be set when transferring from the cloud storage system. 104 105### Case Insensitive ### 106 107If a cloud storage systems is case sensitive then it is possible to 108have two files which differ only in case, e.g. `file.txt` and 109`FILE.txt`. If a cloud storage system is case insensitive then that 110isn't possible. 111 112This can cause problems when syncing between a case insensitive 113system and a case sensitive system. The symptom of this is that no 114matter how many times you run the sync it never completes fully. 115 116The local filesystem and SFTP may or may not be case sensitive 117depending on OS. 118 119 * Windows - usually case insensitive, though case is preserved 120 * OSX - usually case insensitive, though it is possible to format case sensitive 121 * Linux - usually case sensitive, but there are case insensitive file systems (e.g. FAT formatted USB keys) 122 123Most of the time this doesn't cause any problems as people tend to 124avoid files whose name differs only by case even on case sensitive 125systems. 126 127### Duplicate files ### 128 129If a cloud storage system allows duplicate files then it can have two 130objects with the same name. 131 132This confuses rclone greatly when syncing - use the `rclone dedupe` 133command to rename or remove duplicates. 134 135### Restricted filenames ### 136 137Some cloud storage systems might have restrictions on the characters 138that are usable in file or directory names. 139When `rclone` detects such a name during a file upload, it will 140transparently replace the restricted characters with similar looking 141Unicode characters. To handle the different sets of restricted characters 142for different backends, rclone uses something it calls [encoding](#encoding). 143 144This process is designed to avoid ambiguous file names as much as 145possible and allow to move files between many cloud storage systems 146transparently. 147 148The name shown by `rclone` to the user or during log output will only 149contain a minimal set of [replaced characters](#restricted-characters) 150to ensure correct formatting and not necessarily the actual name used 151on the cloud storage. 152 153This transformation is reversed when downloading a file or parsing 154`rclone` arguments. For example, when uploading a file named `my file?.txt` 155to Onedrive, it will be displayed as `my file?.txt` on the console, but 156stored as `my file？.txt` to Onedrive (the `?` gets replaced by the similar 157looking `？` character, the so-called "fullwidth question mark"). 158The reverse transformation allows to read a file `unusual/name.txt` 159from Google Drive, by passing the name `unusual／name.txt` on the command line 160(the `/` needs to be replaced by the similar looking `／` character). 161 162#### Caveats {#restricted-filenames-caveats} 163 164The filename encoding system works well in most cases, at least 165where file names are written in English or similar languages. 166You might not even notice it: It just works. In some cases it may 167lead to issues, though. E.g. when file names are written in Chinese, 168or Japanese, where it is always the Unicode fullwidth variants of the 169punctuation marks that are used. 170 171On Windows, the characters `:`, `*` and `?` are examples of restricted 172characters. If these are used in filenames on a remote that supports it, 173Rclone will transparently convert them to their fullwidth Unicode 174variants `＊`, `？` and `：` when downloading to Windows, and back again 175when uploading. This way files with names that are not allowed on Windows 176can still be stored. 177 178However, if you have files on your Windows system originally with these same 179Unicode characters in their names, they will be included in the same conversion 180process. E.g. if you create a file in your Windows filesystem with name 181`Test：1.jpg`, where `：` is the Unicode fullwidth colon symbol, and use 182rclone to upload it to Google Drive, which supports regular `:` (halfwidth 183question mark), rclone will replace the fullwidth `:` with the 184halfwidth `:` and store the file as `Test:1.jpg` in Google Drive. Since 185both Windows and Google Drive allows the name `Test：1.jpg`, it would 186probably be better if rclone just kept the name as is in this case. 187 188With the opposite situation; if you have a file named `Test:1.jpg`, 189in your Google Drive, e.g. uploaded from a Linux system where `:` is valid 190in file names. Then later use rclone to copy this file to your Windows 191computer you will notice that on your local disk it gets renamed 192to `Test：1.jpg`. The original filename is not legal on Windows, due to 193the `:`, and rclone therefore renames it to make the copy possible. 194That is all good. However, this can also lead to an issue: If you already 195had a *different* file named `Test：1.jpg` on Windows, and then use rclone 196to copy either way. Rclone will then treat the file originally named 197`Test:1.jpg` on Google Drive and the file originally named `Test：1.jpg` 198on Windows as the same file, and replace the contents from one with the other. 199 200Its virtually impossible to handle all cases like these correctly in all 201situations, but by customizing the [encoding option](#encoding), changing the 202set of characters that rclone should convert, you should be able to 203create a configuration that works well for your specific situation. 204See also the [example](/overview/#encoding-example-windows) below. 205 206(Windows was used as an example of a file system with many restricted 207characters, and Google drive a storage system with few.) 208 209#### Default restricted characters {#restricted-characters} 210 211The table below shows the characters that are replaced by default. 212 213When a replacement character is found in a filename, this character 214will be escaped with the `‛` character to avoid ambiguous file names. 215(e.g. a file named `␀.txt` would shown as `‛␀.txt`) 216 217Each cloud storage backend can use a different set of characters, 218which will be specified in the documentation for each backend. 219 220| Character | Value | Replacement | 221| --------- |:-----:|:-----------:| 222| NUL | 0x00 | ␀ | 223| SOH | 0x01 | ␁ | 224| STX | 0x02 | ␂ | 225| ETX | 0x03 | ␃ | 226| EOT | 0x04 | ␄ | 227| ENQ | 0x05 | ␅ | 228| ACK | 0x06 | ␆ | 229| BEL | 0x07 | ␇ | 230| BS | 0x08 | ␈ | 231| HT | 0x09 | ␉ | 232| LF | 0x0A | ␊ | 233| VT | 0x0B | ␋ | 234| FF | 0x0C | ␌ | 235| CR | 0x0D | ␍ | 236| SO | 0x0E | ␎ | 237| SI | 0x0F | ␏ | 238| DLE | 0x10 | ␐ | 239| DC1 | 0x11 | ␑ | 240| DC2 | 0x12 | ␒ | 241| DC3 | 0x13 | ␓ | 242| DC4 | 0x14 | ␔ | 243| NAK | 0x15 | ␕ | 244| SYN | 0x16 | ␖ | 245| ETB | 0x17 | ␗ | 246| CAN | 0x18 | ␘ | 247| EM | 0x19 | ␙ | 248| SUB | 0x1A | ␚ | 249| ESC | 0x1B | ␛ | 250| FS | 0x1C | ␜ | 251| GS | 0x1D | ␝ | 252| RS | 0x1E | ␞ | 253| US | 0x1F | ␟ | 254| / | 0x2F | ／ | 255| DEL | 0x7F | ␡ | 256 257The default encoding will also encode these file names as they are 258problematic with many cloud storage systems. 259 260| File name | Replacement | 261| --------- |:-----------:| 262| . | ． | 263| .. | ．． | 264 265#### Invalid UTF-8 bytes {#invalid-utf8} 266 267Some backends only support a sequence of well formed UTF-8 bytes 268as file or directory names. 269 270In this case all invalid UTF-8 bytes will be replaced with a quoted 271representation of the byte value to allow uploading a file to such a 272backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`. 273 274A common source of invalid UTF-8 bytes are local filesystems, that store 275names in a different encoding than UTF-8 or UTF-16, like latin1. See the 276[local filenames](/local/#filenames) section for details. 277 278#### Encoding option {#encoding} 279 280Most backends have an encoding option, specified as a flag 281`--backend-encoding` where `backend` is the name of the backend, or as 282a config parameter `encoding` (you'll need to select the Advanced 283config in `rclone config` to see it). 284 285This will have default value which encodes and decodes characters in 286such a way as to preserve the maximum number of characters (see 287above). 288 289However this can be incorrect in some scenarios, for example if you 290have a Windows file system with Unicode fullwidth characters 291`＊`, `？` or `：`, that you want to remain as those characters on the 292remote rather than being translated to regular (halfwidth) `*`, `?` and `:`. 293 294The `--backend-encoding` flags allow you to change that. You can 295disable the encoding completely with `--backend-encoding None` or set 296`encoding = None` in the config file. 297 298Encoding takes a comma separated list of encodings. You can see the 299list of all possible values by passing an invalid value to this 300flag, e.g. `--local-encoding "help"`. The command `rclone help flags encoding` 301will show you the defaults for the backends. 302 303| Encoding | Characters | 304| --------- | ---------- | 305| Asterisk | `*` | 306| BackQuote | `` ` `` | 307| BackSlash | `\` | 308| Colon | `:` | 309| CrLf | CR 0x0D, LF 0x0A | 310| Ctl | All control characters 0x00-0x1F | 311| Del | DEL 0x7F | 312| Dollar | `$` | 313| Dot | `.` or `..` as entire string | 314| DoubleQuote | `"` | 315| Hash | `#` | 316| InvalidUtf8 | An invalid UTF-8 character (e.g. latin1) | 317| LeftCrLfHtVt | CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string | 318| LeftPeriod | `.` on the left of a string | 319| LeftSpace | SPACE on the left of a string | 320| LeftTilde | `~` on the left of a string | 321| LtGt | `<`, `>` | 322| None | No characters are encoded | 323| Percent | `%` | 324| Pipe | \| | 325| Question | `?` | 326| RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | 327| RightPeriod | `.` on the right of a string | 328| RightSpace | SPACE on the right of a string | 329| SingleQuote | `'` | 330| Slash | `/` | 331| SquareBracket | `[`, `]` | 332 333##### Encoding example: FTP 334 335To take a specific example, the FTP backend's default encoding is 336 337 --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot" 338 339However, let's say the FTP server is running on Windows and can't have 340any of the invalid Windows characters in file names. You are backing 341up Linux servers to this FTP server which do have those characters in 342file names. So you would add the Windows set which are 343 344 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot 345 346to the existing ones, giving: 347 348 Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace 349 350This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file. 351 352##### Encoding example: Windows 353 354As a nother example, take a Windows system where there is a file with 355name `Test：1.jpg`, where `：` is the Unicode fullwidth colon symbol. 356When using rclone to copy this to a remote which supports `:`, 357the regular (halfwidth) colon (such as Google Drive), you will notice 358that the file gets renamed to `Test:1.jpg`. 359 360To avoid this you can change the set of characters rclone should convert 361for the local filesystem, using command-line argument `--local-encoding`. 362Rclone's default behavior on Windows corresponds to 363 364``` 365--local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot" 366``` 367 368If you want to use fullwidth characters `：`, `＊` and `？` in your filenames 369without rclone changing them when uploading to a remote, then set the same as 370the default value but without `Colon,Question,Asterisk`: 371 372``` 373--local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot" 374``` 375 376Alternatively, you can disable the conversion of any characters with `--local-encoding None`. 377 378Instead of using command-line argument `--local-encoding`, you may also set it 379as [environment variable](/docs/#environment-variables) `RCLONE_LOCAL_ENCODING`, 380or [configure](/docs/#configure) a remote of type `local` in your config, 381and set the `encoding` option there. 382 383The risk by doing this is that if you have a filename with the regular (halfwidth) 384`:`, `*` and `?` in your cloud storage, and you try to download 385it to your Windows filesystem, this will fail. These characters are not 386valid in filenames on Windows, and you have told rclone not to work around 387this by converting them to valid fullwidth variants. 388 389### MIME Type ### 390 391MIME types (also known as media types) classify types of documents 392using a simple text classification, e.g. `text/html` or 393`application/pdf`. 394 395Some cloud storage systems support reading (`R`) the MIME type of 396objects and some support writing (`W`) the MIME type of objects. 397 398The MIME type can be important if you are serving files directly to 399HTTP from the storage system. 400 401If you are copying from a remote which supports reading (`R`) to a 402remote which supports writing (`W`) then rclone will preserve the MIME 403types. Otherwise they will be guessed from the extension, or the 404remote itself may assign the MIME type. 405 406## Optional Features ## 407 408All rclone remotes support a base command set. Other features depend 409upon backend specific capabilities. 410 411| Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | LinkSharing | About | EmptyDir | 412| ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------:|:-----:|:--------:| 413| 1Fichier | No | Yes | Yes | No | No | No | No | Yes | No | Yes | 414| Amazon Drive | Yes | No | Yes | Yes | No | No | No | No | No | Yes | 415| Amazon S3 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | 416| Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | 417| Box | Yes | Yes | Yes | Yes | Yes ‡‡ | No | Yes | Yes | Yes | Yes | 418| Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | Yes | No | No | Yes | 419| Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes | 420| Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | Yes | 421| FTP | No | No | Yes | Yes | No | No | Yes | No | No | Yes | 422| Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No | 423| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 424| Google Photos | No | No | No | No | No | No | No | No | No | No | 425| HDFS | Yes | No | No | No | No | No | Yes | No | Yes | Yes | 426| HTTP | No | No | No | No | No | No | No | No | No | Yes | 427| Hubic | Yes † | Yes | No | No | No | Yes | Yes | No | Yes | No | 428| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | 429| Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 430| Mega | Yes | No | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 431| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | 432| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No | 433| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 434| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | 435| OpenStack Swift | Yes † | Yes | No | No | No | Yes | Yes | No | Yes | No | 436| pCloud | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | 437| premiumize.me | Yes | No | Yes | Yes | No | No | No | Yes | Yes | Yes | 438| put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | Yes | Yes | 439| QingStor | No | Yes | No | No | Yes | Yes | No | No | No | No | 440| Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | 441| SFTP | No | No | Yes | Yes | No | No | Yes | No | Yes | Yes | 442| SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | 443| Tardigrade | Yes † | No | No | No | No | Yes | Yes | No | No | No | 444| Uptobox | No | Yes | Yes | Yes | No | No | No | No | No | No | 445| WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No | Yes | Yes | 446| Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | 447| Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | Yes | Yes | 448| The local filesystem | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | 449 450### Purge ### 451 452This deletes a directory quicker than just deleting all the files in 453the directory. 454 455† Note Swift, Hubic, and Tardigrade implement this in order to delete 456directory markers but they don't actually have a quicker way of deleting 457files other than deleting them individually. 458 459‡ StreamUpload is not supported with Nextcloud 460 461### Copy ### 462 463Used when copying an object to and from the same remote. This known 464as a server-side copy so you can copy a file without downloading it 465and uploading it again. It is used if you use `rclone copy` or 466`rclone move` if the remote doesn't support `Move` directly. 467 468If the server doesn't support `Copy` directly then for copy operations 469the file is downloaded then re-uploaded. 470 471### Move ### 472 473Used when moving/renaming an object on the same remote. This is known 474as a server-side move of a file. This is used in `rclone move` if the 475server doesn't support `DirMove`. 476 477If the server isn't capable of `Move` then rclone simulates it with 478`Copy` then delete. If the server doesn't support `Copy` then rclone 479will download the file and re-upload it. 480 481### DirMove ### 482 483This is used to implement `rclone move` to move a directory if 484possible. If it isn't then it will use `Move` on each file (which 485falls back to `Copy` then download and upload - see `Move` section). 486 487### CleanUp ### 488 489This is used for emptying the trash for a remote by `rclone cleanup`. 490 491If the server can't do `CleanUp` then `rclone cleanup` will return an 492error. 493 494‡‡ Note that while Box implements this it has to delete every file 495individually so it will be slower than emptying the trash via the WebUI 496 497### ListR ### 498 499The remote supports a recursive list to list all the contents beneath 500a directory quickly. This enables the `--fast-list` flag to work. 501See the [rclone docs](/docs/#fast-list) for more details. 502 503### StreamUpload ### 504 505Some remotes allow files to be uploaded without knowing the file size 506in advance. This allows certain operations to work without spooling the 507file to local disk first, e.g. `rclone rcat`. 508 509### LinkSharing ### 510 511Sets the necessary permissions on a file or folder and prints a link 512that allows others to access them, even if they don't have an account 513on the particular cloud provider. 514 515### About ### 516 517Rclone `about` prints quota information for a remote. Typical output 518includes bytes used, free, quota and in trash. 519 520If a remote lacks about capability `rclone about remote:`returns 521an error. 522 523Backends without about capability cannot determine free space for an 524rclone mount, or use policy `mfs` (most free space) as a member of an 525rclone union remote. 526 527See [rclone about command](https://rclone.org/commands/rclone_about/) 528 529### EmptyDir ### 530 531The remote supports empty directories. See [Limitations](/bugs/#limitations) 532 for details. Most Object/Bucket based remotes do not support this. 533
