1---
2title: Upgrading
3weight: 250
4---
5
6# Upgrading Grafana Loki
7
8Every attempt is made to keep Grafana Loki backwards compatible, such that upgrades should be low risk and low friction.
9
10Unfortunately Loki is software and software is hard and sometimes we are forced to make decisions between ease of use and ease of maintenance.
11
12If we have any expectation of difficulty upgrading we will document it here.
13
14As more versions are released it becomes more likely unexpected problems arise moving between multiple versions at once.
15If possible try to stay current and do sequential updates. If you want to skip versions, try it in a development environment before attempting to upgrade production.
16
17
18## Master / Unreleased
19
20### Loki
21
22#### Distributor now stores ring in memory by default instead of Consul
23
24PR [4440](https://github.com/grafana/loki/pull/4440) **DylanGuedes**: Config: Override distributor's default ring KV store
25
26This change sets `inmemory` as the new default storage for the Distributor ring (previously `consul`).
27The motivation is making the Distributor easier to run with default configs, by not requiring Consul anymore.
28In any case, if you prefer to use Consul as the ring storage, you can set it by using the following config:
29
30```yaml
31distributor:
32  ring:
33    kvstore:
34      store: consul
35```
36
37#### Memberlist config now automatically applies to all non-configured rings
38PR [4400](https://github.com/grafana/loki/pull/4400) **trevorwhitney**: Config: automatically apply memberlist config too all rings when provided
39
40This change affects the behavior of the ingester, distributor, and ruler rings. Previously, if you wanted to use memberlist for all of these rings, you
41had to provide a `memberlist` configuration as well as specify `store: memberlist` for the `kvstore` of each of the rings you wanted to use memberlist.
42For example, your configuration might look something like this:
43
44```yaml
45memberlist:
46  join_members:
47    - loki.namespace.svc.cluster.local
48distributor:
49  ring:
50    kvstore:
51      store: memberlist
52ingester:
53    lifecycler:
54      ring:
55        kvstore:
56          store: memberlist
57ruler:
58  ring:
59    kvstore:
60      store: memberlist
61```
62
63Now, if your provide a `memberlist` configuration with at least one `join_members`, loki will default all rings to use a `kvstore` of type `memberlist`.
64You can change this behavior by overriding specific configurations. For example, if you wanted to use `consul` for you `ruler` rings, but `memberlist`
65for the `ingester` and `distributor`, you could do so with the following config:
66
67```yaml
68memberlist:
69  join_members:
70    - loki.namespace.svc.cluster.local
71ruler:
72  ring:
73    kvstore:
74      store: consul
75      consul:
76        host: consul.namespace.svc.cluster.local:8500
77```
78
79#### Changed defaults for some GRPC server settings
80* [4435](https://github.com/grafana/loki/pull/4435) **trevorwhitney**: Change default values for two GRPC settings so querier can connect to frontend/scheduler
81
82This changes two default values, `grpc_server_min_time_between_pings` and `grpc_server_ping_without_stream_allowed` used by the GRPC server.
83
84*Previous Values*:
85```
86server:
87  grpc_server_min_time_between_pings: '5m'
88  grpc_server_ping_without_stream_allowed: false
89```
90
91*New Values*:
92```
93server:
94  grpc_server_min_time_between_pings: '10s'
95  grpc_server_ping_without_stream_allowed: true
96```
97
98Please manually provide the values of `5m` and `true` (respectively) in your config if you rely on those values.
99
100-_add changes here which are unreleased_
101
102### Loki Config
103
104#### Change of some default limits to common values
105
106PR [4415](https://github.com/grafana/loki/pull/4415) **DylanGuedes**: the default value of some limits were changed to protect users from overwhelming their cluster with ingestion load caused by relying on default configs.
107
108We suggest you double check if the following parameters are
109present in your Loki config: `ingestion_rate_strategy`, `max_global_streams_per_user`
110`max_query_length` `max_query_parallelism` `max_streams_per_user`
111`reject_old_samples` `reject_old_samples_max_age`. If they are not present, we recommend you double check that the new values will not negatively impact your system. The changes are:
112
113| config | new default | old default |
114| --- | --- | --- |
115| ingestion_rate_strategy | "global" | "local" |
116| max_global_streams_per_user | 5000 | 0 (no limit) |
117| max_query_length | "721h" | "0h" (no limit) |
118| max_query_parallelism | 32 | 14 |
119| max_streams_per_user | 0 (no limit) | 10000 |
120| reject_old_samples | true | false |
121| reject_old_samples_max_age | "168h" | "336h" |
122
123## 2.3.0
124
125### Loki
126
127#### Query restriction introduced for queries which do not have at least one equality matcher
128
129PR [3216](https://github.com/grafana/loki/pull/3216) **sandeepsukhani**: check for stream selectors to have at least one equality matcher
130
131This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:
132
133`{namespace=~".*"}`
134
135This query will now be rejected, however there are several ways to modify it for it to succeed:
136
137Add at least one equals label matcher:
138
139`{cluster="us-east-1",namespace=~".*"}`
140
141Use `.+` instead of `.*`
142
143`{namespace=~".+"}`
144
145This difference may seem subtle but if we break it down `.` matches any character, `*` matches zero or more of the preceding character and `+` matches one or more of the preceding character. The `.*` case will match empty values where `.+` will not, this is the important difference. `{namespace=""}` is an invalid request (unless you add another equals label matcher like the example above)
146
147The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.
148
149
150## 2.2.0
151
152### Loki
153
154**Be sure to upgrade to 2.0 or 2.1 BEFORE upgrading to 2.2**
155
156In Loki 2.2 we changed the internal version of our chunk format from v2 to v3, this is a transparent change and is only relevant if you every try to _downgrade_ a Loki installation. We incorporated the code to read v3 chunks in 2.0.1 and 2.1, as well as 2.2 and any future releases.
157
158**If you upgrade to 2.2+ any chunks created can only be read by 2.0.1, 2.1 and 2.2+**
159
160This makes it important to first upgrade to 2.0, 2.0.1, or 2.1 **before** upgrading to 2.2 so that if you need to rollback for any reason you can do so easily.
161
162**Note:** 2.0 and 2.0.1 are identical in every aspect except 2.0.1 contains the code necessary to read the v3 chunk format. Therefor if you are on 2.0 and ugrade to 2.2, if you want to rollback, you must rollback to 2.0.1.
163
164### Loki Config
165
166**Read this if you use the query-frontend and have `sharded_queries_enabled: true`**
167
168We discovered query scheduling related to sharded queries over long time ranges could lead to unfair work scheduling by one single query in the per tenant work queue.
169
170The `max_query_parallelism` setting is designed to limit how many split and sharded units of 'work' for a single query are allowed to be put into the per tenant work queue at one time. The previous behavior would split the query by time using the `split_queries_by_interval` and compare this value to `max_query_parallelism` when filling the queue, however with sharding enabled, every split was then sharded into 16 additional units of work after the `max_query_parallelism` limit was applied.
171
172In 2.2 we changed this behavior to apply the `max_query_parallelism` after splitting _and_ sharding a query resulting a more fair and expected queue scheduling per query.
173
174**What this means** Loki will be putting much less work into the work queue per query if you are using the query frontend and have sharding_queries_enabled (which you should).  **You may need to increase your `max_query_parallelism` setting if you are noticing slower query performance** In practice, you may not see a difference unless you were running a cluster with a LOT of queriers or queriers with a very high `parallelism` frontend_worker setting.
175
176You could consider multiplying your current `max_query_parallelism` setting by 16 to obtain the previous behavior, though in practice we suspect few people would really want it this high unless you have a significant querier worker pool.
177
178**Also be aware to make sure `max_outstanding_per_tenant` is always greater than `max_query_parallelism` or large queries will automatically fail with a 429 back to the user.**
179
180
181
182### Promtail
183
184For 2.0 we eliminated the long deprecated `entry_parser` configuration in Promtail configs, however in doing so we introduced a very confusing and erroneous default behavior:
185
186If you did not specify a `pipeline_stages` entry you would be provided with a default which included the `docker` pipeline stage.  This can lead to some very confusing results.
187
188In [3404](https://github.com/grafana/loki/pull/3404), we corrected this behavior
189
190**If you are using docker, and any of your `scrape_configs` are missing a `pipeline_stages` definition**, you should add the following to obtain the correct behaviour:
191
192```yaml
193pipeline_stages:
194  - docker: {}
195```
196
197## 2.1.0
198
199The upgrade from 2.0.0 to 2.1.0 should be fairly smooth, please be aware of these two things:
200
201### Helm charts have moved!
202
203Helm charts are now located at: https://github.com/grafana/helm-charts/
204
205The helm repo URL is now: https://grafana.github.io/helm-charts
206
207### Fluent Bit plugin renamed
208
209Fluent bit officially supports Loki as an output plugin now! WoooHOOO!
210
211However this created a naming conflict with our existing output plugin (the new native output uses the name `loki`) so we have renamed our plugin.
212
213In time our plan is to deprecate and eliminate our output plugin in favor of the native Loki support. However until then you can continue using the plugin with the following change:
214
215Old:
216
217```
218[Output]
219    Name loki
220```
221
222New:
223
224```
225[Output]
226    Name grafana-loki
227```
228
229## 2.0.0
230
231This is a major Loki release and there are some very important upgrade considerations.
232For the most part, there are very few impactful changes and for most this will be a seamless upgrade.
233
2342.0.0 Upgrade Topics:
235
236* [IMPORTANT If you are using a docker image, read this!](#important-if-you-are-using-a-docker-image-read-this)
237* [IMPORTANT boltdb-shipper upgrade considerations](#important-boltdb-shipper-upgrade-considerations)
238* [IMPORTANT results_cachemax_freshness removed from yaml config](#important-results_cachemax_freshness-removed-from-yaml-config)
239* [Promtail removed entry_parser config](#promtail-config-removed)
240* [If you would like to use the new single store index and v11 schema](#upgrading-schema-to-use-boltdb-shipper-andor-v11-schema)
241
242### **IMPORTANT If you are using a docker image, read this!**
243
244(This includes, Helm, Tanka, docker-compose etc.)
245
246The default config file in the docker image, as well as the default helm values.yaml and jsonnet for Tanka all specify a schema definition to make things easier to get started.
247
248>**If you have not specified your own config file with your own schema definition (or you do not have a custom schema definition in your values.yaml), upgrading to 2.0 will break things!**
249
250In 2.0 the defaults are now v11 schema and the `boltdb-shipper` index type.
251
252
253If you are using an index type of `aws`, `bigtable`, or `cassandra` this means you have already defined a custom schema and there is _nothing_ further you need to do regarding the schema.
254You can consider however adding a new schema entry to use the new `boltdb-shipper` type if you want to move away from these separate index stores and instead use just one object store.
255
256#### What to do
257
258The minimum action required is to create a config which specifies the schema to match what the previous defaults were.
259
260(Keep in mind this will only tell Loki to use the old schema default, if you would like to upgrade to v11 and/or move to the single store boltdb-shipper, [see below](#upgrading-schema-to-use-boltdb-shipper-andor-v11-schema))
261
262There are three places we have hard coded the schema definition:
263
264##### Helm
265
266Helm has shipped with the same internal schema in the values.yaml file for a very long time.
267
268If you are providing your own values.yaml file then there is no _required_ action because you already have a schema definition.
269
270**If you are not providing your own values.yaml file, you will need to make one!**
271
272We suggest using the included [values.yaml file from the 1.6.0 tag](https://raw.githubusercontent.com/grafana/loki/v1.6.0/production/helm/loki/values.yaml)
273
274This matches what the default values.yaml file had prior to 2.0 and is necessary for Loki to work post 2.0
275
276As mentioned above, you should also consider looking at moving to the v11 schema and boltdb-shipper [see below](#upgrading-schema-to-use-boltdb-shipper-andor-v11-schema) for more information.
277
278##### Tanka
279
280This likely only affects a small portion of tanka users because the default schema config for Loki was forcing `GCS` and `bigtable`.
281
282**If your `main.jsonnet` (or somewhere in your manually created jsonnet) does not have a schema config section then you will need to add one like this!**
283
284```jsonnet
285{
286  _config+:: {
287    using_boltdb_shipper: false,
288    loki+: {
289      schema_config+: {
290        configs: [{
291          from: '2018-04-15',
292          store: 'bigtable',
293          object_store: 'gcs',
294          schema: 'v11',
295          index: {
296            prefix: '%s_index_' % $._config.table_prefix,
297            period: '168h',
298          },
299        }],
300      },
301    },
302  }
303}
304```
305
306>**NOTE** If you had set `index_period_hours` to a value other than 168h (the previous default) you must update this in the above config `period:` to match what you chose.
307
308>**NOTE** We have changed the default index store to `boltdb-shipper` it's important to add `using_boltdb_shipper: false,` until you are ready to change (if you want to change)
309
310Changing the jsonnet config to use the `boltdb-shipper` type is the same as [below](#upgrading-schema-to-use-boltdb-shipper-andor-v11-schema) where you need to add a new schema section.
311
312**HOWEVER** Be aware when you change `using_boltdb_shipper: true` the deployment type for the ingesters and queriers will change to statefulsets! Statefulsets are required for the ingester and querier using boltdb-shipper.
313
314##### Docker (e.g. docker-compose)
315
316For docker related cases you will have to mount a Loki config file separate from what's shipped inside the container
317
318I would recommend taking the previous default file from the [1.6.0 tag on github](https://raw.githubusercontent.com/grafana/loki/v1.6.0/cmd/loki/loki-docker-config.yaml)
319
320How you get this mounted and in use by Loki might vary based on how you are using the image, but this is a common example:
321
322```shell
323docker run -d --name=loki --mount type=bind,source="path to loki-config.yaml",target=/etc/loki/local-config.yaml
324```
325
326The Loki docker image is expecting to find the config file at `/etc/loki/local-config.yaml`
327
328
329### IMPORTANT: boltdb-shipper upgrade considerations.
330
331Significant changes have taken place between 1.6.0 and 2.0.0 for boltdb-shipper index type, if you are already running this index and are upgrading some extra caution is warranted.
332
333Please strongly consider taking a complete backup of the `index` directory in your object store, this location might be slightly different depending on what store you use.
334It should be a folder named index with a bunch of folders inside with names like `index_18561`,`index_18560`...
335
336The chunks directory should not need any special backups.
337
338If you have an environment to test this in please do so before upgrading against critical data.
339
340There are 2 significant changes warranting the backup of this data because they will make rolling back impossible:
341* A compactor is included which will take existing index files and compact them to one per day and remove non compacted files
342* All index files are now gzipped before uploading
343
344The second part is important because 1.6.0 does not understand how to read the gzipped files, so any new files uploaded or any files compacted become unreadable to 1.6.0 or earlier.
345
346_THIS BEING SAID_ we are not expecting problems, our testing so far has not uncovered any problems, but some extra precaution might save data loss in unforeseen circumstances!
347
348Please report any problems via GitHub issues or reach us on the #loki slack channel.
349
350**Note if are using boltdb-shipper and were running with high availability and separate filesystems**
351
352This was a poorly documented and even more experimental mode we toyed with using boltdb-shipper. For now we removed the documentation and also any kind of support for this mode.
353
354To use boltdb-shipper in 2.0 you need a shared storage (S3, GCS, etc), the mode of running with separate filesystem stores in HA using a ring is not officially supported.
355
356We didn't do anything explicitly to limit this functionality however we have not had any time to actually test this which is why we removed the docs and are listing it as not supported.
357
358#### If running in microservices, deploy ingesters before queriers
359
360Ingesters now expose a new RPC method that queriers use when the index type is `boltdb-shipper`.
361Queriers generally roll out faster than ingesters, so if new queriers query older ingesters using the new RPC, the queries would fail.
362To avoid any query downtime during the upgrade, rollout ingesters before queriers.
363
364#### If running the compactor, ensure it has delete permissions for the object storage.
365
366The compactor is an optional but suggested component that combines and deduplicates the boltdb-shipper index files. When compacting index files, the compactor writes a new file and deletes unoptimized files. Ensure that the compactor has appropriate permissions for deleting files, for example, s3:DeleteObject permission for AWS S3.
367
368### IMPORTANT: `results_cache.max_freshness` removed from YAML config
369
370The `max_freshness` config from `results_cache` has been removed in favour of another flag called `max_cache_freshness_per_query` in `limits_config` which has the same effect.
371If you happen to have `results_cache.max_freshness` set please use `limits_config.max_cache_freshness_per_query` YAML config instead.
372
373### Promtail config removed
374
375The long deprecated `entry_parser` config in Promtail has been removed, use [pipeline_stages]({{< relref "../clients/promtail/configuration/#pipeline_stages" >}}) instead.
376
377### Upgrading schema to use boltdb-shipper and/or v11 schema
378
379If you would also like to take advantage of the new Single Store (boltdb-shipper) index, as well as the v11 schema if you aren't already using it.
380
381You can do this by adding a new schema entry.
382
383Here is an example:
384
385```yaml
386schema_config:
387  configs:
388    - from: 2018-04-15           ①
389      store: boltdb              ①④
390      object_store: filesystem   ①④
391      schema: v11                ②
392      index:
393        prefix: index_           ①
394        period: 168h             ①
395    - from: 2020-10-24           ③
396      store: boltdb-shipper
397      object_store: filesystem   ④
398      schema: v11
399      index:
400        prefix: index_
401        period: 24h              ⑤
402```
403① Make sure all of these match your current schema config
404② Make sure this matches your previous schema version, Helm for example is likely v9
405③ Make sure this is a date in the **FUTURE** keep in mind Loki only knows UTC so make sure it's a future UTC date
406④ Make sure this matches your existing config (e.g. maybe you were using gcs for your object_store)
407⑤ 24h is required for boltdb-shipper
408
409There are more examples on the [Storage description page]({{< relref "../storage/_index.md#examples" >}}) including the information you need to setup the `storage` section for boltdb-shipper.
410
411
412## 1.6.0
413
414### Important: Ksonnet port changed and removed NET_BIND_SERVICE capability from Docker image
415
416In 1.5.0 we changed the Loki user to not run as root which created problems binding to port 80.
417To address this we updated the docker image to add the NET_BIND_SERVICE capability to the loki process
418which allowed Loki to bind to port 80 as a non root user, so long as the underlying system allowed that
419linux capability.
420
421This has proved to be a problem for many reasons and in PR [2294](https://github.com/grafana/loki/pull/2294/files)
422the capability was removed.
423
424It is now no longer possible for the Loki to be started with a port less than 1024 with the published docker image.
425
426The default for Helm has always been port 3100, and Helm users should be unaffect unless they changed the default.
427
428**Ksonnet users however should closely check their configuration, in PR 2294 the loki port was changed from 80 to 3100**
429
430
431### IMPORTANT: If you run Loki in microservices mode, special rollout instructions
432
433A new ingester GRPC API has been added allowing to speed up metric queries, to ensure a rollout without query errors **make sure you upgrade all ingesters first.**
434Once this is done you can then proceed with the rest of the deployment, this is to ensure that queriers won't look for an API not yet available.
435
436If you roll out everything at once, queriers with this new code will attempt to query ingesters which may not have the new method on the API and queries will fail.
437
438This will only affect reads(queries) and not writes and only for the duration of the rollout.
439
440### IMPORTANT: Scrape config changes to both Helm and Ksonnet will affect labels created by Promtail
441
442PR [2091](https://github.com/grafana/loki/pull/2091) Makes several changes to the Promtail scrape config:
443
444````
445This is triggered by https://github.com/grafana/jsonnet-libs/pull/261
446
447The above PR changes the instance label to be actually unique within
448a scrape config. It also adds a pod and a container target label
449so that metrics can easily be joined with metrics from cAdvisor, KSM,
450and the Kubelet.
451
452This commit adds the same to the Loki scrape config. It also removes
453the container_name label. It is the same as the container label
454and was already added to Loki previously. However, the
455container_name label is deprecated and has disappeared in K8s 1.16,
456so that it will soon become useless for direct joining.
457````
458
459TL;DR
460
461The following label have been changed in both the Helm and Ksonnet Promtail scrape configs:
462
463`instance` -> `pod`
464`container_name` -> `container`
465
466
467### Experimental boltdb-shipper changes
468
469PR [2166](https://github.com/grafana/loki/pull/2166) now forces the index to have a period of exactly `24h`:
470
471Loki will fail to start with an error if the active schema or upcoming schema are not set to a period of `24h`
472
473You can add a new schema config like this:
474
475```yaml
476schema_config:
477  configs:
478    - from: 2020-01-01      <----- This is your current entry, date will be different
479      store: boltdb-shipper
480      object_store: aws
481      schema: v11
482      index:
483        prefix: index_
484        period: 168h
485    - from: [INSERT FUTURE DATE HERE]   <----- Add another entry, set a future date
486      store: boltdb-shipper
487      object_store: aws
488      schema: v11
489      index:
490        prefix: index_
491        period: 24h   <--- This must be 24h
492```
493If you are not on `schema: v11` this would be a good opportunity to make that change _in the new schema config_ also.
494
495**NOTE** If the current time in your timezone is after midnight UTC already, set the date one additional day forward.
496
497There was also a significant overhaul to how boltdb-shipper internals, this should not be visible to a user but as this
498feature is experimental and under development bug are possible!
499
500The most noticeable change if you look in the storage, Loki no longer updates an existing file and instead creates a
501new index file every 15mins, this is an important move to make sure objects in the object store are immutable and
502will simplify future operations like compaction and deletion.
503
504### Breaking CLI flags changes
505
506The following CLI flags where changed to improve consistency, they are not expected to be widely used
507
508```diff
509- querier.query_timeout
510+ querier.query-timeout
511
512- distributor.extra-query-delay
513+ querier.extra-query-delay
514
515- max-chunk-batch-size
516+ store.max-chunk-batch-size
517
518- ingester.concurrent-flushed
519+ ingester.concurrent-flushes
520```
521
522### Loki Canary metric name changes
523
524When adding some new features to the canary we realized the existing metrics were not compliant with standards for counter names, the following metrics have been renamed:
525
526```nohighlight
527loki_canary_total_entries               ->      loki_canary_entries_total
528loki_canary_out_of_order_entries        ->      loki_canary_out_of_order_entries_total
529loki_canary_websocket_missing_entries   ->      loki_canary_websocket_missing_entries_total
530loki_canary_missing_entries             ->      loki_canary_missing_entries_total
531loki_canary_unexpected_entries          ->      loki_canary_unexpected_entries_total
532loki_canary_duplicate_entries           ->      loki_canary_duplicate_entries_total
533loki_canary_ws_reconnects               ->      loki_canary_ws_reconnects_total
534loki_canary_response_latency            ->      loki_canary_response_latency_seconds
535```
536
537### Ksonnet Changes
538
539In `production/ksonnet/loki/config.libsonnet` the variable `storage_backend` used to have a default value of `'bigtable,gcs'`.
540This has been changed to providing no default and will error if not supplied in your environment jsonnet,
541here is an example of what you should add to have the same behavior as the default (namespace and cluster should already be defined):
542
543```jsonnet
544_config+:: {
545    namespace: 'loki-dev',
546    cluster: 'us-central1',
547    storage_backend: 'gcs,bigtable',
548```
549
550Defaulting to `gcs,bigtable` was confusing for anyone using ksonnet with other storage backends as it would manifest itself with obscure bigtable errors.
551
552## 1.5.0
553
554Note: The required upgrade path outlined for version 1.4.0 below is still true for moving to 1.5.0 from any release older than 1.4.0 (e.g. 1.3.0->1.5.0 needs to also look at the 1.4.0 upgrade requirements).
555
556### Breaking config changes!
557
558Loki 1.5.0 vendors Cortex v1.0.0 (congratulations!), which has a [massive list of changes](https://cortexmetrics.io/docs/changelog/#1-0-0-2020-04-02).
559
560While changes in the command line flags affect Loki as well, we usually recommend people to use configuration file instead.
561
562Cortex has done lot of cleanup in the configuration files, and you are strongly urged to take a look at the [annotated diff for config file](https://cortexmetrics.io/docs/changelog/#config-file-breaking-changes) before upgrading to Loki 1.5.0.
563
564Following fields were removed from YAML configuration completely: `claim_on_rollout` (always true), `normalise_tokens` (always true).
565
566#### Test Your Config
567
568To see if your config needs to change, one way to quickly test is to download a 1.5.0 (or newer) binary from the [release page](https://github.com/grafana/loki/releases/tag/v1.5.0)
569
570Then run the binary providing your config file `./loki-linux-amd64 -config.file=myconfig.yaml`
571
572If there are configs which are no longer valid you will see errors immediately:
573
574```shell
575./loki-linux-amd64 -config.file=loki-local-config.yaml
576failed parsing config: loki-local-config.yaml: yaml: unmarshal errors:
577  line 35: field dynamodbconfig not found in type aws.StorageConfig
578```
579
580Referencing the [list of diffs](https://cortexmetrics.io/docs/changelog/#config-file-breaking-changes) I can see this config changed:
581
582```diff
583-  dynamodbconfig:
584+  dynamodb:
585```
586
587Also several other AWS related configs changed and would need to udpate those as well.
588
589
590### Loki Docker Image User and File Location Changes
591
592To improve security concerns, in 1.5.0 the Docker container no longer runs the loki process as `root` and instead the process runs as user `loki` with UID `10001` and GID `10001`
593
594This may affect people in a couple ways:
595
596#### Loki Port
597
598If you are running Loki with a config that opens a port number above 1024 (which is the default, 3100 for HTTP and 9095 for GRPC) everything should work fine in regards to ports.
599
600If you are running Loki with a config that opens a port number less than 1024 Linux normally requires root permissions to do this, HOWEVER in the Docker container we run `setcap cap_net_bind_service=+ep /usr/bin/loki`
601
602This capability lets the loki process bind to a port less than 1024 when run as a non root user.
603
604Not every environment will allow this capability however, it's possible to restrict this capability in linux.  If this restriction is in place, you will be forced to run Loki with a config that has HTTP and GRPC ports above 1024.
605
606#### Filesystem
607
608**Please note the location Loki is looking for files with the provided config in the docker image has changed**
609
610In 1.4.0 and earlier the included config file in the docker container was using directories:
611
612```
613/tmp/loki/index
614/tmp/loki/chunks
615```
616
617In 1.5.0 this has changed:
618
619```
620/loki/index
621/loki/chunks
622```
623
624This will mostly affect anyone using docker-compose or docker to run Loki and are specifying a volume to persist storage.
625
626**There are two concerns to track here, one is the correct ownership of the files and the other is making sure your mounts updated to the new location.**
627
628One possible upgrade path would look like this:
629
630If I were running Loki with this command `docker run -d --name=loki --mount source=loki-data,target=/tmp/loki -p 3100:3100 grafana/loki:1.4.0`
631
632This would mount a docker volume named `loki-data` to the `/tmp/loki` folder which is where Loki will persist the `index` and `chunks` folder in 1.4.0
633
634To move to 1.5.0 I can do the following (please note that your container names and paths and volumes etc may be different):
635
636```
637docker stop loki
638docker rm loki
639docker run --rm --name="loki-perm" -it --mount source=loki-data,target=/mnt ubuntu /bin/bash
640cd /mnt
641chown -R 10001:10001 ./*
642exit
643docker run -d --name=loki --mount source=loki-data,target=/loki -p 3100:3100 grafana/loki:1.5.0
644```
645
646Notice the change in the `target=/loki` for 1.5.0 to the new data directory location specified in the [included Loki config file](https://github.com/grafana/loki/tree/master/cmd/loki/loki-docker-config.yaml).
647
648The intermediate step of using an ubuntu image to change the ownership of the Loki files to the new user might not be necessary if you can easily access these files to run the `chown` command directly.
649That is if you have access to `/var/lib/docker/volumes` or if you mounted to a different local filesystem directory, you can change the ownership directly without using a container.
650
651
652### Loki Duration Configs
653
654If you get an error like:
655
656```nohighlight
657 ./loki-linux-amd64-1.5.0 -log.level=debug -config.file=/etc/loki/config.yml
658failed parsing config: /etc/loki/config.yml: not a valid duration string: "0"
659```
660
661This is because of some underlying changes that no longer allow durations without a unit.
662
663Unfortunately the yaml parser doesn't give a line number but it's likely to be one of these two:
664
665```yaml
666chunk_store_config:
667  max_look_back_period: 0s # DURATION VALUES MUST HAVE A UNIT EVEN IF THEY ARE ZERO
668
669table_manager:
670  retention_deletes_enabled: false
671  retention_period: 0s # DURATION VALUES MUST HAVE A UNIT EVEN IF THEY ARE ZERO
672```
673
674### Promtail Config Changes
675
676The underlying backoff library used in Promtail had a config change which wasn't originally noted in the release notes:
677
678If you get this error:
679
680```nohighlight
681Unable to parse config: /etc/promtail/promtail.yaml: yaml: unmarshal errors:
682  line 3: field maxbackoff not found in type util.BackoffConfig
683  line 4: field maxretries not found in type util.BackoffConfig
684  line 5: field minbackoff not found in type util.BackoffConfig
685```
686
687The new values are:
688
689```yaml
690min_period:
691max_period:
692max_retries:
693```
694
695## 1.4.0
696
697Loki 1.4.0 vendors Cortex v0.7.0-rc.0 which contains [several breaking config changes](https://github.com/cortexproject/cortex/blob/v0.7.0-rc.0/CHANGELOG).
698
699One such config change which will affect Loki users:
700
701In the [cache_config](../../configuration#cache_config):
702
703`defaul_validity` has changed to `default_validity`
704
705Also in the unlikely case you were configuring your schema via arguments and not a config file, this is no longer supported.  This is not something we had ever provided as an option via docs and is unlikely anyone is doing, but worth mentioning.
706
707The other config changes should not be relevant to Loki.
708
709### Required Upgrade Path
710
711The newly vendored version of Cortex removes code related to de-normalized tokens in the ring. What you need to know is this:
712
713*Note:* A "shared ring" as mentioned below refers to using *consul* or *etcd* for values in the following config:
714
715```yaml
716kvstore:
717  # The backend storage to use for the ring. Supported values are
718  # consul, etcd, inmemory
719  store: <string>
720```
721
722- Running without using a shared ring (inmemory): No action required
723- Running with a shared ring and upgrading from v1.3.0 -> v1.4.0: No action required
724- Running with a shared ring and upgrading from any version less than v1.3.0 (e.g. v1.2.0) -> v1.4.0: **ACTION REQUIRED**
725
726There are two options for upgrade if you are not on version 1.3.0 and are using a shared ring:
727
728- Upgrade first to v1.3.0 **BEFORE** upgrading to v1.4.0
729
730OR
731
732**Note:** If you are running a single binary you only need to add this flag to your single binary command.
733
7341. Add the following configuration to your ingesters command: `-ingester.normalise-tokens=true`
7351. Restart your ingesters with this config
7361. Proceed with upgrading to v1.4.0
7371. Remove the config option (only do this after everything is running v1.4.0)
738
739**Note:** It's also possible to enable this flag via config file, see the [`lifecycler_config`](https://github.com/grafana/loki/tree/v1.3.0/docs/configuration#lifecycler_config) configuration option.
740
741If using the Helm Loki chart:
742
743```yaml
744extraArgs:
745  ingester.normalise-tokens: true
746```
747
748If using the Helm Loki-Stack chart:
749
750```yaml
751loki:
752  extraArgs:
753    ingester.normalise-tokens: true
754```
755
756#### What will go wrong
757
758If you attempt to add a v1.4.0 ingester to a ring created by Loki v1.2.0 or older which does not have the commandline argument `-ingester.normalise-tokens=true` (or configured via [config file](https://github.com/grafana/loki/tree/v1.3.0/docs/configuration#lifecycler_config)), the v1.4.0 ingester will remove all the entries in the ring for all the other ingesters as it cannot "see" them.
759
760This will result in distributors failing to write and a general ingestion failure for the system.
761
762If this happens to you, you will want to rollback your deployment immediately. You need to remove the v1.4.0 ingester from the ring ASAP, this should allow the existing ingesters to re-insert their tokens.  You will also want to remove any v1.4.0 distributors as they will not understand the old ring either and will fail to send traffic.
763