# Getting Started with IndexedDB for Big Data Storage

#### A powerful database right there in your browser

An introduction to IndexedDB

It’s not always necessary to send a user’s data to the server: you could choose to store some information in the browser. It’s ideal for device-specific settings such as interface configurations (e.g. light/dark modes) or private data which should not be transmitted across the network (e.g. an encryption key).

Consider the [Performance API](https://blog.asayer.io/how-to-evaluate-site-speed-with-the-performance-api) for timing network and page events. You *could* upload all data the moment it’s available but that’s a considerable volume of information which, ironically, would hit page performance. A better option would be to store it locally, process statistics using a [Web Worker](https://developer.mozilla.org/docs/Web/API/Web_Workers_API), and upload the results when the browser is less busy.

Two cross-browser client-side storage APIs are available:

1.  [Web Storage](https://developer.mozilla.org/docs/Web/API/Web_Storage_API)  
    Synchronous name-value pair storage saved permanently (`localStore`) or for the current session (`sessionStore`). Browsers permit up to 5MB of Web Storage per domain.
2.  [IndexedDB](https://developer.mozilla.org/docs/Web/API/IndexedDB_API)  
    An asynchronous NoSQL-like name-value store which can save data, files, and blobs. At least 1GB should be available per domain and it can reach up to 60% of the remaining disk space.

> *Another storage option,* [*WebSQL*](https://www.w3.org/TR/webdatabase/)*, is available in some editions of Chrome and Safari. However, it has a 5MB limit, is inconsistent, and was deprecated in 2010.*

In this tutorial we’ll store timing data for the page and all assets. Web Storage space can be slow and too limiting so IndexedDB is the best option.

[**All example code is available on Github**](https://github.com/craigbuckler/asayer-indexeddb) should you want to try IndexedDB on your own site.

### What is IndexedDB?

IndexedDB was first implemented in 2011 and became a W3C standard in January 2015. It has [good browser support](https://caniuse.com/indexeddb) although its callback and events-based API seems clunky now we have ES2015+. This article demonstrates how to write a Promise-based wrapper so you can use chaining and `async`/ `await`.

Note the following IndexedDB terms:

*   **database** — a top-level store. A domain can create any number of IndexedDB databases but it’s unusual to see more than one. Only pages within the same domain can access the database.
*   **object store** — a name/value store for related data items. It’s similar to a collection in MongoDB or tables in a relational database.
*   **key** — a unique name used to reference every record (value) in the object store. It can be generated using an **autoIncrement** number or set to any unique value within the record.
*   **index** — another way to organize data in an object store. Search queries can only examine the key or an index.
*   **schema** — the definition of object stores, keys, and indexes.
*   **version** — a version number (integer) assigned to a schema. IndexedDB offers automated versioning so you can update databases to the latest schema.
*   **operation** — database activities such as creating, reading, updating, or deleting records.
*   **transaction** — a set of one or more operations. A transaction guarantees all its operations either succeed or fail. It cannot fail some and not others.
*   **cursor** — a way to iterate over records without having to load all into memory at once.

### Developing and debugging a database

In this tutorial, you will create an IndexedDB database named `performance`. It contains two object stores:

1.  **navigation**  
    This stores [page navigation timings](https://blog.asayer.io/how-to-evaluate-site-speed-with-the-performance-api#navigation-timing) information (redirects, DNS lookups, page loading, file sizes, load events, etc). A date will be added to use as the **key**.
2.  **resource**  
    This stores [resource timings](https://blog.asayer.io/how-to-evaluate-site-speed-with-the-performance-api#resource-timing) information (timings for other resources such as images, stylesheets, scripts, Ajax calls etc.) A date will be added, but two or more assets could load at the same time so an auto-incrementing ID will be used as the **key**. Indexes will be created for the date and the name (the resource’s URL).

All Chrome-based browsers have an **Application** tab where you can examine storage space, artificially limit the capacity, and wipe all data. The **IndexedDB** entry in the Storage tree allows you to view, update, and delete object stores, indexes, and individual records. Firefox’s panel is named **Storage**.

You can also run your application in incognito mode so all data is deleted once the browser window is closed.

### Connecting to an IndexedDB database

A wrapper class created in `indexeddb.js` checks for IndexedDB support using:

if ('indexedDB' in window) // ...

It then opens a database connection using `[indexedDB.open()](https://developer.mozilla.org/docs/Web/API/IDBFactory/open)` by passing:

1.  the database name, and
2.  an optional version integer.

const dbOpen = indexedDB.open('performance', 1);

Three important event handler functions must be defined:

1.  `dbOp en.onerror` runs when an IndexedDB connection cannot be established.
2.  `dbOpen.onupgradeneeded` runs when the version required ( `1`) is greater than the current version ( `0` when the database is not defined). A handler function must run IndexedDB methods such as `[createObjectStore()](https://developer.mozilla.org/docs/Web/API/IDBDatabase/createObjectStore)` and `[createIndex()](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/createIndex)` to create the storage structures.
3.  `dbOpen.onsuccess` runs when the connection has been established and any upgrades have completed. The connection object in `dbOpen.result` is used in all subsequent data operations. It is assigned to `this.db` in the wrapper class.

The wrapper constructor code:

// IndexedDB wrapper class: indexeddb.js  
export class IndexedDB {

  // connect to IndexedDB database  
  constructor(dbName, dbVersion, dbUpgrade) {

  return new Promise((resolve, reject) => {

    // connection object  
    this.db = null;

    // no support  
    if (!('indexedDB' in window)) reject('not supported');

    // open database  
    const dbOpen = indexedDB.open(dbName, dbVersion);

    if (dbUpgrade) {

      // database upgrade event  
      dbOpen.onupgradeneeded = e => {  
        dbUpgrade(dbOpen.result, e.oldVersion, e.newVersion);  
      };

    }

    // success event handler  
    dbOpen.onsuccess = () => {  
      this.db = dbOpen.result;  
      resolve( this );  
    };

    // failure event handler  
    dbOpen.onerror = e => {  
      reject(\`IndexedDB error: ${ e.target.errorCode }\`);  
    };

   });

  }

  // more methods coming later...

}

A `performance.js` script loads this module and instantiates a new IndexedDB object named `perfDB` after the page has loaded. It passes the database name ( `performance`), version ( `1`), and an upgrade function. The `indexeddb.js` constructor calls the upgrade function with the database connection object, the current database version, and the new version:

// performance.js  
import { IndexedDB } from './indexeddb.js';

window.addEventListener('load', async () => {

// IndexedDB connection  
  const perfDB = await new IndexedDB(  
    'performance',  
    1,  
    (db, oldVersion, newVersion) => {

      console.log(\`upgrading database from ${ oldVersion } to ${ newVersion }\`);

      switch (oldVersion) {

        case 0: {

          const  
            navigation = db.createObjectStore('navigation', { keyPath: 'date' }),  
            resource = db.createObjectStore('resource', { keyPath: 'id', autoIncrement: true });

          resource.createIndex('dateIdx', 'date', { unique: false });  
          resource.createIndex('nameIdx', 'name', { unique: false });

      }

    }

  });

  // more code coming later...

});

At some point it will become necessary to change the database schema — perhaps to add new object stores, indexes, or data updates. In that situation, you must increment the version (from `1` to `2`). The next page load will trigger the upgrade handler again so you can add a further block to the `switch` statement, e.g. to create an index named `durationIdx` on the `duration` property in the `resource` object store:

case 1: {  
  const resource = db.transaction.objectStore('resource');  
  resource.createIndex('durationIdx', 'duration', { unique: false });  
}

The usual `break` at the end of each `case` block is omitted. When someone accesses the application for the first time, the `case 0` block will run followed by `case 1` and all subsequent blocks. Anyone already on version `1` would run the updates starting at the `case 1`. IndexedDB schema update methods include:

*   `[createIndex()](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/createIndex)`
*   `[deleteIndex()](https://developer.mozilla.org/docs/Web/API/IDBObjectStore/deleteIndex)`
*   `[createObjectStore()](https://developer.mozilla.org/docs/Web/API/IDBDatabase/createObjectStore)`
*   `[deleteObjectStore()](https://developer.mozilla.org/docs/Web/API/IDBDatabase/deleteObjectStore)`
*   `[deleteDatabase()](https://developer.mozilla.org/docs/Web/API/IDBFactory/deleteDatabase)`

Everyone who loads the page will be on the same version — *unless they have the app running in two or more tabs*. To avoid conflicts, the database connection `onversionchange` handler can be added to `indexeddb.js` which prompts the user to reload the page:

// version change handler  
dbOpen.onversionchange = () => {

dbOpen.close();  
  alert('Database upgrade required - reloading...');  
  location.reload();

};

You can now add the `performance.js` script to a page and run it to check that object stores and indexes are created (DevTools **Application** or **Storage** panels):

<script type="module" src="./performance.js"></script>

### Record performance statistics

All IndexedDB operations are wrapped in a transaction. The following process is used:

1.  Create a database transaction object. This defines one or more object stores (single string or array of strings) and the access type: `"readonly"` for fetching data, or `"readwrite"` for inserts and updates.
2.  Create a reference to an `[objectStore()](https://developer.mozilla.org/docs/Web/API/IDBTransaction/objectStore)` within the scope of the transaction.
3.  Run any number of (inserts only) or methods (inserts and updates).

Add a new `update()` method to the `IndexedDB` class in `indexeddb.js`:

// store item  
update(storeName, value, overwrite = false) {

  return new Promise((resolve, reject) => {

    // new transaction  
    const  
      transaction = this.db.transaction(storeName, 'readwrite'),  
      store = transaction.objectStore(storeName);

    // ensure values are in array  
    value = Array.isArray(value) ? value : \[ value \];

    // write all values  
    value.forEach(v => {  
      if (overwrite) store.put(v);  
      else store.add(v);  
    });

    transaction.oncomplete = () => {  
      resolve(true); // success  
    };

    transaction.onerror = () => {  
      reject(transaction.error); // failure  
    };

  });

}

This adds or updates (if the `overwrite` parameter is `true`) one or more values in the named store and wraps the whole transaction in a Promise. The `transaction.oncomplete` event handler runs when the transaction auto-commits at the end of the function and all database operations are complete. A `transaction.onerror` handler reports errors.

> *IndexedDB events bubble up from the operation to the transaction, to the store, and to the database. You could create a single* `*onerror*` *handler on the database which receives all errors. Like DOM events, propagation can be stopped with* `*event.stopPropagation()*`*.*

The `performance.js` script can now report page navigation metrics:

// record page navigation information  
  const  
    date = new Date(),

    nav = Object.assign(  
      { date },  
      performance.getEntriesByType('navigation')\[0\].toJSON()  
    );

await perfDB.update('navigation', nav);

and resource metrics:

const res = performance.getEntriesByType('resource').map(  
  r => Object.assign({ date }, r.toJSON())  
);

await perfDB.update('resource', res);

In both cases, a `date` property is added to the cloned timing objects so it becomes possible to search for data within a specific period.

### Frontend Monitoring

[OpenReplay](https://github.com/openreplay/openreplay) is a developer-oriented, open-source session replay alternative to FullStory, LogRocket and Hotjar. It’s like having your browser’s DevTools open while looking over your user’s shoulder.

[OpenReplay](https://github.com/openreplay/openreplay) is self-hosted so you have complete control over your data and costs.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1660399277343/RKoUa2SKV.png)

Happy debugging, for modern frontend teams — [Start monitoring your web app for free](https://github.com/openreplay/openreplay).

### Reading performance records

IndexedDB searching is rudimentary compared to other databases. You can only fetch records by their key or an indexed value. You cannot use equivalents of SQL `JOIN` or functions such as `AVERAGE()` and `SUM()`. All record processing must be handled with JavaScript code; a background [Web Worker](https://developer.mozilla.org/docs/Web/API/Web_Workers_API) thread could be a practical option.

You can retrieve a single record by passing its key to an object store or index’s and defining an `onsuccess` handler:

// EXAMPLE CODE  
const

  // new readonly transaction  
  transaction = db.transaction('resource', 'readonly'),

  // get resource object store  
  resource = transaction.objectStore('resource'),

  // fetch record 1  
  request = resource.get(1);

// request complete  
request.onsuccess = () => {  
  console.log('result:', request.result);  
};

// request failed  
request.onerror = () => {  
  console.log('failed:', request.error);  
};

Similar methods include:

*   `[.count(query)](https://developer.mozilla.org/docs/Web/API/IDBIndex/getAllKeys)` — the number of matching records
*   `[.getAll(query)](https://developer.mozilla.org/docs/Web/API/IDBIndex/getAll)` — an array of matching records
*   `[.getKey(query)](https://developer.mozilla.org/docs/Web/API/IDBIndex/getKey)` — a matching key (rather than the value assigned to that key)
*   `[.getAllKeys(query)](https://developer.mozilla.org/docs/Web/API/IDBIndex/getAllKeys)` — an array of matching keys

The *query* can also be a [KeyRange](https://developer.mozilla.org/docs/Web/API/IDBKeyRange) argument to find records within a range, e.g. `IDBKeyRange.bound(1, 10)` returns all records with key between 1 and 10 inclusive:

request = resource.getAll( IDBKeyRange.bound(1, 10) );

KeyRange options:

*   `[IDBKeyRange.lowerBound(X)](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/lowerBound)` — greater than or equal to `X`
*   `[IDBKeyRange.upperBound(X)](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/upperBound)` — less than or equal to `Y`
*   `[IDBKeyRange.bound(X,Y)](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/bound)` — between `X` and `Y` inclusive
*   `[IDBKeyRange.only(X)](https://developer.mozilla.org/docs/Web/API/IDBKeyRange/only)` — a single key matching `X`.

The lower, upper, and bound methods have an optional exclusive flag, e.g. `IDBKeyRange.bound(1, 10, true, false)` - keys greater than `1` (but not `1` itself) and less than or equal to `10`.

Reading a whole dataset into an array becomes impossible as the database grows larger. IndexedDB provides *cursors* which can iterate through each record one at a time. The `[.openCursor()](https://developer.mozilla.org/docs/Web/API/IDBIndex/openCursor)` [method](https://developer.mozilla.org/docs/Web/API/IDBIndex/openCursor) is passed a KeyRange and optional direction string (`"next"`, `"nextunique"`, `"prev"`, or `"preunique"`).

Add a new `fetch()` method to the `IndexedDB` class in `indexeddb.js` to search an object store or index with upper and lower bounds with a callback function that is passed the cursor. Two further methods are also required:

1.  `index(storeName, indexName)` - returns either an object store or index on that store, and
2.  `bound(lowerBound, upperBound)` - returns an appropriate KeyRange object.

// get items using cursor  
  fetch(storeName, indexName, lowerBound = null, upperBound = null, callback) {

    const  
      request = this.index(storeName, indexName)  
        .openCursor( this.bound(lowerBound, upperBound) );

    // pass cursor to callback function  
    request.onsuccess = () => {  
      if (callback) callback(request.result);  
    };

    request.onerror = () => {  
      return(request.error); // failure  
    };

  }

  // start a new read transaction on object store or index  
  index(storeName, indexName) {

    const  
      transaction = this.db.transaction(storeName),  
      store = transaction.objectStore(storeName);

    return indexName ? store.index(indexName) : store;

  }

  // get bounding object  
  bound(lowerBound, upperBound) {

    if (lowerBound && upperBound) return IDBKeyRange.bound(lowerBound, upperBound);  
    else if (lowerBound) return IDBKeyRange.lowerBound(lowerBound);  
    else if (upperBound) return IDBKeyRange.upperBound(upperBound);

  }

The `performance.js` script can now retrieve page navigation metrics, e.g. return all `domContentLoadedEventEnd` during June 2021:

// fetch page navigation objects in June 2021  
perfDB.fetch(  
  'navigation',  
  null, // not an index  
  new Date(2021,5,1,10,40,0,0), // lower  
  new Date(2021,6,1,10,40,0,0), // upper  
  cursor => { // callback function

    if (cursor) {  
      console.log(cursor.value.domContentLoadedEventEnd);  
      cursor.continue();  
    }

  }  
);

Similarly, you can calculate the average download time for a specific file and report it back to [OpenReplay](https://github.com/openreplay/openreplay):

// calculate average download time using index  
let  
  filename = '[http://mysite.com/main.css'](http://mysite.com/main.css%27),  
  count = 0,  
  total = 0;

perfDB.fetch(  
    'resource', // object store  
    'nameIdx',  // index  
    filename,   // matching file  
    filename,  
    cursor => { // callback

      if (cursor) {

        count++;  
        total += cursor.value.duration;  
        cursor.continue();

      }  
      else {

        // all records processed  
        if (count) {

          const avgDuration = total / count;

          console.log(\`average duration for ${ filename }: ${ avgDuration } ms\`);

          // report to OpenReplay  
          if (asayer) asayer.event(\`${ filename }\`, { avgDuration });

     }

   }

});

In both cases, the `cursor` object is passed to the callback function where it can:

1.  obtain the record value with `cursor.value`
2.  advance to the next record with `[cursor.continue()](https://developer.mozilla.org/docs/Web/API/IDBCursor/continue)`
3.  move forward `N` records with `[cursor.advance(N)](https://developer.mozilla.org/docs/Web/API/IDBCursor/advance)`
4.  update the record with `[cursor.update(data)](https://developer.mozilla.org/docs/Web/API/IDBCursor/update)`, or
5.  delete the record with `[cursor.delete()](https://developer.mozilla.org/docs/Web/API/IDBCursor/delete)`

`cursor` is `null` when all matching records have been processed.

### Check Remaining Storage Space

Browsers allocate a significant volume of storage to IndexedDB but it will eventually run out. The new Promise-based [StorageManager API](https://developer.mozilla.org/docs/Web/API/StorageManager) can calculate the space remaining for the domain:

(async () => {

  if (!navigator.storage) return;

  const  
    estimate = await navigator.storage.estimate(),

    // calculate remaining storage in MB  
    available = Math.floor((estimate.quota - estimate.usage) / 1024 / 1024);

  console.log(\`${ available } MB remaining\`);

})();

The API is not supported in IE or Safari. As the limit is approached, you could choose to remove older records.

### Conclusion

IndexedDB is one of the older and more complex browser APIs but you can add wrapper methods to adopt Promises and `async`/ `await`. Pre-built libraries such as [idb](https://www.npmjs.com/package/idb) can help if you'd rather not do that yourself.

Despite its drawbacks and some unusual design decisions, IndexedDB remains the fastest and largest browser-based data store.

*Originally published at* [*https://blog.asayer.io*](https://blog.asayer.io/getting-started-with-indexeddb-for-big-data-storage) *on June 3, 2021.*