A development status updateOver the last few months, the IOTA network has seen a significant increase in activity as more and more developers start to implement solutions based on the Tangle. While this is a very promising development, reflecting the increasing adoption of IOTA, it also results in an increase in database size, which may be problematic for nodes with limited hardware resources.The IOTA Foundation has been performing global snapshots on a regular basis, whereby the transaction history is pruned, and the resulting balances are consolidated into a new genesis state that allows nodes to start over, with an empty database. However, this way of dealing with a growing ledger size is becoming more and more impractical as it requires us to:Temporarily stop the coordinator.Generate the snapshot state.Give the community time to verify the generated files.And finally restart the coordinator.The Solution — Local SnapshotsTo solve this problem, we have been working on implementing a feature called Local Snapshots. This has always been a central part of IOTA’s Roadmap. The initial implementation is now being tested internally, and we will keep you up to date with next steps, but first we must review all the implemented changes and gather sufficient metrics about the behaviour of this new feature.Console output during local snapshotWhat does this mean for node operators?Before we dive into the technical aspects of Local Snapshots, we want to give a short summary of the changes that this new feature brings for node operators:When spinning up a new node, it is possible to sync based on a small local snapshot file, which allows nodes to be fully synced within minutes (rather than bootstrapping the node with a large database file).The disk requirements for nodes are massively reduced — in fact we already have nodes running with just a few hundred megabytes of hard disk space.Since there will no longer be a need for global snapshots, nodes could theoretically run for years without maintenance.Nodes should be in a position to handle thousands of transactions per second, without the database size ever becoming a problem.How does it work?To understand the way Local Snapshots work, we first need to clarify a few things about the way the Tangle works:The Tangle is a data structure that has a lot of uncertainty at its tips but gains certainty as time progresses.Consequently, the further you go back in time the less likely it is for an unconfirmed transaction to suddenly become part of consensus. This is the reason why it is necessary to “reattach” transactions if they have been pending for too long.To verify transactions and take part in IOTA’s consensus, it is only necessary to know the recent history of pending transactions and the current state (balances) of the ledger.The basic principle behind Local Snapshots is relatively easy to understand and can be divided into different aspects:Pruning of old transactions and persisting the balancesWe first choose a confirmed transaction that is sufficiently old and use this transaction as an “anchor” for the local snapshot.We then prune all transactions that are directly or indirectly referenced by this transaction and clean up the database accordingly.Before we clean the old transactions we check which balances were affected by them and persist the resulting state of the ledger in a local snapshot file, which is consequently used by IRI as a new starting point.Solid Entry Points (fast sync for new nodes)While pruning old transactions is no problem for nodes that are already fully synced with the network, it poses a problem for new nodes that try to enter the network, since they are no longer able to easily retrieve the complete transaction history dating back to the last global snapshot.Even if we assume that they are able to retrieve the full history by asking permanodes for the missing transactions, it would still take a very long time to catch up with the latest state of the ledger. This problem is not new and one of the reasons why a lot of node operators bootstrap their nodes with a copy of the database from another synchronized node.To solve this problem, we use the local snapshot files not just as a way to persist the state of the node but also to allow new nodes to start their operations based on the exact same file (which can be shared by the community and the IF in regular intervals).To be able to bootstrap a new node with a local snapshot file we need to store a few more details than just the balances:First of all a new syncing node needs to know at which point it can stop solidifying a chain of transactions and just consider the subtangle solid. To be able to do so, we determine which one of the transactions that we deleted, had approvers that did not become orphaned and store their hashes in a list of “solid entry points”.Once a node reaches one of those hashes it stops asking for its approvees and marks the transaction as solid (like the 999999….99 transaction after a global snapshot).This enables us to use local snapshot files as a bootstrap mechanism to get a new node synced very quickly (within minutes), which at the same time is much easier to provide and retrieve than a copy of the whole database.Seen Milestones (even faster sync)While solid entry points allow us to stop the solidification process as soon possible, it can still take a while to learn about all subsequent milestones that happened after our chosen cut-off point.Since we want the local snapshot files to be a viable and efficient way of bootstrapping a new node we also save all subsequent milestone hashes in the local snapshot files, so that new nodes can immediately ask for missing milestones, without having to passively wait for their neighbours to broadcast them.PermanodesSince the pruning of data will be controlled by a simple configuration setting, it will now be possible to run permanodes that keep the full history of transactions, which has so far been impossible due to the fact that global snapshots were a network wide event.The whole procedure of taking local snapshots is then automatically repeated to allow nodes to run with a relatively constant space requirement without unnecessary maintenance.SummaryThe upcoming feature of Local Snapshots will not just solve the space problems that arise with the growing adoption of IOTA, but will also simplify the setup of new nodes and allow organisations and community members to operate permanodes.We will be opening this up for beta testing in the coming weeks. More information will be posted in the #snapshots channel on the IOTA Discord.Coming Up: Local Snapshots was originally published in IOTA on Medium, where people are continuing the conversation by highlighting and responding to this story.