This is part 2 of how and why our system works (part one: The stack discovery) and I’ll try to explain what we are doing with our API. If you have not read the previous post about how our system is configured I advise you to do so as I will be mentioning it in here.Why do we have a min-api?At first all our APIs were written in C# on top of Umbraco. As our API requests grew in number I noticed our main client facing website was slowing down. This only became apparent only after The Mist Ethereum Wallet integrated our API and the load increased 10x overnight. I knew I had to move that load somewhere else to ensure the website was running smoothly and I also knew there was no need to load the full .net framework in order to return mostly cached content.I kept postponing it for a few weeks (this was around March 2016) and I think I was on holiday in Madrid when chayter (our CEO and my co-founder) started calling me in a panic that the main site kept going down. I checked the logs and it was all calls from the Mist client. I told Charlie that we were accidentally DDoS-ing ourselves and I just increased the cache length from 10 seconds to 1 minute. It was a good quick fix but I knew it was just a patch.A few days later I came back from holiday and started working on the Node.Js min-api server. I called it min-api because at first it was just supposed to return small JSON objects for the Mist wallet. As time passed by it became our main API back-end and it’s what we currently use on our website and portfolio at the moment.How does it work?We have in general two types of requests: one of live data (what is the price now) and one for historical data (what was the price last month).The live data is cached for 10 seconds and it comes from our Redis servers (see prev post). The historical data is cached depending on the period requested. The minute data is cached for 30 seconds, the hour data for 10 minutes and the day data for 1 hour. We also have queries that ask for a specific date and that could be cached indefinitely (the price 5 days ago is never going to change).For caching we use node-cache and a series of internal objects. Depending on the pairs requested we try to calculate if we can get the data· directly (eg BTC-USD)· inverting a more popular pair (eg USD-BTC = 1/BTC-USD)· multiplying through BTC (eg REP-USD = REP-BTC * BTC-USD)· dividing through BTC (eg REP-XMR = REP-BTC/XMR-BTC)· invert dividing through BTC if BTC trades in both pairs (eg USD-EUR = BTC-USD/BTC-EURA typical API requestThe server is started and it loads from Redis all the possible pair combinations and it loads from the block explorer database all the available supplies (we need it to calculate market cap). The whole process takes on average around 200 milliseconds. We use forever to make sure the process restarts if there is a bug. On average each process is up for 4–5 days before we restart it. After the data is loaded the server starts listening for SSL connections.A request for live data comes in.We validate the request against the pair combinations we have data for. We try to see if we have direct data or if we can infer the price going through BTC. We also do duplication removal and we standardize the inputs (eg. all uppercase for symbols)Once the request is validated we check to see if we have data in the cache for the request. If we have data we just return that.If we don’t have data we generate all the possible combinations for direct, multiply, divide and invert divide. If the tryConversion flag is set to false, we only try the direct pair and return that.For each combination we check if we have the data in our internal object cache. If we do, we move to the next pair in the request.If we don’t we check to see if we are currently loading it for another client. Since NodeJS only runs on one thread we can avoid doing multiple calls once the cache expires. The first request, after the cache expiry date, sets a flag and everyone that comes after checks the flag. If the data is being loaded they try again in 200 ms for up to 5 seconds otherwise we set the flag and start loading all the possible pair combinations based on the direct pairs we have from exchanges. For example for XMR-USD we also try to load USD-XMR, XMR-BTC and BTC-USD. All the pairs go through the same process. Imagine someone asks for BTC-USD and 1ms after someone asks for XMR-USD, it would not make sense to load the BTC-USD pair twice since both calls would miss the cache.Once we load all the possible combinations we try to be clever and decide which one is the most accurate representation of the price. (see example of code for how we try)After all the data is loaded we then have a display function that just strips out all the non-required fields and actually performs the inversion, multiplication or division.The output of the display function is returned to the client.The process is generally the same for both historical and live data. The main difference is that for historical we look in the PostgreSQL database and we polyfill blank dates. For the live data we look in Redis.In terms of issues, there are a few that we need to address but most of them are not that big and some of them are nice-to-haves.Known bugs and limitationsThe three bugs or limitations that we want to address in the coming weeks are:The historical data makes assumptions about the operation it needs to do based on live data. For example DASH volume has been growing recently on the USD pair but historically its main pair has been BTC. When you request DASH-USD historical we see that the current DASH-USD volume is high and we return only direct data. What we should do is notice that it’s just a recent trend and return DASH-BTC*BTC-USD for historical.There is no way to ask for a conversion directly. You can ask for direct data or you can let us try to be clever and try to go through BTC but can’t say you want the data through ETH conversion for example REP-USD you would want it as REP-ETH * ETH-USD. We should have a conversion pair parameter.There is no way to force a conversion. Say you want historical DASH-USD and you know the DASH-BTC * BTC-USD is the correct way to go, we should let you force the conversion though BTC so you have more control over how the data is calculated. Of course you can call DASH-BTC and then call BTC-USD separately and multiply locally but it’s a bit of a hassle from you side.Nice-to-havesThe big nice to have for me is to improve historical caching. As the title of the API says it, (min-api) this was only meant as a simple price info api but it grew into much more over time. Our caching system was designed for live data and it does not work that well for historical minute, hour and day data. It does a decent job but it could be improved. For example if you call for the last week of hourly data and someone else after you calls for the last day of hourly data, we have two different caches. The way it should work is it should first check if there is data in the cache for a longer time span and if there is it should just return a chuck of that rather than do another DB call for the same data but a smaller sample.As always, thank you for using our API and always feel free to email us if you find bugs or suggestions.1 hour of calls on our API.Our API is under a Creative Commons — Attribution Non-Commercial license. This means that it’s free as long as you attribute it to us and you are not making money from the data. If you are selling a product that uses our data we charge a fee. The idea is that if you start making a lot of money using our data, we would like something in return. If you are just providing a free service, you can use our data freely.The API dissection was originally published in Tales from the Crypto on Medium, where people are continuing the conversation by highlighting and responding to this story.