Around a year ago I was working in banking as a risk analyst. Around that time, I started questioning my goals as a banker, it first seemed like an obvious career choice after years of finance education, but I was looking for something more adventurous.After the financial crisis, the banking sector was mostly about reconciliation and regulation, and I graduated when business was slumping. I joined a financial institution and although I learnt a lot during those years, it appeared to me that I was being taught the standard way of doing things but there was little room for innovation.One day my boss forwarded me an invitation to a FinTech conference in London, the Quantech conference, and he asked me to represent him at the event. So I went and that was the first time I heard about blockchain, bitcoin and smart contracts. After that, it was a fast chain of events: I read everything available about blockchain, went to meetups and conferences and started giving presentations internally about bitcoin and blockchain.This led me to the HackEthon event in September 2016 at Thomson Reuters, where I met Vlad, as CryptoCompare was one of the sponsors. He explained to me that the crypto space is like the wild wild west, information is scattered and there are a lot of scams, but with CryptoCompare he wants to build a trusted source of information that users can go to if they want to learn about a cryptocurrency, mining, exchange or wallet, and many more.After that, I was introduced to Charlie, who co-founded CryptoCompare, and I quickly realised that these are the good guys of the crypto space (and many people confirmed).CryptoCompare wants to deliver consistency, trust and truth for the crypto space.So I joined CryptoCompare in January 2017!It is an exciting space to be in right now, watching governments trying to deal with cryptocurrencies, companies issuing their own share tokens, traders pumping and dumping coins. And indeed, the cryptocurrency market is the wild wild west, but every day is a new learning opportunity. And that’s why it is so important to share the knowledge with others, so the crypto space can become safer.My personal goal at CryptoCompare is to help users become financially literate by building tools for them to analyse and mitigate financial risk.So this was my intro blog post, I’ll come back with more data and finance related posts soon. Happy trading!Why I joined CryptoCompare was originally published in Tales from the Crypto on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few weeks ago I tried to cover our work on the portfolio but I did not explain any of the coding behind it. The best subject to start with would be the foundation layer of our service, the systems behind our pricing and historical data.First of all, it makes sense to give a short overview of what our VMs look like and how they all fit into the overall server architecture.What does our sever architecture look like?Hosting and VM management: Microsoft AzureExchange integration: 60 x Node.js VMs — shared core, 750mb of RAM, 29GB disk space.Long term storage (trades): Azure blob storageHistorical and block explorer data: 4 x PostgreSQL — single core, 1.5GB of RAM, 29GB disk spaceSnapshot data (what is the price now): 2 x Redis — single core, 1.5GB of RAM, 29GB disk spaceStreaming servers: 12 x Node.js + Socket.IO (load balanced)— single core, 1.5GB of RAM, 29GB disk spaceStreaming broadcast server — 1 x Node.js (VM that broadcasts messages from the exchanges to the streamers, lets us scale the streamers with ease) — single core, 3.5Gb of RAM, 50GB disk spaceWidget servers: 2 x Node.js (load balanced — tested to support over 1 million calls per hour, currently at 50,000 on average) — single core, 3.5GB of RAM, 50GB SSD disk spaceAPI servers: 3 x Node.js (load balanced — tested to support over 10 million calls per hour, currently at 1.7 M on average) — single core, 3.5GB of RAM, 50GB SSD disk spaceScript servers: 1x NodeJS — single core, 1.5GB of RAM, 29GB disk spaceData Aggregation Server: 1 x Node.js (creates our aggregated price for all the trades on all the exchanges we integrate with) — single core, 3.5GB of RAM, 50GB SSD disk spaceCMS/front-end web server: 2 x Umbraco (C#) + IIS (load balanced — tested to support over 100,000 page views per hour, currently at 12,000) — 4 cores, 7GB or RAM, 120GB SSD disk spaceImage CDN: Azure blob storageForum, reviews, static content and member data: 1 x Azure SQL ServerForum and member stats: 1 x Redis — single core, 1.5GB or RAM, 29GB disk spaceEmails — SparkPost (forum notifications and daily update email ) and Mandrill (member management emails: registration, activation, password reset)How do we get the data?We have one virtual machine for each exchange we integrate with. Each exchange virtual machine runs the following set of Node.js services:the trades service — this is specific for each exchange and it is either a polling or a web sockets API. The sole purpose of this service is to get trades data and save it in our trades format.the historical data service — reads the trade file and saves minute and hourly data to the historical databasethe snapshot service — reads the trade file and saves current aggregated data to the Redis server (current price, total volume, last trades etc)the streaming service — reads the trade file and streams aggregated data to the streaming serversthe trades archive service — reads the trade files and saves trade data in the azure blob storagethe order book service — connects to each exchange and streams order book data to the streamers (this will be expanded to store the data as well)How does the data reach our members?The front-end Umbraco server has Redis and PostgreSQL integration but most of the data comes from our API servers. We use the same API servers that we have publicly available to everyone, it makes a lot more sense to consume the same APIs as it’s a lot less work in the long run (eat your own dog food). It also helps us find and solve scaling issues as both the API usage and our web servers grow.A lot of the front-end is built using AngularJS and since the beginning, we’ve had to build all our product around APIs. It was one of the best decisions we took early on and it’s helped us be a lot more efficient in our development. It’s also allowed us to share some code between the Node.js back-end servers and the AngularJS front-end.Most of our price pages load with a Redis snapshot of the prices and then connect to our streaming servers and start streaming the rest of the data.Why this stack?Back-end — When we started the website (late 2014) Node.js and AngularJS were gaining a lot of popularity and I wanted to try them out. Without AngularJS I doubt we could have built the website in under 1 year. I’ve previously worked on a similar project and it took us a lot longer just to build the front-end in jQuery (and it was really messy). Before Paul Dobre joined our team the front-end was an afterthought, I did work on it for a bit, but didn’t do a very good job as I’m generally bad at design. Most of the early work went into developing the exchange integration and the data flow. On the Node.js side, we don’t use any frameworks, just vanilla node since most of the code is just used to fetch or retrieve JSON data. Redis and PostgreSQL are just some of the best tools when it comes to snapshot and historical data.Front-end — We chose Umbraco because I find it the easiest CMS to work with from the editor’s point of view. Also, I was comfortable enough with it to make it work for our project and I knew that I would just use it as an API layer for our AngularJS application.Streaming — Socket.IO was really easy to use and it saved me a lot of time. So far, everything has worked really well both in terms of efficiency and scalability. All our external services are load balanced and scalable and our internal servers have a full backup available at a click of the mouse.Next week I’m planning on writing a blog post about how our API works.As always, if you have any questions or suggestions, feel free to comment.The stack discovery was originally published in Tales from the Crypto on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is part 2 of how and why our system works (part one: The stack discovery) and I’ll try to explain what we are doing with our API. If you have not read the previous post about how our system is configured I advise you to do so as I will be mentioning it in here.Why do we have a min-api?At first all our APIs were written in C# on top of Umbraco. As our API requests grew in number I noticed our main client facing website was slowing down. This only became apparent only after The Mist Ethereum Wallet integrated our API and the load increased 10x overnight. I knew I had to move that load somewhere else to ensure the website was running smoothly and I also knew there was no need to load the full .net framework in order to return mostly cached content.I kept postponing it for a few weeks (this was around March 2016) and I think I was on holiday in Madrid when chayter (our CEO and my co-founder) started calling me in a panic that the main site kept going down. I checked the logs and it was all calls from the Mist client. I told Charlie that we were accidentally DDoS-ing ourselves and I just increased the cache length from 10 seconds to 1 minute. It was a good quick fix but I knew it was just a patch.A few days later I came back from holiday and started working on the Node.Js min-api server. I called it min-api because at first it was just supposed to return small JSON objects for the Mist wallet. As time passed by it became our main API back-end and it’s what we currently use on our website and portfolio at the moment.How does it work?We have in general two types of requests: one of live data (what is the price now) and one for historical data (what was the price last month).The live data is cached for 10 seconds and it comes from our Redis servers (see prev post). The historical data is cached depending on the period requested. The minute data is cached for 30 seconds, the hour data for 10 minutes and the day data for 1 hour. We also have queries that ask for a specific date and that could be cached indefinitely (the price 5 days ago is never going to change).For caching we use node-cache and a series of internal objects. Depending on the pairs requested we try to calculate if we can get the data· directly (eg BTC-USD)· inverting a more popular pair (eg USD-BTC = 1/BTC-USD)· multiplying through BTC (eg REP-USD = REP-BTC * BTC-USD)· dividing through BTC (eg REP-XMR = REP-BTC/XMR-BTC)· invert dividing through BTC if BTC trades in both pairs (eg USD-EUR = BTC-USD/BTC-EURA typical API requestThe server is started and it loads from Redis all the possible pair combinations and it loads from the block explorer database all the available supplies (we need it to calculate market cap). The whole process takes on average around 200 milliseconds. We use forever to make sure the process restarts if there is a bug. On average each process is up for 4–5 days before we restart it. After the data is loaded the server starts listening for SSL connections.A request for live data comes in.We validate the request against the pair combinations we have data for. We try to see if we have direct data or if we can infer the price going through BTC. We also do duplication removal and we standardize the inputs (eg. all uppercase for symbols)Once the request is validated we check to see if we have data in the cache for the request. If we have data we just return that.If we don’t have data we generate all the possible combinations for direct, multiply, divide and invert divide. If the tryConversion flag is set to false, we only try the direct pair and return that.For each combination we check if we have the data in our internal object cache. If we do, we move to the next pair in the request.If we don’t we check to see if we are currently loading it for another client. Since NodeJS only runs on one thread we can avoid doing multiple calls once the cache expires. The first request, after the cache expiry date, sets a flag and everyone that comes after checks the flag. If the data is being loaded they try again in 200 ms for up to 5 seconds otherwise we set the flag and start loading all the possible pair combinations based on the direct pairs we have from exchanges. For example for XMR-USD we also try to load USD-XMR, XMR-BTC and BTC-USD. All the pairs go through the same process. Imagine someone asks for BTC-USD and 1ms after someone asks for XMR-USD, it would not make sense to load the BTC-USD pair twice since both calls would miss the cache.Once we load all the possible combinations we try to be clever and decide which one is the most accurate representation of the price. (see example of code for how we try)After all the data is loaded we then have a display function that just strips out all the non-required fields and actually performs the inversion, multiplication or division.The output of the display function is returned to the client.The process is generally the same for both historical and live data. The main difference is that for historical we look in the PostgreSQL database and we polyfill blank dates. For the live data we look in Redis.In terms of issues, there are a few that we need to address but most of them are not that big and some of them are nice-to-haves.Known bugs and limitationsThe three bugs or limitations that we want to address in the coming weeks are:The historical data makes assumptions about the operation it needs to do based on live data. For example DASH volume has been growing recently on the USD pair but historically its main pair has been BTC. When you request DASH-USD historical we see that the current DASH-USD volume is high and we return only direct data. What we should do is notice that it’s just a recent trend and return DASH-BTC*BTC-USD for historical.There is no way to ask for a conversion directly. You can ask for direct data or you can let us try to be clever and try to go through BTC but can’t say you want the data through ETH conversion for example REP-USD you would want it as REP-ETH * ETH-USD. We should have a conversion pair parameter.There is no way to force a conversion. Say you want historical DASH-USD and you know the DASH-BTC * BTC-USD is the correct way to go, we should let you force the conversion though BTC so you have more control over how the data is calculated. Of course you can call DASH-BTC and then call BTC-USD separately and multiply locally but it’s a bit of a hassle from you side.Nice-to-havesThe big nice to have for me is to improve historical caching. As the title of the API says it, (min-api) this was only meant as a simple price info api but it grew into much more over time. Our caching system was designed for live data and it does not work that well for historical minute, hour and day data. It does a decent job but it could be improved. For example if you call for the last week of hourly data and someone else after you calls for the last day of hourly data, we have two different caches. The way it should work is it should first check if there is data in the cache for a longer time span and if there is it should just return a chuck of that rather than do another DB call for the same data but a smaller sample.As always, thank you for using our API and always feel free to email us if you find bugs or suggestions.1 hour of calls on our API.Our API is under a Creative Commons — Attribution Non-Commercial license. This means that it’s free as long as you attribute it to us and you are not making money from the data. If you are selling a product that uses our data we charge a fee. The idea is that if you start making a lot of money using our data, we would like something in return. If you are just providing a free service, you can use our data freely.The API dissection was originally published in Tales from the Crypto on Medium, where people are continuing the conversation by highlighting and responding to this story.

Roughly one year has passed since I joined CryptoCompare and, so far, it has been an incredible journey. Before joining CryptoCompare I had just began dipping my toes in the cryptoworld. Although I had used Bitcoin before, I dismissed it as basically “anonymous internet money” and didn’t do any research on it. One day, however, I found out that I could make some money selling Bitcoin for PayPal in P2P exchanges (an extremely dangerous venture that I would not recommend to anyone). That led me to discover Ethereum and cryptocurrency mining.My adventure with CryptoCompare began with Vlad Cealicu, who I met on an Ethereum mining support group on Skype. At the time, I had begun mining Ethereum and I was spending all my free time reading and watching every piece of information I could find about cryptocurrencies and blockchain technology. He helped me with a bunch of questions I had regarding mining.Eventually, after bugging him for a considerable amount of time with ideas for guides and other stuff I had for CryptoCompare, he introduced me to chayter. A few weeks later I quit my job and started working at CryptoCompare. My job initially was to add and manage the content of our coin pages, a position that has now expanded to pretty much everything. I basically run the company now… No, I’m joking, but since then, and thanks to the help and patience of both Charles and Vlad, I’ve been able to tackle much more exciting tasks. I’m now in charge of writing guides, the daily update, work on planning sections of the website itself and much more.Working for a company in the crypto space allowed me to dedicate all my time to learning about cryptocurrencies. It did not come easy to me, especially since I don’t have any particular education or background in technology, economy or anything relevant to this field. It was challenging at first but it has also put me in a position where I can understand what makes cryptocurrencies tick and I can convey that information to a general audience in a manner that everyone can understand. I’ve also had the opportunity to contribute to other websites like Coremedia and Bitcoinist where I have learned a great deal about cryptocurrencies and about the current alternative, fiat.After more than a year of working at CryptoCompare and with other companies in the cryptosphere, my perception of the world around me has changed. I can now see, understand and interpret current events differently.In this time, I’ve come to the conclusion that what gets people involved with Bitcoin and cryptocurrencies is not necessarily an understanding of how these cryptocurrencies themselves work. Instead, what really matters is an understanding of how the current financial system works, or rather, how it doesn’t.The most interesting aspect of Bitcoin, to me at least, isn’t how it can protect our privacy or how it can be used to send value across the world. They are important features but they are not game changing. What makes Bitcoin important is the fact that it cannot be controlled, censored or issued by any single entity. It makes no difference if you are a government, country, bank or a simple user. That’s what makes Bitcoin amazing. Not the fact that it’s some sort of technological revolution (although, in some sense, it is), but rather that it can help save mankind from its own greed and incompetence.The reason I say my perception of the world has changed is not that I have come to realize how good of a solution Bitcoin is, but rather how big of a problem it fixes. Before Bitcoin, I had no idea how money worked. The more I read the clearer the picture of the current financial system became and the more I realized the true value of Bitcoin.To sum it up, the most important lesson I’ve learned so far is not that Bitcoin can change the world, but rather that the world has to change. That can’t happen while a flick of a pen can magically add new billions of dollars/euros/pounds in circulation, not always for the most noble of reasons.So if you’re trying to explain how Bitcoin works to a friend or co-worker, don’t start with Bitcoin. Start with fiat, because it’s hard to see value in the solution until you understand the problem.What I’ve learned from working at CryptoCompare — and from crypto in general. was originally published in Tales from the Crypto on Medium, where people are continuing the conversation by highlighting and responding to this story.