carlsborg 15 hours ago | next |

The main lichess engine (lila, open source) is a single monolith program that's deployed on a single server. It serves ~5 million games per day. But there are a several other pieces too. They discuss the architecture here https://www.youtube.com/watch?v=crKNBSpO2_I

BTW consider donating if you use lichess.

justinclift 14 hours ago | root | parent | next |

Wow. ~US$40k/mo running costs, with about US$5k/mo for server hosting:

https://lichess.org/costs

It looks like the servers are individually managed via OVH or similar, rather than running their own gear in co-location or similar. Wonder why?

tormeh 7 hours ago | root | parent | next |

Easy: If something is wrong with the physical gear it's OVH's problem rather than theirs. It also means no one has to ever go to the data center which is probably important for a geographically distributed team (I assume they are). Cheap, no-frills cloud is extremely underrated, IMO.

squigz 11 hours ago | root | parent | prev | next |

Surprising numbers, and really goes to show how cheap the hardware/software side is for this sort of thing if you do it right.

I wonder what the "Misc dev salaries" is for - only curious because it's a flat $5k

justinclift 10 hours ago | root | parent |

Heh heh heh.

To me those numbers seem on the high side as I'm (personally) used to (for cheap projects) scavenging together stuff from Ebay before deploying to a data centre. ;)

benmmurphy 5 hours ago | root | parent | prev |

its also crazy how much cheaper it is than AWS. the primary DB is around $500/month with 32 CPU and 256 GB of RAM and 7TB. AWS RDS db.m6gd.8xlarge which is 32 CPU and 128 GB of RAM costs $2150/month before paying for storage as well.

bryan_w 4 hours ago | root | parent |

Yeah, but you get what you pay for. That m6gd.8xlarge would never be subject to such a long network outage as once the hardware fault was detected, it would be moved to another machine

theideaofcoffee 4 hours ago | prev | next |

I guess some of my questions are addressed in the latter half of the post, but I'm still puzzled why a prominent service didn't have a plan for what looked like a run of the mill hardware outage. It's hard to know exactly what happened as I'm having trouble parsing some of the post (what is a 'network connector'? is it a cable? nic?). What were some of the 'increasingly outlandish' workarounds? Are they actually standing up production hosts manually, and was that the cause of a delay or unwillingness to get new hardware goin? I think it would be important to have all of that set down either in documentation or code seeing as most of their technical staff are either volunteers, who may come and go, or part timers. Maybe they did, it's not clear.

It's also weird seeing that they are still waiting on their provider to tell them exactly what was done to the hardware to get it going again, that's usually one of the first things a tech mentions: "ok, we replaced the optics in port 1" or "I replaced that cable after seeing increased error rates", something like that.

holsta 7 hours ago | prev | next |

This response and post-mortem is superior to most commercial services I have seen in recent years.

hyperbovine 6 hours ago | root | parent | next |

That's basically every aspect of their service. The founder Thibault Duplessis is criminally undercompensated (his choice) for running a site that is better designed, faster, and more popular than 99% of commercial websites out there.

agentcoops 5 hours ago | root | parent |

I worked with him once on a job -- incredibly nice guy and obviously talented developer who used to work for the French agency responsible for the Scala Play Framework. https://github.com/lichess-org/lila and https://github.com/lichess-org/scalachess are great resources for anyone ever curious to see a production quality Scala3 web application using Cats and all the properly functional properties of the language.

nomilk 7 hours ago | root | parent | prev | next |

Exact same thought went through my head. Also note in the first few paragraphs they acknowledge the worst impacts to users. That's very selfless - often corporate postmortems downplay the impact, which frustrates users more. Incidentally, a critical service I use (Postmark) had an outage this week and I didn't even hear from them (I found out via a random twitter post). Shows the difference.

CSMastermind 6 hours ago | root | parent |

Presumably because Lichess is free thus doesn't have contractual obligations and SLAs that they'll be sued for breaching.

redbell 6 hours ago | root | parent | prev | next |

> so you, as our beneficiaries and stakeholders, who support us and encourage us — deserve to get clarification on what happened

Is it that complicated for big tech to reply politely with the above statement when they suddenly disable your account for no obvious reason!

mewpmewp2 6 hours ago | root | parent |

It may not be complicated, but it does require caring about what you do and your customers as opposed to going through basic minimum requirements to appear that you are doing something.

It is much more difficult for corporate cogs to have that level of care compared to someone who does their things with passion.

morgante 4 hours ago | root | parent | prev |

The post-mortem is honest, but the infrastructure is well below what I'd expect from commercial services.

If a commercial provider told me they're dependent on a single physical server, with no real path or plans to fail over to another server if they need to, I would consider it extremely negligent.

It's fine to not use big cloud providers, but frankly it's pretty incompetent to not have the ability to quickly deploy to a new server.

lukhas an hour ago | root | parent | next |

We're an understaffed charity.

morgante an hour ago | root | parent |

Yeah I'm not criticizing it as a charity, just pointing out this definitely isn't "superior to most commercial services."

That being said, removing dependence on single hardware nodes isn't something you need a big team for. I've done failover at 1-person startups.

KolmogorovComp an hour ago | root | parent | prev |

And yet even Meta recently had a multiple hours downtime, despite a budget thousands if not million times higher. Would you call them negligent too?

By increasing the complexity you multiply the failure points and increase ongoing maintenance, which is the bottleneck (even more than money) for volunteer-driven projects.

morgante an hour ago | root | parent |

To be clear, you don't need to make it more complex / failure-prone. I didn't say failover needs to be automated.

Kubernetes or complex cloud services are not required to have some basic deployment automation.

You can do it with a simple bash script if you need to. It's just pretty surprising to see the reaction to a hardware failure being to wait around for it to be repaired instead of simply spinning up a new host.

ctippett 6 hours ago | prev | next |

Once the private link was reestablished, could they not have tunneled out to the internet via another server acting as a sort of gateway?

Disclaimer: I'm not a network engineer so I may be misunderstanding the practicality and complexity of such a workaround.