So, a funny thing happened when CCP released the latest CREST update— things got a lot faster. Instead of pulling a single item type per region per request, CREST was now allowing us to pull ALL market orders for a region, up to 10,000 results per request. Needless to say, things started picking up speed.
The downside was that the site wasn’t built to handle this sort of throughput. Log files that would usually be archived and moved every day were suddenly filling up our entire 100+GB partition in a matter of minutes, causing the site to hang. Along with memory cache issues because of the now massive data arrays we were creating with the new volume, Eve-Marketdata’s server stumbled back every time it tried to bring itself back up.
TLDR: We went from sipping from a straw to trying to drink from a fire hose and made a bit of a mess before we got used to the flow. Everything is back up and running now and you’ll notice that we’re now getting MUCH more frequent market updates. Hooray!