Site Downtime/CREST Unleashed

So, a funny thing happened when CCP released the latest CREST update— things got a lot faster. Instead of pulling a single item type per region per request, CREST was now allowing us to pull ALL market orders for a region, up to 10,000 results per request. Needless to say, things started picking up speed.

The downside was that the site wasn’t built to handle this sort of throughput. Log files that would usually be archived and moved every day were suddenly filling up our entire 100+GB partition in a matter of minutes, causing the site to hang. Along with memory cache issues because of the now massive data arrays we were creating with the new volume, Eve-Marketdata’s server stumbled back every time it tried to bring itself back up.


TLDR: We went from sipping from a straw to trying to drink from a fire hose and made a bit of a mess before we got used to the flow. Everything is back up and running now and you’ll notice that we’re now getting MUCH more frequent market updates. Hooray!



13 responses to “Site Downtime/CREST Unleashed

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: