Site Downtime/CREST Unleashed

So, a funny thing happened when CCP released the latest CREST update— things got a lot faster. Instead of pulling a single item type per region per request, CREST was now allowing us to pull ALL market orders for a region, up to 10,000 results per request. Needless to say, things started picking up speed.

The downside was that the site wasn’t built to handle this sort of throughput. Log files that would usually be archived and moved every day were suddenly filling up our entire 100+GB partition in a matter of minutes, causing the site to hang. Along with memory cache issues because of the now massive data arrays we were creating with the new volume, Eve-Marketdata’s server stumbled back every time it tried to bring itself back up.

w1X4f3j

TLDR: We went from sipping from a straw to trying to drink from a fire hose and made a bit of a mess before we got used to the flow. Everything is back up and running now and you’ll notice that we’re now getting MUCH more frequent market updates. Hooray!

Krastor

Advertisements

13 responses to “Site Downtime/CREST Unleashed

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: