Elasticsearch v2 Upgrade #315
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Switch the default API version to v2. This updates the couple places our queries need to be changed for v2 compatibility. v1 mode can still be enabled if you're connecting to an external elasticsearch v1 instance. - Create the elasticsearch scripts directory, which v2 seems to require being present or it will fail to startup. - Startup v2 with "mapper.allow_dots_in_name=true". This allows for easier upgrades from v1 installations where the analytics data in older API Umbrella installations (before v0.12.0) may have contained field names with dots. With this flag enabled we can upgrade this older data directly without having to reindex everything. - Tweak how the test suite clean Elasticsearch data between tests. Elasticsearch imposes a lower per query limit, so our queries with size 100,000 were breaking. Instead, we'll switch this to a scan-and-scroll query, which should be better anyway, since now we don't have to worry about any specific size.
After upgrading to elasticsearch v2, some of our logging tests were consistently failing if they happened to follow a number of other tests. The issue was that if a test previously ran that made a bunch of requests, then Elasticsearch wasn't indexing the data quickly enough, causing rsyslog queue to get backed up. This resulted in the requests from the logging tests not getting indexed into Elasticsearch within the expected time. There were 2 main performance issues with Elasticsearch v2 that led to this: 1. Elasticsearch syncs the data to disk on every request in version 2, which hurts indexing performance. We've switched things back v1's mode of performing asynchronous syncs periodically (so we're trading some safety for speed, but that seems okay for this kind of log data). 2. Updating index mappings in Elasticsearch version 2 is more costly. Our mapping is mostly static unless you enable logging of the "request_query" field, which stores all the request query params as a nested object. We have some tests that generate a bunch of unique, random query parameters (mainly for cache busting), but this leads to a deluge of mapping updates since each new query param seen means the mapping needs to be updated. We recently disabled gathering this "request_query" field by default, but it was still enabled in our test suite by default, since we had some existing tests that relied on this functionality. So to solve the performance issues, we've shifted our test suite to disable gathering "request_query" too. This eliminates all the mapping updates during tests. The ability to enabled "request_query" collection still exists, and the existing tests for this functionality have been retained (just in a more isolated fashion that won't impact other tests). However, given the potential performance issues of enabling this, it might be a good reason to get rid of this functionality altogether. While debugging the performance issues, we've also made a couple tweaks and improvements to our rsyslog setup: - The queue.size wasn't configured, so the memory-portion of the queue was capped at 1,000 by default. This meant the configured highwater and lowwater sizes weren't actually being used. Set a higher queue.size to resolve this. - Enable the impstats plugin to output rsyslog queue stats every minute. This seems generally helpful to have in place to be able to see what rsyslog's up to and whether anything is becoming congested with logging. - More comments to explain some of the more cryptic rsyslog configuration settings.
|
Oh, and one more additional note about Elasticsearch v2 indexing performance: On the initial implementation, we were hitting bottlenecks in our test suite with requests not being indexed into Elasticsearch in a timely manner. However, with some tweaking, indexing performance is back to where it should be. But this might be worth highlighting in case anyone's using an external Elasticsearch instance. The short version:
The longer explanation is in the commit message here: c3afad9 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This upgrades the bundled version of Elasticsearch to v2.4 (up from v1.7). Since Elasticsearch v1.7 will be end-of-life'd next month, we need to get on a newer version. Version 2 should also bring various speed and stability benefits.
We're also upgrading to v2 rather than the latest v5, since v2 has an easier upgrade path. Upgrading from v1 to v2 should just require a full cluster restart, whereas jumping to v5 will require reindexing data. At some point we'll probably need to tackle the v5 upgrade, but that at least becomes easier once everything's on v2 (so any old v1 data can be reindexed in place).
It's also worth noting that based on earlier discussions (eg, #248 (comment)), it seemed like upgrading older databases from v1 to v2 was going to be problematic given that our old data (before API Umbrella v0.12) may have contained field names with dots and Elasticsearch v2 had disabled support for any field names with dots (if you tried to upgrade an existing database with dotted field names, Elasticsearch v2 would refuse to start). Luckily, Elasticsearch 2.4.0 rolls back some of these restrictions so older API Umbrella installations (pre API Umbrella v0.12) where dotted field names may have been present can now be upgraded directly without reindexing all the data.
One final note is that while we're upgrading our default bundled version of Elasticsearch to v2, API Umbrella is still compatible with v1 if you happen to be running an external Elasticsearch cluster (rather than relying on the versions bundled with API Umbrella). If you need to continue connecting to an Elasticsearch v1 cluster, then you'll need to update your
/etc/api-umbrella/api-umbrella.ymlconfig to include:However, since Elasticsearch v1 is end-of-life'd next month, I'm thinking we probably won't support this beyond the next release, unless there's a need.