Using MySQL 5.7 libraries on Heroku

README.md

Edit 2021-02-21: I've confirmed this solution works with both heroku-18 and heroku-20 stacks.

---

What follows is my solution for having mysql2 gem compiled and working against MySQL v5.7 (rather than v8+) on the heroku-20 stack. This is how I've managed to make it work:

• Add the apt buildstack ahead of your standard buildpack (in my case, that's Ruby):

$ heroku buildpacks:add --index 1 heroku-community/apt

• Create an Aptfile in the root of your project, and add the MySQL 5.7 client tools (as a URL for the deb file) to it:

# in Aptfile:
https://cdn.mysql.com/Downloads/MySQL-5.7/libmysqlclient-dev_5.7.32-1ubuntu18.04_amd64.deb

• Ensure that the mysql2 gem compiles using the Apt-provided library (via BUNDLE_BUILD__MYSQL2) and that, when the library loads, it knows how to find libstdc++ (via LD_PRELOAD). I've found that LD_PRELOAD isn't already set to anything in my apps - but yours may be different, so do check before running this command.

$ heroku config:set \
  LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 \
  BUNDLE_BUILD__MYSQL2=--with-mysql-config

• Update your app to use the heroku-20 stack if you haven't already:

$ heroku stack:set heroku-20

• Deploy this Aptfile to your app, and it should work. If the gems are cached from a previous build (for example, if you had already changed the stack and deployed other changes), you will need to clear your build cache prior to deploying, to ensure the mysql2 gem is recompiled.

If anyone has any questions or thoughts about this, do let me know.

@pat Hi! If the mysql2 gem works at runtime and not build time, I would suspect that either:

1. the APT buildpack is not setting the necessary LIBRARY_PATH / LD_LIBRARY_PATH env var in its export script (which defines env vars to carry across from one buildpack to the next during the build; the env vars defined in .profile.d/* are separate and for runtime only)
2. the Ruby buildpack is clobbering those values with its own, or otherwise rearranging the order, such that the system mysql libs (of v8) are being found before those installed by the APT buildpack

To debug I would:

1. Run heroku run env to see what's set at runtime (since that config works)
2. Dump the build-time env vars set by the APT buildpack, by inserting https://github.com/tt/heroku-buildpack-env after the APT buildpack and before the Ruby buildpack
3. Dump the env vars seen during the Ruby buildpack rake step, by printing env during one of those rake tasks

My hunch would be that this might be a Ruby buildpack issue, since at first glance the APT buildpack seems to set the same things at both build and runtime, so if this was an APT buildpack bug (of which the APT buildpack has many!), it wouldn't work at runtime either.

Thanks for the comment @edmorley, greatly appreciated!

As it turns out, I've found my approaches documented above have stopped working, so the gem can't be loaded in either buildtime or runtime. The error is the same as the above LoadError, just with paths adjusted to be in /app. I've tried looking into the env vars, and as best as I can tell the Ruby buildpack doesn't clobber most env variables when compiling gems - though it does prioritise some paths.

Using the env buildpack, the LIBRARY_PATH and LD_LIBRARY_PATH look correct - both refer to the buildtime path for the apt buildpack:

LIBRARY_PATH=/tmp/build_e33615f8/.apt/usr/lib/x86_64-linux-gnu:/tmp/build_e33615f8/.apt/usr/lib/i386-linux-gnu:/tmp/build_e33615f8/.apt/usr/lib:
LD_LIBRARY_PATH=/tmp/build_e33615f8/.apt/usr/lib/x86_64-linux-gnu:/tmp/build_e33615f8/.apt/usr/lib/i386-linux-gnu:/tmp/build_e33615f8/.apt/usr/lib:

They look the same as when the rake task is invoked via the Ruby buildpack:

LIBRARY_PATH=/tmp/build_ceaf7676/.apt/usr/lib/x86_64-linux-gnu:/tmp/build_ceaf7676/.apt/usr/lib/i386-linux-gnu:/tmp/build_ceaf7676/.apt/usr/lib:
LD_LIBRARY_PATH=/tmp/build_ceaf7676/.apt/usr/lib/x86_64-linux-gnu:/tmp/build_ceaf7676/.apt/usr/lib/i386-linux-gnu:/tmp/build_ceaf7676/.apt/usr/lib:

They're also set correctly at runtime:

LIBRARY_PATH=/app/.apt/usr/lib/x86_64-linux-gnu:/app/.apt/usr/lib/i386-linux-gnu:/app/.apt/usr/lib:
LD_LIBRARY_PATH=/app/.apt/usr/lib/x86_64-linux-gnu:/app/.apt/usr/lib/i386-linux-gnu:/app/.apt/usr/lib:

And neither of these variables point to the existing MySQL v8 library in /lib/x86_64-linux-gnu/. I've tried telling the gem compiler to use mysql_config for the right details (and when I've gotten that env variable setting wrong, the gem fails to compile, which suggests that's flowing through fine too). mysql_config points to the apt paths too. 🤷🏻‍♂️

I've just asked for any ideas in the mysql2 repo - I feel it's quite an edge case, but maybe someone there can point me in the right direction. Certainly, any further ideas from yourself would also be very much appreciated!

Made some progress: setting LD_PRELOAD to point to libstdc++ has allowed mysql2 to load successfully. Everything works smoothly on heroku-18, but not yet on heroku-20 - but at least this removes any need for me to use heroku-16.

If I get time, I might take another stab at getting it working on heroku-20, but it's no longer a high priority, so I'm not making any promises 😅

Last week's brain fade: I didn't have the apt buildpack in the app I was migrating (only in my test app) 🤦🏻‍♂️ So, actually: this solution does indeed work on the heroku-20 and heroku-18 stacks.

Hi Pat,

Been a long time since I thought about Thinking Sphinx, setup and stability is top notch. Great work, my project is purring along nicely for years.

Thank you for sending out a message about the change. What was not mentioned, was if upgrading to Thinking Sphinx v5 would also solve the issue? Would it then automatically use Sphinx v3 on Heroku?

Was wondering if upgrading the gems is a more future proof approach. I could not find info if the mysql2 gem has been upgraded to work with MySQL v8 libraries on Heroku-20 stack or at what release of Thinking Sphinx would be compatible with Sphinx v3.

Heroku-18 stack, Sinatra App
ruby 2.4.1p111
mysql2 (0.5.2)
flying-sphinx (2.1.4)
thinking-sphinx (4.3.0)
activerecord (4.2.11.1)

Warm regards,
Conor

Hi Conor, great to know things have been super stable for you :) and those are great questions!

With regards to the mysql2 gem, my understanding is that it can be compiled for either v5.7 or v8 support, but not both. Sphinx prior to the v3.1.1 release only supported MySQL v5.7 client libraries (and I realise it's worth noting, for anyone not aware: Sphinx's main query approach uses the MySQL protocol, hence this dependency).

Flying Sphinx doesn't have Sphinx v3.3.1 available, only v3.1.1 and v3.0.3 of the 3.x releases. Unfortunately, I've found that all v3.x releases do not work well when you're using PostgreSQL as your database and with SQL-backed indices. This is the most common approach for Flying Sphinx customers (understandably, given the ease of using PostgreSQL on Heroku), hence I've held off on adding v3.3.1 to the Flying Sphinx servers.

It's also important to note that - at least at this point - Flying Sphinx's servers are all expecting the MySQL v5.7 protocol, and I'm not sure if v8 will work at all. Managing both protocols at the same time isn't something I've yet looked into, and I strongly recommend sticking with v5.7 protocols (and thus, using this gist's approach).

With all of that in mind, I recommend one of the three options:

• use Sphinx v2.2.11 with SQL-backed indices (i.e. using :with => :active_record in your index definitions). This means the changes outlined in this gist are required, because you must use the MySQL v5.7 client libraries.
• use Manticore, which is a fork of Sphinx and works with Thinking Sphinx automatically. Flying Sphinx has v2.7.5 available, and I'll look into getting a v3 release of that onto the servers as well. Manticore has supported MySQL v8 client libraries since v2.7.2, but as noted above, Flying Sphinx's servers may not work with a v8 approach.
• use Sphinx v2.2.11 or v3.1.1 with real-time indices. This is the approach I use in my own apps, though I recognise the processing time for indexing can be much slower than SQL-backed indices. However, it removes the need to have deltas, and thus regular reprocessing (ts:index) of indices doesn't need to happen so often (ideally the callbacks cover all changes, but it's good to have reprocessing take place nonetheless - especially if you're adding/editing data in ways that don't fire ActiveRecord callbacks).

Thinking Sphinx, even the latest v5 releases, continues to support Sphinx v2.2.11. But upgrading Thinking Sphinx to v5 will not change which version of the MySQL client libraries are required. That's entirely tied to Sphinx/Manticore versions. If/when you do upgrade, please see the documentation about what must be changed - v5 removed implicit callbacks, so you'll want to add them in yourself.

Managing Sphinx/Manticore versions for Flying Sphinx apps is all done via config/thinking_sphinx.yml and the version and engine keys. For example:

production:
  version: 2.7.5
  engine: manticore

# or …

production:
  version: 2.2.11
  engine: sphinx

The defaults are Sphinx and v2.2.11 for all new apps. Older apps may have a different (older) Sphinx version if they've not configured one.

I hope this covers all of your questions - but certainly do ask if there are things I've missed :)

Thank you so much for being so thorough and informative. Makes working with a 2 year old code base easier.

I had forgotten about config/thinking_sphinx.yml, I have upgraded to version 2.2.11, explicitly added "engine:" and integarated the build pack apt to Heroku as mentioned above. Everything is working on Heroku-18 after flushing the cache and redeploying. This way I'll be ready for Heroku-20 in the future.

Don't work too hard on any of those complex integrations, I am a happy Flying Sphinx client using MySQL database and delta indexing.

Hi Pat,

We've a container-stack based Rails Heroku app, and our CI is heroku-16 stack. I've followed the instructions here, set the stack in app.json to heroku-20 for the test env (tried heroku-18 at some point as well) and set the LD_PRELOAD & BUNDLE_BUILD__MYSQL2 ENV variables (in app.json for test env). While the test setup phase is going fine over the CI, some Thinking Sphinx tests are now failing and I'm not sure why. I should mention we've inherited this app and our team has small-to-none experience working with TS, so we've no idea where to look for more exactly. Here's more info:

Ruby:2.7.2-buster
mysql2 0.5.3
pg 1.2.3
thinking-sphinx 5.0.0
flying-sphinx 2.2.0
activerecord 5.2.4.4

 test_0002_Searching by invalid filter type raises error when NOT in production#SomeTest (174.32s)
        ThinkingSphinx::SphinxError expected but nothing was raised.
        test/integration/some_test.rb:31:in `block in <class:SomeTest>'
�[1000D�[K FAIL["test_0001_Searching by invalid filter type returns empty results page when in production", #<Minitest::Reporters::Suite:0x000056397a3864e0 @name="SomeTest">, 178.64281626208685]
 test_0001_Searching by invalid filter type returns empty results page when in production#SomeTest (178.64s)
        Expected at least 1 element matching ".qwerty", found 0..
        Expected 0 to be >= 1.
        test/integration/some_test.rb:26:in `block in <class:SomeTest>'

Tried specifying the engine/version, tests still fail. They passed before doing this update.

I've no idea where to look at this point, so what I guess I'm looking for are some directions, please. Thanks!

Hi Alex - I'm not sure what the cause of this issue is, but perhaps you can share that test (and the corresponding index definitions) with me at http://support.flying-sphinx.com? It'll be easier to discuss the specifics there :)

Hi Pat. I opened a ticket a few days ago, but there's no reply from anyone yet. Hoping to hear from someone soon to get this issue fixed :) Cheers

Quick update, the URL for the download changed. If you get a 404 error from curl (22), you need to update to:

https://dev.mysql.com/get/Downloads/MySQL-5.7/libmysqlclient-dev_5.7.32-1ubuntu18.04_amd64.deb