ESP32 & ESPHome Voice Assistant

esp32-voice.yaml

esphome:
  name: esp32-mic-speaker
  friendly_name: esp32-mic-speaker
  on_boot:
     - priority: -100
       then:
         - wait_until: api.connected
         - delay: 1s
         - if:
             condition:
               switch.is_on: use_wake_word
             then:
               - voice_assistant.start_continuous:

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable logging
logger:

# Enable Home Assistant API
api:

ota:

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Esp32-Mic-Speaker"
    password: "9vYvAFzzPjuc"

i2s_audio:
  i2s_lrclk_pin: GPIO27
  i2s_bclk_pin: GPIO26

microphone:
  - platform: i2s_audio
    id: mic
    adc_type: external
    i2s_din_pin: GPIO13
    pdm: false

speaker:
  - platform: i2s_audio
    id: big_speaker
    dac_type: external
    i2s_dout_pin: GPIO25
    mode: mono

voice_assistant:
  microphone: mic
  use_wake_word: false
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.0
  speaker: big_speaker
  id: assist

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(assist).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(assist).set_use_wake_word(false);

I am having trouble running an INMP441 and the max 98357a amplifier. They are sharing the bus as in the original design here.
The amplifier is running on the left channel (by default), while the microphone is on the right (pulled up to 3.3V).
The wakeword is working but the speaker is giving an error message: "Parent bus is not free".

Running a minimal sketch with only the speaker I can get the media pipeline to work. Announcement is not outputting anything.

Did anyone encounter the "parent bus not free" error?
Did anyone get this to work on a shared bus?