Basics of generating a compile_commands.json file with Bazel

BUILD

py_binary(
  name = 'generate_compile_command',
  srcs = [
    'generate_compile_command.py',
  ],
  deps = [
    '//third_party/bazel:extra_actions_proto_py',
  ],
)

action_listener(
  name = 'generate_compile_commands_listener',
  visibility = ['//visibility:public'],
  mnemonics = [
    'CppCompile',
  ],
  extra_actions = [':generate_compile_commands_action'],
)

extra_action(
  name = 'generate_compile_commands_action',
  tools = [
    ':generate_compile_command',
  ],
  out_templates = [
    '$(ACTION_ID)_compile_command',
  ],
  cmd = '$(location :generate_compile_command) $(EXTRA_ACTION_FILE)' +
        ' $(output $(ACTION_ID)_compile_command)',
)

generate_compile_command.py

# This is the implementation of a Bazel extra_action which genenerates
# _compile_command files for generate_compile_commands.py to consume.

import sys

import third_party.bazel.protos.extra_actions_base_pb2 as extra_actions_base_pb2

def _get_cpp_command(cpp_compile_info):
  compiler = cpp_compile_info.tool
  options = ' '.join(cpp_compile_info.compiler_option)
  source = cpp_compile_info.source_file
  output = cpp_compile_info.output_file
  return '%s %s -c %s -o %s' % (compiler, options, source, output), source

def main(argv):
  action = extra_actions_base_pb2.ExtraActionInfo()
  with open(argv[1], 'rb') as f:
    action.MergeFromString(f.read())
  command, source_file = _get_cpp_command(
      action.Extensions[extra_actions_base_pb2.CppCompileInfo.cpp_compile_info])
  with open(argv[2], 'w') as f:
    f.write(command)
    f.write('\0')
    f.write(source_file)

if __name__ == '__main__':
  sys.exit(main(sys.argv))

generate_compile_commands_json.py

#!/usr/bin/python3

# This reads the _compile_command files :generate_compile_commands_action
# generates a outputs a compile_commands.json file at the top of the source
# tree for things like clang-tidy to read.

# Overall usage directions: run bazel with
# --experimental_action_listener=//tools/actions:generate_compile_commands_listener
# for all the files you want to use clang-tidy with and then run this script.
# Afer that, `clang-tidy build_tests/gflags.cc` should work.

import sys
import pathlib
import os.path
import subprocess

'''
Args:
  path: The pathlib.Path to _compile_command file.
  command_directory: The directory commands are run from.

Returns a string to stick in compile_commands.json.
'''
def _get_command(path, command_directory):
  with path.open('r') as f:
    contents = f.read().split('\0')
    if len(contents) != 2:
      # Old/incomplete file or something; silently ignore it.
      return None
    return '''{
        "directory": "%s",
        "command": "%s",
        "file": "%s",
      },''' % (command_directory, contents[0].replace('"', '\\"'), contents[1])

'''
Args:
  path: A directory pathlib.Path to look for _compile_command files under.
  command_directory: The directory commands are run from.

Yields strings to stick in compile_commands.json.
'''
def _get_compile_commands(path, command_directory):
  for f in path.iterdir():
    if f.is_dir():
      yield from _get_compile_commands(f, command_directory)
    elif f.name.endswith('_compile_command'):
      command = _get_command(f, command_directory)
      if command:
        yield command

def main(argv):
  source_path = os.path.join(os.path.dirname(__file__), '../..')
  action_outs = os.path.join(source_path,
                             'bazel-bin/../extra_actions',
                             'tools/actions/generate_compile_commands_action')
  command_directory = subprocess.check_output(
      ('bazel', 'info', 'execution_root'),
      cwd=source_path).decode('utf-8').rstrip()
  commands = _get_compile_commands(pathlib.Path(action_outs), command_directory)
  with open(os.path.join(source_path, 'compile_commands.json'), 'w') as f:
    f.write('[')
    for command in commands:
      f.write(command)
    f.write(']')

if __name__ == '__main__':
  sys.exit(main(sys.argv))

I had quite some trouble to understand where which file goes and what to do with them, so here is how I got it to work: (under Linux / Arch)
(and see the file-tree at the bottom to make it clearer where which file goes)

in your WORKSPACE root, create /tools/actions folder(s)
create all files from above in there.

download / create the file from (github bazel source) bazel/src/main/protobuf/extra_actions_base.proto (anywhere really, we don't need the proto file afterwards)
and run protoc extra_actions_base.proto --python_out=.
(you need to have protocol buffers installed on your system)

create a new folder (relative to thoe WORKSPACE root) /third_party/bazel/protos and move the Python file we just generated (extra_actions_base_pb2.py) there.
in /third_party/bazel (not /protos) we now create a BUILD file with the following content:

licenses(["notice"])

py_library(
    name = "extra_actions_proto_py",
    srcs = ["protos/extra_actions_base_pb2.py"],
    visibility = ["//visibility:public"],
)

now make sure your python has google.protobuf installed ( [sudo] pip install protobuf )

now if you run the bazel build command, i.e.
bazel build --experimental_action_listener=//tools/actions:generate_compile_commands_listener main:hello-world
it should work, i.e. not throw any errors.

Then do (from WORKSPACE root) cd tools/actions and from this folder you run the _json script:
python generate_compile_commands_json.py

Now there should be a compile_commands.json in your WORKSPACE root folder. Congratulations, it worked! :)

Below a file tree to make it easier to see where all the files are. The project is just the bazel C++ tutorial.

.
├── compile_commands.json
├── extra_actions_base.proto  <- This can be anywhere and can be deleted as soon as we have the .py file
├── gtest.BUILD
├── lib
│   ├── BUILD
│   ├── hello-greet.cc
│   └── hello-greet.h
├── main
│   ├── BUILD
│   ├── hello-time.cc
│   ├── hello-time.h
│   ├── hello-world.cc
├── test
│   ├── BUILD
│   └── hello-test.cc
├── third_party
│   └── bazel
│       ├── BUILD
│       └── protos
│           └── extra_actions_base_pb2.py
├── tools
│   └── actions
│       ├── BUILD
│       ├── generate_compile_command.py
│       └── generate_compile_commands_json.py
└── WORKSPACE

Hope it helps someone ¯\(ツ)/¯

@mmlac Thanks.

Added that document as an example into the discussion in https://docs.google.com/document/d/1QKT7sxS9DzQspni-QZajocozmAad1Cjvs7jv-X_0Of0/edit?usp=sharing.

@bsilver8192, @mmlac Many thanks for these explanations.
I embedded the codes in a bash script for ease of use. This is available in my GitHub repo.

Self-promoting plug here: I wrote Bazel rules to generate the compilation database. It is not perfect but will work for most cases, and requires much less setup, as simple as copying a file and running a script. And will work even if the code does not compile.

https://github.com/grailbio/bazel-compilation-database

I had quite some trouble to understand where which file goes and what to do with them, so here is how I got it to work: (under Linux / Arch)
(and see the file-tree at the bottom to make it clearer where which file goes)

in your WORKSPACE root, create /tools/actions folder(s)
create all files from above in there.

download / create the file from (github bazel source) bazel/src/main/protobuf/extra_actions_base.proto (anywhere really, we don't need the proto file afterwards)
and run protoc extra_actions_base.proto --python_out=.
(you need to have protocol buffers installed on your system)

create a new folder (relative to thoe WORKSPACE root) /third_party/bazel/protos and move the Python file we just generated (extra_actions_base_pb2.py) there.
in /third_party/bazel (not /protos) we now create a BUILD file with the following content:

licenses(["notice"])

py_library(
    name = "extra_actions_proto_py",
    srcs = ["protos/extra_actions_base_pb2.py"],
    visibility = ["//visibility:public"],
)

now make sure your python has google.protobuf installed ( [sudo] pip install protobuf )

now if you run the bazel build command, i.e.
bazel build --experimental_action_listener=//tools/actions:generate_compile_commands_listener main:hello-world
it should work, i.e. not throw any errors.

Then do (from WORKSPACE root) cd tools/actions and from this folder you run the _json script:
python generate_compile_commands_json.py

Now there should be a compile_commands.json in your WORKSPACE root folder. Congratulations, it worked! :)

Below a file tree to make it easier to see where all the files are. The project is just the bazel C++ tutorial.

.
├── compile_commands.json
├── extra_actions_base.proto  <- This can be anywhere and can be deleted as soon as we have the .py file
├── gtest.BUILD
├── lib
│   ├── BUILD
│   ├── hello-greet.cc
│   └── hello-greet.h
├── main
│   ├── BUILD
│   ├── hello-time.cc
│   ├── hello-time.h
│   ├── hello-world.cc
├── test
│   ├── BUILD
│   └── hello-test.cc
├── third_party
│   └── bazel
│       ├── BUILD
│       └── protos
│           └── extra_actions_base_pb2.py
├── tools
│   └── actions
│       ├── BUILD
│       ├── generate_compile_command.py
│       └── generate_compile_commands_json.py
└── WORKSPACE

Hope it helps someone ¯_(ツ)_/¯

Hi @mmlac !
I am trying to follow this. My system uses Python 2.7 version and hence does not support "yield from" command in generate_compile_commands.json .
Is there an alternate to that line of code which will be compatible on python 2.7.

Thank you ! Would really appreciate your help.

Unfortunately, the example code produces compile commands that confuse (modern?) compilers. Specifically line 13 in generate_compile_command.py results in duplicate "-o" and "-c" options which both gcc (version 10.2.0) and clang (version 11.0.0) complain about.

Also, the result is not valid JSON but that doesn't seem to bother ccls or clangd 😄

Here's the code I'm using right now. I might refactor this into a dedicated repository at some point 😄
https://github.com/kdungs/adventofcode/tree/main/2020/tools/actions

---

My system uses Python 2.7 version and hence does not support "yield from"

IMHO you should use Python 3 🙈 But until then,

for foo in bar:
    yield foo

should do the trick.

For anybody else looking at this: I (the author of the gist) have since moved to https://github.com/grailbio/bazel-compilation-database. The aspect-based approach is definitively better IMHO with the improved CC Starlark API since I threw this together. It has better fidelity and it's less brittle. You can see a little fix from me at grailbio/bazel-compilation-database@9682280, and then it works great on my codebases.

Thanks a lot for the clarification 😄

It's good to have this here for posterity since this gist is referenced in a lot of places and one of the first results in any search related to Bazel and compile commands.

Happy holidays!

For anyone landing here, we'd highly recommend using https://github.com/hedronvision/bazel-compile-commands-extractor, just released.

[It's fast and less brittle thanks to an aquery-based approach that directly asks Bazel what build commands it would run. The grail bio/bazel-compilation-database creator had some nice things to say about it and passing the torch. (See here for more)]

This works great with bazel 0.5.3 on Ubuntu 18.04.

Yesterday I tried Ubuntu 16.04, to install the Python 2.7's package protobuf, you need to do it as follows:

sudo apt install python-pip
sudo pip install protobuf=3.17.3

Otherwise the default installed protobuf with version 3.19+ is not working.

@bsilver8192 could you add a license to this gist? (preferably MIT)