↤ Data Replication Design Spectrum ↑ Blog ↑ S3-Compatible Cloud Storage Cost Calculator ↦

Calling OCaml from C

The official docs on this subject is Interfacing C with OCaml, which is a useful read, though it focuses more on making C callable from OCaml. It’s also missing some collective guidance on how to interact with OCaml values from C and not offend the garbage collector when the C function isn’t being called by OCaml. So, these are my rough notes on how to get all the pieces to fit together. This is subtweeting the ongoing Building BerkeleyDB series, as my personal BerkeleyDB re-implementation is in OCaml.

Basics

I’m going to assume a minimal directory structure, which looks like:

project/
├── bin
│   ├── dune
│   └── main.ml
├── dune-project
├── lib
│   ├── dune
│   ├── ocaml_library.ml
├── project.opam

In this, you have some library ocaml_library, and the goal is to expose some methods it defines to C while still allowing the rest of your OCaml main/tests/etc. modules to work. We’ll accomplish this by adding a new folder for the bindings:

bindings/
├── bindings
    ├── capi.ml
    ├── cstub.c
    └── dune

This new dune file should contain the below:[1] [1]: This was mostly copied from an example on an OCaml forums post.

dune
(executable
 (name capi) (1)
 (libraries ocaml_library) (2)
 (foreign_stubs (language c) (names cstub)) (3)
 (flags :standard -linkall)
 (modes (native shared_object)))

(install (4)
 (section lib)
 (files
  (capi.so as capi-1.0.0.so))) (5)
1 capi here means dune expects a capi.ml file by that exact name.
2 Or whatever your library module name is which defines the public API you wish to export to C. Only code (transitively) included in the specified modules here will be included in the generated .so file.
3 cstubs here means dune expects a cstubs.c file by that exact name.
4 The install section is skippable, and means you’ll need a .opam and/or dune-project file as well.
5 The major point of having this is that install lets you rename the .so, otherwise its name will be your module name + .so.

capi.ml is where you’ll leverage the Callback library to register the functions that you wish to make accessible to C.

capi.ml
let () = Callback.register "ocaml_library_func1" Ocaml_library.func1 (1)
let () =
  let func2_as_string str = (2)
    str |> Bigstring.of_string |> Ocaml_library.func2 |> Bigstring.to_string
  in
  Callback.register "ocaml_library_func2" func2_as_string
1 Associate each function you wish to expose with a unique string.
2 Accessing and manipulating primitive types is easier than OCaml types, so if there’s easy opportunities to turn types into int/string/etc., then it’s sometimes worth the small wrapper function to do so.

cstubs.c is where you’ll implement the C half that knows how to invoke the registered OCaml functions.

The first part we’ll need to ensure is that the OCaml runtime is initialized. If there’s no clean singular entrypoint, then perform a (thread-safe) initialization check within each function exposed.

cstubs.c
#include <caml/alloc.h>
#include <caml/mlvalues.h>
#include <caml/callback.h>
#include <caml/backtrace.h>
#include <caml/printexc.h>

void __caml_init() {
    // Or pthread_once if you need it threadsafe.
    static int once = 0;

    if (once == 0) {
        // Fake an argv by prentending we're an executable `./ocaml_startup`.
        char* argv[] = {"ocaml_startup", NULL};

        // Initialize the OCaml runtime
        caml_startup(argv);
        // If you care about speed, omit this.  But for debugging, it's nice.
        // It's required for the backtrace printing below.
        caml_record_backtraces(1);

        once = 1;
    }
}

We’re also going to use some wrappers around invoking OCaml functions. Classically, one used the caml_invoke API to invoke an OCaml function, which did a sketchy and unsafe encoding when the invoked code used raise or failwith to encode that the result was an exception. OCaml 5.3 introduced a better variant of this API: caml_callback_res. We’ll use a small wrapper around this to turn the caml_result into a value, and print the exception and die if there was an exception.

cstubs.c
// One can get this definition via defining CAML_INTERNALS instead, and
// both methods feel sketchy.
extern void caml_print_exception_backtrace(void);

// You're suppose to use caml_get_value_or_raise, but that raises back
// _into_ OCaml, which isn't what we're trying to do here.
value value_of_result(caml_result result) {
  if (caml_result_is_exception(result)) {
    char *exn_msg = caml_format_exception(result.data);

    fflush(stdout);
    fprintf(stderr, "\nOCaml raised: %s\n", exn_msg);
    caml_print_exception_backtrace();
    fflush(stderr);
    caml_stat_free(exn_msg);
    abort();
  }

  return result.data;
}

value caml_invoke(value func, value arg) {
  return value_of_result(caml_callback_res(func, arg));
}
value caml_invoke2(value func, value arg1, value arg2) {
  return value_of_result(caml_callback2_res(func, arg1, arg2));
}
value caml_invoke3(value func, value arg1, value arg2, value arg3) {
  return value_of_result(caml_callback3_res(func, arg1, arg2, arg3));
}
value caml_invokeN(value func, int n, value args[]) {
  return value_of_result(caml_callbackN_res(func, n, args));
}

Now, with all the preamble out of the way, let’s do the minimal wrapping of an OCaml function. Let’s assume we’re trying to wrap a trivial function of type () -> ():

let library_func1 () = ()
let () = Callback.register "Ocaml_library_func1" library_func1
cstubs.c
void ocaml_library_func1() {
  // Ensure the OCaml runtime is initialized before we invoke anything.
  __caml_init();

  // Fetch the function we registered via Callback.
  static const value* _Ocaml_library_func1 = NULL;
  if (_Ocaml_library_func1 == NULL)
    _Ocaml_library_func1 = caml_named_value("Ocaml_library_func1"); (1)

  // Invoke the function, supplying () as the argument.
  value result = caml_invoke(*_Ocaml_library_func1, Val_unit); (2)
}
1 The unique string for the function.
2 caml_invoke is the cornerstone of this post, as it’s our way to invoke an OCaml function from C.

You should now be able to run dune build or dune install, and see your capi.so file generated! nm -D capi.so will let you double check that ocaml_library_func1 is an exported symbol.

As an unexpected bonus, there’s integration between GDB and OCaml, such that if you manage to segfault OCaml and have backtraces enabled via ocaml_record_backtraces(1), then GDB will tell you the backtrace of the OCaml code as well:

Program received signal SIGSEGV, Segmentation fault.
camlBdb__Cursor.next_2427 () at lib/cursor.ml:20
(gdb) bt
#0  camlBdb__Cursor.next_2427 () at lib/cursor.ml:20
#1  0x00007ffff6c28b30 in camlDune__exe__Capi.cursor_get_2618 () at bindings/capi.ml:25
#2  <signal handler called>
#3  0x00007ffff71ba142 in caml_invoke (closure=<optimized out>, arg=<optimized out>)
    at runtime/callback.c:208
#4  0x00007ffff71ba677 in caml_callbackN_exn (closure=<optimized out>, narg=<optimized out>,
    narg@entry=4, args=<optimized out>, args@entry=0x7fffffffcd80) at runtime/callback.c:294
#5  0x00007ffff71ba689 in caml_callbackN_res (closure=<optimized out>, narg=narg@entry=4,
    args=args@entry=0x7fffffffcd80) at runtime/callback.c:315
#6  0x00007ffff71b3104 in __cursor_get (cursor=0x4ab030, key=0x7fffffffcee8, val=0x7fffffffcec0,
    flags=16) at cstub.c:114

Primitive Types

All OCaml values are represented as one type, named value, which is the size of a pointer, which I’ll claim is 64bits. If the lowest bit of the value is a 1, then it is a 63-bit integer (with value v>>1). If the lowest get of the value is a 0, then it is a pointer to a boxed value.

Boxes all have the same format. There’s a header:

mlvalues.h
/*
For 16-bit and 32-bit architectures:
     +--------+-------+-----+
     | wosize | color | tag |
     +--------+-------+-----+
bits  31    10 9     8 7   0

For 64-bit architectures:

     +----------+--------+-------+-----+
     | reserved | wosize | color | tag |
     +----------+--------+-------+-----+
bits  63    64-R 63-R  10 9     8 7   0

  wosize: Size (in words) of the "fields" part.
  color: The value of the color field of the header.
         This is for use only by the GC.
  tag: The value of the tag field of the header.
*/

And then the header is followed by an array value[] of size wosize fields. All types are contained in this box representation:

Even strings are put into this same format, but treating a string as a tuple of strlen(str)/sizeof(value) fields, with the last byte of the last word telling you how much padding to substract. So the full boxed format of a string looks lke

+--------+-------+-----+------------------------+----+--------+
| wosize | color | tag | string of wosize words | \0 | offset |
+--------+-------+-----+------------------------+----+--------+

and they compute the size via:

size_t string_length(Block *blk) {
  size_t bytesize = blk->wosize * sizeof(value);
  return bytesize - ((char*)blk->fields)[bytesize-1];
}

Thus demonstrating their complete commitment to the one true block format. There are still a few exceptions. The most common is that an array of floats or a record of only floats have a special tag which tells the VM the the array of values is actually unboxed floats, so that math over an array of N floats doesn’t involve N indirect memory accesses. Some other internal things have special representations indicated by a special tag value, like closures, lazy values, or objects, and those are largely opaque to the C API.

If you want to read more about this, go read Real World OCaml: Runtime Memory Layout.

To convert to/from the C native and OCaml value representations, there’s a long list of macros, which are of the form ToType_FromType. OCaml style is to write conversions from X to Y as Y_of_X, and they just removed the "of". In the macros, value is named val, and otherwise, they’re relatively easy to guess. Some examples are:

If you have autocomplete in your editor, it’s relatively easy to just lean on it to guess the right conversion macro rather than looking it up.

However, don’t put conversation macros on the left hand side of an assignment. Code like Double_val(x) = 0.0 will work, but is frowned upon. If you want to store into a value, there’s macros that start with Store_ instead. In this case, Store_double_val(x, 0.0).

For constructing and accessing boxed values, OCaml’s C API also offers a good amount of help:

We’ll see much more of field accessing in a bit.

Garbage Collection Safety

In our minimal example, we’ve ignored all interactions with the garbage collector. This is fine, as the returned () from func1 is immediately garbage anyway, so it’s fine for it to be GC’d at any point. However, as boxes are heap allocated and used for every type except int, we’ll need to be GC-friendly in how we store values.

Let’s assume the function we’d like to wrap looks like:

let make_plural (str : string) : string = str + "s"
let () = Callback.register "make_plural" make_plural

This is something less trivially safe for garbage collection, and we also get an excuse to go through string allocation and handling!

For allocating a string, there’s two options:

And for extracting data out of a string, mlsize_t caml_string_length (value) returns the length of the string, and String_val(value) is a macro which returns the pointer to the beginning of the string.

To prevent accidents, it’s also nice to assert on the tag type of returned values when possible, so that it’s obvious if the types don’t line up across OCaml and C. For strings, that looks like assert(Tag_val(val) == String_tag).

And now, the garbage collection safe pattern:

cstubs.c
char* ocaml_library_func2(char* str_in) {
  __caml_init();

  CAMLparam0(); (1)

  static const value* _Ocaml_make_plural = NULL;
  if (_Ocaml_make_plural == NULL)
    _Ocaml_make_plural = caml_named_value("ocaml_make_plural");

  value ocaml_str_in = caml_copy_string(str_in); (2)

  CAMLlocal1(result); (3)
  result = caml_invoke2(*_Ocaml_make_plural, ocaml_str_in);

  assert(Tag_val(result) == String_tag);
  size_t result_len = caml_string_length(result);
  char* str_out = malloc(result_len);
  memcpy(str_out, String_val(result), result_len);

  CAMLreturnT(char*, str_out); (4)
}
1 Start all functions with CAMLparam0(). The 0 is that the C function takes 0 value arguments given by the OCaml runtime. This is mostly meant for C functions called from OCaml, which isn’t what we’re doing, so it’ll always be 0.
2 We don’t know the string size, so let OCaml do the strlen() call.
3 Use CAMLlocal*() to create locals which are GC-safe. CAMLlocal1(result); is equivalent to value result;, but GC-safe. The number can range from 1 through 5.
4 Use CAMLreturnT instead of return. First argument is your return type, second is the return expression. Most other example code shows CAMLreturn(val), which is equivalent to CAMLreturnT(value, val). Except we aren’t a C function being called from OCaml, so we probably never want to return a value.

This idiom provides a way to ensure that values returned from OCaml stay alive during the local scope of the function. To allow them to stay alive past the end of the function scope, then they need to be registered as a GC root with the OCaml runtime. There’s two ways of registering GC roots offered: caml_register_global_root(value*) and caml_register_generational_global_root(value*). The difference is in how often the pointed-to value will be mutated. If nearly never, then use the latter generational variant. If the pointed-to value is expected to change, then use the former not-generational variant. Both forms of GC roots are un-registered via caml_remove_global_root(value*).

In both cases, the expected usage is to register the GC root immediately after a valid value has been written to the location, and one must not call any other OCaml runtime or allocation function in between. As an example, we have a function which allocates a non-trivial OCaml object, and associated functions to get information about it:

capi.ml
(* Our non-trivial pseudo-object. *)
type t = { s : string }

let make_t_obj () = { s = "hello" }
let () =  Callback.register "make_t_obj" make_t_obj

let t_get_s obj = obj.s
let () = Callback.register "t_get_s" t_get_s

We’d then expose this in C as something like:

cstubs.c
typedef void* ocaml_obj_t; (1)

ocaml_obj_t make_t_obj() {
  __caml_init();
  CAMLparam0();

  static const value* _ocaml_make_t_obj = NULL;
  if (_ocaml_make_t_obj == NULL)
    _ocaml_make_t_obj = caml_named_value("make_t_obj");

  CAMLlocal1(result);
  result = caml_invoke2(*_ocaml_make_t_obj, Val_unit);

  ocaml_obj_t *ocs = malloc(sizeof(ocaml_obj_t));
  *((value*)ocs) = result;
  caml_register_generational_global_root((value*)ocs); (2)

  CAMLreturnT(ocaml_obj_t*, ocs);
}

char* ocaml_obj_t_get_s(ocaml_obj_t* obj) {
  CAMLparam0(); (3)

  static const value* _ocaml_t_get_s = NULL;
  if (_ocaml_t_get_s == NULL)
    _ocaml_t_get_s = caml_named_value("t_get_s");

  CAMLlocal1(result);
  result = caml_invoke2(*_ocaml_t_get_s, *((value*)obj));
  assert(Tag_val(result) == String_tag);

  size_t result_len = caml_string_length(result);
  char* str_out = malloc(result_len);
  memcpy(str_out, String_val(result), result_len);

  CAMLreturnT(char*, str_out);
}

void free_ocaml_obj_t(ocaml_obj_t* obj) {
    caml_remove_global_root(obj); (4)
    free(ocs);
}
1 Expose the ocaml object under some opaque type. We’ll cast it back to value* when needed, but this prevent anything else from knowing it’s an OCaml value.
2 We know our ocaml_obj_t is something written to only once, so the generational variant is appropriate here.
3 obj is already a GC root, so there’s no need to CAMLparam1(obj). Also, note that one wouldn’t call this function without already having called make_t_obj(), so there’s no need to repeat the __caml_init() check.
4 Remove the GC root as part of the normal C flow of destroying and freeing the object.

Tuples aka Product Types

We’ll start with the simplest of structured types: the tuple.

let swap_pair (x,y : int*int) : int*int = (y,x)
let () = Callback.register "swap_pair" swap_pair
typedef struct IntPair {
  int x;
  int y;
} IntPair;

IntPair swap_pair(IntPair pair) {
  __caml_init();
  CAMLparam0();

  CAMLlocal2(result, camlpair);
  camlpair = caml_alloc(2, 0); (1)
  Store_field(camlpair, 0, Val_int(pair.x)); (2)
  Store_field(camlpair, 1, Val_int(pair.y));

  static const value* _ocaml_swap_pair = NULL;
  if (_ocaml_swap_pair == NULL)
    _ocaml_swap_pair = caml_named_value("swap_pair");
  result = caml_invoke(_ocaml_swap_pair, camlpair);

  IntPair ret_pair;
  ret_pair.x = Int_val(Field(result, 0)); (3)
  ret_pair.y = Int_val(Field(result, 1));

  CAMLreturnT(IntPair, ret_pair);
}
1 We’re using the long way here of allocating an empty box with two fields and a tag of zero.
2 Use Store_field to place x and y into fields 0 and 1 of the tuple.
3 Extracting the fields back into x and y is done by accessing field 0 and 1 of the returned tuple.

Record Types

OCaml lets you define records to give names to tuples. Records are represented as a block with a tag of 0, and then each item of the record is a field. It’s a tuple! I’m really appreciating the One True Value Representation that OCaml has going on.

This means that if we return to our example of swapping the two elements:

type coordinate = {
  x : int;
  y : int;
}

let swap_xy (c : coordinate) : coordinate = { x = c.y; y = c.x }

Then the C code for it is the exact same as the swap_tuple implementation.

Variants aka Sum Types

Suppose we have a Dr. Seuss function in OCaml:

type fishtype =
| RedFish of int
| BlueFish of int

let swap_fish (fish : fishtype) : fishtype =
  match fish with
  | RedFish(n) -> BlueFish(n)
  | BlueFish(n) -> RedFish(n)
let () = Callback.register "swap_fish" swap_fish

And our goal is to wrap this into a reasonable C API:

typedef struct FishType {
  enum { REDFISH, BLUEFISH } color;
  int count;
} FishType;

FishType swap_fish(FishType fishtype);

The novel problem here is the sum type. OCaml’s sum types have their constructors internally partitioned into two lists: those that have arguments, and those that do not. Constructors with arguments are represented as a block: a tag (integer) plus a tuple. Constructors without arguments are represented as an integer literal. Each list is constructed top down, and assigned its tag or integer literal starting from zero and increasing by one. Exactly like a C enum. OCaml’s value representation steals the lowest bit to identify integer constants versus a pointer to an allocation, so there’s already a way to identify which list the integer value belongs to. OCaml’s C API calls a tag a tag_t. We continue to use the same field accessor and setter macros.

So the implementation would look like:

FishType swap_fish(FishType fishtype) {
  __caml_init();
  CAMLparam0();

  CAMLlocal2(result, camlfish);
  camlfish = caml_alloc(1, fishtype.color); (1)
  Store_field(camlfish, 0, Val_int(fishtype.count)); (2)

  static const value* _ocaml_swap_fish = NULL;
  if (_ocaml_swap_fish == NULL)
    _ocaml_swap_fish = caml_named_value("swap_fish");
  result = caml_invoke(_ocaml_swap_fish, camlfish);

  FishType ret_fish;
  ret_fish.color = Tag_val(result); (3)
  ret_fish.count = Int_val(Field(result, 0)); (4)

  CAMLreturnT(FishType, ret_fishtype);
}
1 The enum order matches the OCaml fishtypes order, so we can directly use the enum value as the tag value.
2 Both fish colors have one element, so we set the count the same for both.
3 If different constructors take different numbers of arguments, one would instead dispatch on the tag accordingly.
4 Extract the value out of the variant, and convert it to a native int.

Option

An option is just another Sum type:

type 'a option =
| None        (* Int 0 *)
| Some of 'a  (* Tag 0 *)

However, they’re common enough that the C API also gives a few special macros for dealing with options. This time, let’s implement a lifted fun x -> x + 1 function:

capi.ml
let maybe_add_one (x : int option) : int option =
  match x with
  | None -> None
  | Some(v) -> Some(v+1)

let () = Callback.register "maybe_add_one" maybe_add_one

Our C wrapper will look similar, but note that we get some nice and descriptive names to use instead:

cstubs.c
typedef struct optional_integer_t {
  bool present;
  int value;
} optional_integer_t;

optional_integer_t maybe_add_one(optional_integer_t opt_int) {
  __caml_init();
  CAMLparam0();

  static const value* _ocaml_maybe_integer = NULL;
  if (_ocaml_maybe_add_one == NULL)
    _ocaml_maybe_add_one = caml_named_value("maybe_add_one");

  CAMLlocal1(result, input);
  if (opt_int.present) {
    input = caml_alloc_1(Tag_some, Val_int(opt_int.value)); (1)
  } else {
    input = Val_none; (2)
  }
  result = caml_invoke2(*_ocaml_maybe_add_one, input);
  optional_integer_t ret_value;

  if (Is_none(result)) { (3)
    ret_value.present = false;
  } else {
    ret_value.present = true;
    ret_value.value = Int_val(Some_val(result)); (4)
  }

  CAMLreturnT(optional_integer_t, ret_value);
}
1 Tag_some instead of the magic constant 0. Also note the nice caml_alloc_1 form!
2 Val_none instead of the magic Val_int(0).
3 Is_none tests for none, instead of !Is_block(result) or == Val_int(0).
4 Some_Val instead of Field(result, 0).

Nothing life-changing, but the readability is nice!

If you’re wondering, despite OCaml’s result being also a nice and convenient type provided by the standard library, there’s no convenience functions in mlvalues.h for unpacking a result. It’s just Ok is tag 0, Error is tag 1, and you get the value with Field(result, 0) either way.

Lists

Simiarly, a list is implemented as a variant:

type 'a list =
| EmptyList             (* Int 0 *)
| Cons of 'a * 'a list  (* Tag 0 *)

And again similarly, we have a couple nice names given to each case:

Skipping the full example this time, constructing [1; 2; 3] in C would look like:

value int_list =
  caml_alloc_2(Tag_cons, Int_val(1),
    caml_alloc_2(Tag_cons, Int_val(2),
      caml_alloc_2(Tag_cons, Int_val(3),
        Val_emptylist)));
↤ Data Replication Design Spectrum ↑ Blog ↑ S3-Compatible Cloud Storage Cost Calculator ↦

See discussion of this page on Reddit, Hacker News, and Lobsters.