For fun I thought I would see how hard it is to write an engine for OpenSSL. There are several existing ones that you can look at. I started by seeing how the opensc engine worked. This code shows the first step.

#include <stdio.h>
#include <string.h>
#include <openssl/crypto.h>
#include <openssl/objects.h>
#include <openssl/engine.h>

static int bind_fn(ENGINE * e, const char *id)
{
  if (!ENGINE_set_id(e, "simple") ||
      !ENGINE_set_name(e, "simple engine")) {
    return 0;
  } else {
    return 1;
  }
}

IMPLEMENT_DYNAMIC_CHECK_FN();
IMPLEMENT_DYNAMIC_BIND_FN(bind_fn);

Compile it like this

gcc -c -fpic simple_engine.c
gcc -shared -o simple_engine.so simple_engine.o

Make openssl.cnf look like this

openssl_conf            = openssl_def

[openssl_def]
engines = engine_section

[engine_section]
simple = simple_section

[simple_section]
engine_id = simple
dynamic_path = /path/to/simple_engine.so
init = 0

[req]
distinguished_name = req_distinguished_name

[req_distinguished_name]

Run OpenSSL and see your results

$ openssl engine
(padlock) VIA PadLock (no-RNG, no-ACE)
(dynamic) Dynamic engine loading support
(simple) simple engine

Of course it doesn’t do anything useful yet. But it is a start.