The Relicans

Kirk Haines
Kirk Haines

Posted on • Updated on

Writing Crystal Bindings for the New Relic C SDK

Background

As stated in my previous post, New Relic directly supports 8 languages with agent libraries for integrating with your own software in order to add observability into your systems -- C/C++, Go, Java, .Net, Node.js, PHP, Python, and Ruby. In addition, there are some languages that have open-source, as-is language support from New Relic, like Elixir, and some languages that have done the same thing that this article is discussing in order to make O11y with New Relic available to it, like Rust

For languages that support binding to a C library, those bindings represent the easiest way to add that language to the list of languages that one can instrument with New Relic.

The Crystal language has excellent first-class support for binding with C libraries, and with its recently announced 1.0 release, there will be a growing number of people using it. This seems like an excellent time to introduce the start of a library to support the use of New Relic for observability within your Crystal software, whether they be web apps or non-web apps.

This article is going to walk through the process of creating that initial shard to support New Relic, from the start through the implementation to the release of a public version 0.1.0.

The Starting Point

There are two ways to write support for New Relic for a language. The first is to implement all of the low-level API interactions directly in the language. This is how most of the New Relic agent libraries for the languages that are supported work. So, for example, the New Relic Agent for Ruby implements all of the New Relic API communications in Ruby.

This is arguably the most ideal approach, as the semantics of the interactions with the API can be most easily tuned to the relevant language, and users of the language that the agent is written for can also directly participate in the maintenance and growth of the agent itself. This is also arguably the more difficult approach, as a lot of the most basic capabilities of the agent have to be developed from scratch.

The other approach, and the one that I am using for this initial release of the NewRelic.cr library, is to leverage the New Relic C SDK for all of the heavy lifting. The Crystal code will be the glue between Crystal and the C SDK, implementing a more Crystal-like interface to it, but all of the work will be handled within the C SDK and the accompanying New Relic Daemon.

Creating A Shard

This is covered in greater detail in the Crystal documentation, but Crystal provides a built-in tool to make it easy to get started with writing a new shard:

crystal init lib NewRelic.cr
Enter fullscreen mode Exit fullscreen mode

This will create a starting point for our shard, complete with everything that we need to hit the ground running.

After doing all of the Git things to set up a new repository with the stub created by crystal init as its foundation, I added the New Relic C SDK as a submodule, since in this first iteration, the Crystal language support for New Relic will be built on top of that SDK.

git submodule add git@github.com:newrelic/c-sdk.git
Enter fullscreen mode Exit fullscreen mode

It's Time To Have Some Fun!

Referring back to the earlier article about the SDK, one can see that the first step to using it is to create a configuration data structure using the newrelic_create_app_config() function. So, let's start there and see what it looks like to write some Crystal to wrap that function and to make it callable.

The library wrapping features of Crystal are first-class members of the language, which means that one can write all of the glue to connect Crystal to a give C library in Crystal -- there is no need to drop to C or some other intermediary language.

@[Link("newrelic")]
lib NewRelic
end
Enter fullscreen mode Exit fullscreen mode

That code tells Crystal to link to libnewrelic.so, and to create a binding (which will, from a language usage perspective, seem like a module that has class methods defined on it for one to use).

Writing A Binding

By itself, there isn't anything useful there, though. Crystal doesn't know how to call any of the functions defined within the library. It needs some more glue:

lib NewRelic
  fun newrelic_create_app_config(app_name : LibC::Char*, license_key : LibC::Char*)
end
Enter fullscreen mode Exit fullscreen mode

That new line tells crystal that there is a function, newrelic_create_app_config(), that takes two arguments, app_name and license_key. In the C prototype for that function, the argument types are specified as a const char *, with is a pointer to an array of characters.

In Crystal, that would be the same as a pointer to an array of UInt8 (8-bit unsigned integers), but Crystal provides a number of type wrappings in its own LibC binding, in order to make things look and read more clearly for the developer. LibC::Char is one of those things.

Technically, that is all that is necessary in order to make this first function callable from within Crystal.

The Return Types Gotcha

There is a gotcha, though. If you look at the prototype for that config function, it is clear that it returns something. It returns a pointer to a newrelic_app_config_t structure.

If you try to write some Crystal to do that:

config = NewRelic.newrelic_create_app_config("experiment",apikey)
Enter fullscreen mode Exit fullscreen mode

It will fail to compile because the signature defined in the C binding for the library does not indicate any sort of return type for the function. A function defined in a fun call, with no return type specified, is assumed to be a Void function. That is, it does not return a value.

From the point of view of the Crystal binding to the SDK, though, Crystal doesn't know anything about the type definition for the structure that is to be returned. It has to be given this information before the function definition that will allow this function to be called can be completed.

A peek at the definition of this structure in the C SDK might be helpful (comments eliminated for brevity):

typedef struct _newrelic_app_config_t {
  char app_name[255];
  char license_key[255];
  char redirect_collector[100];
  char log_filename[512];
  newrelic_loglevel_t log_level;
  newrelic_transaction_tracer_config_t transaction_tracer;
  newrelic_datastore_segment_config_t datastore_tracer;
  newrelic_distributed_tracing_config_t distributed_tracing;
  newrelic_span_event_config_t span_events;
} newrelic_app_config_t;
Enter fullscreen mode Exit fullscreen mode

Crystal Can Do C Structs!

This is starting to get complicated. In Crystal, one can define a C level struct inside of a lib by doing this:

struct NewrelicAppConfigT
  app_name : LibC::Char[255]
  license_key : LibC::Char[255]
  redirect_collector : LibC::Char[100]
  log_filename : LibC::Char[512]
end
Enter fullscreen mode Exit fullscreen mode

Everything there is very direct until one gets down to that log_level field. It has a type of newrelic_loglevel_t, which Crystal also does not know about. So, we can define that, too. In looking at the C SDK for that typedef, it can be seen that it is a simple enum.

typedef enum _newrelic_loglevel_t {
  NEWRELIC_LOG_ERROR,
  NEWRELIC_LOG_WARNING,
  NEWRELIC_LOG_INFO,
  NEWRELIC_LOG_DEBUG,
} newrelic_loglevel_t;
Enter fullscreen mode Exit fullscreen mode

And Enums!

Crystal can support this directly, too. Remember that in C, an enum just assigns a monotonically increasing series of numbers, starting from 0, to each element.

enum NewrelicLoglevelT
  NewrelicLogError   = 0
  NewrelicLogWarning = 1
  NewrelicLogInfo    = 2
  NewrelicLogDebug   = 3
end
Enter fullscreen mode Exit fullscreen mode

A Little Closer...

With that defined, we are a little closer to having a complete NewrelicAppConfigT structure:

struct NewrelicAppConfigT
  app_name : LibC::Char[255]
  license_key : LibC::Char[255]
  redirect_collector : LibC::Char[100]
  log_filename : LibC::Char[512]
  log_level : NewrelicLogLevelT
end
Enter fullscreen mode Exit fullscreen mode

Referring back to the C header for the structure, though, there are several more data types to define. While it is certainly not a difficult task to manually define all of the rest of these, it is tedious to do so.

A Shortcut Around The Boilerplate

However, there does exist a very useful utility that can automate a lot of this work. This utility, crystal-lib, can autogenerate bindings. It is considered experimental, and there are cases where it will not work right, as well as cases where even if it does work right, what is generated may need to be altered before it is really ready for use, but even in those cases, this utility can save a lot of tedious typing while building a binding to a C library.

It works on the New Relic C SDK, although it does not work perfectly. There are some things, such as cases where it declares a type that is better declared as an alias but using it is still a huge win.

The end result for the above set of definitions that I had originally started writing by hand, looks similar to this:

lib NewRelicExt
  enum X_NewrelicLoglevelT
    NewrelicLogError   = 0
    NewrelicLogWarning = 1
    NewrelicLogInfo    = 2
    NewrelicLogDebug   = 3
  end
  alias LoglevelT = X_NewrelicLoglevelT

  struct X_NewrelicTransactionTracerConfigT
    enabled : Bool
    threshold : TransactionTracerThresholdT
    duration_us : TimeUsT
    stack_trace_threshold_us : TimeUsT
    datastore_reporting : X_NewrelicTransactionTracerConfigTDatastoreReporting
  end
  type TransactionTracerConfigT = X_NewrelicTransactionTracerConfigT

  struct X_NewrelicDatastoreSegmentConfigT
    instance_reporting : Bool
    database_name_reporting : Bool
  end
  type DatastoreSegmentConfigT = X_NewrelicDatastoreSegmentConfigT

  struct X_NewrelicDistributedTracingConfigT
    enabled : Bool
  end
  type DistributedTracingConfigT = X_NewrelicDistributedTracingConfigT

  struct X_NewrelicSpanEventConfigT
    enabled : Bool
  end
  type SpanEventConfigT = X_NewrelicSpanEventConfigT

  struct X_NewrelicAppConfigT
    app_name : LibC::Char[255]
    license_key : LibC::Char[255]
    redirect_collector : LibC::Char[100]
    log_filename : LibC::Char[512]
    log_level : LoglevelT
    transaction_tracer : TransactionTracerConfigT
    datastore_tracer : DatastoreSegmentConfigT
    distributed_tracing : DistributedTracingConfigT
    span_events : SpanEventConfigT
  end
  type AppConfigT = X_NewrelicAppConfigT
end
Enter fullscreen mode Exit fullscreen mode

And with all of those structs and types defined, almost automatically, a fully working binding to the newrelic_create_app_config() function can be written.

  fun create_app_config = newrelic_create_app_config(app_name : LibC::Char*, license_key : LibC::Char*) : AppConfigT*
Enter fullscreen mode Exit fullscreen mode

Et Voila! A Working Binding?

In this example, the name of the function on the Crystal side is shortened. Since the lib is already NewRelic, it didn't make any sense to keep the newrelic_ in the function name, too; that would result in invocations that looked like NewRelic.newrelic_create_app_config(app_name, license_key). With the code above, it is NewRelic.create_app_config(app_name, license_key).

The final result of this first pass at creating bindings for the C SDK resulted in a nominally working library that more or less exactly preserved the C library idioms.

Translating the sample program from the C SDK Guide to Crystal results in something that works, but doesn't completely look like idiomatic Crystal, though it is a lot more concise than the C language version:

require "new_relic"

license_key = File.read("apikey")
config = NewRelic::Config.new("experiment", license_key)

raise "There is a problem with the license key format. Please check it again." if config == Pointer(NewRelicExt::AppConfigT).null
raise "Error configuring the logfile." if (!NewRelic.configure_log("./c_sdk.log", NewRelic::LoglevelT::NewrelicLogInfo))
raise "Error connecting to the New Relic Daemon." if (!NewRelic.init(nil, 0))

app = NewRelic.create_app(config, 10000)
NewRelic.destroy_app_config(pointerof(config))

txn = NewRelic.start_web_transaction(app, "Sample")
seg = NewRelic.start_segment(txn, "Segment1", "Test")

sleep 2

NewRelic.end_segment(txn, pointerof(seg))
NewRelic.end_transaction(pointerof(txn))
NewRelic.destroy_app(pointerof(app))
Enter fullscreen mode Exit fullscreen mode

It Could Be A Lot More Idiomatic

This is cool. It is enough that one can use the C SDK to write working instrumentation of Crystal software, and expect that it will work. The C SDK does have some hard edges, however, and the overall usage idioms create code that does not look a lot like idiomatic Crystal. Fortunately, it is a direct process to take the above code and to wrap it in code that presents the end-user with more comfortable, more familiar idioms of usage.

require "new_relic"

NewRelic.new do |app|
  app.transaction("Sample") do |txn|
    txn.segment(label: "Segment1", category: "Test") do
      sleep 2
    end
  end
ens
Enter fullscreen mode Exit fullscreen mode

The above example assumes that there is a newrelic.yml somewhere within the search path for the application which provides the license key and application name configuration, at a minimum. This file should conform to the general guidelines for that file as described in the New Relic Ruby Agent documentation.

The Second Star To The Right, And Straight Ahead Until APM!

The next step in this adventure is to work through creating that much nicer idiomatic Crystal interface for instrumenting code, and then to use that interface to build support for APM with some of the systems that people are, right now, using in production with Crystal, such as Lucky, Kemal, and Athena.

The current progress on this Shard can be found at https://github.com/wyhaines/NewRelic.cr.


I stream on Twitch for The Relicans. Stop by and follow me at https://www.twitch.tv/wyhaines, and feel free to drop in any time. In addition to whatever I happen to be working on that day, I'm always happy to field questions or to talk about anything that I may have written.

Discussion (2)

Collapse
wyhaines profile image
Kirk Haines Author

Nim is a very interesting language indeed. It looks a lot like Python, though they also draw inspiration from other languages like Modula and Ada. Unlike Crystal, Nim directly targets C as its backend, whereas Crystal, like Rust, is built on top of the LLVM. One of the other interesting differences with the Nim compiler is that it can also target Javascript, which means that one can write Nim code that can be compiled to run within a browser. That feature is a very interesting one, as it allows Nim code to be written to target both the front end side of an application and the API side of an application.

Collapse
vulogov profile image
Vladimir Ulogov

Interesting how it is became. For the Python fans, there is a Nim (Python-like-to-C compiler), for Ruby fans, there are Crystal.