Implementing ARA

Preparing Your Implementation: Studying SDK Examples And Existing Products

Properly implementing ARA is typically a non-trivial task that requires careful preparation to be successful. Before starting out with designing and coding the ARA adoption of your product, we recommend that in addition to reading the ARA Design Overview, developers get familiar with existing ARA products to get a better feel of how ARA works from a users perspective.
Melodyne is a particularly useful example because it implements nearly all ARA features and provides extensive online documentation from a user's perspective online here. This documentation is not limited to plug-in features, but also shows how the various hosts interact with ARA plug-ins to provide a deep and unique user experience.

Next, we suggest to spend some time experimenting with the accompanying example code to see the API in action and get a feeling for the API and the typical call sequences.
The examples will also continue being useful during coding and testing - having full source code of both the host and the plug-in side examples allows to step through the flow of the program on both sides. Each developer can follow how their calls are typically processed on the other side of the API to catch any mismatches in the interpretation of the ARA model graph and its associated content.

Mini Host

The mini host example is a great starting point for understanding the basic management and lifetime of the ARA entities. It's written in pure C and walks very briefly through the core functionality of the ARA API. It is more educational than actually useful for debugging.

Test Host and Test Plug-In

We've also provided a much more elaborate test host which will put an ARA plug-in through various test scenarios, using typical host call sequences and patterns and responding to notifications from the plug-in.
Of course there's also a matching test plug-in which demonstrates how to iterate over host content, read and analyze audio source samples, and expose analysis results back to the host. It also handles persistence and demonstrates a playback renderer implementation. The test plug-in is accompanied by a ARATestChunkWriter tool for creating ARA audio file chunks for the plug-in.
Both the plug-in and host samples contain checks and assertions to ensure the other party plays by the rules, and should be used often to catch issues during the development process.

JUCE_ARA ARAPluginDemo

The ARATestPlugIn that ships with the ARA SDK already covers a wide range of ARA features, but is limited to console output and lacks a proper user interface. To address this, we've provided an additional experimental JUCE_ARA plug-in demo which is a plug-in with a UI that aims to visualize the ARA graph structure and its associated content in a representation that is close to actual ARA products.
Even without looking at the source code, this will be a great tool for host and plug-in developers alike to conveniently test how the ARA model graph and its associated content data will be interpreted by plug-ins and to experiment with the ARA UI integration features.
Since it requires using a custom fork of the extensive JUCE library, this demo is not directly included with the ARA SDK, but it contains the install_JUCE_ARA.cmake script to conveniently clone the corresponding Git repository, after which you can build and launch the Projucer to export a project for your IDE of choice via Open Example > Plugins > ARAPluginDemo.

Interaction With Existing Products

When developing and testing, it's also wise to regularly review your product's behavior with existing ARA enabled products, both from an end user's point of view and from a development perspective.
Plug-in developers should carefully use multiple ARA enabled DAWs when implementing and testing their products, as hosts often exhibit different usage patterns and call sequences that will need to be handled smoothly.
For host developers, Melodyne is a great benchmark of ARA feature functionality and should be loaded often to ensure a proper host implementation. Melodyne also supports ARA assertions, even in release builds, making it a great tool for detecting API errors.

Integrating The ARA SDK Into Your Products

While the ARA API has been defined in C for maximum compatibility, writing low level C code is not convenient. In order to ease ARA development and help creating robust plug-ins and hosts, the ARA library provides several layers of reusable C++ code on top of the underlying bare-bones C API.

Each layer of the ARA SDK provides its own level of abstraction and convenience. Picking the API level that's right for you is an important decision. The following overview of each layer shall help you with this task, as should the example code in Examples which illustrates how each layer is used in practice.

ARAInterface, Debug

The lowest level of abstraction is the "raw" C API contract defined in ARAInterface.h. This layer is header-only and merely defines the C structures that are exchanged between the API partners. It does not contain any executable code, and the abstract definitions therein are designed to be as flexible and interoperable as possible to support a large variety of architectures, languages and compilers.

The accompanying ARADebug.h helpers provided with the SDK are compatible with this lowest C layer; they only add minimal standard C and OS dependencies and ease implementing ARA assertions.

The mini host example is written in C using the raw API and the ARADebug utilities to illustrate coding directly against this lowest layer of the SDK.

If you're not using C++, or are stuck with a very old C++ compiler, then this layer is your only choice. Otherwise, there should be only very few reasons to stick to this lowest level.

C++ Dispatcher, Utilities

The C++ dispatch level defines minimal abstract classes and helper templates for both hosts and plug-ins that encapsulate the underlying C ARA interface. It does not provide any validation or other functional code beyond forwarding the C API calls into the C++ world. Compared to ARAInterface, it merely adds dependencies to a few C++11 standard headers.

As an optional addition to the basic dispatching, `Utilities` provide various useful helpers designed to work in tandem with the dispatcher-based code.
The Debug folder also contains additional code for debugging when using these classes.

The dispatch level is typically used by host vendors, since the host code structures differ widely so that higher abstractions hardly apply.
The ARA test host makes use of the host dispatch classes as well.

For plug-in developers, starting at this layer may be convenient if the code base already contains C++ classes representing ARA objects, or if they find that the higher ARAPlug level has more abstraction than necessary or applicable.

ARAPlug

The ARAPlug layer provides a whole suite of C++ classes that represent every component of the ARA model graph, along with its document controller and representations of the ARA plug-in instance roles. It also provides many convenience classes and utility functions that simplify various tasks like archiving, content reading or sending update notifications to the host.

It is the highest in terms of abstraction and convenience and is used to for the ARA test plug-in included with the SDK. We recommend that new ARA plug-in developers start with this layer.
Please see the `ARAPlug` documentation for more information.

JUCE_ARA

JUCE is a popular C++ framework that is seeing wide-spread adoption in the audio developer community. Because our goal is to make ARA development easy and robust for as many developers as possible, we've teamed up with SoundRadix to created an experimental JUCE_ARA fork that drafts support for building ARA-enabled plug-ins with JUCE.
With version 7, the JUCE team has reviewed this fork and created a proper ARA integration in JUCE. The JUCE_ARA fork has been updated to use this official integration and now merely enhances the JUCE 7 ARAPluginDemo example with several more complex features which are beyond the scope of a simple example, but will be very useful when debugging ARA host applications. The fork also adds experimental implementations of draft API from the ARA SDK that is not reflected on the JUCE mainline yet. Note that Celemony is not endorsing the use of JUCE by providing this fork. The ARA integration in JUCE is merely an adapter, it does not provide any features that would be relevant when using ARA with a different framework. Consequently, the decision whether or not to use JUCE for any given project should be made independently of JUCE_ARA.

Mapping The Internal Model To ARA

The ARA model defines abstractions that allow for mapping a wide range of host and plug-in models to it. In several cases though, the mapping may not be entirely straight forward - there will likely be some "impedance mismatches".
The general guideline here should be for the host to configure and for the plug-in to interpret the ARA model not driven by their actual internal data model, but rather by the user representation thereof. This ensures that from a user's perspective, host and plug-in feel "in sync" as tightly as possible.

Dealing With Overlapping Playback Regions

An example for this approach is dealing with region overlaps. Hosts typically define a z-order between regions, and depending on the host implementation and potentially user preferences, overlapping regions will sound concurrently, or only the top-most region will be audible in the overlap range (possibly using a cross fade range at the border where both the obscured and the top-most region are playing).
In the latter case, when configuring the ARA playback regions the host should restrict the borders of the partially covered region(s) to those areas that are not fully obscured by other regions and thus actually do sound when playing back. Note that depending on the position and duration of the overlap(s), this may include slicing one partially covered region in the host into two or more regions at ARA API level. This way, the plug-in will only display and render audio material that is actually audible in the host.

Modelling Audio Modifications In The Host

Another area that tends to be no straight match is that hosts often either lack a dedicated audio modification abstraction, or use a somewhat different separation between playback region and audio modification.
If the modification abstraction is not part of the host model, then the typical approach is to use a fixed 1:1 relationship between audio source and audio modification, which results in any modification edit to be reflected in all playback regions that cover the edited modification range - the regions are "aliases", not distinct copies.

If the host model includes some concept of multiple modifications per audio file, there must be some logic implemented to decide whether a newly created region shall be using an existing modification, or whether a new modification should be created. In the latter case, there's the option to clone an existing modification or create a new one from the original audio source state.
To come up with a proper pattern to address those concerns in a given host, it is often valuable to compare this to the MIDI capabilities of the host as reference. A typical pattern is to create aliases per default, but provide an option to turn an alias into a distinct copy (modifier upon creation, or conversion command).
An alternate approach is to conceptually treat every region as an independent copy. In that case, the relationship between playback regions and modifications is 1:1. This pattern is especially viable if ARA's separation between modification and region does not map well to the host model.

Another consideration to keep in mind is that compared to plain audio files, ARA audio modifications are not limited in the time range they cover. Users can freely copy or move the original material around, this extends beyond start and end of the underlying audio source.
It is therefore desirable to allow ARA users to extend the playback region borders beyond the range of the audio source, instead of applying the typical border restriction implemented for regular audio files. Depending on whether this restriction is implemented at control layer or in the model, it needs to be considered either when mapping the model or when adjusting the user interface for ARA, see below.

Choosing An Appropriate Region Sequence Representation

Region sequences typically map to arrangement tracks or lanes, but as with the other mapping topics discussed above this may not always be a direct match. For example, some hosts allow for storing several alternate versions of the arrangement per lane, and let the user pick one version that is currently editable and played back.
In order for each track version to preserve its distinct ARA edit state, hosts must use separate audio modifications per version (while still sharing the underlying audio sources). When duplicating versions, the affected audio modifications will be cloned.
In order to prevent inactive versions from cluttering the plug-in UI, hosts should only expose a single region sequence per versioned track, containing only the current version's playback regions. If the active version is switched, the region sequences remain as-is, but all the playback regions are replaced as needed.

A similar pattern applies when comping: there should be a single region sequence representing the resulting comp lane, holding playback regions based on the selection made on the per-take lanes. This may be different from some host implementations where the resulting comp is merely a UI entity, and the internal engine just plays the selected regions from the take lanes.

Configuring The Rendering

Setting Up An ARA Playback Renderer

The setup process for an ARA-enabled plug-in instance is not much different from setting up a non-ARA plug-in instance, but there's some considerations worth mentioning, mostly related to the fact that the audio source replaces the realtime input of the plug-in.

The established plug-in APIs usually define two states for a plug-in: a setup state where certain configurations such as the maximum render block size may be changed, but no rendering may occur, and a render state where the configuration is fixed but rendering may occur.
In most APIs, changing the I/O configuration is restricted to the setup state. Assigning playback regions to an ARA playback renderer plug-in instance can be considered an I/O change too, and thus is always restricted to the setup state. (Note that this is different for editor renderers, see below.)

Since ARA playback renderers have no realtime inputs, it would seem appropriate to suppress these in the supported I/O configurations published through the underlying companion APIs. However, given that preview renderers do use their inputs, and that a plug-in instance can be both playback and editor renderer, the I/O configurations made available by the plug-in must not depend upon ARA being enabled or not or on the assigned ARA instance roles.
Instead, for plug-in instances that do only playback but no editor rendering, ARA establishes the rule that the main inputs are simply never used - plug-ins never read them, and hosts do not need to supply a meaningful signal.

Ignoring the realtime inputs also means that ARA playback renderers should not incur any processing latency, because it can be completely compensated for inside the plug-in. Plug-ins must make sure to report this correctly through the companion API, and hosts should read the latency only after establishing the ARA binding.

Further, being independent of realtime input means that ARA playback renderers can be processed ahead of the actual playback location using rather large buffers to reduce CPU load, because the larger latency introduced by this can be fully compensated for. Note that this also includes proper visual latency compensation as supported by VST3's IAudioPresentationLatency or by Audio Units's
kAudioUnitProperty_PresentationLatency.

Using larger buffers is particularly useful when a plug-in internally applies some sort of transformation into the frequency domain, because it then needs to apply internal buffering for the transformation and will cause high spikes of CPU load in all render slices where it can process a full transformation buffer, but will hardly do any work in all other render slices. Using render slice sizes that are about as large as the plug-in's internal transformation buffer therefore leads to a much steadier overall CPU load.
It is therefore highly recommended to process ARA-enabled playback with render slice sizes between 1024 and 4096 samples because these sizes are typically used internally in frequency domain related DSP algorithms.

Following these considerations, a host might even decide to render ARA playback renderers inside its file I/O thread (much like a realtime MP3 decoder), instead of performing the rendering from the realtime audio processing thread. Doing this is certainly valid and only requires to properly flag non-realtime usage to the plug-in, using VST's kVstProcessLevelOffline or Audio Units's
kAudioUnitProperty_OfflineRender etc.

Preview Rendering

In addition to the actual audio output signals, ARA allows for temporary, auxiliary signals to be produced while editing the data. These signals are audible clues when performing the model edits, aiding and speeding up the editing process. They typically are only generated when song playback is stopped, in order not to corrupt potential bounces. A prominent example is Melodyne playing back a note when it is grabbed with the mouse and dragged up and down, so its new pitch is immediately audible. Another of these preview features is that it allows for optionally playing back a chord when the user selects it in its chord track. It also features a metronome in its audio source tempo definition editor.
These three examples show that there generally are two classes of these signals: sounds that are associated with a given playback region or region sequence, which should accordingly be routed through the same effect chain as used during playback of that region or sequence respectively, and song-global sounds that are not associated with a particular mixer channel, such as the aforementioned chord track preview or metronome. Some hosts, most prominently Cubase, feature an explicit monitoring channel for such signals with separate routing capabilities.

ARA 2.0 defines a dedicated editor renderer role that enables the host to set up these scenarios as supported: preview renderers can be set up with a set of playback regions or region sequences defined by the host. For a song-global preview renderer, that set remains empty. Hosts shall set up the preview so that it is unambiguous, i.e. all the sets should be fully disjoint, and only one set should be empty (song-global renderer).
Due to their very different nature, editor renderers comply to a different set of rules compared to playback renderers. In terms of signal flow, preview renderers add their signal to any input signal that the host may provide - or if a given plug-in instance is both playback and editor renderer at the same time, to the playback renderer output.
Since any preview signal is only temporary, drop-outs are acceptable if it eases the implementation. Further, generating the signal is a task in the plug-in that is fully transparent to the host. Therefore, the responsibility to properly react to any ARA model graph edits or signal routing changes (expressed by modifying the set of playback regions or region sequences of a given editor renderer) in a thread-safe fashion is entirely placed on the plug-in - the host can make any of such changes without toggling the plug-in temporarily from render state to setup state, as would be required for playback renderers. An easy way to deal with that requirement on the plug-in side is to simply cease and later resume any preview rendering whenever such a change to the model or the routing occurs.

Conversion Between Audio Source Format And Song Playback Configuration

The audio signal in any given audio source does not necessarily have to match the settings that the user has configured for playback of the song. The channel count or spacial channel layout may be different, or the source might be recorded at a different sample rate. Accordingly, channel format conversion and/or sample rate conversion (SRC) algorithms must be applied when processing the audio.
Some hosts deal with this issue by converting all audio files to the desired format upon importing them into a project, but others keep the audio files in their original format and perform on-the-fly conversion during playback.
Per default, these hosts should provide the audio source signal in its original format to any ARA plug-in in the model graph. This minimizes any artifacts that the conversions introduce, ensuring plug-ins will have the highest quality signal available for analysis and therefore yield the best results. It also prevents potential re-analysis and the according potential loss of edit data when the playback configuration is changed, which is especially important when dealing with ARA audio file chunks.

Plug-ins must therefore implement an appropriate handling of audio sources with different sample rates or channel formats. Some plug-ins such as Melodyne are capable of integrating the SRC into their DSP algorithms, thereby reducing the overall artifacts. On the other end of the spectrum, some plug-ins may not implement SRC at all, instead choosing to render silence in case of a sample rate mismatch and informing the user that any SRC must be explicitly applied in the host. Note that the host does not know nor need to care about the plug-ins SRC implementation, it will always present the situation "as is" and rely on the plug-in to deal with it appropriately.

If the plug-in does implement SRC, users should be aware that non-ARA audio sources or audio sources processed by different ARA plug-ins will be converted using different algorithms. This can result in differing levels of quality, or even changes in phase in extreme cases, which is problematic in applications where phase alignment is crucial.
In such cases, it is up to the plug-in vendor to educate users about the issue. For example, the plug-in could detect whether SRC is required, and instruct the user to either add the plug-in to all phase-aligned recordings to achieve consistent conversion, or to explicitly convert them all in the host before proceeding, whatever yields the preferred results.

Playback Region Head And Tail Times

Companion APIs allow plug-ins to publish a "tail time" that informs hosts of a signal that will be appended beyond the end of the input signal. The value of the tail time typically depends on the settings that the user has dialed in. While it defaults to 0, it may be several seconds long in some use cases.
ARA adopts this concept for each region. Since ARA is a random access API, it also extends this to include a matching head time before the start of the region. Since ARA has the entire model down to the samples available, the head and tail times may not only depend on the user settings, but also on the arrangement and the content of the audio files. If changes in any of that calculation parameters result in different times, the plug-in will signal that change through a playback region content update notification to the host.
When rendering playback regions, host must take the head and tail time into account to allow the playback renderers to generate proper output. Head and tail times are a prerequisite for the ARA'scontent based fades" feature, see below.

Dealing With Denormals

When processing audio samples, de-normalized floating point values may cause severe performance penalties. A common way to handle this is to switch off denormals in the CPU, as they are irrelevant for the audible results of the rendering. A plug-in that uses this technique must potentially perform the switch whenever the host calls into the plug-in, and must switch back before exiting. To avoid this considerable overhead, it is recommended that host applications generally turn off denormals, so that plug-ins do not need to perform this switching at the begin and end of each render call.

Caching Especially CPU-intense DSP

In some cases, the processing demand for certain ARA edits may be so expensive that it cannot be rendered in realtime. To still allow for proper realtime playback, the processing results (or some intermediate data) must be cached.

If such caches can be created within a reasonably short amount of time, the plug-in can manage this data in a typical system-wide cache folder that will keep recently used data around until a certain threshold is exceeded. When loading a project that references data no longer part of the cache, the plug-in will recreate the data during the unarchiving process (for which the host will provide a proper progress bar). This approach is implemented e.g. by Melodyne's polyphonic audio processing.

There are however rare cases when calculating the render output is so expensive that it will stall the system for several minutes per affected audio modification. In this scenario, the cached data should preferably be kept alive as long as the project that uses the data is being worked on. This is best achieved by placing the cache inside the host's project folder, i.e. a folder specific to the host document that can be used by plug-ins when storing data.
Finding this folder can be done using special companion API functions. The specific details of these functions vary depending on the companion API being used, but for example PreSonus provides an IContextInfoProvider extension to VST3 that can be downloaded from their Developer Website.

Analyzing Audio Material

What Can Be Analyzed?

ARA plug-ins typically need some initial analysis phase to build an internal model of the content of a given audio source before users can edit the DSP accordingly. If parts of the model can be mapped to the ARA content types, then the plug-in may export the analyzed data to the host, allowing it to be used as analysis engine - the ARAFactory publishes a list of content types that can be provided through analysis.

Note that any analysis performed by the plug-in may fail to provide meaningful results for a specific audio source and a given content type, e.g. if the signal is very noisy, or if it does not contain any information of that type. In this case, a plug-in may indicate that no content reading is possible, or it may fall back to some default state and keep providing "initial grade" content despite an analysis was performed - the host must deal properly with both cases.

Manual Adjustments

Some plug-ins such as Melodyne allow for manual adjustments of the analysis results in their UI. When such edits happen, the host will be notified and content grade will be updated accordingly (see below).

In addition to the analyzable content, a plug-in may provide means for the user to define more content information for the audio source in its UI (e.g. specifying a key signature). While not determined through analysis and therefore not listed in the ARAFactory, such data will still be communicated to the host after the user has edited it, and the host can sync accordingly - listening to all relevant content changes should therefore be done regardless of a given plug-in's analysis capabilities.

Triggering Explicit Analysis

Depending on the design of the plug-in, the analysis of a given content type may be started automatically by the plug-in when creating an audio source, or it may be postponed until the user explicitly triggers it on demand throughout its UI. In cases where a host wants to use the analysis capabilities of a plug-in without user interaction, it must therefore explicitly request an analysis. (Host developers can test both behaviors with the ARA Test Plug-In, see the ARA_ALWAYS_PERFORM_ANALYSIS define in ARATestDocumentController.cpp.)

If user interaction in the host is blocked while the analysis is running, progress information should be provided to the user - plug-ins must support this if their analysis is taking a non-trivial amount of time.

In addition to progress information, a call is available to determine whether or not the analysis for a given content type is still in progress. This call is particularly relevant if the user can perform edits in the plug-in UI while the host waits for the analysis results - it may be possible for the user to restart the analysis with different settings there.

Algorithm Selection

Plug-ins often provide sets of pre-configured analysis and other processing parameters configured for specific materials, such as distinguishing between percussive material versus tonal sounds with clearly perceived pitches. ARA 2.0 allows for exporting these of algorithms to the host so it can configure this appropriately when adding a new audio source - either through UI or through context, such as when recording a new take of a piece of music that has already been analyzed before.

Utilizing Content Exchange

Content information exchange is a key component of the relationship between ARA host and plug-in, and properly utilizing this information can yield powerful results. Content information resulting from plug-in analysis can be used by the host to adjust project settings to match the content, and plug-ins can use information provided by the host to adjust their transformation accordingly. Also, any information readily available in the host does not need to be analyzed by the plug-in.

Content information shared between plug-in and host will be flagged with a "content grade" to indicate it's quality. It will be evaluated when deciding about how to use the information received from the API partner, as illustrated in the examples below.

When implementing content readers, it is important to be aware of potential rounding issues when converting between internal data structures and the continuous time used in the ARA API - both when calculating time stamps and when evaluating the optional content time range.

Musical Timing Information

A proper description of the musical timing (tempo and bar signatures) and how it changes over time is the foundation of many musical applications. Uses for this data range from presenting a natural editing grid to applying quantization or aligning and time-stretching some already existing recording to fit into some different piece of music.

Most hosts define a project-wide musical timing to which the entire arrangement is optionally aligned to. They typically provide a sophisticated UI to edit this timing. In ARA, this data is modeled as part of the musical context, and plug-ins can read it from the host on demand.

In order to align any audio signal musically meaningfully within this arrangement, the musical timing of the underlying audio source must also be known. If audio material is recorded in the host, the musical timing at the time of the recording should thus be stored alongside the audio.If pre-recorded audio material is imported into the host, it may also contain proper timing information. Otherwise, the host may prompt the user to specify the tempo and bar signatures manually, or it may implement automatic tempo and bar signature analysis.The latter option is non-trivial and involves very elaborate DSP, so it may be desirable to delegate this implementation to specialized ARA plug-ins such as Melodyne - the host can test if this feature is available in a given plug-in by evaluating the its ARAFactory::analyzeableContentTypes.
To start the analysis process, the host would first request the analysis of the audio source, then wait until the plug-in acknowledges the completion of the analysis. After checking that the analysis was successful, the host can then read the content information from the plug-in.

Content Grade Examples

After the plug-in has analyzed a given audio source, the content grade of the plug-in data will typically be `kARAContentGradeDetected`, indicating that the data results from an automatic detection without further user interaction.
This means that the results may not be entirely reliable, since the user has not reviewed them yet - applying automatic transformations such as time stretching to make the audio align with the musical timing of the song may thus yield undesirable results. Some host (e.g. Studio One) will therefore suppress such automatic adjustments at this early point - if so, the host or the plug-in may choose to indicate that the relevant information is available for user review (e.g. Melodyne utilizes a flashing tempo text field for this).

Should the user then review and potentially modify the timing definition in plug-in's UI, the grade will transition to `kARAContentGradeAdjusted`. The plug-in will notify the host about this change and the host can update its representation and will now consider that data fully valid and use it to its full potential.

This user approval is not required to be done on the plug-in side - if the host features some user interface for specifying audio source tempo information, the user can make adjustments there and the host will communicate this to the plug-in so it can update, exactly mirroring the use case described previously.

This symmetry in using content readers also extends to hosts that feature built-in tempo detection - when the plug-in queries the timing of a given audio source, the host can return a data of with the detected grade. The plug-in can then decide whether this information is plausible or whether it should try to re-detect the tempo. If however the tempo was available in the host because the user entered it, the grade would be adjusted and and the plug-in would accept this and skip its built-in detection.

Once host and plug-in agree on the musical timing, the plug-in can be instructed to apply time stretching when rendering playback regions as detailed in the next section. The host can further implement other features based on this information, such as allowing to set the project time line to match the tempo of an audio source etc.
Note that the user may be able to further edit the audio source timing definition in either the plug-in or the host, so both sides should continuously listen to further change notifications from the other side.

There are two more content grades defined in the ARA API. First, if a content reader is requested before the initial analysis has completed, or if an analysis failed to provide meaningful results, the content information will receive a grade of `kARAContentGradeInitial` to indicate that the data is not actually related to the content but merely represents some reasonable default value (e.g. a tempo of 120 BPM).

Finally, there are scenarios where the audio sources have been carefully preprocessed by some content producer, as is the case with some pre-packaged content libraries. Here the content information can be assumed to be fully correct and not to be changed by the user, which is represented by the `kARAContentGradeApproved`.

The content grades apply universally to all content types described in the ARA API, not just to musical timing information. The musical timing content types however are the most widely supported types on both sides of the API and thus serve best as examples illustrating how their content grade affects the actual user workflow on both the host and the plug-in side.

Notes And How Playback Transformations Affect Content Data

Another example of utilizing content information is performing an "Audio to MIDI" conversion. This is done by using notes detected by the plug-ins analysis to create a new MIDI clip in the host, representing the notes being played in the original audio source. This content type is referred to in code as ARAContentNote. Like in the previous example, this content information will be given a grade depending on whether any notes were actually detected or what kind of adjustments the user has made on their own.

Note content serves well for illustrating the effects of the transformations applied by the plug-in at the different parts of the ARA model graph. At audio source level, the returned notes would describe what was played in the original recording. If the application needs access to this original data, it is available here.

When reading at audio modification level, the data would be still in the original timing context, but include all edits that the user has made when rearranging the audio data inside the plug-in. This may be the least valuable information, since these user edits where made within the playback context, and may not be meaningful when being used in the original audio source context.

Reading per playback region yields the notes as rendered when playing back the arrangement, so if MIDI export is for example implemented by dragging audio regions in the host to MIDI tracks, this level of content reading must be used. Further, some hosts optionally draw MIDI-like representations of this content information on top of any audio region that is being associated with an ARA plug-in that delivers note content information.

Chords, Key Signatures And Other Content Types

The third important group of content types is the related to musical pitches and the harmonies they form, most prominently the chord progression in musical timing, and the overall key the notes are sounding in, available through `kARAContentTypeSheetChords` and `kARAContentTypeKeySignatures` respectively.
If the host provides this information in its musical context, it can be used by plug-ins to adjust their pitch editing of notes according to the harmonic structure of the song. This powerful feature can be seen e.g. when using Melodyne in Studio One or Cubase - notes detected by Melodyne can be pitched so that they follow the key scale and/or the chord progression of the song, allowing the host's chord track to affect the audio transformations in Melodyne.

There are several more content types available that are no discussed in detail here, for example related to tuning/intonation. More content types are likely to be added as the ARA API evolves. See Content Readers And Content Events in ARAInterface.h for a complete list and detailed description of each type.

Manipulating The Timing

A key benefit of ARA compared to established plug-in APIs is that because the musical timing information is provided both on audio source level and for the entire song timeline, hosts can use any ARA plug-in that advertises the time-stretch `kARAPlaybackTransformationTimestretch` capability in their ARAFactory::supportedPlaybackTransformationFlags as a very flexible time stretch engine.

In the most simple use case such plug-ins can perform a classic, linear time-stretching whenever different different durations are provided for playback regions in playback vs. modification. This would ignore the musical synchronization capabilities of ARA, and would be a typical behavior for plain sample editors that do not use a musical representation internally.

Hosts that deal with musical timing information however can behave much smarter. They can optionally quantize both playback region placement and stretching so that the musical timing in the song and the audio source are appropriately musically synchronized. Note that if the tempo in either domain is not constant, then the time stretching is likely non-constant across the duration of the playback region.
Consider the following example (each line represents a beat drawn in time-linear):

song timing:    |      |      |      |      |  (constant tempo)
source timing:     |     |     |   |   |       (tempo is faster in the second half)

If a constant stretch factor is applied, this prevents the beats from aligning. However by tracking the tempo relationship between the source and the song and accordingly stretching the source material more in the second half where its tempo is faster, perfect alignment is possible:

song timing:    |      |      |      |      |  (constant tempo)
equal stretch:  |       |       |     |     |  (still faster in the second half, by same ratio)
tempo-adjusted: |      |      |      |      |  (constant tempo)

Note that this example is simplified, real-world calculations cannot assume constant tempo for either timeline and also need to properly reflect potential offsets between the first beat of the song/the audio source and the start of the song/the audio source.

Practical use cases often are not as extreme as our contrived example, and depending on the music a "perfect" sync may not be desired - it's a popular production workflow to add a slightly sloppy instrument to an otherwise fully quantized arrangement to make it more interesting and lively.
Hosts therefore typically allow users to choose between both mapping techniques, and they can communicate this choice to the plug-in by toggling the flag `kARAPlaybackTransformationTimestretchReflectingTempo` in the region's ARAPlaybackRegionProperties::transformationFlags.

Note that when mapping each individual note between both time domains, the plug-in may choose to apply any algorithm suitable. It may even offer the user an option to choose some sort of groove pattern that is to be taken into account when calculating the mapping. If a plug-in offers such an option, it must do so at audio modification level.

Finally, it should be pointed out that in addition to adjusting audio source data to the song timeline, the opposite feature may be desired as well: extracting the timing information from an audio recording and applying that to the song so that this audio recording becomes the new timing reference and other audio can align to it. This might for example be implemented by dragging playback regions onto the song timeline - the timing in the target ranges is replaced by the respective underlying audio source timing.

Content Based Fades

The head and tail time of playback regions can be leveraged by plug-ins to implement content based fades at the region borders. Instead of fading the overall signal, plug-ins can re-interpret these border fades depending on the type of content that will be playing when the fade occurs. This is best explained with an example:

This shows a region being sliced at a time that causes notes to be cut off or start after their transient. Instead of applying a traditional overall fade-out/fade-in, the plug-in could reflect this in its playback model so that each note that intersects with the transition is either completely played in the tail of the first or the head of the second region, so no fades are necessary at all.
This concept could be extended further - the plug-in could actually modify the notes that are intersected by the borders so that they start / stop as close as possible to the region boundaries, further improving the quality of the fade in a way that a traditional crossfade could not do. Head and tail time should always be kept at the minimum time that is required for achieving such a natural sounding processing of the region borders.
Content-based fades can be particularly strong when comping. Picking notes from either the first or the second region allows to deal with situations where overall fades are bound to create artifacts, such as when the notes in the second region are played slightly later than in the first.
The actual implementation of content based fades is entirely dependent on the internal processing algorithms of the plug-in. If those algorithms do not naturally provide an alternative to traditional overall fades, the plug-in should not implement this feature, and the host will then just use existing fade-based implementation used also for non-ARA regions.

Managing ARA Archives

An important aspect to consider when implementing persistency is dealing properly with the various persistent IDs in the ARA API and their associated versioning. There are three IDs involved: the ARAFactory::factoryID, the ARAFactory::documentArchiveID and the plug-in ID as defined via the companion API.

The factoryID serves a runtime ID of the plug-in - it can be used to uniquely identify a specific version of a specific ARA plug-in, for example when copy/pasting plug-in state between documents. It also enables the host to determine whether the same ARA plug-in is installed across different companion APIs, such as an Audio Unit and VST3 version of the same plug-in. This allows to filter duplicate plug-ins and choose only a single companion API for each ARA plug-in as detailed later in Choosing Companion APIs.
As long as thefactoryID has not changed, the host-observable ARA behavior of the plug-in has not changed - e.g. it has the same analysis capabilities, supports the same archives etc. Hosts can therefore choose to store such information about the plug-in in a cache to avoid fully loading all plug-ins at each application startup, and update this cache whenever the factoryID changes.

The documentArchiveID uniquely identifies opaque archives of ARA plug-in state and must be stored by the host alongside the raw archive bytes whenever calling ARADocumentControllerInterface::storeObjectsToArchive() (or the legacy ARADocumentControllerInterface::storeDocumentToArchive()). The host should also store the user-readable meta information about the plug-in (ARAFactory::plugInName, ARAFactory::manufacturerName, ARAFactory::informationURL and ARAFactory::version) so it can later provide a proper error dialog should the archive be loaded on a system where the plug-in is missing or outdated, informing the user which version of which plug-in is needed and where to get it.

If a plug-in is updated in a way that makes its archived state incompatible with previous versions, both the factoryID and the documentArchiveID must be adjusted. Further, the previous documentArchiveID must be added to the list of ARAFactory::compatibleDocumentArchiveIDs and proper import code must be written to read existing archives in that previous format.
If the new version of the plug-in is installed, the host can detect that change via the updated factoryID. It will then read the new ARAFactory data and recognize the new documentArchiveID and the updated list of compatibleDocumentArchiveIDs. If a project is loaded that contains an old archive, the host will find the old ID in the list of compatible IDs and therefore knows that the plug-in will safely migrate the data.

A documentArchiveID is not necessarily unique to a specific plug-in. It is valid for any plug-in to be able to read or write archives that can also be processed by another plug-in. This allows for migration paths between different plug-ins, not just between updates of the same plug-in. It however also implies that the archive ID is not suitable to identify a given ARA plug-in - instead, the companion API provides proper IDs to handle that task.

To sum things up, let's step through the project load sequence from a host's perspective. The project archive typically contains a list of archived ARA document states, including the associated meta information from the ARAFactory described above as well as the companion API plug-in ID. For each of these states, the host first checks via the companion API whether the plug-in is currently installed in the system. If it is, the host then validates that the installed version can in fact read the archive by comparing its documentArchiveID and compatibleDocumentArchiveIDs with the document archive ID stored in the project. If ID is not listed, the installed plug-in is outdated and the host will show an appropriate error message.
If no plug-in has been found through the companion API, the host compares the project's document archive ID against the documentArchiveID and compatibleDocumentArchiveIDs of all installed ARA plug-ins. If another plug-in supports reading archives with the given IDs, the host can proceed to migrate the project to the new plug-in by loading the document archive via the new plug-in's document controller, and also switching all insert points in the arrangement that used the previous plug-in to the new one so that they can be bound properly to the migrated document controller.
With the ARA document controller and its associated document state restored, the host will then create companion API instances to fulfill the various required instance roles and bind them to the document controller. Both kARAPlaybackRendererRole and kARAEditorRendererRole are transient, so no state must be saved or restored for these via the companion API. Instances with the kARAEditorViewRole however may use that state to store customizable view configuration, so their state must be persisted and restored via the companion API.

Partial Persistency

Partial persistence is a feature added in ARA 2.0 that allows to limit storing or restoring objects to a subset of the current ARA graph by providing an optional filter to the store/restore operation. This can be used to export groups of objects from one document and import them into another - as an example, a dragged audio source from one document could be stored in a temporary archive and restored into a different document onto which it is dropped. Partial persistency also enables hosts to split the potentially large archive of a document into smaller chunks, which may be useful in data base or network sync contexts, and is used when reading ARA audio file chunks.

Copying ARA Data Between Documents

The most common use case for filtering the scope of archiving operations is implementing a feature that allows users to copy data between ARA documents, typically via copy/paste commands or drag and drop operations. This requires both an 'ARAStoreObjectsFilter' to store only the selected objects from the source document, and an ARARestoreObjectsFilter to initialize the newly created target objects.

When performing this operation, it is important that hosts manage the persistent IDs correctly, i.e. make sure that the IDs used in the source document while storing do not collide with IDs used in the target document. If there is a conflict, then hosts must map the stored IDs to new unique IDs and export this mapping to the plug-in in the 'ARARestoreObjectsFilter'. To see the filters in action, study the testDragAndDrop() function in the ARA Test Host.

Audio File Chunks

Another use case for partial persistence is importing WAV or AIFF files that contain ARA audio file chunks. These chunks allow for embedding the archive of an audio source state into the audio files that contain the audio samples for the source via the the iXML standard.

There are currently two main use cases for these chunks. First, for plug-ins which may require extensive analysis or manual editing of the audio source internal model, this is a way for content providers to ship audio content with these tasks already accomplished, so that user can go ahead and use the content with the supported plug-ins immediately without going to these time-consuming processes.
For example, 3rd party audio samples provider could create a loop library with polyphonic detection data for Melodyne and other plug-ins provided up-front in the iXML chunks, so user can load these loops and directly adjust the loops to their song's harmonies without further preparation.

The other major use case are plug-ins which generate audio content, e.g by de-mixing audio material or by exporting audio from one plug-in to another. An example here is SpectraLayers, which allows user to drag layers out of the plug-in into the host to process them separately from the remaining audio signal. The layer's samples are rendered into a new audio file, which is then passed to the host's file drop handler. The host imports the file and parses the iXML chunk to find that it should automatically add SpectraLayers to the playback region created upon file import.

Note also that in both cases, the audio files are not created by the host but rather by code that is provided by the plug-in developer, either as part of the plug-in or as standalone application. The test plug-in from the SDK provides a dedicated ARATestChunkWriter tool for the plug-in, while Melodyne allows for storing such chunks from its regular standalone version.
To keep the chunks small and to speed up file loading, plug-in developers may accordingly choose to use a different archive format compared to ARA document archives, as done in Melodyne.

Companion API Considerations

Choosing Companion APIs

Supporting multiple companion APIs not only adds to the code which might introduce more bugs, it also adds another dimension to the already large test matrix for ARA enabled products and places a long-time maintenance burden on the developers.
To mitigate this effect as much as possible, it is strongly recommended that hosts which support multiple plug-in APIs when not using ARA limit this choice to only a single, preferably cross-platform companion API when implementing ARA.
This strategy requires on the other hand that plug-in developers should strife to implement ARA companion API support for all APIs that they support without ARA, enabling hosts to actually make the above restriction without excluding any specific plug-in from being used with ARA.

VST3: setActive() vs. setProcessing()

Both VST3 hosts and plug-ins should be aware of the difference between IAudioProcessor::setProcessing and IComponent::setActive when dealing with plug-in instances. As per the VST3 documentation, the setProcessing function should be limited to light-weight operations that reset the internal processing state, whereas setActive is used for heavy-weight initialization and memory allocations.
Unfortunately, when VST3 was initially established there where some confusions about these calls, which have resulted in some plug-ins not resetting properly upon setProcessing, and some hosts in turn calling setActive instead to work around issues with resetting plug-ins.
When adding ARA support, plug-ins are required to properly implement the VST3 specification, and hosts will accordingly rely on ARA plug-ins responding properly to setProcessing. Doing otherwise and sticking with setActive for reset can cause severe performance issues with large songs with several thousand playback regions, because accordingly as many playback renderer instances will be performing their memory allocations and complex initialization upon each reset, e.g. each when starting playback.

Audio Unit: Optimizing Buffer Allocation

The Audio Unit API uses a very complex I/O setup scenario with various connection options. While plug-ins must be prepared to handle all these options, hosts typically only use a specific one for all Audio Units. To allow optimization for the concrete connection in use, the Audio Unit specification includes the property kAudioUnitProperty_ShouldAllocateBuffer for both inputs and outputs.
Since ARA implementations typically rely on a high number of plug-in instances (typically one playback renderer instance per playback region, plus editor rendering instances), it is important that hosts enable this optimization by configuring the ARA Audio Unit instances accordingly.
Plug-in implementations must evaluate the property accordingly, which will happen automatically if the implementation is based on the Core Audio Utility Classes (like done in the SDK test plug-in), but may require extra coding otherwise.

User Interface Considerations

For the better part of it, ARA relies on the companion API regarding how to create and use the graphical user interface of a plug-in. There are however a few things to consider in order to support the full ARA experience.

View Embedding

In many ways, ARA-enabled regions work similar to MIDI regions. It is therefore often desirable to embed the UI of an ARA plug-in into the main window, similar to the MIDI editors in many hosts. This is possible without further API changes across all currently supported companion APIs, and currently implemented e.g. in Studio One and Cubase.
Note that some hosts rely on custom extensions of the companion APIs for view embedding, for example Studio One - to indicate that view embedding is appropriate for any given plug-in (regardless of ARA), PreSonus has defined the custom, easy-to-implement Presonus::IPlugInViewEmbedding interface as part of their PreSonus Plug-In Extensions to VST3. This extension also been adopted by Cakewalk SONAR.

Reflecting Arrangement Selection In The Plug-In

Further following the MIDI editor analogy, those editors tend to optionally update the contents of their view based the selection in the arrange area. Since navigation and selection in the arrange area are core workflows that are well understood by all users, relying on them is preferable to implementing a different workflow for the MIDI editor.
The same considerations hold true for ARA-enabled plug-ins, so the ARA API provides means for the host to communicate the arrangement selection through the ARAEditorViewInterface interface. This selection optionally covers regions, tracks and a time range, so that the various arrangement selection techniques in current hosts can be mapped to a proper ARA 2 representation, enabling plug-ins to follow them in whatever is a suitable way for their UI.
Note that most MIDI editors include the option to "pin" the current view state - ARA plug-ins are expected to implement such a feature too if applicable to their user interface.

Windows High DPI View Scaling

The Windows operating system allows for a per-application scaling of the UI through its High DPI API. To allow proper scaling of plug-in UI in host windows, ARA 2 plug-ins must support the associated view scaling provided through the various companion API, such as VST3's IPlugViewContentScaleSupport interface.

Key Event Handling

An area that is often problematic when using plug-in views via the established APIs is the routing of key commands. While ARA doesn't actually need to define anything special here, all developers shall be reminded to check proper routing of all key events to the plug-in depending on whether or not it has keyboard focus.
In your Cocoa implementation in particular, double-check that you've implemented all methods that handle events properly (most notably -performKeyEquivalent:) and that you're maintaining the first responder properly - see Apple's documentation on the topic.

Future ARA Development

There are several features that have been discussed when developing the API in its current form which were postponed for the time being because defining and implementing them properly needs more input from a wider range of development partners. The goal was to avoid designing API that would not actually be used, or would not work as desired in the various use cases that potential ARA development partners would come up with. As both hosts and plug-ins evolve and more and more developers are adopting the ARA standard, these features may shape up and eventually become part of the next generation of ARA:

  • undo/redo integration with the host
  • support for instrument plug-ins (i.e. no audio sources)
  • support for MIDI plug-ins
  • improved support for "warp-"/"bend-"markers as found in many hosts
  • support for phase-locked editing across concurrently recorded audio sources
  • explicit support for comping/takes
  • running the analysis while recording the audio
  • optional side-chains and realtime input

Developer's feedback about how to design these or other missing features is highly welcome.