Node.JS

This Node.JS guide covers how to get started with Gentrace Observe. Observe monitors your logged requests and tracks speed, cost and aggregate statistics.

The other critical part of our product is Gentrace Evaluate, which automates grading your generative pipeline on a variety of dimensions. We will not cover Evaluate in this guide. Visit here to get started with Evaluate.

Installation

This package is the simplest way to interact with our service in your Node.JS application. We recommend using this TypeScript package for Node.JS environments. Install our core npm package with the following command:

npm install @gentrace/core

If you want to use our provider SDK handlers, you must install our associated plugin SDKs. These SDKs have a direct dependency on the officially supported SDK for their respective providers. We type match the official SDK whenever possible.

# For OpenAI v4 (the new version)
npm install @gentrace/openai@v4

# For OpenAI v3 (the older version)
npm install @gentrace/openai@v3

# For Pinecone v1 (the new version)
npm install @gentrace/pinecone@v1

# For Pinecone v0 (the older version)
npm install @gentrace/pinecone@v0

These packages will only work with Node.JS versions >= 16.16.0.

❗️
Server-side only
Please only use this library on the server-side. Using it on the client-side will reveal your API key.

Quick start (OpenAI Simple SDK)

🚧
Requires @gentrace/openai@v4 or @gentrace/openai@v3
This section requires Gentrace's official OpenAI plugin. The plugin version matches the major version of the official OpenAI Node.JS SDK.

📘
OpenAI v3 SDK deprecated
As of August 16th, 2023, the OpenAI Node v3 (3.X.X) SDK has been deprecated. It's incompatible with the v4 (4.X.X) SDK. We still support both OpenAI SDK versions as separate plugins (@gentrace/openai@v4 and @gentrace/openai@v3).

We designed our SDKs to mostly preserve the original interface to OpenAI's client library. You can simply insert the following lines of code before your OpenAI invocations.

import { init } from "@gentrace/core";
import { OpenAI } from "@gentrace/openai";

// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like OpenAI() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});

const openai = new OpenAI({
  apiKey: process.env.OPENAI_KEY,
});

The OpenAI class is virtually identical to the equivalents in the official SDK.

You can then execute your OpenAI functions against the openai handle directly.

async function createEmbedding() {
  const embeddingResponse = await openai.embeddings.create({
    model: "text-embedding-ada-002",
    input: "Example input",
		// IMPORTANT: Supply a Gentrace Pipeline slug to track this invocation
    pipelineSlug: "create-test-embedding"
  });

  console.log("Pipeline run ID: ", embeddingResponse.pipelineRunId);
}

createEmbedding();

🚧
Provide pipelineSlug to collect data in Gentrace
You should provide a Pipeline slug (pipelineSlug) as a request parameter to any method that you want to instrument. This ID associates OpenAI invocations to that identifier on our service. If you omit the pipelineSlug, we will not track telemetry for that invocation.

The PipelineRun ID provided by the OpenAI create() return value is from Gentrace. Our SDK provides this for you to uniquely associate feedback with AI generated content. If you do not provide a Pipeline slug, the create() functions will not return a PipelineRun ID.

Prompt templates for the `openai.completions.create()` interface

One difference between the Gentrace-instrumented SDK and the official SDK is how prompts are specified for openai.completions.create() requests.

In the official version of the SDK, you must specify the prompt as a string. In our SDK, you must specify a prompt template and it's associated inputs. We use Mustache templating with the Mustache.js library to render the prompt templates.

// ❌ Official SDK invocation
await openai.completion.create({
  ...DEFAULT_PARAMS,
  prompt: "What's the trendiest neighborhood in New York City?"
});

// ✅ Gentrace OpenAI Handle
await openai.completions.create({
  ...DEFAULT_PARAMS,
  promptTemplate: "What's {{ interestingFact }} in {{ location }}?",
  promptInputs: {
    interestingFact: "the trendiest neighborhood",
    location: "New York City"
  },
});

Note: We still allow you to specify the original prompt parameter if you want to incrementally migrate your invocations.

Consult OpenAI's Node.JS SDK documentation for more details to learn more about the original SDK.

Content templates for the `openai.chat.completions.create()` interface

The other difference between the Gentrace-instrumented SDK and the official SDK is how prompts are specified for openAi.chat.completion.create() requests.

In the official version of the SDK, you specify your chat completion input as an object array with role and content key-pairs defined.

// ❌ Official OpenAI SDK invocation
const chatCompletionResponse = await openai.chat.completions.create({
  messages: [
    {
      role: "user",
      content: "Hello Vivek!"
    },
  ],
  model: "gpt-3.5-turbo"
});

In our SDK, if part of the content is dynamically generated, you should instead create contentTemplate and contentInputs key-pairs to separate the static and dynamic information, respectively. This is helpful to better display the generation in our UI and internally track version changes.

We use Mustache templating with the Mustache.js library to render the final content that is sent to OpenAI.

// ✅ Gentrace-instrumented OpenAI SDK
const chatCompletionResponse = await openai.chat.completions.create({
  messages: [
    {
      role: "user",
      contentTemplate: "Hello {{ name }}!",
      contentInputs: { name: "Vivek" },
    },
  ],
  model: "gpt-3.5-turbo",
  pipelineSlug: "testing-pipeline-id",
});

Note: We still allow you to specify the original content key-value pair in the dictionary if you want to incrementally migrate your invocations.

Consult OpenAI's Node.JS SDK documentation for more details to learn more about the original SDK.

Streaming

We transparently wrap OpenAI's Node streaming functionality.

// Imports and initialization truncated

async function main() {
  const streamChat = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'Say this is a test' }],
    stream: true,
  });
  
  // This pipeline run ID actually hasn't been created on the server yet.
  // It's created asynchronously after the final stream event is processed.
  console.log("Pipeline run ID: ", streamChat.pipelineRunId);
  
  for await (const part of streamChat) {
    console.log(part.choices[0]?.delta?.content || '');
  }
  
  // Stream data is coalesced and sent to our server.
}

main();

🚧
Measuring semantics
We measure the total time a generation takes from the first byte received from iterating on the stream to the last event yielded from the stream.

Before sending the streamed events to Gentrace, we coalesce the streamed payload as a single payload to improve readability.

Tracking multiple invocations in one Pipeline

With the functions shown thus far, you can only track a single OpenAI invocation per Pipeline. Check out the advanced section to learn about our methods for tracking multiple invocations across multiple providers (e.g. Pinecone vector query + OpenAI embedding call) in a single Pipeline.

Telemetry support

We automatically capture analytics from these OpenAI SDK methods. We plan to support other methods upon request. In these points, openai is an instance of the OpenAI class.

openai.completions.create()
openai.chat.completions.create()
openai.embeddings.create()

Quick start (Pinecone Simple SDK)

🚧
Requires @gentrace/pinecone@v1 or @gentrace/pinecone@v0
This section requires Gentrace's official Pinecone plugin. The plugin version matches the major version of the official Pinecone Node.JS SDK.

📘
Pinecone v0 deprecated
As of September 7th, 2023, the Pinecone Node v0 (0.X.X) SDK has been deprecated. We recommend that you update your code to v1.
We still support both Pinecone SDK versions as separate plugins (@gentrace/pinecone@v1 and @gentrace/pinecone@v0).

We designed our SDKs to preserve the original interface to Pinecone's client library. Here's how you instantiate the Pinecone client with Gentrace.

import { init } from "@gentrace/core";
import { Pinecone } from "@gentrace/pinecone";

// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like PineconeClient() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});

const pinecone = new Pinecone({
  apiKey: process.env.PINECONE_API_KEY,
  environment: process.env.PINECONE_ENVIRONMENT,
});

The PineconeClient class has an identical interface to the equivalent class in Pinecone's official SDK.

Here's example code that queries for similar vectors in Pinecone.

import { init } from "@gentrace/core";
import { PineconeClient } from "@gentrace/pinecone";
import { DEFAULT_VECTOR } from "../utils";

// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like PineconeClient() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});

const pinecone = new Pinecone({
  apiKey: process.env.PINECONE_API_KEY,
  environment: process.env.PINECONE_ENVIRONMENT,
});

async function queryPineconeIndex() {
  const index = await pinecone.Index("openai-trec");
  
  const queryResponse = await index.query({
    includeValues: true,
    topK: 3,
    vector: DEFAULT_VECTOR,
  });

  console.log("queryResponse", queryResponse);
}

queryPineconeIndex();

You should provide a Pipeline slug (pipelineSlug) as a request parameter to any supported method that you want to instrument. This ID associates Pinecone invocations to that identifier on our service. If you omit the pipelineSlug, we will not track telemetry for that invocation.

The PipelineRun ID provided by the Pinecone Index methods is from Gentrace. Our SDK provides this for you to uniquely associate feedback with AI generated content. If you do not provide a Pipeline slug, the methods will not return a PipelineRun ID.

Telemetry support

We automatically capture analytics only from methods within an Index() class instance. We plan to support other methods upon request.

Index()
- fetch()
- update()
- query()
- upsert()
- delete()

Advanced usage

The SDKs described above are designed for creating single invocations to one provider like OpenAI or Pinecone. We also provide abstractions for chaining multiple invocations together into a single pipeline.

Creating `Pipeline` and `PipelineRuns`

To declare a Pipeline, you must define the configuration (including API keys) for Gentrace and the services you intend to monitor.

import { init, Pipeline } from "@gentrace/core";
import { initPlugin as initOpenAIPlugin } from "@gentrace/openai";
import { initPlugin as initPineconePlugin } from "@gentrace/pinecone";

// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like Pipeline() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});

const openaiPlugin = await initOpenAIPlugin({
  apiKey: process.env.OPENAI_KEY,
});

const pineconePlugin = await initPineconePlugin({
  apiKey: process.env.PINECONE_API_KEY,
  environment: process.env.PINECONE_ENVIRONMENT,
});

const pipeline = new Pipeline({
  slug: "searchCompanyKnowledge",
  plugins: {
    openai: openaiPlugin,
    pinecone: pineconePlugin
  },
});

🚧
Pipelines are global
We designed the Pipeline class to specify the static, global configuration of a pipeline. Then, we expect users to use this global Pipeline reference to create additional PipelineRun instances via pipeline.start(). More on that below.

To create a PipelineRun, invoke the following code. The returned runner allows you to interact with providers like OpenAI and Pinecone.

const runner = await pipeline.start();

To access a handle on a supported provider like OpenAI or Pinecone, invoke the following code.

// For OpenAI
const openAi = runner.openai;

// For Pinecone
const pinecone = runner.pinecone;

You can then access methods for these external services on the handlers. These clients are nearly API-compatible with their equivalent NodeJS SDKs. There are a few key differences we’ll get into later when we cover each provider in detail.

Once you've invoked all the requests you need, you can submit this data to our external provider with the following code. This functionality asynchronously sends the PipelineRun data to our servers and returns a PipelineRun ID that you can send to your client.

const { pipelineRunId } = await runner.submit()

The PipelineRun ID is used to associate user feedback with the generated content. It is important to pass this ID to your client application so that you can effectively link user feedback to the corresponding AI-generated content. To facilitate this association, you can use the browser Feedback SDK.

Full example

Here's a full example of a PipelineRun invocation with multiple calls to OpenAI and Pinecone.

export async function generateKnowledgeResponse(input: string) {
	const runner = pipeline.start();

	// Near type matches for the respective clients
	const openai = runner.openai;
	const pinecone = runner.pinecone;

	const embeddingResponse = await openai.embeddings.create({
		model: 'text-embedding-ada-002',
		input,
	});
	const vectorQuery = // process embedding response

	// getPinecone() returns a near type match for the Pinecone client
	const vectorResponse = await pinecone.Index('main').query(vectorQuery);
	const snippets = // process vectorResponse

	const response = await openai.completions.create({
		model: 'text-davinci-003',
		temperature: 0.7,
		// We do modify OpenAI in one way, splitting prompt into template and inputs
		// this allows us to monitor metrics changes due to the template
		promptTemplate: `Context:\n\n{{ snippets }}\n\n{{ input}}`,
		promptInputs: {
			input,
			snippets,
		},
	});

	// Data is submitted asynchronously
	runner.submit();

	return response;
}

OpenAI (Advanced usage)

🚧
Requires @gentrace/openai@v4 or @gentrace/openai@v3
This section requires Gentrace's official OpenAI plugin. The plugin version matches the major version of the official OpenAI Node.JS SDK.

Our package provides a near type-match for the OpenAI Node.JS SDK. To get an instrumented version of the OpenAI SDK, simply invoke the following code.

const openaiPlugin = await initPlugin({
  apiKey: process.env.OPENAI_KEY,
});

const pipeline = new Pipeline({
  slug: "openai",
  plugins: {
    openai: openaiPlugin,
  },
});

const openai = pipeline.openai;

const embeddingResponse = await openai.embeddings.create({
  model: "text-embedding-ada-002",
  input: "What is Vivek's birthday?",
});

You can then invoke functions against the resulting handle that match the official SDK.

Note that in the Simple SDK, you had to specify a pipelineSlug for your invocations. If you're using the Pipeline object, the Pipeline slug is declared explicitly in the Pipeline object constructor. Similarly, the result of an invocation will not return a PipelineRun ID.

Configuration

To configure Gentrace's Node.JS SDK with OpenAI, you must initialize a plugin using the initPlugin() method exported from every Gentrace plugin. Then, pass the same parameter object that you would pass to the OpenAI constructor as the first parameter to initPlugin().

import { init, Pipeline } from "@gentrace/core";
import { initPlugin } from "@gentrace/openai";

// The provided parameter object has the same types as the OpenAI constructor.
const openaiPlugin = await initPlugin({
  apiKey: process.env.OPENAI_KEY,
});

const pipeline = new Pipeline({
  slug: "searchCompanyKnowledge",
  // ... other configuration
  plugins: {
    openai: openaiPlugin
  }
});

Prompt templates

The only difference between the interface of the Gentrace-instrumented SDK and official SDK is how prompts are specified. Consult the section about prompt templates in the OpenAI Simple SDK quick start guide for more information.

Telemetry support

We automatically capture analytics from these OpenAI SDK methods. We plan to support other methods upon request. In these points, openai is an instance of the OpenAI class.

openai.chat.completions.create()
openai.completions.create()
openai.embeddings.create()

Pinecone (Advanced usage)

🚧
Requires @gentrace/pinecone@v1 or @gentrace/pinecone@v0
This section requires Gentrace's official Pinecone plugin. The plugin version matches the major version of the official Pinecone Node.JS SDK.

Our package provides a type-match for the Pinecone Node.JS SDK. To get an instrumented version of the Pinecone SDK, simply invoke the following code.

const pinecone = pipeline.pinecone;

const vectorResponse = await pinecone.Index('main').query(vectorQuery);
const snippets = // process vectorResponse

You can then invoke functions against the resulting handle that match the official SDK.

Configuration

To configure Gentrace's Node.JS SDK with Pinecone, you must initialize a plugin using the initPlugin() method exported from every Gentrace plugin. Then, pass the same parameter object that you would pass to the PineconeClient constructor as the first parameter to initPlugin().

import { init, Pipeline }  from "@gentrace/core";
import { initPlugin } from "@gentrace/pinecone";

// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like Pipeline() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});

const pineconePlugin = await initPlugin({
	apiKey: process.env.PINECONE_API_KEY,
	environment: process.env.PINECONE_ENVIRONMENT
});

const pipeline = new Pipeline({
  slug: "searchCompanyKnowledge",
  // ... Other configuration
  plugins: {
   	pinecone: pineconePlugin 
  }
});

Telemetry support

We automatically capture analytics only from methods within an Index() class instance. We plan to support other methods upon request.

Index()
- fetch()
- update()
- query()
- upsert()
- delete()

Custom integration

If you need to measure a provider that we don't currently support, follow these steps.

Profile your request
Instantiate a custom StepRun with all the parameters that the constructor requires
Insert the StepRun into the active PipelineRun instance

Here's a code example that monitors completion requests to Anthropic (a popular LLM foundation model provider).

import { init, Pipeline, StepRun } from '@gentrace/core';

const ANTHROPIC_API_URL = 'https://api.anthropic.com/v1/complete';

const ANTHROPIC_PROMPT_BASE = 'Expand on this statement: \n';

// This function globally initializes Gentrace with the appropriate 
// credentials. Constructors like Pipeline() will transparently use
// these credentials to authenticate with Gentrace.
init({
  apiKey: process.env.GENTRACE_API_KEY
});

const pipeline = new Pipeline({
  slug: "anthropic-pipeline",
});

export async function expandStatement(statement: string) {
  const runner = pipeline.start();

  const startTimePerf = new Date();

  const response = await axios.post(ANTHROPIC_API_URL, {
    prompt: ANTHROPIC_PROMPT_BASE + statement,
    model: 'claude-v1',
    max_tokens_to_sample: 300,
  }, { 
    headers: {
      'x-api-key': process.env.ANTHROPIC_API_KEY,
      'content-type': 'application/json',
    }
  });
  
  const endTimePerf = new Date();

  const elapsedTime = endTimePerf - startTimePerf;
  
  // Start and end times must be specified as ISO strings
  const startTime = new Date(startTimePerf).toISOString();
  const endTime = new Date(endTimePerf).toISOString();

  const anthropicStepRun = new StepRun(
    // Identifier for the external provider you interacted with. You can name this
    // whatever you think is helpful to uniquely identify that provider.
    // Warning: we cannot guarantee that our SDK will not conflict with this 
    // identifier in the future.
    "anthropic",

    // Identifier for the invocation. You can name this whatever you think is helpful 
    // to uniquely identify the particular invocation against that provider (in
    // this case a completion). Warning: we cannot guarantee that our SDK will 
    // not conflict with this identifier in the future.
    "anthropic_complete",
    elapsedTime,
    startTime,
    endTime,
    // Inputs
    {
      statement
    },
    // Model params
    {
      prompt: ANTHROPIC_PROMPT_BASE,
      model: "claude-v1",
      max_tokens_to_sample: 300,
    },
    // Outputs
    response.data
  );
  
  runner.addStepRunNode(anthropicStepRun);

  // Data is submitted asynchronously
  runner.submit();

  return response;
}

Installation

Quick start (OpenAI Simple SDK)

Prompt templates for the openai.completions.create() interface

Content templates for the openai.chat.completions.create() interface

Streaming

Tracking multiple invocations in one Pipeline

Telemetry support

Quick start (Pinecone Simple SDK)

Telemetry support

Advanced usage

Creating Pipeline and PipelineRuns

Full example

OpenAI (Advanced usage)

Configuration

Prompt templates

Telemetry support

Pinecone (Advanced usage)

Configuration

Telemetry support

Custom integration

Prompt templates for the `openai.completions.create()` interface

Content templates for the `openai.chat.completions.create()` interface

Creating `Pipeline` and `PipelineRuns`