Share via


Using the Oracle Database Vector Store connector (Preview)

Warning

The Oracle Database Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release.

Warning

The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release.

Warning

The Semantic Kernel Vector Store functionality is in preview, and improvements that require breaking changes may still occur in limited circumstances before release.

Overview

The Oracle Database Vector Store Connector can be used to access and manage data in Oracle Database. The connector has the following characteristics.

Feature Area Support
Collection maps to Oracle database table
Supported key property types
  • short
  • int
  • long
  • string
  • Guid
Supported data property types
  • bool
  • byte
  • short
  • int
  • decimal
  • long
  • float
  • double
  • DateTime
  • DateTimeOffset
  • TimeSpan
  • char
  • char[]
  • byte[]
  • String
  • Guid
  • and nullable type of the above types
Supported vector property types
  • ReadOnlyMemory<float>
  • Embedding<float>
  • float[]
  • ReadOnlyMemory<double>
  • Embedding<double>
  • double[]
  • ReadOnlyMemory<short>
  • Embedding<short>
  • short[]
  • ReadOnlyMemory<byte>
  • Embedding<byte>
  • byte[]
  • BitArray
  • BinaryEmbedding
Supported index types
  • Flat (default)
  • HNSW
  • IVF
Supported distance functions
  • CosineDistance
    • FLOAT32, FLOAT64, and INT8 vector default
  • CosineSimilarity
  • DotProductSimilarity
  • NegativeDotProductSimilarity
  • EuclideanDistance
  • EuclideanSquaredDistance
  • HammingDistance
    • BINARY vector default
  • ManhattanDistance
  • JaccardSimilarity
    To use Jaccard similarity, set the DistanceFunction string to "JACCARD" or "JACCARDSIMILARITY" (for example, DistanceFunction = "JACCARDSIMILARITY"). This value is case sensitive. Jaccard similarity requires BINARY numeric format vectors.
Supported filter clauses
  • ==
  • !=
  • <
  • <=
  • >
  • >=
  • List.Contains()
    • Only when checking if the model property is in the list
Supports zero, one, or multiple vectors in a record Yes
IsIndexed supported? Yes
IsFullTextSearchable supported? No
StorageName supported? Yes
HybridSearch supported? No

Important

Vector data searches require Oracle Database 23ai or higher. All other Oracle connector features are available using Oracle Database 19c or higher.

More information coming soon.

Feature Area Support
Supported filter clauses
  • AnyTagEqualTo
  • EqualTo
Collection maps to SQL database table
Supported key property types
  • short/Short
  • int/Integer
  • long/Long
  • String
  • UUID
Supported data property types
  • byte/Byte
  • short/Short
  • int/Integer
  • long/Long
  • float/Float
  • double/Double
  • decimal/Decimal
  • DateTime
  • OffsetDataTime
  • Timestamp
  • String
  • UUID
  • List<of all above types>
Supported vector property types
  • String
  • Collection<Float>
  • List<Float>
  • Float[]
  • float[]
Supported index types
  • HNSW
  • IVF
Supported distance functions
  • DOT_PRODUCT
  • COSINE_SIMILARITY
  • COSINE_DISTANCE
  • EUCLIDEAN_DISTANCE
Supports multiple vectors in a record Yes
IsIndexed support? Yes
IsFullTextSearchable supported? No
StoragePropertyName supported? No, use @JsonProperty instead
HybridSearch supported? No

Getting started

Add the Oracle Database Vector Store connector NuGet package to your project.

dotnet add package Oracle.VectorData --prerelease

You can add the vector store to the IServiceCollection dependency injection container using extension methods provided by Semantic Kernel. In this case, an instance of the Oracle.VectorData.OracleVectorStore class also gets registered with the container.

using Microsoft.SemanticKernel;
using Oracle.VectorData;
using Microsoft.Extensions.DependencyInjection;

// Using Kernel Builder.
var builder = Kernel.CreateBuilder();
builder.Services.AddOracleVectorStore("<connection string>");
using Microsoft.AspNetCore.Builder;
using Oracle.VectorData;
using Microsoft.Extensions.DependencyInjection;

// Using IServiceCollection with ASP.NET Core.
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOracleVectorStore("<connection string>");

Extension methods that take no parameters are also available. These require an instance of the Oracle.ManagedDataAccess.Client.OracleDataSource class to be separately registered with the dependency injection container.

using Microsoft.SemanticKernel;
using Oracle.VectorData;
using Microsoft.Extensions.DependencyInjection;
using Oracle.ManagedDataAccess.Client;

// Using Kernel Builder.
var kernelBuilder = Kernel.CreateBuilder();
builder.Services.AddSingleton<OracleDataSource>(sp =>
{
    OracleDataSourceBuilder dataSourceBuilder = new("<connection string>");  
    return dataSourceBuilder.Build();
});

builder.Services.AddOracleVectorStore();
using Microsoft.AspNetCore.Builder;
using Oracle.VectorData;
using Microsoft.Extensions.DependencyInjection;
using Oracle.ManagedDataAccess.Client;

// Using IServiceCollection with ASP.NET Core.
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton<OracleDataSource>(sp =>
{
    OracleDataSourceBuilder dataSourceBuilder = new("<connection string>");  
    return dataSourceBuilder.Build();
});

builder.Services.AddOracleVectorStore();

You can construct an Oracle Database Vector Store instance directly with a custom data source or with a connection string.

using Oracle.VectorData;
using Oracle.ManagedDataAccess.Client;

OracleDataSourceBuilder dataSourceBuilder = new("<connection string>");
var dataSource = dataSourceBuilder.Build();

var connection = new OracleVectorStore(dataSource);
using Oracle.VectorData;

var connection = new OracleVectorStore("<connection string>");

It is possible to construct a direct reference to a named collection with a custom data source or with a connection string.

using Oracle.VectorData;
using Oracle.ManagedDataAccess.Client;

OracleDataSourceBuilder dataSourceBuilder = new("<connection string>");
var dataSource = dataSourceBuilder.Build();

var collection = new OracleCollection<string, Hotel>(dataSource, "skhotels");
using Oracle.VectorData;

var collection = new OracleCollection<string, Hotel>("<connection string>", "skhotels");

Getting started

More information coming soon.

Getting started

Set up Oracle Database Vector Store connector.

// Copyright (c) Microsoft. All rights reserved.
package com.microsoft.semantickernel.samples.syntaxexamples.memory;

import com.microsoft.semantickernel.data.jdbc.JDBCVectorStore;
import com.microsoft.semantickernel.data.jdbc.JDBCVectorStoreOptions;
import com.microsoft.semantickernel.data.jdbc.JDBCVectorStoreRecordCollection;
import com.microsoft.semantickernel.data.jdbc.JDBCVectorStoreRecordCollectionOptions;
import com.microsoft.semantickernel.data.jdbc.oracle.OracleVectorStoreQueryProvider;
import com.microsoft.semantickernel.data.vectorstorage.VectorStoreRecordCollection;
import com.microsoft.semantickernel.samples.documentationexamples.data.index.Hotel;
import java.sql.SQLException;
import java.util.Collections;
import oracle.jdbc.datasource.impl.OracleDataSource;

public class VectorStoreWithOracle {

    public static void main(String[] args) throws SQLException {
        System.out.println("==============================================================");
        System.out.println("============== Oracle Vector Store Example ===================");
        System.out.println("==============================================================");

        // Configure the data source
        OracleDataSource dataSource = new OracleDataSource();
        dataSource.setURL("jdbc:oracle:thin:@localhost:1521/FREEPDB1");
        dataSource.setUser("scott");
        dataSource.setPassword("tiger");

        // Build a query provider
        OracleVectorStoreQueryProvider queryProvider = OracleVectorStoreQueryProvider.builder()
            .withDataSource(dataSource)
            .build();

        // Build a vector store
        JDBCVectorStore vectorStore = JDBCVectorStore.builder()
            .withDataSource(dataSource)
            .withOptions(JDBCVectorStoreOptions.builder()
                .withQueryProvider(queryProvider)
                .build())
            .build();

        // Get a collection from the vector store
        VectorStoreRecordCollection<String, Hotel> collection = vectorStore.getCollection(
            "skhotels",
            JDBCVectorStoreRecordCollectionOptions.<Hotel>builder()
                .withRecordClass(Hotel.class)
                .build());

        // Create the collection if it doesn't exist yet.
        collection.createCollectionAsync().block();

        collection.upsertAsync(new Hotel("1",
            "HotelOne",
            "Desc for HotelOne",
            Collections.emptyList(), Collections.emptyList()),
            null)
            .block();

    }

}

Data mapping

The Oracle Database Vector Store connector provides a default mapper when mapping data from the data model to storage. This mapper does a direct conversion of the data model properties list to the Oracle database columns to convert to the storage schema.

The Oracle Database Vector Store connector supports data model annotations and record definitions.Using annotations, the information can be provided to the data model for creating indexes and database column mapping. Using record definitions, the information can be defined and supplied separately from the data model.

The following table shows the default primary key data type mapping between Oracle Database and C#:

C# Data Type Database Type
short/int16 NUMBER(5)
int/int32 NUMBER(10)
long/int64 NUMBER(19)
string NVARCHAR2(2000)
Guid RAW(16)

The following table shows the default data property type mapping, including nullable types:

C# Data Type Database Type
bool BOOLEAN for Oracle Database 23ai and higher
NUMBER(1) for earlier versions
byte NUMBER(3)
short/int16 NUMBER(5)
int/int32 NUMBER(10)
decimal NUMBER(18,2)
long/int64 NUMBER(19)
float BINARY_FLOAT
double BINARY_DOUBLE
DateTime TIMESTAMP(7)
DateTimeOffset TIMESTAMP(7) WITH TIME ZONE
TimeSpan INTERVAL DAY(8) TO SECOND(7)
char NVARCHAR2(1)
char[] NVARCHAR2(2000)
byte[] RAW(2000)
string NVARCHAR2(2000)
Guid RAW(16)

Starting with Oracle Database 23ai, database vectors can be mapped to .NET. data types. Multiple vector columns are supported. The following table shows the default vector property type mapping, including nullable types:

C# Data Type Database Type
  • ReadOnlyMemory<byte>
  • Embedding<System.Byte>
  • BinaryEmbedding
  • Embedding<byte>
  • byte[]
  • System.Byte[]
  • BitArray
VECTOR(dimensions, BINARY)
  • ReadOnlyMemory<short>
  • ReadOnlyMemory<System.Int16>
  • Embedding<short>
  • Embedding<System.Int16>
  • short[]
  • System.Int16[]
VECTOR(dimensions, INT8)
  • ReadOnlyMemory<double>
  • ReadOnlyMemory<System.Double>
  • Embedding<System.Double>
  • Embedding<double>
  • double[]
  • System.Double[]
VECTOR(dimensions, FLOAT64)
  • ReadOnlyMemory<float>
  • ReadOnlyMemory<System.Float>
  • Embedding<float>
  • Embedding<System.Float>
  • float[]
  • System.Float[]
VECTOR(dimensions, FLOAT32)

Property name override

For data properties and vector properties, you can override names to use in storage that are different from the data model property names. The property name override occurs when setting the StorageName option either in the data model properties or record definition.

Here is a data model with StorageName set code sample and how that will be represented in an Oracle SQL command.

using Microsoft.Extensions.VectorData;

public class Hotel
{
    [VectorStoreKey]
    public long HotelId { get; set; }

    [VectorStoreData(StorageName = "hotel_name")]
    public string? HotelName { get; set; }

    [VectorStoreData(StorageName = "hotel_description")]
    public string? Description { get; set; }

    [VectorStoreVector(Dimensions: 384, DistanceFunction = DistanceFunction.CosineDistance)]
    public ReadOnlyMemory<float>? DescriptionEmbedding { get; set; }
}
CREATE TABLE "MYSCHEMA"."Hotels"
  ("HotelId" NUMBER(10),
   "hotel_name" NVARCHAR2(2000),
   "hotel_description" NVARCHAR2(2000),
   "DescriptionEmbedding" VECTOR(384, FLOAT32),
   PRIMARY KEY ( "HotelId" )
);

Learn More

Refer to the following Oracle Database Vector Store connector resources to learn more:

Data mapping

The Oracle Database Vector Store connector provides a default mapper when mapping data from the data model to storage. This mapper does a direct conversion of the data model properties list to the Oracle database columns to convert to the storage schema.

The Oracle Database Vector Store connector supports data model annotations and record definitions.Using annotations, the information can be provided to the data model for creating indexes and database column mapping. Using record definitions, the information can be defined and supplied separately from the data model.

The following table shows the default primary key data type mapping between Oracle Database and Java, along with the corresponding methods to retrieve data from a ResultSet:

Java Type Database Type ResultSet Getter Method
byte/Byte NUMBER(3) resultSet.getByte(name)
short/Short NUMBER(5) resultSet.getShort(name)
int/Integer NUMBER(10) resultSet.getInt(name)
long/Long NUMBER(19) resultSet.getLong(name)
String NVARCHAR2(2000) resultSet.getString(name)
UUID RAW(16) resultSet.getObject(name, java_type)

The following table shows the default data property type mapping along with the corresponding methods to retrieve data from a ResultSet:

Java Type Database Type ResultSet Getter Method
boolean BOOLEAN resultSet.getByte(name)
byte/Byte NUMBER(3) resultSet.getByte(name)
byte[] RAW(2000) resultSet.getBytes(name)
short/Short NUMBER(5) resultSet.getShort(name)
int/Integer NUMBER(10) resultSet.getInt(name)
long/Long NUMBER(19) resultSet.getLong(name)
float/Float BINARY_FLOAT resultSet.getFloat(name)
double/Double BINARY_DOUBLE resultSet.getDouble(name)
BigDecimal NUMBER(18,2) resultSet.getBigDecimal(name)
OffsetDateTime TIMESTAMP(7) WITH TIME ZONE resultSet.getTIMESTAMPTZ(name).offsetDateTimeValue()
String CLOB/NVARCHAR2(%s) resultSet.getString(name)
UUID RAW(16) resultSet.getObject(name, java_type)
List<T> JSON resultSet.getObject(name, java_type) Using ojdbc-extensions-jackson-oson

Starting with Oracle Database 23ai, database vectors can be mapped to Java data types. Multiple vector columns are supported. The following table shows the default vector property type mapping:

Java Type Database Type
String VECTOR(%d, FLOAT32)
Collection<Float> VECTOR(%d, FLOAT32)
List<Float> VECTOR(%d, FLOAT32)
Float[] VECTOR(%d, FLOAT32)
float[] VECTOR(%d, FLOAT32)