How Can We Create Custom Field Types in Payload?

[flowingkhaos]

Hi there!

I am using a postgres adapter to use supabase as DB, and i am working on a RAG system. Which means that i want to store fields of type vector and tsvector into supabase postgres DB. I can't figure out how to accomplish that since it isn't clear how to declare custom field types.

For instance,

  {
  
  //collections/Article.ts
  
  import { CollectionConfig } from 'payload'
  import type { Field } from 'payload'
  

    export const Articles: CollectionConfig = {
     slug: 'articles',
        fields: [
        {
          name: 'embedding',
          type: 'json', // <- i need to change this to a vector type
          required: true,
         },
        ]  as Field[],
  }

In this case, in order to store embeddings in the database articles table, the field type in the payload Article.ts collection file AND the column type in the DB table NEED to match. However as we all know there is a limited selection of types native to payloads framework:

array
blocks
checkbox
json
code
collapsible
date
email
group
number
point
radio
relationship
rich-text
select
tabs
text
textarea
ui
upload

So here I ask: Does anyone know how to declare custom field types?

[DanRibbens]

Hey @flowingkhaos what you're working on sounds very interesting!

@r1tsuu has your answer from this other discussion: #8544 (comment)

You should be able to iterate from that and make the custom db type that way.

[flowingkhaos]

Nice, thanks for the direction!

[flowingkhaos]

My situation was very specific, but not necessarily uncommon. I am using supabase through the postgres adapter. I wanted to create collections with an ' embeddings ' field, so that I can perform RAG on these tables using my own technology.

In that specific instance, supabase expects ' vector ' and ' tsvector ' column types for search functions to work as intended. These types aren't supported in payload. No matter how i flipped it, I wasn't able to create a type match, between the backend and supabase(postgres) for embeddings.

If your field typing issue in payload, is related to storing vectors(embeddings) in collections, here is my solution.

Supabase gives you two project slots by default. Therefore, I encourage AI developers/engineers, that want to use payload as a cms to create content, generate embeddings for it and store it, to do as follows:

• Create a second project in your account, to store copies of each entry for the collection of your choice, with their embeddings added.
• Check the payload-types.ts file, or the collection table in your original supabase project, in order to create its parallel table in the second project. Leave out the fields/columns that aren't necessary to transfer over.
• Create a ' lib ' folder, where you will paste supabase's ' App frameworks ' boiler plate code, in order to configure a second supabase project in your application. (don't forget to set your .env variables)
• Create an ' actions ' folder, where you will write your server action or helper function to generate embeddings for a given field.
• Create a ' hooks ' folder where you will write a payload hook that triggers afterChange. This is where you will implement embedding and storing logic for each entry that is created or saved from the admin ui.

It is not as messy and complicated as it sounds.. The concept is to use two parallel tables that each reside in different supabase projects, to avoid conflict. One for storing cms entries, and one for storing copies with their embeddings. We use payload's hooks to generate an embedding and store the entry in the parallel table every time an entry is created, updated or duplicated. The parallel table will always be in the second supabse project, the one that has copies of your entries with embeddings. don't forget to use a field that is unique, in order to avoid duplicate content in the tables that contains embeddings, since embedding costs money. Slug is commonly used for that purpose. It is far from the most efficient solution, but it fixes the problem, while keeping your structure scalable and somewhat flexible.

Hope that was insightful..

[flowingkhaos]

This is the REVISED, and FINAL solution to this typing and compatibility issue.

i have been working on this issue for a while now. Using payload to manipulate and view data types that aren't supported locally can be messy and hard if your not an experienced dev with a concise goal and approach to your project. It's hard because, most of the time, you are scaffolding payload over an existing structure.. Thankfully, Payload offers you two approaches to deal with data.

Everything Done and Sync. For You, Versus Manual data manipulation

1. You can either use payload out of the box, and have the API ovewrite your DB tables and automatically sync everything from schemas to types etc. This is why payload is very attractive, but the problems start to reveal themselves when you want to manipulate data that is created outside of that environment and also when you want to manipulate unsupported data types and formats.

2. In that case Payload offers you the option to use an ORM and along with their different DB adapters to manually generate and migrate schemas and finally synchronise tables to have a uniform type and schema interface in your application. That solution works so well because you can manage data that was created outside of payload, and inside of it as well equally, and the local API will work as well on any view. But you loose the apeal of having payload handle absolutely everything for your, and you most likely need to learn ORMs.

i didn't want to learn ORMs and i was forced to because of this situation, and i have to admit, i regret not learning those sooner, so in my humble opinion you should learn ORMs if you haven't already. That will really get you out of a pinch, when you need to manipulate objects throughout complex structures, while maintaining their type safety...

So to sum up, if your application structure is complex, and you need special data types like embeddings for example, you should create them outside of payload using an ORM like drizzle and create collections using the dbName property to match the table in the DB and make sure there are no naming conflict between the tables created from drizzle or prisma versus the ones created in payload. For instance, you can't have a postgres schema named "users" using drizzle, and have a payload collection called "users", while expecting to find two different tables, that will create a conflict, since payload doesn't know what table you are reffering to. so for example, what i do is i create an "Admin" collection using payload for the admin dashboard, then i create a "Users" schema in my application using drizzle for accessing the application front-end. the i migrate and sync everything using payload and drizzle CLI commands.

It sounds complicated but in reality it is so easy once you get it, the docs aren't bad, they're just lacking a bit. Read those docs to fully understand whats going on under the hood:

database overview
postgres adapter
collections

Here is some extra info to give you direction:

your app might look like this:

• /src/app: Next.js application routes and pages
• /src/collections: Payload CMS collection definitions
• /src/components: React components for UI elements
• /src/lib/db: Database schema, migrations, and queries

this is what a collection looks like when created inside of payload:

import type { CollectionConfig } from 'payload';
import { isAdmin, isAdminFieldLevel } from '@/access/isAdmin';
import { isAdminOrSelf } from '@/access/isAdminOrSelf';

export const Admins: CollectionConfig = {
  slug: 'admins',
  admin: {
    useAsTitle: 'username',
  },
  fields: [
    {
      type: 'row',
      fields: [
        {
          name: 'username',
          type: 'text',
          required: true,
        },
        // Email added by default
      ],
    },
  ],
};

this is what a collection looks like when the table was created outside of payload:

// collections/User.ts
import { CollectionConfig } from 'payload';

export const Users: CollectionConfig = {
  slug: 'users',
  admin: {
    useAsTitle: 'email',
  },
  fields: [
    {
      name: 'id',
      type: 'text',
      admin: {
        readOnly: true,
      },
    },
    {
      name: 'email',
      type: 'email',
      required: true,
    },
    {
      name: 'password',
      type: 'text',
      admin: {
        readOnly: true,
        position: 'sidebar',
        description: 'Password is stored securely and cannot be edited here',
      },
    },
  ],
  // Set custom database name to match existing schema
  dbName: 'User',
};

here is what a config file looks like when using this hybrid approach with tables coming from both inside and outside of payload:

// storage-adapter-import-placeholder
import { postgresAdapter } from '@payloadcms/db-postgres'
import { payloadCloudPlugin } from '@payloadcms/payload-cloud'
import { lexicalEditor } from '@payloadcms/richtext-lexical'
import path from 'path'
import { buildConfig } from 'payload'
import { fileURLToPath } from 'url'
import sharp from 'sharp'

//?? Import your collection configs
import { Admins } from './collections/Admin'; // i import regular collections created by payload as we usually would
import { Media } from './collections/Media';
import { 
  Users,
} from './collections'; // i put external tables in an index.ts file in the collections folder so i can identify them quickly

//?? Import your schema from the existing drizzle schema
import { 
  user, 
} from '@/lib/db/schema';

const filename = fileURLToPath(import.meta.url)
const dirname = path.dirname(filename)

export default buildConfig({
  admin: {
    user: Admins.slug,
    importMap: {
      baseDir: path.resolve(dirname),
    },
  },
  email: brevoAdapter(),
  collections: [
    // tables created from payload
    Admins, 
    Media,
    // external tables created from any ORM
    Users,
  ],
  editor: lexicalEditor(),
  secret: (process.env.PAYLOAD_SECRET as string) ?? '',
  typescript: {
    outputFile: path.resolve(dirname, 'payload-types.ts'),
  },
  db: postgresAdapter({
    pool: {
      connectionString: (process.env.DATABASE_URI as string) || '',
    },
    push: false, // Important! We don't want to modify our existing schema that were created outside of payload using drizzle or prisma
    beforeSchemaInit: [
      ({ schema, adapter }) => {
        return {
          ...schema,
          tables: {
            ...schema.tables,
            // Add your existing drizzle tables to the schema
            User: user,
          },
        };
      },
    ],
  }),
  sharp,
  plugins: [
    payloadCloudPlugin(),
    // storage-adapter-placeholder
  ],
})

this is what a drizzle schema looks like in src/lib/db/schema.ts:

import type { InferSelectModel } from 'drizzle-orm';
import {
  pgTable,
  varchar,
  timestamp,
  json,
  uuid,
  text,
  primaryKey,
  foreignKey,
  boolean,
} from 'drizzle-orm/pg-core';

export const user = pgTable('User', {
  id: uuid('id').primaryKey().notNull().defaultRandom(),
  email: varchar('email', { length: 64 }).notNull(),
  password: varchar('password', { length: 64 }),
});

export type User = InferSelectModel<typeof user>;

CLI commands:

• pnpm db:generate: Generate Drizzle schema
• pnpm db:migrate: Run migrations
• pnpm generate:types: Generate Payload types
• pnpm generate:importmap: Generate import map

I really hope that could help. Cheers.