KeyedVectors | Wisty.js

A reusable class storing words embeddings for functions and class that needs it.

Hierarchy

KeyedVectors

Index

Constructors

constructor

Properties

Methods

Constructors

constructor

new KeyedVectors(__namedParameters: { cased: boolean; loaderFunction: loaderFunction; maxDistance: number; size: number; tokenization: "word" | "byte_pair"; unknownKey: string }): KeyedVectors

- Defined in tools/keyed_vectors.ts:30
Build a KeyedVector.

Parameters
- __namedParameters: { cased: boolean; loaderFunction: loaderFunction; maxDistance: number; size: number; tokenization: "word" | "byte_pair"; unknownKey: string }
  - cased: boolean
  - loaderFunction: loaderFunction
  - maxDistance: number
  - size: number
  - tokenization: "word" | "byte_pair"
  - unknownKey: string
Returns KeyedVectors

Properties

Private cased

cased: boolean

Private loaderFunction

loaderFunction: () => Promise<string>

Type declaration

- (): Promise<string>
- Returns Promise<string>

Private maxDistance

maxDistance: number

Readonly size

size: number

Private tokenization

tokenization: "word" | "byte_pair"

Private trie

trie: Trie

Private unknownKey

unknownKey: string

Private vectors

vectors: {}

Type declaration

[key: string]: number[]

Methods

Private bytePairTokenize

bytePairTokenize(text: string): string[]

- Defined in tools/keyed_vectors.ts:136
Tokenize a string based on the vocabulary.

Parameters
- text: string
  
  A non tokenized text string.
Returns string[]

get

get(key: string): tf.Tensor1D

- Defined in tools/keyed_vectors.ts:90
Return the vector associated with a key. If the key is not part of the vocabulary, it will use a similar key according to the leveinshtein distance. If no similar keys are below maxDistance, it will return the unknown key vector or undefined.

Parameters
- key: string
Returns tf.Tensor1D

isLoaded

isLoaded(): boolean

- Defined in tools/keyed_vectors.ts:79
Check if the word embeddings were loaded.

Returns boolean

keys

keys(): string[]

- Defined in tools/keyed_vectors.ts:60
Return every keys stored as an array.

Returns string[]

load

load(): Promise<void>

- Defined in tools/keyed_vectors.ts:67
Load the word embeddings.

Returns Promise<void>

tokenize

tokenize(text: string): string[]

- Defined in tools/keyed_vectors.ts:147
Tokenize a string based on the settings.

Parameters
- text: string
  
  A raw text string.
Returns string[]

Private wordTokenize

wordTokenize(text: string): string[]

- Defined in tools/keyed_vectors.ts:127
Tokenize a string at each non-word character.

Parameters
- text: string
  
  A non tokenized text string.
Returns string[]

Generated using TypeDoc