llm-code-highlighter

llm-code-highlighter is a TypeScript library for creating succinct repository highlights, based on a simplified version of Paul Gauthier's repomap technique as outlined in the Aider Chat docs and implemented by the code in the aider and grep-ast repos.

This typescript version of the original python code was created with assistance from ChatGPT 4. Every line of code was manually curated (by me, @restlessronin 😇).

Installation

At the moment, this plugin is only being actively developed against the Continue repository. If you are interested in other use cases, please file an issue on the repository and I'll see what I can do to accommodate.

To install llm-code-highlighter, you can use npm or yarn:

npm install llm-code-highlighter

Usage

To use llm-code-highlighter in your TypeScript project, you need to import the required functions:

import { getHighlightsThatFit, getOutlines, ILLMContextSizer } from 'llm-code-highlighter';

getHighlightsThatFit

This function identifies highlights within the code by selecting high-ranked tags from "chat" and "other" source files. It then determines and returns the maximum number of highlighted lines, exclusively from "other" sources, that can be included within the token budget defined by the ContextSizer.

const contextSizer = {
  fits(content: string): boolean {
    return content.length <= 100;
  },
} as ILLMContextSizer;

const chatSources = [
  {
    relPath: 'chat1.js',
    code: `
          console.log(add(1, 2));
        `,
  },
  {
    relPath: 'chat2.js',
    code: `
          console.log(multiply(3, 1));
        `,
  },
];

const otherSources = [
  {
    relPath: 'file1.js',
    code: `
function add(a, b) {
  return a + b;
}
console.log(add(1, 2));
`,
  },
  {
    relPath: 'file2.js',
    code: `
function subtract(a, b) {
  return a - b;
}
console.log(subtract(3, 1));
`,
  },
  {
    relPath: 'file3.js',
    code: `
function multiply(a, b) {
  return a * b;
}
console.log(multiply(2, 3));
`,
  },
];

const result = await getHighlightsThatFit(contextSizer, chatSources, otherSources);
console.log(result);
// Output:
// file1.js
// ⋮...
// █function add(a, b) {
// ⋮...
//
// file3.js
// ⋮...
// █function multiply(a, b) {
// ⋮...

getOutlines

This function generates an outline for a set of files by only displaying the definition lines. It takes an array of objects, each containing the path and source code for a file. The function generates outlines for each of the files, concatenates all of them, and returns the result as a single string.

const sources = [
  {
    relPath: 'file1.js',
    code: `
function add(a, b) {
  return a + b;
}`,
  },
  {
    relPath: 'file2.js',
    code: `
function subtract(a, b) {
  return a - b;
}`,
  },
];

const outlines = await getOutlines(sources);

console.log(outlines);
// Output:
// file1.js
// █function add(a, b) {
// ⋮...
//
// file2.js
//  █function subtract(a, b) {
// ⋮...

Please refer to the source code for more details and options.