Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[csv-loader] Support for generating unique id's #49

Open
marceloverdijk opened this issue Oct 24, 2024 · 1 comment
Open

[csv-loader] Support for generating unique id's #49

marceloverdijk opened this issue Oct 24, 2024 · 1 comment

Comments

@marceloverdijk
Copy link

Hi,

First of all thank you for this project.

I have some csv files without a defined id column, nor a column which values are unique itself.
E.g. a csv with a year and month column.

Would it be an idea to provide a (new) CSVLoaderOptions option to automatically generate an id?
Or to have a Function option to generate an id for each row?
E.g. (row) => `${year}-${month}`

@marceloverdijk
Copy link
Author

marceloverdijk commented Oct 24, 2024

I was looking at the papaparse options and thought the step might be useful for this:

  loader: csvLoader({
    fileName: 'data/my-data.csv',
    idField: 'id',
    parserOptions: {
      step: (result) => {
        const row = result.data;
        row.id = `${row.year}-${row.month};
      },
    },
  }),

However that option is omitted unfortunately:

https://github.com/ascorbic/astro-loaders/blob/main/packages/csv/src/csv-loader.ts#L25-L28

  parserOptions?: Omit<
    Papa.ParseConfig,
    "header" | "dynamicTyping" | "transformHeader" | "step" | "complete"
  >;

for the other options I understand why they are omitted, but not for the step:

    const csvStream = Papa.parse(Papa.NODE_STREAM_INPUT, {
      dynamicTyping: true,
      ...parserOptions,
      header: true,
      transformHeader: transformHeader === false ? undefined : transformHeader,
    });

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant