Use a DAO to govern LLM training data, Part 3: From IPFS to the knowledge base

This article discusses how to set up AWS components like Lambda functions and API Gateway to copy data from the InterPlanetary File System (IPFS) to an Amazon S3 bucket, and then trigger ingestion of that data into an Amazon Bedrock knowledge base for training a large language model (LLM).

Specifically, the article covers:

• Setting up a Lambda function (s32kb) to refresh the knowledge base when new content is added to the S3 bucket
• Creating an S3 trigger to invoke the s32kb Lambda function
• Setting up another Lambda function (ipfs2s3) to upload content from IPFS to the S3 bucket
• Creating an API Gateway to invoke the ipfs2s3 Lambda function
• Testing the setup by uploading sample files and verifying the knowledge base ingestion