Overview
In this example we’ll be using a number of different tools and features to:- Scrape the content of the top 3 articles from Hacker News
- Summarize each article
- Email the summaries to yourself
- Schedules to run the task every weekday at 9 AM
- Batch Triggering to run separate child tasks for each article while the parent task waits for them all to complete
- idempotencyKey to prevent tasks being triggered multiple times
- BrowserBase to proxy the scraping of the Hacker News articles
- Puppeteer to scrape the articles linked from Hacker News
- OpenAI to summarize the articles
- Resend to send a nicely formatted email summary
WEB SCRAPING: When web scraping, you MUST use a proxy to comply with our terms of service. Direct scraping of third-party websites without the site owner’s permission using Trigger.dev Cloud is prohibited and will result in account suspension. See this example which uses a proxy.
Prerequisites
- A project with Trigger.dev initialized
- Puppeteer installed on your machine
- A BrowserBase account
- An OpenAI account
- A Resend account
Build configuration
First up, add these build settings to yourtrigger.config.ts
file:
trigger.config.ts
Environment variables
Set the following environment variable in your local.env
file to run this task locally. And before deploying your task, set them in the Trigger.dev dashboard or using the SDK:
Task code
trigger/scrape-hacker-news.ts
Create your email template using React Email
To prevent the main example from becoming too cluttered, we’ll create a separate file for our email template. It’s formatted using React Email components so you’ll need to install the package to use it. Notice how this file is imported into the main task code and passed to Resend to send the email.summarize-hn-email.tsx
Local development
To test this example task locally, be sure to install any packages from the build extensions you added to yourtrigger.config.ts
file to your local machine. In this case, you need to install .