feat: add 12 official Apify agent-skills for web scraping & data extraction (#165)
* feat: add 12 official Apify skills for web scraping and data extraction Add the complete Apify agent-skills collection as official vendor skills, bringing the total skill count from 954 to 966. New skills: - apify-actor-development: Develop, debug, and deploy Apify Actors - apify-actorization: Convert existing projects into Apify Actors - apify-audience-analysis: Audience demographics across social platforms - apify-brand-reputation-monitoring: Track reviews, ratings, and sentiment - apify-competitor-intelligence: Analyze competitor strategies and pricing - apify-content-analytics: Track engagement metrics and campaign ROI - apify-ecommerce: E-commerce data scraping for pricing intelligence - apify-influencer-discovery: Find and evaluate influencers - apify-lead-generation: B2B/B2C lead generation from multiple platforms - apify-market-research: Market conditions and geographic opportunities - apify-trend-analysis: Discover emerging trends across platforms - apify-ultimate-scraper: Universal AI-powered web scraper Existing skill fixes: - design-orchestration: Add missing description, fix markdown list spacing - multi-agent-brainstorming: Add missing description, fix markdown list spacing Registry and documentation updates: - Update skill count to 966+ across README.md, README.vi.md - Add Apify to official sources in SOURCES.md and all README variants - Register new skills in catalog.json, skills_index.json, bundles.json, aliases.json - Update CATALOG.md category counts (data-ai: 152, infrastructure: 95) Validation script improvements: - Raise description length limit from 200 to 1024 characters - Add empty description validation check - Apply PEP 8 formatting (line length, spacing, trailing whitespace) * refactor: truncate skill descriptions in SKILL.md files and revert description length validation to 200 characters. * feat: Add `apify-ultimate-scraper` to data-ai and move `apify-lead-generation` from business to general categories.
This commit is contained in:
Ahmed Rehan
GitHub
@@ -30,7 +30,7 @@ If this project helps you, you can [support it here](https://buymeacoffee.com/si
|
||||
- ⚪ **OpenCode** (Open-source CLI)
|
||||
- 🌸 **AdaL CLI** (Self-evolving Coding Agent)
|
||||
|
||||
This repository provides essential skills to transform your AI assistant into a **full-stack digital agency**, including official capabilities from **Anthropic**, **OpenAI**, **Google**, **Microsoft**, **Supabase**, and **Vercel Labs**.
|
||||
This repository provides essential skills to transform your AI assistant into a **full-stack digital agency**, including official capabilities from **Anthropic**, **OpenAI**, **Google**, **Microsoft**, **Supabase**, **Apify**, and **Vercel Labs**.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -472,6 +472,7 @@ This collection would not be possible without the incredible work of the Claude
|
||||
- **[supabase/agent-skills](https://github.com/supabase/agent-skills)**: Supabase official skills - Postgres Best Practices.
|
||||
- **[microsoft/skills](https://github.com/microsoft/skills)**: Official Microsoft skills - Azure cloud services, Bot Framework, Cognitive Services, and enterprise development patterns across .NET, Python, TypeScript, Go, Rust, and Java.
|
||||
- **[google-gemini/gemini-skills](https://github.com/google-gemini/gemini-skills)**: Official Gemini skills - Gemini API, SDK and model interactions.
|
||||
- **[apify/agent-skills](https://github.com/apify/agent-skills)**: Official Apify skills - Web scraping, data extraction and automation.
|
||||
|
||||
### Community Contributors
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
"agent-orchestration-optimize": "agent-orchestration-multi-agent-optimize",
|
||||
"android-jetpack-expert": "android-jetpack-compose-expert",
|
||||
"api-testing-mock": "api-testing-observability-api-mock",
|
||||
"apify-brand-monitoring": "apify-brand-reputation-monitoring",
|
||||
"templates": "app-builder/templates",
|
||||
"application-performance-optimization": "application-performance-performance-optimization",
|
||||
"azure-ai-dotnet": "azure-ai-agents-persistent-dotnet",
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
"api-security-best-practices",
|
||||
"api-security-testing",
|
||||
"api-testing-observability-api-mock",
|
||||
"apify-actorization",
|
||||
"app-store-optimization",
|
||||
"appdeploy",
|
||||
"application-performance-performance-optimization",
|
||||
@@ -385,6 +386,10 @@
|
||||
"airflow-dag-patterns",
|
||||
"analytics-tracking",
|
||||
"angular-ui-patterns",
|
||||
"apify-actor-development",
|
||||
"apify-content-analytics",
|
||||
"apify-ecommerce",
|
||||
"apify-ultimate-scraper",
|
||||
"appdeploy",
|
||||
"azure-ai-document-intelligence-dotnet",
|
||||
"azure-ai-document-intelligence-ts",
|
||||
@@ -489,6 +494,7 @@
|
||||
"agent-evaluation",
|
||||
"airflow-dag-patterns",
|
||||
"api-testing-observability-api-mock",
|
||||
"apify-brand-reputation-monitoring",
|
||||
"application-performance-performance-optimization",
|
||||
"aws-serverless",
|
||||
"azd-deployment",
|
||||
|
||||
@@ -3,16 +3,16 @@
|
||||
We believe in giving credit where credit is due.
|
||||
If you recognize your work here and it is not properly attributed, please open an Issue.
|
||||
|
||||
| Skill / Category | Original Source | License | Notes |
|
||||
| :-------------------------- | :----------------------------------------------------------------- | :------------- | :---------------------------- |
|
||||
| `cloud-penetration-testing` | [HackTricks](https://book.hacktricks.xyz/) | MIT / CC-BY-SA | Adapted for agentic use. |
|
||||
| `active-directory-attacks` | [HackTricks](https://book.hacktricks.xyz/) | MIT / CC-BY-SA | Adapted for agentic use. |
|
||||
| `owasp-top-10` | [OWASP](https://owasp.org/) | CC-BY-SA | Methodology adapted. |
|
||||
| `burp-suite-testing` | [PortSwigger](https://portswigger.net/burp) | N/A | Usage guide only (no binary). |
|
||||
| `crewai` | [CrewAI](https://github.com/joaomdmoura/crewAI) | MIT | Framework guides. |
|
||||
| `langgraph` | [LangGraph](https://github.com/langchain-ai/langgraph) | MIT | Framework guides. |
|
||||
| `react-patterns` | [React Docs](https://react.dev/) | CC-BY | Official patterns. |
|
||||
| **All Official Skills** | [Anthropic / Google / OpenAI / Microsoft / Supabase / Vercel Labs] | Proprietary | Usage encouraged by vendors. |
|
||||
| Skill / Category | Original Source | License | Notes |
|
||||
| :-------------------------- | :------------------------------------------------------------------------- | :------------- | :---------------------------- |
|
||||
| `cloud-penetration-testing` | [HackTricks](https://book.hacktricks.xyz/) | MIT / CC-BY-SA | Adapted for agentic use. |
|
||||
| `active-directory-attacks` | [HackTricks](https://book.hacktricks.xyz/) | MIT / CC-BY-SA | Adapted for agentic use. |
|
||||
| `owasp-top-10` | [OWASP](https://owasp.org/) | CC-BY-SA | Methodology adapted. |
|
||||
| `burp-suite-testing` | [PortSwigger](https://portswigger.net/burp) | N/A | Usage guide only (no binary). |
|
||||
| `crewai` | [CrewAI](https://github.com/joaomdmoura/crewAI) | MIT | Framework guides. |
|
||||
| `langgraph` | [LangGraph](https://github.com/langchain-ai/langgraph) | MIT | Framework guides. |
|
||||
| `react-patterns` | [React Docs](https://react.dev/) | CC-BY | Official patterns. |
|
||||
| **All Official Skills** | [Anthropic / Google / OpenAI / Microsoft / Supabase / Apify / Vercel Labs] | Proprietary | Usage encouraged by vendors. |
|
||||
|
||||
## Skills from VoltAgent/awesome-agent-skills
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@
|
||||
Các trợ lý AI (như Claude Code, Cursor, hoặc Gemini) rất thông minh, nhưng chúng thiếu các **công cụ chuyên biệt**. Chúng không biết "Quy trình Triển khai" của công ty bạn hoặc cú pháp cụ thể cho "AWS CloudFormation".
|
||||
**Skills** là các tệp markdown nhỏ dạy cho chúng cách thực hiện những tác vụ cụ thể này một cách chính xác trong mọi lần thực thi.
|
||||
...
|
||||
Repository này cung cấp các kỹ năng thiết yếu để biến trợ lý AI của bạn thành một **đội ngũ chuyên gia số toàn năng**, bao gồm các khả năng chính thức từ **Anthropic**, **OpenAI**, **Google**, **Supabase**, và **Vercel Labs**.
|
||||
Repository này cung cấp các kỹ năng thiết yếu để biến trợ lý AI của bạn thành một **đội ngũ chuyên gia số toàn năng**, bao gồm các khả năng chính thức từ **Anthropic**, **OpenAI**, **Google**, **Supabase**, **Apify**, và **Vercel Labs**.
|
||||
...
|
||||
Cho dù bạn đang sử dụng **Gemini CLI**, **Claude Code**, **Codex CLI**, **Cursor**, **GitHub Copilot**, **Antigravity**, hay **OpenCode**, những kỹ năng này được thiết kế để có thể sử dụng ngay lập tức và tăng cường sức mạnh cho trợ lý AI của bạn.
|
||||
|
||||
@@ -40,17 +40,17 @@ Repository này tập hợp những khả năng tốt nhất từ khắp cộng
|
||||
|
||||
Repository được tổ chức thành các lĩnh vực chuyên biệt để biến AI của bạn thành một chuyên gia trên toàn bộ vòng đời phát triển phần mềm:
|
||||
|
||||
| Danh mục | Trọng tâm | Ví dụ kỹ năng |
|
||||
| :--- | :--- | :--- |
|
||||
| Kiến trúc (52) | Thiết kế hệ thống, ADRs, C4 và các mẫu có thể mở rộng | `architecture`, `c4-context`, `senior-architect` |
|
||||
| Kinh doanh (35) | Tăng trưởng, định giá, CRO, SEO và thâm nhập thị trường | `copywriting`, `pricing-strategy`, `seo-audit` |
|
||||
| Dữ liệu & AI (81) | Ứng dụng LLM, RAG, agents, khả năng quan sát, phân tích | `rag-engineer`, `prompt-engineer`, `langgraph` |
|
||||
| Phát triển (72) | Làm chủ ngôn ngữ, mẫu thiết kế framework, chất lượng code | `typescript-expert`, `python-patterns`, `react-patterns` |
|
||||
| Tổng quát (95) | Lập kế hoạch, tài liệu, vận hành sản phẩm, viết bài, hướng dẫn | `brainstorming`, `doc-coauthoring`, `writing-plans` |
|
||||
| Hạ tầng (72) | DevOps, cloud, serverless, triển khai, CI/CD | `docker-expert`, `aws-serverless`, `vercel-deployment` |
|
||||
| Bảo mật (107) | AppSec, pentesting, phân tích lỗ hổng, tuân thủ | `api-security-best-practices`, `sql-injection-testing`, `vulnerability-scanner` |
|
||||
| Kiểm thử (21) | TDD, thiết kế kiểm thử, sửa lỗi, quy trình QA | `test-driven-development`, `testing-patterns`, `test-fixing` |
|
||||
| Quy trình (17) | Tự động hóa, điều phối, công việc, agents | `workflow-automation`, `inngest`, `trigger-dev` |
|
||||
| Danh mục | Trọng tâm | Ví dụ kỹ năng |
|
||||
| :---------------- | :------------------------------------------------------------- | :------------------------------------------------------------------------------ |
|
||||
| Kiến trúc (52) | Thiết kế hệ thống, ADRs, C4 và các mẫu có thể mở rộng | `architecture`, `c4-context`, `senior-architect` |
|
||||
| Kinh doanh (35) | Tăng trưởng, định giá, CRO, SEO và thâm nhập thị trường | `copywriting`, `pricing-strategy`, `seo-audit` |
|
||||
| Dữ liệu & AI (81) | Ứng dụng LLM, RAG, agents, khả năng quan sát, phân tích | `rag-engineer`, `prompt-engineer`, `langgraph` |
|
||||
| Phát triển (72) | Làm chủ ngôn ngữ, mẫu thiết kế framework, chất lượng code | `typescript-expert`, `python-patterns`, `react-patterns` |
|
||||
| Tổng quát (95) | Lập kế hoạch, tài liệu, vận hành sản phẩm, viết bài, hướng dẫn | `brainstorming`, `doc-coauthoring`, `writing-plans` |
|
||||
| Hạ tầng (72) | DevOps, cloud, serverless, triển khai, CI/CD | `docker-expert`, `aws-serverless`, `vercel-deployment` |
|
||||
| Bảo mật (107) | AppSec, pentesting, phân tích lỗ hổng, tuân thủ | `api-security-best-practices`, `sql-injection-testing`, `vulnerability-scanner` |
|
||||
| Kiểm thử (21) | TDD, thiết kế kiểm thử, sửa lỗi, quy trình QA | `test-driven-development`, `testing-patterns`, `test-fixing` |
|
||||
| Quy trình (17) | Tự động hóa, điều phối, công việc, agents | `workflow-automation`, `inngest`, `trigger-dev` |
|
||||
|
||||
## Bộ sưu tập Tuyển chọn
|
||||
|
||||
@@ -119,6 +119,7 @@ Bộ sưu tập này sẽ không thể hình thành nếu không có công việ
|
||||
- **[vercel-labs/agent-skills](https://github.com/vercel-labs/agent-skills)**: Skills chính thức của Vercel Labs - Thực hành tốt nhất cho React, Hướng dẫn thiết kế Web.
|
||||
- **[openai/skills](https://github.com/openai/skills)**: Danh mục skill của OpenAI Codex - Các kỹ năng của Agent, Trình tạo Skill, Lập kế hoạch Súc tích.
|
||||
- **[supabase/agent-skills](https://github.com/supabase/agent-skills)**: Skills chính thức của Supabase - Thực hành tốt nhất cho Postgres.
|
||||
- **[apify/agent-skills](https://github.com/apify/agent-skills)**: Skills chính thức của Apify - Web scraping, data extraction and automation.
|
||||
|
||||
### Những người đóng góp từ Cộng đồng
|
||||
|
||||
|
||||
218
skills/apify-actor-development/SKILL.md
Normal file
218
skills/apify-actor-development/SKILL.md
Normal file
@@ -0,0 +1,218 @@
|
||||
---
|
||||
name: apify-actor-development
|
||||
description: "Develop, debug, and deploy Apify Actors - serverless cloud programs for web scraping, automation, and data processing. Use when creating new Actors, modifying existing ones, or troubleshooting Acto..."
|
||||
---
|
||||
|
||||
# Apify Actor Development
|
||||
|
||||
**Important:** Before you begin, fill in the `generatedBy` property in the meta section of `.actor/actor.json`. Replace it with the tool and model you're currently using, such as "Claude Code with Claude Sonnet 4.5". This helps Apify monitor and improve AGENTS.md for specific AI tools and models.
|
||||
|
||||
## What are Apify Actors?
|
||||
|
||||
Actors are serverless programs inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems. They're packaged as Docker images and run in isolated containers in the cloud.
|
||||
|
||||
**Core Concepts:**
|
||||
- Accept well-defined JSON input
|
||||
- Perform isolated tasks (web scraping, automation, data processing)
|
||||
- Produce structured JSON output to datasets and/or store data in key-value stores
|
||||
- Can run from seconds to hours or even indefinitely
|
||||
- Persist state and can be restarted
|
||||
|
||||
## Prerequisites & Setup (MANDATORY)
|
||||
|
||||
Before creating or modifying actors, verify that `apify` CLI is installed `apify --help`.
|
||||
|
||||
If it is not installed, use one of these methods (listed in order of preference):
|
||||
|
||||
```bash
|
||||
# Preferred: install via a package manager (provides integrity checks)
|
||||
npm install -g apify-cli
|
||||
|
||||
# Or (Mac): brew install apify-cli
|
||||
```
|
||||
|
||||
> **Security note:** Do NOT install the CLI by piping remote scripts to a shell
|
||||
> (e.g. `curl … | bash` or `irm … | iex`). Always use a package manager.
|
||||
|
||||
When the apify CLI is installed, check that it is logged in with:
|
||||
|
||||
```bash
|
||||
apify info # Should return your username
|
||||
```
|
||||
|
||||
If it is not logged in, check if the `APIFY_TOKEN` environment variable is defined (if not, ask the user to generate one on https://console.apify.com/settings/integrations and then define `APIFY_TOKEN` with it).
|
||||
|
||||
Then authenticate using one of these methods:
|
||||
|
||||
```bash
|
||||
# Option 1 (preferred): The CLI automatically reads APIFY_TOKEN from the environment.
|
||||
# Just ensure the env var is exported and run any apify command — no explicit login needed.
|
||||
|
||||
# Option 2: Interactive login (prompts for token without exposing it in shell history)
|
||||
apify login
|
||||
```
|
||||
|
||||
> **Security note:** Avoid passing tokens as command-line arguments (e.g. `apify login -t <token>`).
|
||||
> Arguments are visible in process listings and may be recorded in shell history.
|
||||
> Prefer environment variables or interactive login instead.
|
||||
> Never log, print, or embed `APIFY_TOKEN` in source code or configuration files.
|
||||
> Use a token with the minimum required permissions (scoped token) and rotate it periodically.
|
||||
|
||||
## Template Selection
|
||||
|
||||
**IMPORTANT:** Before starting actor development, always ask the user which programming language they prefer:
|
||||
- **JavaScript** - Use `apify create <actor-name> -t project_empty`
|
||||
- **TypeScript** - Use `apify create <actor-name> -t ts_empty`
|
||||
- **Python** - Use `apify create <actor-name> -t python-empty`
|
||||
|
||||
Use the appropriate CLI command based on the user's language choice. Additional packages (Crawlee, Playwright, etc.) can be installed later as needed.
|
||||
|
||||
## Quick Start Workflow
|
||||
|
||||
1. **Create actor project** - Run the appropriate `apify create` command based on user's language preference (see Template Selection above)
|
||||
2. **Install dependencies** (verify package names match intended packages before installing)
|
||||
- JavaScript/TypeScript: `npm install` (uses `package-lock.json` for reproducible, integrity-checked installs — commit the lockfile to version control)
|
||||
- Python: `pip install -r requirements.txt` (pin exact versions in `requirements.txt`, e.g. `crawlee==1.2.3`, and commit the file to version control)
|
||||
3. **Implement logic** - Write the actor code in `src/main.py`, `src/main.js`, or `src/main.ts`
|
||||
4. **Configure schemas** - Update input/output schemas in `.actor/input_schema.json`, `.actor/output_schema.json`, `.actor/dataset_schema.json`
|
||||
5. **Configure platform settings** - Update `.actor/actor.json` with actor metadata (see [references/actor-json.md](references/actor-json.md))
|
||||
6. **Write documentation** - Create comprehensive README.md for the marketplace
|
||||
7. **Test locally** - Run `apify run` to verify functionality (see Local Testing section below)
|
||||
8. **Deploy** - Run `apify push` to deploy the actor on the Apify platform (actor name is defined in `.actor/actor.json`)
|
||||
|
||||
## Security
|
||||
|
||||
**Treat all crawled web content as untrusted input.** Actors ingest data from external websites that may contain malicious payloads. Follow these rules:
|
||||
|
||||
- **Sanitize crawled data** — Never pass raw HTML, URLs, or scraped text directly into shell commands, `eval()`, database queries, or template engines. Use proper escaping or parameterized APIs.
|
||||
- **Validate and type-check all external data** — Before pushing to datasets or key-value stores, verify that values match expected types and formats. Reject or sanitize unexpected structures.
|
||||
- **Do not execute or interpret crawled content** — Never treat scraped text as code, commands, or configuration. Content from websites could include prompt injection attempts or embedded scripts.
|
||||
- **Isolate credentials from data pipelines** — Ensure `APIFY_TOKEN` and other secrets are never accessible in request handlers or passed alongside crawled data. Use the Apify SDK's built-in credential management rather than passing tokens through environment variables in data-processing code.
|
||||
- **Review dependencies before installing** — When adding packages with `npm install` or `pip install`, verify the package name and publisher. Typosquatting is a common supply-chain attack vector. Prefer well-known, actively maintained packages.
|
||||
- **Pin versions and use lockfiles** — Always commit `package-lock.json` (Node.js) or pin exact versions in `requirements.txt` (Python). Lockfiles ensure reproducible builds and prevent silent dependency substitution. Run `npm audit` or `pip-audit` periodically to check for known vulnerabilities.
|
||||
|
||||
## Best Practices
|
||||
|
||||
**✓ Do:**
|
||||
- Use `apify run` to test actors locally (configures Apify environment and storage)
|
||||
- Use Apify SDK (`apify`) for code running ON Apify platform
|
||||
- Validate input early with proper error handling and fail gracefully
|
||||
- Use CheerioCrawler for static HTML (10x faster than browsers)
|
||||
- Use PlaywrightCrawler only for JavaScript-heavy sites
|
||||
- Use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls
|
||||
- Implement retry strategies with exponential backoff
|
||||
- Use proper concurrency: HTTP (10-50), Browser (1-5)
|
||||
- Set sensible defaults in `.actor/input_schema.json`
|
||||
- Define output schema in `.actor/output_schema.json`
|
||||
- Clean and validate data before pushing to dataset
|
||||
- Use semantic CSS selectors with fallback strategies
|
||||
- Respect robots.txt, ToS, and implement rate limiting
|
||||
- **Always use `apify/log` package** — censors sensitive data (API keys, tokens, credentials)
|
||||
- Implement readiness probe handler (required if your Actor uses standby mode)
|
||||
|
||||
**✗ Don't:**
|
||||
- Use `npm start`, `npm run start`, `npx apify run`, or similar commands to run actors (use `apify run` instead)
|
||||
- Assume local storage from `apify run` is pushed to or visible in the Apify Console — it is local-only; deploy with `apify push` and run on the platform to see results in the Console
|
||||
- Rely on `Dataset.getInfo()` for final counts on Cloud
|
||||
- Use browser crawlers when HTTP/Cheerio works
|
||||
- Hard code values that should be in input schema or environment variables
|
||||
- Skip input validation or error handling
|
||||
- Overload servers - use appropriate concurrency and delays
|
||||
- Scrape prohibited content or ignore Terms of Service
|
||||
- Store personal/sensitive data unless explicitly permitted
|
||||
- Use deprecated options like `requestHandlerTimeoutMillis` on CheerioCrawler (v3.x)
|
||||
- Use `additionalHttpHeaders` - use `preNavigationHooks` instead
|
||||
- Pass raw crawled content into shell commands, `eval()`, or code-generation functions
|
||||
- Use `console.log()` or `print()` instead of the Apify logger — these bypass credential censoring
|
||||
- Disable standby mode without explicit permission
|
||||
|
||||
## Logging
|
||||
|
||||
See [references/logging.md](references/logging.md) for complete logging documentation including available log levels and best practices for JavaScript/TypeScript and Python.
|
||||
|
||||
Check `usesStandbyMode` in `.actor/actor.json` - only implement if set to `true`.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
apify run # Run Actor locally
|
||||
apify login # Authenticate account
|
||||
apify push # Deploy to Apify platform (uses name from .actor/actor.json)
|
||||
apify help # List all commands
|
||||
```
|
||||
|
||||
**IMPORTANT:** Always use `apify run` to test actors locally. Do not use `npm run start`, `npm start`, `yarn start`, or other package manager commands - these will not properly configure the Apify environment and storage.
|
||||
|
||||
## Local Testing
|
||||
|
||||
When testing an actor locally with `apify run`, provide input data by creating a JSON file at:
|
||||
|
||||
```
|
||||
storage/key_value_stores/default/INPUT.json
|
||||
```
|
||||
|
||||
This file should contain the input parameters defined in your `.actor/input_schema.json`. The actor will read this input when running locally, mirroring how it receives input on the Apify platform.
|
||||
|
||||
**IMPORTANT - Local storage is NOT synced to the Apify Console:**
|
||||
- Running `apify run` stores all data (datasets, key-value stores, request queues) **only on your local filesystem** in the `storage/` directory.
|
||||
- This data is **never** automatically uploaded or pushed to the Apify platform. It exists only on your machine.
|
||||
- To verify results on the Apify Console, you must deploy the Actor with `apify push` and then run it on the platform.
|
||||
- Do **not** rely on checking the Apify Console to verify results from local runs — instead, inspect the local `storage/` directory or check the Actor's log output.
|
||||
|
||||
## Standby Mode
|
||||
|
||||
See [references/standby-mode.md](references/standby-mode.md) for complete standby mode documentation including readiness probe implementation for JavaScript/TypeScript and Python.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
.actor/
|
||||
├── actor.json # Actor config: name, version, env vars, runtime
|
||||
├── input_schema.json # Input validation & Console form definition
|
||||
└── output_schema.json # Output storage and display templates
|
||||
src/
|
||||
└── main.js/ts/py # Actor entry point
|
||||
storage/ # Local-only storage (NOT synced to Apify Console)
|
||||
├── datasets/ # Output items (JSON objects)
|
||||
├── key_value_stores/ # Files, config, INPUT
|
||||
└── request_queues/ # Pending crawl requests
|
||||
Dockerfile # Container image definition
|
||||
```
|
||||
|
||||
## Actor Configuration
|
||||
|
||||
See [references/actor-json.md](references/actor-json.md) for complete actor.json structure and configuration options.
|
||||
|
||||
## Input Schema
|
||||
|
||||
See [references/input-schema.md](references/input-schema.md) for input schema structure and examples.
|
||||
|
||||
## Output Schema
|
||||
|
||||
See [references/output-schema.md](references/output-schema.md) for output schema structure, examples, and template variables.
|
||||
|
||||
## Dataset Schema
|
||||
|
||||
See [references/dataset-schema.md](references/dataset-schema.md) for dataset schema structure, configuration, and display properties.
|
||||
|
||||
## Key-Value Store Schema
|
||||
|
||||
See [references/key-value-store-schema.md](references/key-value-store-schema.md) for key-value store schema structure, collections, and configuration.
|
||||
|
||||
|
||||
## Apify MCP Tools
|
||||
|
||||
If MCP server is configured, use these tools for documentation:
|
||||
|
||||
- `search-apify-docs` - Search documentation
|
||||
- `fetch-apify-docs` - Get full doc pages
|
||||
|
||||
Otherwise, the MCP Server url: `https://mcp.apify.com/?tools=docs`.
|
||||
|
||||
## Resources
|
||||
|
||||
- [docs.apify.com/llms.txt](https://docs.apify.com/llms.txt) - Apify quick reference documentation
|
||||
- [docs.apify.com/llms-full.txt](https://docs.apify.com/llms-full.txt) - Apify complete documentation
|
||||
- [https://crawlee.dev/llms.txt](https://crawlee.dev/llms.txt) - Crawlee quick reference documentation
|
||||
- [https://crawlee.dev/llms-full.txt](https://crawlee.dev/llms-full.txt) - Crawlee complete documentation
|
||||
- [whitepaper.actor](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete Actor specification
|
||||
66
skills/apify-actor-development/references/actor-json.md
Normal file
66
skills/apify-actor-development/references/actor-json.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Actor Configuration (actor.json)
|
||||
|
||||
The `.actor/actor.json` file contains the Actor's configuration including metadata, schema references, and platform settings.
|
||||
|
||||
## Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"name": "project-name",
|
||||
"title": "Project Title",
|
||||
"description": "Actor description",
|
||||
"version": "0.0",
|
||||
"meta": {
|
||||
"templateId": "template-id",
|
||||
"generatedBy": "<FILL-IN-TOOL-AND-MODEL>"
|
||||
},
|
||||
"input": "./input_schema.json",
|
||||
"output": "./output_schema.json",
|
||||
"storages": {
|
||||
"dataset": "./dataset_schema.json"
|
||||
},
|
||||
"dockerfile": "../Dockerfile"
|
||||
}
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"name": "project-cheerio-crawler-javascript",
|
||||
"title": "Project Cheerio Crawler Javascript",
|
||||
"description": "Crawlee and Cheerio project in javascript.",
|
||||
"version": "0.0",
|
||||
"meta": {
|
||||
"templateId": "js-crawlee-cheerio",
|
||||
"generatedBy": "Claude Code with Claude Sonnet 4.5"
|
||||
},
|
||||
"input": "./input_schema.json",
|
||||
"output": "./output_schema.json",
|
||||
"storages": {
|
||||
"dataset": "./dataset_schema.json"
|
||||
},
|
||||
"dockerfile": "../Dockerfile"
|
||||
}
|
||||
```
|
||||
|
||||
## Properties
|
||||
|
||||
- `actorSpecification` (integer, required) - Version of actor specification (currently 1)
|
||||
- `name` (string, required) - Actor identifier (lowercase, hyphens allowed)
|
||||
- `title` (string, required) - Human-readable title displayed in UI
|
||||
- `description` (string, optional) - Actor description for marketplace
|
||||
- `version` (string, required) - Semantic version number
|
||||
- `meta` (object, optional) - Metadata about actor generation
|
||||
- `templateId` (string) - ID of template used to create the actor
|
||||
- `generatedBy` (string) - Tool and model name that generated/modified the actor (e.g., "Claude Code with Claude Sonnet 4.5")
|
||||
- `input` (string, optional) - Path to input schema file
|
||||
- `output` (string, optional) - Path to output schema file
|
||||
- `storages` (object, optional) - Storage schema references
|
||||
- `dataset` (string) - Path to dataset schema file
|
||||
- `keyValueStore` (string) - Path to key-value store schema file
|
||||
- `dockerfile` (string, optional) - Path to Dockerfile
|
||||
|
||||
**Important:** Always fill in the `generatedBy` property with the tool and model you're currently using (e.g., "Claude Code with Claude Sonnet 4.5") to help Apify improve documentation.
|
||||
209
skills/apify-actor-development/references/dataset-schema.md
Normal file
209
skills/apify-actor-development/references/dataset-schema.md
Normal file
@@ -0,0 +1,209 @@
|
||||
# Dataset Schema Reference
|
||||
|
||||
The dataset schema defines how your Actor's output data is structured, transformed, and displayed in the Output tab in the Apify Console.
|
||||
|
||||
## Examples
|
||||
|
||||
### JavaScript and TypeScript
|
||||
|
||||
Consider an example Actor that calls `Actor.pushData()` to store data into dataset:
|
||||
|
||||
```javascript
|
||||
import { Actor } from 'apify';
|
||||
// Initialize the JavaScript SDK
|
||||
await Actor.init();
|
||||
|
||||
/**
|
||||
* Actor code
|
||||
*/
|
||||
await Actor.pushData({
|
||||
numericField: 10,
|
||||
pictureUrl: 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png',
|
||||
linkUrl: 'https://google.com',
|
||||
textField: 'Google',
|
||||
booleanField: true,
|
||||
dateField: new Date(),
|
||||
arrayField: ['#hello', '#world'],
|
||||
objectField: {},
|
||||
});
|
||||
|
||||
// Exit successfully
|
||||
await Actor.exit();
|
||||
```
|
||||
|
||||
### Python
|
||||
|
||||
Consider an example Actor that calls `Actor.push_data()` to store data into dataset:
|
||||
|
||||
```python
|
||||
# Dataset push example (Python)
|
||||
import asyncio
|
||||
from datetime import datetime
|
||||
from apify import Actor
|
||||
|
||||
async def main():
|
||||
await Actor.init()
|
||||
|
||||
# Actor code
|
||||
await Actor.push_data({
|
||||
'numericField': 10,
|
||||
'pictureUrl': 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png',
|
||||
'linkUrl': 'https://google.com',
|
||||
'textField': 'Google',
|
||||
'booleanField': True,
|
||||
'dateField': datetime.now().isoformat(),
|
||||
'arrayField': ['#hello', '#world'],
|
||||
'objectField': {},
|
||||
})
|
||||
|
||||
# Exit successfully
|
||||
await Actor.exit()
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
To set up the Actor's output tab UI, reference a dataset schema file in `.actor/actor.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"name": "book-library-scraper",
|
||||
"title": "Book Library Scraper",
|
||||
"version": "1.0.0",
|
||||
"storages": {
|
||||
"dataset": "./dataset_schema.json"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then create the dataset schema in `.actor/dataset_schema.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"fields": {},
|
||||
"views": {
|
||||
"overview": {
|
||||
"title": "Overview",
|
||||
"transformation": {
|
||||
"fields": [
|
||||
"pictureUrl",
|
||||
"linkUrl",
|
||||
"textField",
|
||||
"booleanField",
|
||||
"arrayField",
|
||||
"objectField",
|
||||
"dateField",
|
||||
"numericField"
|
||||
]
|
||||
},
|
||||
"display": {
|
||||
"component": "table",
|
||||
"properties": {
|
||||
"pictureUrl": {
|
||||
"label": "Image",
|
||||
"format": "image"
|
||||
},
|
||||
"linkUrl": {
|
||||
"label": "Link",
|
||||
"format": "link"
|
||||
},
|
||||
"textField": {
|
||||
"label": "Text",
|
||||
"format": "text"
|
||||
},
|
||||
"booleanField": {
|
||||
"label": "Boolean",
|
||||
"format": "boolean"
|
||||
},
|
||||
"arrayField": {
|
||||
"label": "Array",
|
||||
"format": "array"
|
||||
},
|
||||
"objectField": {
|
||||
"label": "Object",
|
||||
"format": "object"
|
||||
},
|
||||
"dateField": {
|
||||
"label": "Date",
|
||||
"format": "date"
|
||||
},
|
||||
"numericField": {
|
||||
"label": "Number",
|
||||
"format": "number"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"fields": {},
|
||||
"views": {
|
||||
"<VIEW_NAME>": {
|
||||
"title": "string (required)",
|
||||
"description": "string (optional)",
|
||||
"transformation": {
|
||||
"fields": ["string (required)"],
|
||||
"unwind": ["string (optional)"],
|
||||
"flatten": ["string (optional)"],
|
||||
"omit": ["string (optional)"],
|
||||
"limit": "integer (optional)",
|
||||
"desc": "boolean (optional)"
|
||||
},
|
||||
"display": {
|
||||
"component": "table (required)",
|
||||
"properties": {
|
||||
"<FIELD_NAME>": {
|
||||
"label": "string (optional)",
|
||||
"format": "text|number|date|link|boolean|image|array|object (optional)"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Properties
|
||||
|
||||
### Dataset Schema Properties
|
||||
|
||||
- `actorSpecification` (integer, required) - Specifies the version of dataset schema structure document (currently only version 1)
|
||||
- `fields` (JSONSchema object, required) - Schema of one dataset object (use JsonSchema Draft 2020-12 or compatible)
|
||||
- `views` (DatasetView object, required) - Object with API and UI views description
|
||||
|
||||
### DatasetView Properties
|
||||
|
||||
- `title` (string, required) - Visible in UI Output tab and API
|
||||
- `description` (string, optional) - Only available in API response
|
||||
- `transformation` (ViewTransformation object, required) - Data transformation applied when loading from Dataset API
|
||||
- `display` (ViewDisplay object, required) - Output tab UI visualization definition
|
||||
|
||||
### ViewTransformation Properties
|
||||
|
||||
- `fields` (string[], required) - Fields to present in output (order matches column order)
|
||||
- `unwind` (string[], optional) - Deconstructs nested children into parent object
|
||||
- `flatten` (string[], optional) - Transforms nested object into flat structure
|
||||
- `omit` (string[], optional) - Removes specified fields from output
|
||||
- `limit` (integer, optional) - Maximum number of results (default: all)
|
||||
- `desc` (boolean, optional) - Sort order (true = newest first)
|
||||
|
||||
### ViewDisplay Properties
|
||||
|
||||
- `component` (string, required) - Only `table` is available
|
||||
- `properties` (Object, optional) - Keys matching `transformation.fields` with ViewDisplayProperty values
|
||||
|
||||
### ViewDisplayProperty Properties
|
||||
|
||||
- `label` (string, optional) - Table column header
|
||||
- `format` (string, optional) - One of: `text`, `number`, `date`, `link`, `boolean`, `image`, `array`, `object`
|
||||
66
skills/apify-actor-development/references/input-schema.md
Normal file
66
skills/apify-actor-development/references/input-schema.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# Input Schema Reference
|
||||
|
||||
The input schema defines the input parameters for an Actor. It's a JSON object comprising various field types supported by the Apify platform.
|
||||
|
||||
## Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "<INPUT-SCHEMA-TITLE>",
|
||||
"type": "object",
|
||||
"schemaVersion": 1,
|
||||
"properties": {
|
||||
/* define input fields here */
|
||||
},
|
||||
"required": []
|
||||
}
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "E-commerce Product Scraper Input",
|
||||
"type": "object",
|
||||
"schemaVersion": 1,
|
||||
"properties": {
|
||||
"startUrls": {
|
||||
"title": "Start URLs",
|
||||
"type": "array",
|
||||
"description": "URLs to start scraping from (category pages or product pages)",
|
||||
"editor": "requestListSources",
|
||||
"default": [{ "url": "https://example.com/category" }],
|
||||
"prefill": [{ "url": "https://example.com/category" }]
|
||||
},
|
||||
"followVariants": {
|
||||
"title": "Follow Product Variants",
|
||||
"type": "boolean",
|
||||
"description": "Whether to scrape product variants (different colors, sizes)",
|
||||
"default": true
|
||||
},
|
||||
"maxRequestsPerCrawl": {
|
||||
"title": "Max Requests per Crawl",
|
||||
"type": "integer",
|
||||
"description": "Maximum number of pages to scrape (0 = unlimited)",
|
||||
"default": 1000,
|
||||
"minimum": 0
|
||||
},
|
||||
"proxyConfiguration": {
|
||||
"title": "Proxy Configuration",
|
||||
"type": "object",
|
||||
"description": "Proxy settings for anti-bot protection",
|
||||
"editor": "proxy",
|
||||
"default": { "useApifyProxy": false }
|
||||
},
|
||||
"locale": {
|
||||
"title": "Locale",
|
||||
"type": "string",
|
||||
"description": "Language/country code for localized content",
|
||||
"default": "cs",
|
||||
"enum": ["cs", "en", "de", "sk"],
|
||||
"enumTitles": ["Czech", "English", "German", "Slovak"]
|
||||
}
|
||||
},
|
||||
"required": ["startUrls"]
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,129 @@
|
||||
# Key-Value Store Schema Reference
|
||||
|
||||
The key-value store schema organizes keys into logical groups called collections for easier data management.
|
||||
|
||||
## Examples
|
||||
|
||||
### JavaScript and TypeScript
|
||||
|
||||
Consider an example Actor that calls `Actor.setValue()` to save records into the key-value store:
|
||||
|
||||
```javascript
|
||||
import { Actor } from 'apify';
|
||||
// Initialize the JavaScript SDK
|
||||
await Actor.init();
|
||||
|
||||
/**
|
||||
* Actor code
|
||||
*/
|
||||
await Actor.setValue('document-1', 'my text data', { contentType: 'text/plain' });
|
||||
|
||||
await Actor.setValue(`image-${imageID}`, imageBuffer, { contentType: 'image/jpeg' });
|
||||
|
||||
// Exit successfully
|
||||
await Actor.exit();
|
||||
```
|
||||
|
||||
### Python
|
||||
|
||||
Consider an example Actor that calls `Actor.set_value()` to save records into the key-value store:
|
||||
|
||||
```python
|
||||
# Key-Value Store set example (Python)
|
||||
import asyncio
|
||||
from apify import Actor
|
||||
|
||||
async def main():
|
||||
await Actor.init()
|
||||
|
||||
# Actor code
|
||||
await Actor.set_value('document-1', 'my text data', content_type='text/plain')
|
||||
|
||||
image_id = '123' # example placeholder
|
||||
image_buffer = b'...' # bytes buffer with image data
|
||||
await Actor.set_value(f'image-{image_id}', image_buffer, content_type='image/jpeg')
|
||||
|
||||
# Exit successfully
|
||||
await Actor.exit()
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
To configure the key-value store schema, reference a schema file in `.actor/actor.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"name": "data-collector",
|
||||
"title": "Data Collector",
|
||||
"version": "1.0.0",
|
||||
"storages": {
|
||||
"keyValueStore": "./key_value_store_schema.json"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Then create the key-value store schema in `.actor/key_value_store_schema.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"actorKeyValueStoreSchemaVersion": 1,
|
||||
"title": "Key-Value Store Schema",
|
||||
"collections": {
|
||||
"documents": {
|
||||
"title": "Documents",
|
||||
"description": "Text documents stored by the Actor",
|
||||
"keyPrefix": "document-"
|
||||
},
|
||||
"images": {
|
||||
"title": "Images",
|
||||
"description": "Images stored by the Actor",
|
||||
"keyPrefix": "image-",
|
||||
"contentTypes": ["image/jpeg"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"actorKeyValueStoreSchemaVersion": 1,
|
||||
"title": "string (required)",
|
||||
"description": "string (optional)",
|
||||
"collections": {
|
||||
"<COLLECTION_NAME>": {
|
||||
"title": "string (required)",
|
||||
"description": "string (optional)",
|
||||
"key": "string (conditional - use key OR keyPrefix)",
|
||||
"keyPrefix": "string (conditional - use key OR keyPrefix)",
|
||||
"contentTypes": ["string (optional)"],
|
||||
"jsonSchema": "object (optional)"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Properties
|
||||
|
||||
### Key-Value Store Schema Properties
|
||||
|
||||
- `actorKeyValueStoreSchemaVersion` (integer, required) - Version of key-value store schema structure document (currently only version 1)
|
||||
- `title` (string, required) - Title of the schema
|
||||
- `description` (string, optional) - Description of the schema
|
||||
- `collections` (Object, required) - Object where each key is a collection ID and value is a Collection object
|
||||
|
||||
### Collection Properties
|
||||
|
||||
- `title` (string, required) - Collection title shown in UI tabs
|
||||
- `description` (string, optional) - Description appearing in UI tooltips
|
||||
- `key` (string, conditional) - Single specific key for this collection
|
||||
- `keyPrefix` (string, conditional) - Prefix for keys included in this collection
|
||||
- `contentTypes` (string[], optional) - Allowed content types for validation
|
||||
- `jsonSchema` (object, optional) - JSON Schema Draft 07 format for `application/json` content type validation
|
||||
|
||||
Either `key` or `keyPrefix` must be specified for each collection, but not both.
|
||||
50
skills/apify-actor-development/references/logging.md
Normal file
50
skills/apify-actor-development/references/logging.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Actor Logging Reference
|
||||
|
||||
## JavaScript and TypeScript
|
||||
|
||||
**ALWAYS use the `apify/log` package for logging** - This package contains critical security logic including censoring sensitive data (Apify tokens, API keys, credentials) to prevent accidental exposure in logs.
|
||||
|
||||
### Available Log Levels in `apify/log`
|
||||
|
||||
The Apify log package provides the following methods for logging:
|
||||
|
||||
- `log.debug()` - Debug level logs (detailed diagnostic information)
|
||||
- `log.info()` - Info level logs (general informational messages)
|
||||
- `log.warning()` - Warning level logs (warning messages for potentially problematic situations)
|
||||
- `log.warningOnce()` - Warning level logs (same warning message logged only once)
|
||||
- `log.error()` - Error level logs (error messages for failures)
|
||||
- `log.exception()` - Exception level logs (for exceptions with stack traces)
|
||||
- `log.perf()` - Performance level logs (performance metrics and timing information)
|
||||
- `log.deprecated()` - Deprecation level logs (warnings about deprecated code)
|
||||
- `log.softFail()` - Soft failure logs (non-critical failures that don't stop execution, e.g., input validation errors, skipped items)
|
||||
- `log.internal()` - Internal level logs (internal/system messages)
|
||||
|
||||
### Best Practices
|
||||
|
||||
- Use `log.debug()` for detailed operation-level diagnostics (inside functions)
|
||||
- Use `log.info()` for general informational messages (API requests, successful operations)
|
||||
- Use `log.warning()` for potentially problematic situations (validation failures, unexpected states)
|
||||
- Use `log.error()` for actual errors and failures
|
||||
- Use `log.exception()` for caught exceptions with stack traces
|
||||
|
||||
## Python
|
||||
|
||||
**ALWAYS use `Actor.log` for logging** - This logger contains critical security logic including censoring sensitive data (Apify tokens, API keys, credentials) to prevent accidental exposure in logs.
|
||||
|
||||
### Available Log Levels
|
||||
|
||||
The Apify Actor logger provides the following methods for logging:
|
||||
|
||||
- `Actor.log.debug()` - Debug level logs (detailed diagnostic information)
|
||||
- `Actor.log.info()` - Info level logs (general informational messages)
|
||||
- `Actor.log.warning()` - Warning level logs (warning messages for potentially problematic situations)
|
||||
- `Actor.log.error()` - Error level logs (error messages for failures)
|
||||
- `Actor.log.exception()` - Exception level logs (for exceptions with stack traces)
|
||||
|
||||
### Best Practices
|
||||
|
||||
- Use `Actor.log.debug()` for detailed operation-level diagnostics (inside functions)
|
||||
- Use `Actor.log.info()` for general informational messages (API requests, successful operations)
|
||||
- Use `Actor.log.warning()` for potentially problematic situations (validation failures, unexpected states)
|
||||
- Use `Actor.log.error()` for actual errors and failures
|
||||
- Use `Actor.log.exception()` for caught exceptions with stack traces
|
||||
49
skills/apify-actor-development/references/output-schema.md
Normal file
49
skills/apify-actor-development/references/output-schema.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Output Schema Reference
|
||||
|
||||
The Actor output schema builds upon the schemas for the dataset and key-value store. It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results.
|
||||
|
||||
## Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"actorOutputSchemaVersion": 1,
|
||||
"title": "<OUTPUT-SCHEMA-TITLE>",
|
||||
"properties": {
|
||||
/* define your outputs here */
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```json
|
||||
{
|
||||
"actorOutputSchemaVersion": 1,
|
||||
"title": "Output schema of the files scraper",
|
||||
"properties": {
|
||||
"files": {
|
||||
"type": "string",
|
||||
"title": "Files",
|
||||
"template": "{{links.apiDefaultKeyValueStoreUrl}}/keys"
|
||||
},
|
||||
"dataset": {
|
||||
"type": "string",
|
||||
"title": "Dataset",
|
||||
"template": "{{links.apiDefaultDatasetUrl}}/items"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Output Schema Template Variables
|
||||
|
||||
- `links` (object) - Contains quick links to most commonly used URLs
|
||||
- `links.publicRunUrl` (string) - Public run url in format `https://console.apify.com/view/runs/:runId`
|
||||
- `links.consoleRunUrl` (string) - Console run url in format `https://console.apify.com/actors/runs/:runId`
|
||||
- `links.apiRunUrl` (string) - API run url in format `https://api.apify.com/v2/actor-runs/:runId`
|
||||
- `links.apiDefaultDatasetUrl` (string) - API url of default dataset in format `https://api.apify.com/v2/datasets/:defaultDatasetId`
|
||||
- `links.apiDefaultKeyValueStoreUrl` (string) - API url of default key-value store in format `https://api.apify.com/v2/key-value-stores/:defaultKeyValueStoreId`
|
||||
- `links.containerRunUrl` (string) - URL of a webserver running inside the run in format `https://<containerId>.runs.apify.net/`
|
||||
- `run` (object) - Contains information about the run same as it is returned from the `GET Run` API endpoint
|
||||
- `run.defaultDatasetId` (string) - ID of the default dataset
|
||||
- `run.defaultKeyValueStoreId` (string) - ID of the default key-value store
|
||||
61
skills/apify-actor-development/references/standby-mode.md
Normal file
61
skills/apify-actor-development/references/standby-mode.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# Actor Standby Mode Reference
|
||||
|
||||
## JavaScript and TypeScript
|
||||
|
||||
- **NEVER disable standby mode (`usesStandbyMode: false`) in `.actor/actor.json` without explicit permission** - Actor Standby mode solves this problem by letting you have the Actor ready in the background, waiting for the incoming HTTP requests. In a sense, the Actor behaves like a real-time web server or standard API server instead of running the logic once to process everything in batch. Always keep `usesStandbyMode: true` unless there is a specific documented reason to disable it
|
||||
- **ALWAYS implement readiness probe handler for standby Actors** - Handle the `x-apify-container-server-readiness-probe` header at GET / endpoint to ensure proper Actor lifecycle management
|
||||
|
||||
You can recognize a standby Actor by checking the `usesStandbyMode` property in `.actor/actor.json`. Only implement the readiness probe if this property is set to `true`.
|
||||
|
||||
### Readiness Probe Implementation Example
|
||||
|
||||
```javascript
|
||||
// Apify standby readiness probe at root path
|
||||
app.get('/', (req, res) => {
|
||||
res.writeHead(200, { 'Content-Type': 'text/plain' });
|
||||
if (req.headers['x-apify-container-server-readiness-probe']) {
|
||||
res.end('Readiness probe OK\n');
|
||||
} else {
|
||||
res.end('Actor is ready\n');
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Detect the `x-apify-container-server-readiness-probe` header in incoming requests
|
||||
- Respond with HTTP 200 status code for both readiness probe and normal requests
|
||||
- This enables proper Actor lifecycle management in standby mode
|
||||
|
||||
## Python
|
||||
|
||||
- **NEVER disable standby mode (`usesStandbyMode: false`) in `.actor/actor.json` without explicit permission** - Actor Standby mode solves this problem by letting you have the Actor ready in the background, waiting for the incoming HTTP requests. In a sense, the Actor behaves like a real-time web server or standard API server instead of running the logic once to process everything in batch. Always keep `usesStandbyMode: true` unless there is a specific documented reason to disable it
|
||||
- **ALWAYS implement readiness probe handler for standby Actors** - Handle the `x-apify-container-server-readiness-probe` header at GET / endpoint to ensure proper Actor lifecycle management
|
||||
|
||||
You can recognize a standby Actor by checking the `usesStandbyMode` property in `.actor/actor.json`. Only implement the readiness probe if this property is set to `true`.
|
||||
|
||||
### Readiness Probe Implementation Example
|
||||
|
||||
```python
|
||||
# Apify standby readiness probe
|
||||
from http.server import SimpleHTTPRequestHandler
|
||||
|
||||
class GetHandler(SimpleHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
# Handle Apify standby readiness probe
|
||||
if 'x-apify-container-server-readiness-probe' in self.headers:
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'Readiness probe OK')
|
||||
return
|
||||
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'Actor is ready')
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- Detect the `x-apify-container-server-readiness-probe` header in incoming requests
|
||||
- Respond with HTTP 200 status code for both readiness probe and normal requests
|
||||
- This enables proper Actor lifecycle management in standby mode
|
||||
184
skills/apify-actorization/SKILL.md
Normal file
184
skills/apify-actorization/SKILL.md
Normal file
@@ -0,0 +1,184 @@
|
||||
---
|
||||
name: apify-actorization
|
||||
description: "Convert existing projects into Apify Actors - serverless cloud programs. Actorize JavaScript/TypeScript (SDK with Actor.init/exit), Python (async context manager), or any language (CLI wrapper). Us..."
|
||||
---
|
||||
|
||||
# Apify Actorization
|
||||
|
||||
Actorization converts existing software into reusable serverless applications compatible with the Apify platform. Actors are programs packaged as Docker images that accept well-defined JSON input, perform an action, and optionally produce structured JSON output.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Run `apify init` in project root
|
||||
2. Wrap code with SDK lifecycle (see language-specific section below)
|
||||
3. Configure `.actor/input_schema.json`
|
||||
4. Test with `apify run --input '{"key": "value"}'`
|
||||
5. Deploy with `apify push`
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Converting an existing project to run on Apify platform
|
||||
- Adding Apify SDK integration to a project
|
||||
- Wrapping a CLI tool or script as an Actor
|
||||
- Migrating a Crawlee project to Apify
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Verify `apify` CLI is installed:
|
||||
|
||||
```bash
|
||||
apify --help
|
||||
```
|
||||
|
||||
If not installed:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://apify.com/install-cli.sh | bash
|
||||
|
||||
# Or (Mac): brew install apify-cli
|
||||
# Or (Windows): irm https://apify.com/install-cli.ps1 | iex
|
||||
# Or: npm install -g apify-cli
|
||||
```
|
||||
|
||||
Verify CLI is logged in:
|
||||
|
||||
```bash
|
||||
apify info # Should return your username
|
||||
```
|
||||
|
||||
If not logged in, check if `APIFY_TOKEN` environment variable is defined. If not, ask the user to generate one at https://console.apify.com/settings/integrations, then:
|
||||
|
||||
```bash
|
||||
apify login -t $APIFY_TOKEN
|
||||
```
|
||||
|
||||
## Actorization Checklist
|
||||
|
||||
Copy this checklist to track progress:
|
||||
|
||||
- [ ] Step 1: Analyze project (language, entry point, inputs, outputs)
|
||||
- [ ] Step 2: Run `apify init` to create Actor structure
|
||||
- [ ] Step 3: Apply language-specific SDK integration
|
||||
- [ ] Step 4: Configure `.actor/input_schema.json`
|
||||
- [ ] Step 5: Configure `.actor/output_schema.json` (if applicable)
|
||||
- [ ] Step 6: Update `.actor/actor.json` metadata
|
||||
- [ ] Step 7: Test locally with `apify run`
|
||||
- [ ] Step 8: Deploy with `apify push`
|
||||
|
||||
## Step 1: Analyze the Project
|
||||
|
||||
Before making changes, understand the project:
|
||||
|
||||
1. **Identify the language** - JavaScript/TypeScript, Python, or other
|
||||
2. **Find the entry point** - The main file that starts execution
|
||||
3. **Identify inputs** - Command-line arguments, environment variables, config files
|
||||
4. **Identify outputs** - Files, console output, API responses
|
||||
5. **Check for state** - Does it need to persist data between runs?
|
||||
|
||||
## Step 2: Initialize Actor Structure
|
||||
|
||||
Run in the project root:
|
||||
|
||||
```bash
|
||||
apify init
|
||||
```
|
||||
|
||||
This creates:
|
||||
- `.actor/actor.json` - Actor configuration and metadata
|
||||
- `.actor/input_schema.json` - Input definition for the Apify Console
|
||||
- `Dockerfile` (if not present) - Container image definition
|
||||
|
||||
## Step 3: Apply Language-Specific Changes
|
||||
|
||||
Choose based on your project's language:
|
||||
|
||||
- **JavaScript/TypeScript**: See [js-ts-actorization.md](references/js-ts-actorization.md)
|
||||
- **Python**: See [python-actorization.md](references/python-actorization.md)
|
||||
- **Other Languages (CLI-based)**: See [cli-actorization.md](references/cli-actorization.md)
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| Language | Install | Wrap Code |
|
||||
|----------|---------|-----------|
|
||||
| JS/TS | `npm install apify` | `await Actor.init()` ... `await Actor.exit()` |
|
||||
| Python | `pip install apify` | `async with Actor:` |
|
||||
| Other | Use CLI in wrapper script | `apify actor:get-input` / `apify actor:push-data` |
|
||||
|
||||
## Steps 4-6: Configure Schemas
|
||||
|
||||
See [schemas-and-output.md](references/schemas-and-output.md) for detailed configuration of:
|
||||
- Input schema (`.actor/input_schema.json`)
|
||||
- Output schema (`.actor/output_schema.json`)
|
||||
- Actor configuration (`.actor/actor.json`)
|
||||
- State management (request queues, key-value stores)
|
||||
|
||||
Validate schemas against `@apify/json_schemas` npm package.
|
||||
|
||||
## Step 7: Test Locally
|
||||
|
||||
Run the actor with inline input (for JS/TS and Python actors):
|
||||
|
||||
```bash
|
||||
apify run --input '{"startUrl": "https://example.com", "maxItems": 10}'
|
||||
```
|
||||
|
||||
Or use an input file:
|
||||
|
||||
```bash
|
||||
apify run --input-file ./test-input.json
|
||||
```
|
||||
|
||||
**Important:** Always use `apify run`, not `npm start` or `python main.py`. The CLI sets up the proper environment and storage.
|
||||
|
||||
## Step 8: Deploy
|
||||
|
||||
```bash
|
||||
apify push
|
||||
```
|
||||
|
||||
This uploads and builds your actor on the Apify platform.
|
||||
|
||||
## Monetization (Optional)
|
||||
|
||||
After deploying, you can monetize your actor in the Apify Store. The recommended model is **Pay Per Event (PPE)**:
|
||||
|
||||
- Per result/item scraped
|
||||
- Per page processed
|
||||
- Per API call made
|
||||
|
||||
Configure PPE in the Apify Console under Actor > Monetization. Charge for events in your code with `await Actor.charge('result')`.
|
||||
|
||||
Other options: **Rental** (monthly subscription) or **Free** (open source).
|
||||
|
||||
## Pre-Deployment Checklist
|
||||
|
||||
- [ ] `.actor/actor.json` exists with correct name and description
|
||||
- [ ] `.actor/actor.json` validates against `@apify/json_schemas` (`actor.schema.json`)
|
||||
- [ ] `.actor/input_schema.json` defines all required inputs
|
||||
- [ ] `.actor/input_schema.json` validates against `@apify/json_schemas` (`input.schema.json`)
|
||||
- [ ] `.actor/output_schema.json` defines output structure (if applicable)
|
||||
- [ ] `.actor/output_schema.json` validates against `@apify/json_schemas` (`output.schema.json`)
|
||||
- [ ] `Dockerfile` is present and builds successfully
|
||||
- [ ] `Actor.init()` / `Actor.exit()` wraps main code (JS/TS)
|
||||
- [ ] `async with Actor:` wraps main code (Python)
|
||||
- [ ] Inputs are read via `Actor.getInput()` / `Actor.get_input()`
|
||||
- [ ] Outputs use `Actor.pushData()` or key-value store
|
||||
- [ ] `apify run` executes successfully with test input
|
||||
- [ ] `generatedBy` is set in actor.json meta section
|
||||
|
||||
## Apify MCP Tools
|
||||
|
||||
If MCP server is configured, use these tools for documentation:
|
||||
|
||||
- `search-apify-docs` - Search documentation
|
||||
- `fetch-apify-docs` - Get full doc pages
|
||||
|
||||
Otherwise, the MCP Server url: `https://mcp.apify.com/?tools=docs`.
|
||||
|
||||
## Resources
|
||||
|
||||
- [Actorization Academy](https://docs.apify.com/academy/actorization) - Comprehensive guide
|
||||
- [Apify SDK for JavaScript](https://docs.apify.com/sdk/js) - Full SDK reference
|
||||
- [Apify SDK for Python](https://docs.apify.com/sdk/python) - Full SDK reference
|
||||
- [Apify CLI Reference](https://docs.apify.com/cli) - CLI commands
|
||||
- [Actor Specification](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete specification
|
||||
81
skills/apify-actorization/references/cli-actorization.md
Normal file
81
skills/apify-actorization/references/cli-actorization.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# CLI-Based Actorization
|
||||
|
||||
For languages without an SDK (Go, Rust, Java, etc.), create a wrapper script that uses the Apify CLI.
|
||||
|
||||
## Create Wrapper Script
|
||||
|
||||
Create `start.sh` in project root:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Get input from Apify key-value store
|
||||
INPUT=$(apify actor:get-input)
|
||||
|
||||
# Parse input values (adjust based on your input schema)
|
||||
MY_PARAM=$(echo "$INPUT" | jq -r '.myParam // "default"')
|
||||
|
||||
# Run your application with the input
|
||||
./your-application --param "$MY_PARAM"
|
||||
|
||||
# If your app writes to a file, push it to key-value store
|
||||
# apify actor:set-value OUTPUT --contentType application/json < output.json
|
||||
|
||||
# Or push structured data to dataset
|
||||
# apify actor:push-data '{"result": "value"}'
|
||||
```
|
||||
|
||||
## Update Dockerfile
|
||||
|
||||
Reference the [cli-start template Dockerfile](https://github.com/apify/actor-templates/blob/master/templates/cli-start/Dockerfile) which includes the `ubi` utility for installing binaries from GitHub releases.
|
||||
|
||||
```dockerfile
|
||||
FROM apify/actor-node:20
|
||||
|
||||
# Install ubi for easy GitHub release installation
|
||||
RUN curl --silent --location \
|
||||
https://raw.githubusercontent.com/houseabsolute/ubi/master/bootstrap/bootstrap-ubi.sh | sh
|
||||
|
||||
# Install your CLI tool from GitHub releases (example)
|
||||
# RUN ubi --project your-org/your-tool --in /usr/local/bin
|
||||
|
||||
# Or install apify-cli and jq manually
|
||||
RUN npm install -g apify-cli
|
||||
RUN apt-get update && apt-get install -y jq
|
||||
|
||||
# Copy your application
|
||||
COPY . .
|
||||
|
||||
# Build your application if needed
|
||||
# RUN ./build.sh
|
||||
|
||||
# Make start script executable
|
||||
RUN chmod +x start.sh
|
||||
|
||||
# Run the wrapper script
|
||||
CMD ["./start.sh"]
|
||||
```
|
||||
|
||||
## Testing CLI-Based Actors
|
||||
|
||||
For CLI-based actors (shell wrapper scripts), you may need to test the underlying application directly with mock input, as `apify run` requires a Node.js or Python entry point.
|
||||
|
||||
Test your wrapper script locally:
|
||||
|
||||
```bash
|
||||
# Set up mock input
|
||||
export INPUT='{"myParam": "test-value"}'
|
||||
|
||||
# Run wrapper script
|
||||
./start.sh
|
||||
```
|
||||
|
||||
## CLI Commands Reference
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `apify actor:get-input` | Get input JSON from key-value store |
|
||||
| `apify actor:set-value KEY` | Store value in key-value store |
|
||||
| `apify actor:push-data JSON` | Push data to dataset |
|
||||
| `apify actor:get-value KEY` | Retrieve value from key-value store |
|
||||
111
skills/apify-actorization/references/js-ts-actorization.md
Normal file
111
skills/apify-actorization/references/js-ts-actorization.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# JavaScript/TypeScript Actorization
|
||||
|
||||
## Install the Apify SDK
|
||||
|
||||
```bash
|
||||
npm install apify
|
||||
```
|
||||
|
||||
## Wrap Main Code with Actor Lifecycle
|
||||
|
||||
```javascript
|
||||
import { Actor } from 'apify';
|
||||
|
||||
// Initialize connection to Apify platform
|
||||
await Actor.init();
|
||||
|
||||
// ============================================
|
||||
// Your existing code goes here
|
||||
// ============================================
|
||||
|
||||
// Example: Get input from Apify Console or API
|
||||
const input = await Actor.getInput();
|
||||
console.log('Input:', input);
|
||||
|
||||
// Example: Your crawler or processing logic
|
||||
// const crawler = new PlaywrightCrawler({ ... });
|
||||
// await crawler.run([input.startUrl]);
|
||||
|
||||
// Example: Push results to dataset
|
||||
// await Actor.pushData({ result: 'data' });
|
||||
|
||||
// ============================================
|
||||
// End of your code
|
||||
// ============================================
|
||||
|
||||
// Graceful shutdown
|
||||
await Actor.exit();
|
||||
```
|
||||
|
||||
## Key Points
|
||||
|
||||
- `Actor.init()` configures storage to use Apify API when running on platform
|
||||
- `Actor.exit()` handles graceful shutdown and cleanup
|
||||
- Both calls must be awaited
|
||||
- Local execution remains unchanged - the SDK automatically detects the environment
|
||||
|
||||
## Crawlee Projects
|
||||
|
||||
Crawlee projects require minimal changes - just wrap with Actor lifecycle:
|
||||
|
||||
```javascript
|
||||
import { Actor } from 'apify';
|
||||
import { PlaywrightCrawler } from 'crawlee';
|
||||
|
||||
await Actor.init();
|
||||
|
||||
// Get and validate input
|
||||
const input = await Actor.getInput();
|
||||
const {
|
||||
startUrl = 'https://example.com',
|
||||
maxItems = 100,
|
||||
} = input ?? {};
|
||||
|
||||
let itemCount = 0;
|
||||
|
||||
const crawler = new PlaywrightCrawler({
|
||||
requestHandler: async ({ page, request, pushData }) => {
|
||||
if (itemCount >= maxItems) return;
|
||||
|
||||
const title = await page.title();
|
||||
await pushData({ url: request.url, title });
|
||||
itemCount++;
|
||||
},
|
||||
});
|
||||
|
||||
await crawler.run([startUrl]);
|
||||
|
||||
await Actor.exit();
|
||||
```
|
||||
|
||||
## Express/HTTP Servers
|
||||
|
||||
For web servers, use standby mode in actor.json:
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"name": "my-api",
|
||||
"usesStandbyMode": true
|
||||
}
|
||||
```
|
||||
|
||||
Then implement readiness probe. See [standby-mode.md](../../apify-actor-development/references/standby-mode.md).
|
||||
|
||||
## Batch Processing Scripts
|
||||
|
||||
```javascript
|
||||
import { Actor } from 'apify';
|
||||
|
||||
await Actor.init();
|
||||
|
||||
const input = await Actor.getInput();
|
||||
const items = input.items || [];
|
||||
|
||||
for (const item of items) {
|
||||
const result = processItem(item);
|
||||
await Actor.pushData(result);
|
||||
}
|
||||
|
||||
await Actor.exit();
|
||||
```
|
||||
95
skills/apify-actorization/references/python-actorization.md
Normal file
95
skills/apify-actorization/references/python-actorization.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# Python Actorization
|
||||
|
||||
## Install the Apify SDK
|
||||
|
||||
```bash
|
||||
pip install apify
|
||||
```
|
||||
|
||||
## Wrap Main Function with Actor Context Manager
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from apify import Actor
|
||||
|
||||
async def main() -> None:
|
||||
async with Actor:
|
||||
# ============================================
|
||||
# Your existing code goes here
|
||||
# ============================================
|
||||
|
||||
# Example: Get input from Apify Console or API
|
||||
actor_input = await Actor.get_input()
|
||||
print(f'Input: {actor_input}')
|
||||
|
||||
# Example: Your crawler or processing logic
|
||||
# crawler = PlaywrightCrawler(...)
|
||||
# await crawler.run([actor_input.get('startUrl')])
|
||||
|
||||
# Example: Push results to dataset
|
||||
# await Actor.push_data({'result': 'data'})
|
||||
|
||||
# ============================================
|
||||
# End of your code
|
||||
# ============================================
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Key Points
|
||||
|
||||
- `async with Actor:` handles both initialization and cleanup
|
||||
- Automatically manages platform event listeners and graceful shutdown
|
||||
- Local execution remains unchanged - the SDK automatically detects the environment
|
||||
|
||||
## Crawlee Python Projects
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from apify import Actor
|
||||
from crawlee.playwright_crawler import PlaywrightCrawler
|
||||
|
||||
async def main() -> None:
|
||||
async with Actor:
|
||||
# Get and validate input
|
||||
actor_input = await Actor.get_input() or {}
|
||||
start_url = actor_input.get('startUrl', 'https://example.com')
|
||||
max_items = actor_input.get('maxItems', 100)
|
||||
|
||||
item_count = 0
|
||||
|
||||
async def request_handler(context):
|
||||
nonlocal item_count
|
||||
if item_count >= max_items:
|
||||
return
|
||||
|
||||
title = await context.page.title()
|
||||
await context.push_data({'url': context.request.url, 'title': title})
|
||||
item_count += 1
|
||||
|
||||
crawler = PlaywrightCrawler(request_handler=request_handler)
|
||||
await crawler.run([start_url])
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Batch Processing Scripts
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from apify import Actor
|
||||
|
||||
async def main() -> None:
|
||||
async with Actor:
|
||||
actor_input = await Actor.get_input() or {}
|
||||
items = actor_input.get('items', [])
|
||||
|
||||
for item in items:
|
||||
result = process_item(item)
|
||||
await Actor.push_data(result)
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
```
|
||||
140
skills/apify-actorization/references/schemas-and-output.md
Normal file
140
skills/apify-actorization/references/schemas-and-output.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Schemas and Output Configuration
|
||||
|
||||
## Input Schema
|
||||
|
||||
Map your application's inputs to `.actor/input_schema.json`. Validate against the JSON Schema from the `@apify/json_schemas` npm package (`input.schema.json`).
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "My Actor Input",
|
||||
"type": "object",
|
||||
"schemaVersion": 1,
|
||||
"properties": {
|
||||
"startUrl": {
|
||||
"title": "Start URL",
|
||||
"type": "string",
|
||||
"description": "The URL to start processing from",
|
||||
"editor": "textfield",
|
||||
"prefill": "https://example.com"
|
||||
},
|
||||
"maxItems": {
|
||||
"title": "Max Items",
|
||||
"type": "integer",
|
||||
"description": "Maximum number of items to process",
|
||||
"default": 100,
|
||||
"minimum": 1
|
||||
}
|
||||
},
|
||||
"required": ["startUrl"]
|
||||
}
|
||||
```
|
||||
|
||||
### Mapping Guidelines
|
||||
|
||||
- Command-line arguments → input schema properties
|
||||
- Environment variables → input schema or Actor env vars in actor.json
|
||||
- Config files → input schema with object/array types
|
||||
- Flatten deeply nested structures for better UX
|
||||
|
||||
## Output Schema
|
||||
|
||||
Define output structure in `.actor/output_schema.json`. Validate against the JSON Schema from the `@apify/json_schemas` npm package (`output.schema.json`).
|
||||
|
||||
### For Table-Like Data (Multiple Items)
|
||||
|
||||
- Use `Actor.pushData()` (JS) or `Actor.push_data()` (Python)
|
||||
- Each item becomes a row in the dataset
|
||||
|
||||
### For Single Files or Blobs
|
||||
|
||||
- Use key-value store: `Actor.setValue()` / `Actor.set_value()`
|
||||
- Get the public URL and include it in the dataset:
|
||||
|
||||
```javascript
|
||||
// Store file with public access
|
||||
await Actor.setValue('report.pdf', pdfBuffer, { contentType: 'application/pdf' });
|
||||
|
||||
// Get the public URL
|
||||
const storeInfo = await Actor.openKeyValueStore();
|
||||
const publicUrl = `https://api.apify.com/v2/key-value-stores/${storeInfo.id}/records/report.pdf`;
|
||||
|
||||
// Include URL in dataset output
|
||||
await Actor.pushData({ reportUrl: publicUrl });
|
||||
```
|
||||
|
||||
### For Multiple Files with a Common Prefix (Collections)
|
||||
|
||||
```javascript
|
||||
// Store multiple files with a prefix
|
||||
for (const [name, data] of files) {
|
||||
await Actor.setValue(`screenshots/${name}`, data, { contentType: 'image/png' });
|
||||
}
|
||||
// Files are accessible at: .../records/screenshots%2F{name}
|
||||
```
|
||||
|
||||
## Actor Configuration (actor.json)
|
||||
|
||||
Configure `.actor/actor.json`. Validate against the JSON Schema from the `@apify/json_schemas` npm package (`actor.schema.json`).
|
||||
|
||||
```json
|
||||
{
|
||||
"actorSpecification": 1,
|
||||
"name": "my-actor",
|
||||
"title": "My Actor",
|
||||
"description": "Brief description of what the actor does",
|
||||
"version": "1.0.0",
|
||||
"meta": {
|
||||
"templateId": "ts_empty",
|
||||
"generatedBy": "Claude Code with Claude Opus 4.5"
|
||||
},
|
||||
"input": "./input_schema.json",
|
||||
"dockerfile": "../Dockerfile"
|
||||
}
|
||||
```
|
||||
|
||||
**Important:** Fill in the `generatedBy` property with the tool/model used.
|
||||
|
||||
## State Management
|
||||
|
||||
### Request Queue - For Pausable Task Processing
|
||||
|
||||
The request queue works for any task processing, not just web scraping. Use a dummy URL with custom `uniqueKey` and `userData` for non-URL tasks:
|
||||
|
||||
```javascript
|
||||
const requestQueue = await Actor.openRequestQueue();
|
||||
|
||||
// Add tasks to the queue (works for any processing, not just URLs)
|
||||
await requestQueue.addRequest({
|
||||
url: 'https://placeholder.local', // Dummy URL for non-scraping tasks
|
||||
uniqueKey: `task-${taskId}`, // Unique identifier for deduplication
|
||||
userData: { itemId: 123, action: 'process' }, // Your custom task data
|
||||
});
|
||||
|
||||
// Process tasks from the queue (with Crawlee)
|
||||
const crawler = new BasicCrawler({
|
||||
requestQueue,
|
||||
requestHandler: async ({ request }) => {
|
||||
const { itemId, action } = request.userData;
|
||||
// Process your task using userData
|
||||
await processTask(itemId, action);
|
||||
},
|
||||
});
|
||||
await crawler.run();
|
||||
|
||||
// Or manually consume without Crawlee:
|
||||
let request;
|
||||
while ((request = await requestQueue.fetchNextRequest())) {
|
||||
await processTask(request.userData);
|
||||
await requestQueue.markRequestHandled(request);
|
||||
}
|
||||
```
|
||||
|
||||
### Key-Value Store - For Checkpoint State
|
||||
|
||||
```javascript
|
||||
// Save state
|
||||
await Actor.setValue('STATE', { processedCount: 100 });
|
||||
|
||||
// Restore state on restart
|
||||
const state = await Actor.getValue('STATE') || { processedCount: 0 };
|
||||
```
|
||||
121
skills/apify-audience-analysis/SKILL.md
Normal file
121
skills/apify-audience-analysis/SKILL.md
Normal file
@@ -0,0 +1,121 @@
|
||||
---
|
||||
name: apify-audience-analysis
|
||||
description: Understand audience demographics, preferences, behavior patterns, and engagement quality across Facebook, Instagram, YouTube, and TikTok.
|
||||
---
|
||||
|
||||
# Audience Analysis
|
||||
|
||||
Analyze and understand your audience using Apify Actors to extract follower demographics, engagement patterns, and behavior data from multiple platforms.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Identify audience analysis type (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the analysis script
|
||||
- [ ] Step 5: Summarize findings
|
||||
```
|
||||
|
||||
### Step 1: Identify Audience Analysis Type
|
||||
|
||||
Select the appropriate Actor based on analysis needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Facebook follower demographics | `apify/facebook-followers-following-scraper` | FB followers/following lists |
|
||||
| Facebook engagement behavior | `apify/facebook-likes-scraper` | FB post likes analysis |
|
||||
| Facebook video audience | `apify/facebook-reels-scraper` | FB Reels viewers |
|
||||
| Facebook comment analysis | `apify/facebook-comments-scraper` | FB post/video comments |
|
||||
| Facebook content engagement | `apify/facebook-posts-scraper` | FB post engagement metrics |
|
||||
| Instagram audience sizing | `apify/instagram-profile-scraper` | IG profile demographics |
|
||||
| Instagram location-based | `apify/instagram-search-scraper` | IG geo-tagged audience |
|
||||
| Instagram tagged network | `apify/instagram-tagged-scraper` | IG tag network analysis |
|
||||
| Instagram comprehensive | `apify/instagram-scraper` | Full IG audience data |
|
||||
| Instagram API-based | `apify/instagram-api-scraper` | IG API access |
|
||||
| Instagram follower counts | `apify/instagram-followers-count-scraper` | IG follower tracking |
|
||||
| Instagram comment export | `apify/export-instagram-comments-posts` | IG comment bulk export |
|
||||
| Instagram comment analysis | `apify/instagram-comment-scraper` | IG comment sentiment |
|
||||
| YouTube viewer feedback | `streamers/youtube-comments-scraper` | YT comment analysis |
|
||||
| YouTube channel audience | `streamers/youtube-channel-scraper` | YT channel subscribers |
|
||||
| TikTok follower demographics | `clockworks/tiktok-followers-scraper` | TT follower lists |
|
||||
| TikTok profile analysis | `clockworks/tiktok-profile-scraper` | TT profile demographics |
|
||||
| TikTok comment analysis | `clockworks/tiktok-comments-scraper` | TT comment engagement |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `apify/facebook-followers-following-scraper`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Findings
|
||||
|
||||
After completion, report:
|
||||
- Number of audience members/profiles analyzed
|
||||
- File location and name
|
||||
- Key demographic insights
|
||||
- Suggested next steps (deeper analysis, segmentation)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
363
skills/apify-audience-analysis/reference/scripts/run_actor.js
Normal file
363
skills/apify-audience-analysis/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-audience-analysis-1.0.1';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
121
skills/apify-brand-reputation-monitoring/SKILL.md
Normal file
121
skills/apify-brand-reputation-monitoring/SKILL.md
Normal file
@@ -0,0 +1,121 @@
|
||||
---
|
||||
name: apify-brand-reputation-monitoring
|
||||
description: "Track reviews, ratings, sentiment, and brand mentions across Google Maps, Booking.com, TripAdvisor, Facebook, Instagram, YouTube, and TikTok. Use when user asks to monitor brand reputation, analyze..."
|
||||
---
|
||||
|
||||
# Brand Reputation Monitoring
|
||||
|
||||
Scrape reviews, ratings, and brand mentions from multiple platforms using Apify Actors.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Determine data source (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the monitoring script
|
||||
- [ ] Step 5: Summarize results
|
||||
```
|
||||
|
||||
### Step 1: Determine Data Source
|
||||
|
||||
Select the appropriate Actor based on user needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Google Maps reviews | `compass/crawler-google-places` | Business reviews, ratings |
|
||||
| Google Maps review export | `compass/Google-Maps-Reviews-Scraper` | Dedicated review scraping |
|
||||
| Booking.com hotels | `voyager/booking-scraper` | Hotel data, scores |
|
||||
| Booking.com reviews | `voyager/booking-reviews-scraper` | Detailed hotel reviews |
|
||||
| TripAdvisor reviews | `maxcopell/tripadvisor-reviews` | Attraction/restaurant reviews |
|
||||
| Facebook reviews | `apify/facebook-reviews-scraper` | Page reviews |
|
||||
| Facebook comments | `apify/facebook-comments-scraper` | Post comment monitoring |
|
||||
| Facebook page metrics | `apify/facebook-pages-scraper` | Page ratings overview |
|
||||
| Facebook reactions | `apify/facebook-likes-scraper` | Reaction type analysis |
|
||||
| Instagram comments | `apify/instagram-comment-scraper` | Comment sentiment |
|
||||
| Instagram hashtags | `apify/instagram-hashtag-scraper` | Brand hashtag monitoring |
|
||||
| Instagram search | `apify/instagram-search-scraper` | Brand mention discovery |
|
||||
| Instagram tagged posts | `apify/instagram-tagged-scraper` | Brand tag tracking |
|
||||
| Instagram export | `apify/export-instagram-comments-posts` | Bulk comment export |
|
||||
| Instagram comprehensive | `apify/instagram-scraper` | Full Instagram monitoring |
|
||||
| Instagram API | `apify/instagram-api-scraper` | API-based monitoring |
|
||||
| YouTube comments | `streamers/youtube-comments-scraper` | Video comment sentiment |
|
||||
| TikTok comments | `clockworks/tiktok-comments-scraper` | TikTok sentiment |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `compass/crawler-google-places`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Results
|
||||
|
||||
After completion, report:
|
||||
- Number of reviews/mentions found
|
||||
- File location and name
|
||||
- Key fields available
|
||||
- Suggested next steps (sentiment analysis, filtering)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-brand-reputation-monitoring-1.1.1';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
131
skills/apify-competitor-intelligence/SKILL.md
Normal file
131
skills/apify-competitor-intelligence/SKILL.md
Normal file
@@ -0,0 +1,131 @@
|
||||
---
|
||||
name: apify-competitor-intelligence
|
||||
description: Analyze competitor strategies, content, pricing, ads, and market positioning across Google Maps, Booking.com, Facebook, Instagram, YouTube, and TikTok.
|
||||
---
|
||||
|
||||
# Competitor Intelligence
|
||||
|
||||
Analyze competitors using Apify Actors to extract data from multiple platforms.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Identify competitor analysis type (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the analysis script
|
||||
- [ ] Step 5: Summarize findings
|
||||
```
|
||||
|
||||
### Step 1: Identify Competitor Analysis Type
|
||||
|
||||
Select the appropriate Actor based on analysis needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Competitor business data | `compass/crawler-google-places` | Location analysis |
|
||||
| Competitor contact discovery | `poidata/google-maps-email-extractor` | Email extraction |
|
||||
| Feature benchmarking | `compass/google-maps-extractor` | Detailed business data |
|
||||
| Competitor review analysis | `compass/Google-Maps-Reviews-Scraper` | Review comparison |
|
||||
| Hotel competitor data | `voyager/booking-scraper` | Hotel benchmarking |
|
||||
| Hotel review comparison | `voyager/booking-reviews-scraper` | Review analysis |
|
||||
| Competitor ad strategies | `apify/facebook-ads-scraper` | Ad creative analysis |
|
||||
| Competitor page metrics | `apify/facebook-pages-scraper` | Page performance |
|
||||
| Competitor content analysis | `apify/facebook-posts-scraper` | Post strategies |
|
||||
| Competitor reels performance | `apify/facebook-reels-scraper` | Reels analysis |
|
||||
| Competitor audience analysis | `apify/facebook-comments-scraper` | Comment sentiment |
|
||||
| Competitor event monitoring | `apify/facebook-events-scraper` | Event tracking |
|
||||
| Competitor audience overlap | `apify/facebook-followers-following-scraper` | Follower analysis |
|
||||
| Competitor review benchmarking | `apify/facebook-reviews-scraper` | Review comparison |
|
||||
| Competitor ad monitoring | `apify/facebook-search-scraper` | Ad discovery |
|
||||
| Competitor profile metrics | `apify/instagram-profile-scraper` | Profile analysis |
|
||||
| Competitor content monitoring | `apify/instagram-post-scraper` | Post tracking |
|
||||
| Competitor engagement analysis | `apify/instagram-comment-scraper` | Comment analysis |
|
||||
| Competitor reel performance | `apify/instagram-reel-scraper` | Reel metrics |
|
||||
| Competitor growth tracking | `apify/instagram-followers-count-scraper` | Follower tracking |
|
||||
| Comprehensive competitor data | `apify/instagram-scraper` | Full analysis |
|
||||
| API-based competitor analysis | `apify/instagram-api-scraper` | API access |
|
||||
| Competitor video analysis | `streamers/youtube-scraper` | Video metrics |
|
||||
| Competitor sentiment analysis | `streamers/youtube-comments-scraper` | Comment sentiment |
|
||||
| Competitor channel metrics | `streamers/youtube-channel-scraper` | Channel analysis |
|
||||
| TikTok competitor analysis | `clockworks/tiktok-scraper` | TikTok data |
|
||||
| Competitor video strategies | `clockworks/tiktok-video-scraper` | Video analysis |
|
||||
| Competitor TikTok profiles | `clockworks/tiktok-profile-scraper` | Profile data |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `compass/crawler-google-places`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Findings
|
||||
|
||||
After completion, report:
|
||||
- Number of competitors analyzed
|
||||
- File location and name
|
||||
- Key competitive insights
|
||||
- Suggested next steps (deeper analysis, benchmarking)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-competitor-intelligence-1.0.1';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
120
skills/apify-content-analytics/SKILL.md
Normal file
120
skills/apify-content-analytics/SKILL.md
Normal file
@@ -0,0 +1,120 @@
|
||||
---
|
||||
name: apify-content-analytics
|
||||
description: Track engagement metrics, measure campaign ROI, and analyze content performance across Instagram, Facebook, YouTube, and TikTok.
|
||||
---
|
||||
|
||||
# Content Analytics
|
||||
|
||||
Track and analyze content performance using Apify Actors to extract engagement metrics from multiple platforms.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Identify content analytics type (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the analytics script
|
||||
- [ ] Step 5: Summarize findings
|
||||
```
|
||||
|
||||
### Step 1: Identify Content Analytics Type
|
||||
|
||||
Select the appropriate Actor based on analytics needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Post engagement metrics | `apify/instagram-post-scraper` | Post performance |
|
||||
| Reel performance | `apify/instagram-reel-scraper` | Reel analytics |
|
||||
| Follower growth tracking | `apify/instagram-followers-count-scraper` | Growth metrics |
|
||||
| Comment engagement | `apify/instagram-comment-scraper` | Comment analysis |
|
||||
| Hashtag performance | `apify/instagram-hashtag-scraper` | Branded hashtags |
|
||||
| Mention tracking | `apify/instagram-tagged-scraper` | Tag tracking |
|
||||
| Comprehensive metrics | `apify/instagram-scraper` | Full data |
|
||||
| API-based analytics | `apify/instagram-api-scraper` | API access |
|
||||
| Facebook post performance | `apify/facebook-posts-scraper` | Post metrics |
|
||||
| Reaction analysis | `apify/facebook-likes-scraper` | Engagement types |
|
||||
| Facebook Reels metrics | `apify/facebook-reels-scraper` | Reels performance |
|
||||
| Ad performance tracking | `apify/facebook-ads-scraper` | Ad analytics |
|
||||
| Facebook comment analysis | `apify/facebook-comments-scraper` | Comment engagement |
|
||||
| Page performance audit | `apify/facebook-pages-scraper` | Page metrics |
|
||||
| YouTube video metrics | `streamers/youtube-scraper` | Video performance |
|
||||
| YouTube Shorts analytics | `streamers/youtube-shorts-scraper` | Shorts performance |
|
||||
| TikTok content metrics | `clockworks/tiktok-scraper` | TikTok analytics |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `apify/instagram-post-scraper`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Findings
|
||||
|
||||
After completion, report:
|
||||
- Number of content pieces analyzed
|
||||
- File location and name
|
||||
- Key performance insights
|
||||
- Suggested next steps (deeper analysis, content optimization)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
363
skills/apify-content-analytics/reference/scripts/run_actor.js
Normal file
363
skills/apify-content-analytics/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-content-analytics-1.0.0';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
263
skills/apify-ecommerce/SKILL.md
Normal file
263
skills/apify-ecommerce/SKILL.md
Normal file
@@ -0,0 +1,263 @@
|
||||
---
|
||||
name: apify-ecommerce
|
||||
description: "Scrape e-commerce data for pricing intelligence, customer reviews, and seller discovery across Amazon, Walmart, eBay, IKEA, and 50+ marketplaces. Use when user asks to monitor prices, track competi..."
|
||||
---
|
||||
|
||||
# E-commerce Data Extraction
|
||||
|
||||
Extract product data, prices, reviews, and seller information from any e-commerce platform using Apify's E-commerce Scraping Tool.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `.env` file with `APIFY_TOKEN` (at `~/.claude/.env`)
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
|
||||
## Workflow Selection
|
||||
|
||||
| User Need | Workflow | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Track prices, compare products | Workflow 1: Products & Pricing | Price monitoring, MAP compliance, competitor analysis. Add AI summary for insights. |
|
||||
| Analyze reviews (sentiment or quality) | Workflow 2: Reviews | Brand perception, customer sentiment, quality issues, defect patterns |
|
||||
| Find sellers across stores | Workflow 3: Sellers | Unauthorized resellers, vendor discovery via Google Shopping |
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Select workflow and determine data source
|
||||
- [ ] Step 2: Configure Actor input
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the extraction script
|
||||
- [ ] Step 5: Summarize results
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow 1: Products & Pricing
|
||||
|
||||
**Use case:** Extract product data, prices, and stock status. Track competitor prices, detect MAP violations, benchmark products, or research markets.
|
||||
|
||||
**Best for:** Pricing analysts, product managers, market researchers.
|
||||
|
||||
### Input Options
|
||||
|
||||
| Input Type | Field | Description |
|
||||
|------------|-------|-------------|
|
||||
| Product URLs | `detailsUrls` | Direct URLs to product pages (use object format) |
|
||||
| Category URLs | `listingUrls` | URLs to category/search result pages |
|
||||
| Keyword Search | `keyword` + `marketplaces` | Search term across selected marketplaces |
|
||||
|
||||
### Example - Product URLs
|
||||
```json
|
||||
{
|
||||
"detailsUrls": [
|
||||
{"url": "https://www.amazon.com/dp/B09V3KXJPB"},
|
||||
{"url": "https://www.walmart.com/ip/123456789"}
|
||||
],
|
||||
"additionalProperties": true
|
||||
}
|
||||
```
|
||||
|
||||
### Example - Keyword Search
|
||||
```json
|
||||
{
|
||||
"keyword": "Samsung Galaxy S24",
|
||||
"marketplaces": ["www.amazon.com", "www.walmart.com"],
|
||||
"additionalProperties": true,
|
||||
"maxProductResults": 50
|
||||
}
|
||||
```
|
||||
|
||||
### Optional: AI Summary
|
||||
|
||||
Add these fields to get AI-generated insights:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `fieldsToAnalyze` | Data points to analyze: `["name", "offers", "brand", "description"]` |
|
||||
| `customPrompt` | Custom analysis instructions |
|
||||
|
||||
**Example with AI summary:**
|
||||
```json
|
||||
{
|
||||
"keyword": "robot vacuum",
|
||||
"marketplaces": ["www.amazon.com"],
|
||||
"maxProductResults": 50,
|
||||
"additionalProperties": true,
|
||||
"fieldsToAnalyze": ["name", "offers", "brand"],
|
||||
"customPrompt": "Summarize price range and identify top brands"
|
||||
}
|
||||
```
|
||||
|
||||
### Output Fields
|
||||
- `name` - Product name
|
||||
- `url` - Product URL
|
||||
- `offers.price` - Current price
|
||||
- `offers.priceCurrency` - Currency code (may vary by seller region)
|
||||
- `brand.slogan` - Brand name (nested in object)
|
||||
- `image` - Product image URL
|
||||
- Additional seller/stock info when `additionalProperties: true`
|
||||
|
||||
> **Note:** Currency may vary in results even for US searches, as prices reflect different seller regions.
|
||||
|
||||
---
|
||||
|
||||
## Workflow 2: Customer Reviews
|
||||
|
||||
**Use case:** Extract reviews for sentiment analysis, brand perception monitoring, or quality issue detection.
|
||||
|
||||
**Best for:** Brand managers, customer experience teams, QA teams, product managers.
|
||||
|
||||
### Input Options
|
||||
|
||||
| Input Type | Field | Description |
|
||||
|------------|-------|-------------|
|
||||
| Product URLs | `reviewListingUrls` | Product pages to extract reviews from |
|
||||
| Keyword Search | `keywordReviews` + `marketplacesReviews` | Search for product reviews by keyword |
|
||||
|
||||
### Example - Extract Reviews from Product
|
||||
```json
|
||||
{
|
||||
"reviewListingUrls": [
|
||||
{"url": "https://www.amazon.com/dp/B09V3KXJPB"}
|
||||
],
|
||||
"sortReview": "Most recent",
|
||||
"additionalReviewProperties": true,
|
||||
"maxReviewResults": 500
|
||||
}
|
||||
```
|
||||
|
||||
### Example - Keyword Search
|
||||
```json
|
||||
{
|
||||
"keywordReviews": "wireless earbuds",
|
||||
"marketplacesReviews": ["www.amazon.com"],
|
||||
"sortReview": "Most recent",
|
||||
"additionalReviewProperties": true,
|
||||
"maxReviewResults": 200
|
||||
}
|
||||
```
|
||||
|
||||
### Sort Options
|
||||
- `Most recent` - Latest reviews first (recommended)
|
||||
- `Most relevant` - Platform default relevance
|
||||
- `Most helpful` - Highest voted reviews
|
||||
- `Highest rated` - 5-star reviews first
|
||||
- `Lowest rated` - 1-star reviews first
|
||||
|
||||
> **Note:** The `sortReview: "Lowest rated"` option may not work consistently across all marketplaces. For quality analysis, collect a large sample and filter by rating in post-processing.
|
||||
|
||||
### Quality Analysis Tips
|
||||
- Set high `maxReviewResults` for statistical significance
|
||||
- Look for recurring keywords: "broke", "defect", "quality", "returned"
|
||||
- Filter results by rating if sorting doesn't work as expected
|
||||
- Cross-reference with competitor products for benchmarking
|
||||
|
||||
---
|
||||
|
||||
## Workflow 3: Seller Intelligence
|
||||
|
||||
**Use case:** Find sellers across stores, discover unauthorized resellers, evaluate vendor options.
|
||||
|
||||
**Best for:** Brand protection teams, procurement, supply chain managers.
|
||||
|
||||
> **Note:** This workflow uses Google Shopping to find sellers across stores. Direct seller profile URLs are not reliably supported.
|
||||
|
||||
### Input Configuration
|
||||
```json
|
||||
{
|
||||
"googleShoppingSearchKeyword": "Nike Air Max 90",
|
||||
"scrapeSellersFromGoogleShopping": true,
|
||||
"countryCode": "us",
|
||||
"maxGoogleShoppingSellersPerProduct": 20,
|
||||
"maxGoogleShoppingResults": 100
|
||||
}
|
||||
```
|
||||
|
||||
### Options
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `googleShoppingSearchKeyword` | Product name to search |
|
||||
| `scrapeSellersFromGoogleShopping` | Set to `true` to extract sellers |
|
||||
| `scrapeProductsFromGoogleShopping` | Set to `true` to also extract product details |
|
||||
| `countryCode` | Target country (e.g., `us`, `uk`, `de`) |
|
||||
| `maxGoogleShoppingSellersPerProduct` | Max sellers per product |
|
||||
| `maxGoogleShoppingResults` | Total result limit |
|
||||
|
||||
---
|
||||
|
||||
## Supported Marketplaces
|
||||
|
||||
### Amazon (20+ regions)
|
||||
`www.amazon.com`, `www.amazon.co.uk`, `www.amazon.de`, `www.amazon.fr`, `www.amazon.it`, `www.amazon.es`, `www.amazon.ca`, `www.amazon.com.au`, `www.amazon.co.jp`, `www.amazon.in`, `www.amazon.com.br`, `www.amazon.com.mx`, `www.amazon.nl`, `www.amazon.pl`, `www.amazon.se`, `www.amazon.ae`, `www.amazon.sa`, `www.amazon.sg`, `www.amazon.com.tr`, `www.amazon.eg`
|
||||
|
||||
### Major US Retailers
|
||||
`www.walmart.com`, `www.costco.com`, `www.costco.ca`, `www.homedepot.com`
|
||||
|
||||
### European Retailers
|
||||
`allegro.pl`, `allegro.cz`, `allegro.sk`, `www.alza.cz`, `www.alza.sk`, `www.alza.de`, `www.alza.at`, `www.alza.hu`, `www.kaufland.de`, `www.kaufland.pl`, `www.kaufland.cz`, `www.kaufland.sk`, `www.kaufland.at`, `www.kaufland.fr`, `www.kaufland.it`, `www.cdiscount.com`
|
||||
|
||||
### IKEA (40+ country/language combinations)
|
||||
Supports all major IKEA regional sites with multiple language options.
|
||||
|
||||
### Google Shopping
|
||||
Use for seller discovery across multiple stores.
|
||||
|
||||
---
|
||||
|
||||
## Running the Extraction
|
||||
|
||||
### Step 1: Set Skill Path
|
||||
```bash
|
||||
SKILL_PATH=~/.claude/skills/apify-ecommerce
|
||||
```
|
||||
|
||||
### Step 2: Run Script
|
||||
|
||||
**Quick answer (display in chat):**
|
||||
```bash
|
||||
node --env-file=~/.claude/.env $SKILL_PATH/reference/scripts/run_actor.js \
|
||||
--actor "apify/e-commerce-scraping-tool" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV export:**
|
||||
```bash
|
||||
node --env-file=~/.claude/.env $SKILL_PATH/reference/scripts/run_actor.js \
|
||||
--actor "apify/e-commerce-scraping-tool" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_filename.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON export:**
|
||||
```bash
|
||||
node --env-file=~/.claude/.env $SKILL_PATH/reference/scripts/run_actor.js \
|
||||
--actor "apify/e-commerce-scraping-tool" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_filename.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 3: Summarize Results
|
||||
|
||||
Report:
|
||||
- Number of items extracted
|
||||
- File location (if exported)
|
||||
- Key insights based on workflow:
|
||||
- **Products:** Price range, outliers, MAP violations
|
||||
- **Reviews:** Average rating, sentiment trends, quality issues
|
||||
- **Sellers:** Seller count, unauthorized sellers found
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Solution |
|
||||
|-------|----------|
|
||||
| `APIFY_TOKEN not found` | Ensure `~/.claude/.env` contains `APIFY_TOKEN=your_token` |
|
||||
| `Actor not found` | Verify Actor ID: `apify/e-commerce-scraping-tool` |
|
||||
| `Run FAILED` | Check Apify console link in error output |
|
||||
| `Timeout` | Reduce `maxProductResults` or increase `--timeout` |
|
||||
| `No results` | Verify URLs are valid and accessible |
|
||||
| `Invalid marketplace` | Check marketplace value matches supported list exactly |
|
||||
3
skills/apify-ecommerce/reference/scripts/package.json
Normal file
3
skills/apify-ecommerce/reference/scripts/package.json
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"type": "module"
|
||||
}
|
||||
369
skills/apify-ecommerce/reference/scripts/run_actor.js
Normal file
369
skills/apify-ecommerce/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,369 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output data.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-ecommerce-1.0.0';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., apify/e-commerce-scraping-tool) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 products
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "apify/e-commerce-scraping-tool" \\
|
||||
--input '{"keyword": "bluetooth headphones", "marketplaces": ["www.amazon.com"], "maxProductResults": 10}'
|
||||
|
||||
# Export prices to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "apify/e-commerce-scraping-tool" \\
|
||||
--input '{"detailsUrls": ["https://amazon.com/dp/B09V3KXJPB"]}' \\
|
||||
--output prices.csv --format csv
|
||||
|
||||
# Export reviews to JSON
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "apify/e-commerce-scraping-tool" \\
|
||||
--input '{"reviewListingUrls": ["https://amazon.com/dp/B09V3KXJPB"], "maxReviewResults": 100}' \\
|
||||
--output reviews.json --format json
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
118
skills/apify-influencer-discovery/SKILL.md
Normal file
118
skills/apify-influencer-discovery/SKILL.md
Normal file
@@ -0,0 +1,118 @@
|
||||
---
|
||||
name: apify-influencer-discovery
|
||||
description: Find and evaluate influencers for brand partnerships, verify authenticity, and track collaboration performance across Instagram, Facebook, YouTube, and TikTok.
|
||||
---
|
||||
|
||||
# Influencer Discovery
|
||||
|
||||
Discover and analyze influencers across multiple platforms using Apify Actors.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Determine discovery source (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the discovery script
|
||||
- [ ] Step 5: Summarize results
|
||||
```
|
||||
|
||||
### Step 1: Determine Discovery Source
|
||||
|
||||
Select the appropriate Actor based on user needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Influencer profiles | `apify/instagram-profile-scraper` | Profile metrics, bio, follower counts |
|
||||
| Find by hashtag | `apify/instagram-hashtag-scraper` | Discover influencers using specific hashtags |
|
||||
| Reel engagement | `apify/instagram-reel-scraper` | Analyze reel performance and engagement |
|
||||
| Discovery by niche | `apify/instagram-search-scraper` | Search for influencers by keyword/niche |
|
||||
| Brand mentions | `apify/instagram-tagged-scraper` | Track who tags brands/products |
|
||||
| Comprehensive data | `apify/instagram-scraper` | Full profile, posts, comments analysis |
|
||||
| API-based discovery | `apify/instagram-api-scraper` | Fast API-based data extraction |
|
||||
| Engagement analysis | `apify/export-instagram-comments-posts` | Export comments for sentiment analysis |
|
||||
| Facebook content | `apify/facebook-posts-scraper` | Analyze Facebook post performance |
|
||||
| Micro-influencers | `apify/facebook-groups-scraper` | Find influencers in niche groups |
|
||||
| Influential pages | `apify/facebook-search-scraper` | Search for influential pages |
|
||||
| YouTube creators | `streamers/youtube-channel-scraper` | Channel metrics and subscriber data |
|
||||
| TikTok influencers | `clockworks/tiktok-scraper` | Comprehensive TikTok data extraction |
|
||||
| TikTok (free) | `clockworks/free-tiktok-scraper` | Free TikTok data extractor |
|
||||
| Live streamers | `clockworks/tiktok-live-scraper` | Discover live streaming influencers |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `apify/instagram-profile-scraper`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Results
|
||||
|
||||
After completion, report:
|
||||
- Number of influencers found
|
||||
- File location and name
|
||||
- Key metrics available (followers, engagement rate, etc.)
|
||||
- Suggested next steps (filtering, outreach, deeper analysis)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
363
skills/apify-influencer-discovery/reference/scripts/run_actor.js
Normal file
363
skills/apify-influencer-discovery/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-influencer-discovery-1.0.0';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
120
skills/apify-lead-generation/SKILL.md
Normal file
120
skills/apify-lead-generation/SKILL.md
Normal file
@@ -0,0 +1,120 @@
|
||||
---
|
||||
name: apify-lead-generation
|
||||
description: "Generates B2B/B2C leads by scraping Google Maps, websites, Instagram, TikTok, Facebook, LinkedIn, YouTube, and Google Search. Use when user asks to find leads, prospects, businesses, build lead lis..."
|
||||
---
|
||||
|
||||
# Lead Generation
|
||||
|
||||
Scrape leads from multiple platforms using Apify Actors.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Determine lead source (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the lead finder script
|
||||
- [ ] Step 5: Summarize results
|
||||
```
|
||||
|
||||
### Step 1: Determine Lead Source
|
||||
|
||||
Select the appropriate Actor based on user needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Local businesses | `compass/crawler-google-places` | Restaurants, gyms, shops |
|
||||
| Contact enrichment | `vdrmota/contact-info-scraper` | Emails, phones from URLs |
|
||||
| Instagram profiles | `apify/instagram-profile-scraper` | Influencer discovery |
|
||||
| Instagram posts/comments | `apify/instagram-scraper` | Posts, comments, hashtags, places |
|
||||
| Instagram search | `apify/instagram-search-scraper` | Places, users, hashtags discovery |
|
||||
| TikTok videos/hashtags | `clockworks/tiktok-scraper` | Comprehensive TikTok data extraction |
|
||||
| TikTok hashtags/profiles | `clockworks/free-tiktok-scraper` | Free TikTok data extractor |
|
||||
| TikTok user search | `clockworks/tiktok-user-search-scraper` | Find users by keywords |
|
||||
| TikTok profiles | `clockworks/tiktok-profile-scraper` | Creator outreach |
|
||||
| TikTok followers/following | `clockworks/tiktok-followers-scraper` | Audience analysis, segmentation |
|
||||
| Facebook pages | `apify/facebook-pages-scraper` | Business contacts |
|
||||
| Facebook page contacts | `apify/facebook-page-contact-information` | Extract emails, phones, addresses |
|
||||
| Facebook groups | `apify/facebook-groups-scraper` | Buying intent signals |
|
||||
| Facebook events | `apify/facebook-events-scraper` | Event networking, partnerships |
|
||||
| Google Search | `apify/google-search-scraper` | Broad lead discovery |
|
||||
| YouTube channels | `streamers/youtube-scraper` | Creator partnerships |
|
||||
| Google Maps emails | `poidata/google-maps-email-extractor` | Direct email extraction |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `compass/crawler-google-places`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Results
|
||||
|
||||
After completion, report:
|
||||
- Number of leads found
|
||||
- File location and name
|
||||
- Key fields available
|
||||
- Suggested next steps (filtering, enrichment)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
363
skills/apify-lead-generation/reference/scripts/run_actor.js
Normal file
363
skills/apify-lead-generation/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-lead-generation-1.1.11';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
119
skills/apify-market-research/SKILL.md
Normal file
119
skills/apify-market-research/SKILL.md
Normal file
@@ -0,0 +1,119 @@
|
||||
---
|
||||
name: apify-market-research
|
||||
description: Analyze market conditions, geographic opportunities, pricing, consumer behavior, and product validation across Google Maps, Facebook, Instagram, Booking.com, and TripAdvisor.
|
||||
---
|
||||
|
||||
# Market Research
|
||||
|
||||
Conduct market research using Apify Actors to extract data from multiple platforms.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Identify market research type (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the analysis script
|
||||
- [ ] Step 5: Summarize findings
|
||||
```
|
||||
|
||||
### Step 1: Identify Market Research Type
|
||||
|
||||
Select the appropriate Actor based on research needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Market density | `compass/crawler-google-places` | Location analysis |
|
||||
| Geospatial analysis | `compass/google-maps-extractor` | Business mapping |
|
||||
| Regional interest | `apify/google-trends-scraper` | Trend data |
|
||||
| Pricing and demand | `apify/facebook-marketplace-scraper` | Market pricing |
|
||||
| Event market | `apify/facebook-events-scraper` | Event analysis |
|
||||
| Consumer needs | `apify/facebook-groups-scraper` | Group research |
|
||||
| Market landscape | `apify/facebook-pages-scraper` | Business pages |
|
||||
| Business density | `apify/facebook-page-contact-information` | Contact data |
|
||||
| Cultural insights | `apify/facebook-photos-scraper` | Visual research |
|
||||
| Niche targeting | `apify/instagram-hashtag-scraper` | Hashtag research |
|
||||
| Hashtag stats | `apify/instagram-hashtag-stats` | Market sizing |
|
||||
| Market activity | `apify/instagram-reel-scraper` | Activity analysis |
|
||||
| Market intelligence | `apify/instagram-scraper` | Full data |
|
||||
| Product launch research | `apify/instagram-api-scraper` | API access |
|
||||
| Hospitality market | `voyager/booking-scraper` | Hotel data |
|
||||
| Tourism insights | `maxcopell/tripadvisor-reviews` | Review analysis |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `compass/crawler-google-places`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Findings
|
||||
|
||||
After completion, report:
|
||||
- Number of results found
|
||||
- File location and name
|
||||
- Key market insights
|
||||
- Suggested next steps (deeper analysis, validation)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
363
skills/apify-market-research/reference/scripts/run_actor.js
Normal file
363
skills/apify-market-research/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-market-research-1.0.0';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
122
skills/apify-trend-analysis/SKILL.md
Normal file
122
skills/apify-trend-analysis/SKILL.md
Normal file
@@ -0,0 +1,122 @@
|
||||
---
|
||||
name: apify-trend-analysis
|
||||
description: Discover and track emerging trends across Google Trends, Instagram, Facebook, YouTube, and TikTok to inform content strategy.
|
||||
---
|
||||
|
||||
# Trend Analysis
|
||||
|
||||
Discover and track emerging trends using Apify Actors to extract data from multiple platforms.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Identify trend type (select Actor)
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the analysis script
|
||||
- [ ] Step 5: Summarize findings
|
||||
```
|
||||
|
||||
### Step 1: Identify Trend Type
|
||||
|
||||
Select the appropriate Actor based on research needs:
|
||||
|
||||
| User Need | Actor ID | Best For |
|
||||
|-----------|----------|----------|
|
||||
| Search trends | `apify/google-trends-scraper` | Google Trends data |
|
||||
| Hashtag tracking | `apify/instagram-hashtag-scraper` | Hashtag content |
|
||||
| Hashtag metrics | `apify/instagram-hashtag-stats` | Performance stats |
|
||||
| Visual trends | `apify/instagram-post-scraper` | Post analysis |
|
||||
| Trending discovery | `apify/instagram-search-scraper` | Search trends |
|
||||
| Comprehensive tracking | `apify/instagram-scraper` | Full data |
|
||||
| API-based trends | `apify/instagram-api-scraper` | API access |
|
||||
| Engagement trends | `apify/export-instagram-comments-posts` | Comment tracking |
|
||||
| Product trends | `apify/facebook-marketplace-scraper` | Marketplace data |
|
||||
| Visual analysis | `apify/facebook-photos-scraper` | Photo trends |
|
||||
| Community trends | `apify/facebook-groups-scraper` | Group monitoring |
|
||||
| YouTube Shorts | `streamers/youtube-shorts-scraper` | Short-form trends |
|
||||
| YouTube hashtags | `streamers/youtube-video-scraper-by-hashtag` | Hashtag videos |
|
||||
| TikTok hashtags | `clockworks/tiktok-hashtag-scraper` | Hashtag content |
|
||||
| Trending sounds | `clockworks/tiktok-sound-scraper` | Audio trends |
|
||||
| TikTok ads | `clockworks/tiktok-ads-scraper` | Ad trends |
|
||||
| Discover page | `clockworks/tiktok-discover-scraper` | Discover trends |
|
||||
| Explore trends | `clockworks/tiktok-explore-scraper` | Explore content |
|
||||
| Trending content | `clockworks/tiktok-trends-scraper` | Viral content |
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `apify/google-trends-scraper`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Findings
|
||||
|
||||
After completion, report:
|
||||
- Number of results found
|
||||
- File location and name
|
||||
- Key trend insights
|
||||
- Suggested next steps (deeper analysis, content opportunities)
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
363
skills/apify-trend-analysis/reference/scripts/run_actor.js
Normal file
363
skills/apify-trend-analysis/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-trend-analysis-1.0.0';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
230
skills/apify-ultimate-scraper/SKILL.md
Normal file
230
skills/apify-ultimate-scraper/SKILL.md
Normal file
@@ -0,0 +1,230 @@
|
||||
---
|
||||
name: apify-ultimate-scraper
|
||||
description: "Universal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, Google Maps, Google Search, Google Trends, Booking.com, and TripAdvisor. Use for lead gener..."
|
||||
---
|
||||
|
||||
# Universal Web Scraper
|
||||
|
||||
AI-driven data extraction from 55+ Actors across all major platforms. This skill automatically selects the best Actor for your task.
|
||||
|
||||
## Prerequisites
|
||||
(No need to check it upfront)
|
||||
|
||||
- `.env` file with `APIFY_TOKEN`
|
||||
- Node.js 20.6+ (for native `--env-file` support)
|
||||
- `mcpc` CLI tool: `npm install -g @apify/mcpc`
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist and track progress:
|
||||
|
||||
```
|
||||
Task Progress:
|
||||
- [ ] Step 1: Understand user goal and select Actor
|
||||
- [ ] Step 2: Fetch Actor schema via mcpc
|
||||
- [ ] Step 3: Ask user preferences (format, filename)
|
||||
- [ ] Step 4: Run the scraper script
|
||||
- [ ] Step 5: Summarize results and offer follow-ups
|
||||
```
|
||||
|
||||
### Step 1: Understand User Goal and Select Actor
|
||||
|
||||
First, understand what the user wants to achieve. Then select the best Actor from the options below.
|
||||
|
||||
#### Instagram Actors (12)
|
||||
|
||||
| Actor ID | Best For |
|
||||
|----------|----------|
|
||||
| `apify/instagram-profile-scraper` | Profile data, follower counts, bio info |
|
||||
| `apify/instagram-post-scraper` | Individual post details, engagement metrics |
|
||||
| `apify/instagram-comment-scraper` | Comment extraction, sentiment analysis |
|
||||
| `apify/instagram-hashtag-scraper` | Hashtag content, trending topics |
|
||||
| `apify/instagram-hashtag-stats` | Hashtag performance metrics |
|
||||
| `apify/instagram-reel-scraper` | Reels content and metrics |
|
||||
| `apify/instagram-search-scraper` | Search users, places, hashtags |
|
||||
| `apify/instagram-tagged-scraper` | Posts tagged with specific accounts |
|
||||
| `apify/instagram-followers-count-scraper` | Follower count tracking |
|
||||
| `apify/instagram-scraper` | Comprehensive Instagram data |
|
||||
| `apify/instagram-api-scraper` | API-based Instagram access |
|
||||
| `apify/export-instagram-comments-posts` | Bulk comment/post export |
|
||||
|
||||
#### Facebook Actors (14)
|
||||
|
||||
| Actor ID | Best For |
|
||||
|----------|----------|
|
||||
| `apify/facebook-pages-scraper` | Page data, metrics, contact info |
|
||||
| `apify/facebook-page-contact-information` | Emails, phones, addresses from pages |
|
||||
| `apify/facebook-posts-scraper` | Post content and engagement |
|
||||
| `apify/facebook-comments-scraper` | Comment extraction |
|
||||
| `apify/facebook-likes-scraper` | Reaction analysis |
|
||||
| `apify/facebook-reviews-scraper` | Page reviews |
|
||||
| `apify/facebook-groups-scraper` | Group content and members |
|
||||
| `apify/facebook-events-scraper` | Event data |
|
||||
| `apify/facebook-ads-scraper` | Ad creative and targeting |
|
||||
| `apify/facebook-search-scraper` | Search results |
|
||||
| `apify/facebook-reels-scraper` | Reels content |
|
||||
| `apify/facebook-photos-scraper` | Photo extraction |
|
||||
| `apify/facebook-marketplace-scraper` | Marketplace listings |
|
||||
| `apify/facebook-followers-following-scraper` | Follower/following lists |
|
||||
|
||||
#### TikTok Actors (14)
|
||||
|
||||
| Actor ID | Best For |
|
||||
|----------|----------|
|
||||
| `clockworks/tiktok-scraper` | Comprehensive TikTok data |
|
||||
| `clockworks/free-tiktok-scraper` | Free TikTok extraction |
|
||||
| `clockworks/tiktok-profile-scraper` | Profile data |
|
||||
| `clockworks/tiktok-video-scraper` | Video details and metrics |
|
||||
| `clockworks/tiktok-comments-scraper` | Comment extraction |
|
||||
| `clockworks/tiktok-followers-scraper` | Follower lists |
|
||||
| `clockworks/tiktok-user-search-scraper` | Find users by keywords |
|
||||
| `clockworks/tiktok-hashtag-scraper` | Hashtag content |
|
||||
| `clockworks/tiktok-sound-scraper` | Trending sounds |
|
||||
| `clockworks/tiktok-ads-scraper` | Ad content |
|
||||
| `clockworks/tiktok-discover-scraper` | Discover page content |
|
||||
| `clockworks/tiktok-explore-scraper` | Explore content |
|
||||
| `clockworks/tiktok-trends-scraper` | Trending content |
|
||||
| `clockworks/tiktok-live-scraper` | Live stream data |
|
||||
|
||||
#### YouTube Actors (5)
|
||||
|
||||
| Actor ID | Best For |
|
||||
|----------|----------|
|
||||
| `streamers/youtube-scraper` | Video data and metrics |
|
||||
| `streamers/youtube-channel-scraper` | Channel information |
|
||||
| `streamers/youtube-comments-scraper` | Comment extraction |
|
||||
| `streamers/youtube-shorts-scraper` | Shorts content |
|
||||
| `streamers/youtube-video-scraper-by-hashtag` | Videos by hashtag |
|
||||
|
||||
#### Google Maps Actors (4)
|
||||
|
||||
| Actor ID | Best For |
|
||||
|----------|----------|
|
||||
| `compass/crawler-google-places` | Business listings, ratings, contact info |
|
||||
| `compass/google-maps-extractor` | Detailed business data |
|
||||
| `compass/Google-Maps-Reviews-Scraper` | Review extraction |
|
||||
| `poidata/google-maps-email-extractor` | Email discovery from listings |
|
||||
|
||||
#### Other Actors (6)
|
||||
|
||||
| Actor ID | Best For |
|
||||
|----------|----------|
|
||||
| `apify/google-search-scraper` | Google search results |
|
||||
| `apify/google-trends-scraper` | Google Trends data |
|
||||
| `voyager/booking-scraper` | Booking.com hotel data |
|
||||
| `voyager/booking-reviews-scraper` | Booking.com reviews |
|
||||
| `maxcopell/tripadvisor-reviews` | TripAdvisor reviews |
|
||||
| `vdrmota/contact-info-scraper` | Contact enrichment from URLs |
|
||||
|
||||
---
|
||||
|
||||
#### Actor Selection by Use Case
|
||||
|
||||
| Use Case | Primary Actors |
|
||||
|----------|---------------|
|
||||
| **Lead Generation** | `compass/crawler-google-places`, `poidata/google-maps-email-extractor`, `vdrmota/contact-info-scraper` |
|
||||
| **Influencer Discovery** | `apify/instagram-profile-scraper`, `clockworks/tiktok-profile-scraper`, `streamers/youtube-channel-scraper` |
|
||||
| **Brand Monitoring** | `apify/instagram-tagged-scraper`, `apify/instagram-hashtag-scraper`, `compass/Google-Maps-Reviews-Scraper` |
|
||||
| **Competitor Analysis** | `apify/facebook-pages-scraper`, `apify/facebook-ads-scraper`, `apify/instagram-profile-scraper` |
|
||||
| **Content Analytics** | `apify/instagram-post-scraper`, `clockworks/tiktok-scraper`, `streamers/youtube-scraper` |
|
||||
| **Trend Research** | `apify/google-trends-scraper`, `clockworks/tiktok-trends-scraper`, `apify/instagram-hashtag-stats` |
|
||||
| **Review Analysis** | `compass/Google-Maps-Reviews-Scraper`, `voyager/booking-reviews-scraper`, `maxcopell/tripadvisor-reviews` |
|
||||
| **Audience Analysis** | `apify/instagram-followers-count-scraper`, `clockworks/tiktok-followers-scraper`, `apify/facebook-followers-following-scraper` |
|
||||
|
||||
---
|
||||
|
||||
#### Multi-Actor Workflows
|
||||
|
||||
For complex tasks, chain multiple Actors:
|
||||
|
||||
| Workflow | Step 1 | Step 2 |
|
||||
|----------|--------|--------|
|
||||
| **Lead enrichment** | `compass/crawler-google-places` → | `vdrmota/contact-info-scraper` |
|
||||
| **Influencer vetting** | `apify/instagram-profile-scraper` → | `apify/instagram-comment-scraper` |
|
||||
| **Competitor deep-dive** | `apify/facebook-pages-scraper` → | `apify/facebook-posts-scraper` |
|
||||
| **Local business analysis** | `compass/crawler-google-places` → | `compass/Google-Maps-Reviews-Scraper` |
|
||||
|
||||
#### Can't Find a Suitable Actor?
|
||||
|
||||
If none of the Actors above match the user's request, search the Apify Store directly:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call search-actors keywords:="SEARCH_KEYWORDS" limit:=10 offset:=0 category:="" | jq -r '.content[0].text'
|
||||
```
|
||||
|
||||
Replace `SEARCH_KEYWORDS` with 1-3 simple terms (e.g., "LinkedIn profiles", "Amazon products", "Twitter").
|
||||
|
||||
### Step 2: Fetch Actor Schema
|
||||
|
||||
Fetch the Actor's input schema and details dynamically using mcpc:
|
||||
|
||||
```bash
|
||||
export $(grep APIFY_TOKEN .env | xargs) && mcpc --json mcp.apify.com --header "Authorization: Bearer $APIFY_TOKEN" tools-call fetch-actor-details actor:="ACTOR_ID" | jq -r ".content"
|
||||
```
|
||||
|
||||
Replace `ACTOR_ID` with the selected Actor (e.g., `compass/crawler-google-places`).
|
||||
|
||||
This returns:
|
||||
- Actor description and README
|
||||
- Required and optional input parameters
|
||||
- Output fields (if available)
|
||||
|
||||
### Step 3: Ask User Preferences
|
||||
|
||||
Before running, ask:
|
||||
1. **Output format**:
|
||||
- **Quick answer** - Display top few results in chat (no file saved)
|
||||
- **CSV** - Full export with all fields
|
||||
- **JSON** - Full export in JSON format
|
||||
2. **Number of results**: Based on character of use case
|
||||
|
||||
### Step 4: Run the Script
|
||||
|
||||
**Quick answer (display in chat, no file):**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT'
|
||||
```
|
||||
|
||||
**CSV:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.csv \
|
||||
--format csv
|
||||
```
|
||||
|
||||
**JSON:**
|
||||
```bash
|
||||
node --env-file=.env ${CLAUDE_PLUGIN_ROOT}/reference/scripts/run_actor.js \
|
||||
--actor "ACTOR_ID" \
|
||||
--input 'JSON_INPUT' \
|
||||
--output YYYY-MM-DD_OUTPUT_FILE.json \
|
||||
--format json
|
||||
```
|
||||
|
||||
### Step 5: Summarize Results and Offer Follow-ups
|
||||
|
||||
After completion, report:
|
||||
- Number of results found
|
||||
- File location and name
|
||||
- Key fields available
|
||||
- **Suggested follow-up workflows** based on results:
|
||||
|
||||
| If User Got | Suggest Next |
|
||||
|-------------|--------------|
|
||||
| Business listings | Enrich with `vdrmota/contact-info-scraper` or get reviews |
|
||||
| Influencer profiles | Analyze engagement with comment scrapers |
|
||||
| Competitor pages | Deep-dive with post/ad scrapers |
|
||||
| Trend data | Validate with platform-specific hashtag scrapers |
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
`APIFY_TOKEN not found` - Ask user to create `.env` with `APIFY_TOKEN=your_token`
|
||||
`mcpc not found` - Ask user to install `npm install -g @apify/mcpc`
|
||||
`Actor not found` - Check Actor ID spelling
|
||||
`Run FAILED` - Ask user to check Apify console link in error output
|
||||
`Timeout` - Reduce input size or increase `--timeout`
|
||||
363
skills/apify-ultimate-scraper/reference/scripts/run_actor.js
Normal file
363
skills/apify-ultimate-scraper/reference/scripts/run_actor.js
Normal file
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Apify Actor Runner - Runs Apify actors and exports results.
|
||||
*
|
||||
* Usage:
|
||||
* # Quick answer (display in chat, no file saved)
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
*
|
||||
* # Export to file
|
||||
* node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}' --output leads.csv --format csv
|
||||
*/
|
||||
|
||||
import { parseArgs } from 'node:util';
|
||||
import { writeFileSync, statSync } from 'node:fs';
|
||||
|
||||
// User-Agent for tracking skill usage in Apify analytics
|
||||
const USER_AGENT = 'apify-agent-skills/apify-ultimate-scraper-1.3.0';
|
||||
|
||||
// Parse command-line arguments
|
||||
function parseCliArgs() {
|
||||
const options = {
|
||||
actor: { type: 'string', short: 'a' },
|
||||
input: { type: 'string', short: 'i' },
|
||||
output: { type: 'string', short: 'o' },
|
||||
format: { type: 'string', short: 'f', default: 'csv' },
|
||||
timeout: { type: 'string', short: 't', default: '600' },
|
||||
'poll-interval': { type: 'string', default: '5' },
|
||||
help: { type: 'boolean', short: 'h' },
|
||||
};
|
||||
|
||||
const { values } = parseArgs({ options, allowPositionals: false });
|
||||
|
||||
if (values.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (!values.actor) {
|
||||
console.error('Error: --actor is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!values.input) {
|
||||
console.error('Error: --input is required');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
return {
|
||||
actor: values.actor,
|
||||
input: values.input,
|
||||
output: values.output,
|
||||
format: values.format || 'csv',
|
||||
timeout: parseInt(values.timeout, 10),
|
||||
pollInterval: parseInt(values['poll-interval'], 10),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`
|
||||
Apify Actor Runner - Run Apify actors and export results
|
||||
|
||||
Usage:
|
||||
node --env-file=.env scripts/run_actor.js --actor ACTOR_ID --input '{}'
|
||||
|
||||
Options:
|
||||
--actor, -a Actor ID (e.g., compass/crawler-google-places) [required]
|
||||
--input, -i Actor input as JSON string [required]
|
||||
--output, -o Output file path (optional - if not provided, displays quick answer)
|
||||
--format, -f Output format: csv, json (default: csv)
|
||||
--timeout, -t Max wait time in seconds (default: 600)
|
||||
--poll-interval Seconds between status checks (default: 5)
|
||||
--help, -h Show this help message
|
||||
|
||||
Output Formats:
|
||||
JSON (all data) --output file.json --format json
|
||||
CSV (all data) --output file.csv --format csv
|
||||
Quick answer (no --output) - displays top 5 in chat
|
||||
|
||||
Examples:
|
||||
# Quick answer - display top 5 in chat
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}'
|
||||
|
||||
# Export all data to CSV
|
||||
node --env-file=.env scripts/run_actor.js \\
|
||||
--actor "compass/crawler-google-places" \\
|
||||
--input '{"searchStringsArray": ["coffee shops"], "locationQuery": "Seattle, USA"}' \\
|
||||
--output leads.csv --format csv
|
||||
`);
|
||||
}
|
||||
|
||||
// Start an actor run and return { runId, datasetId }
|
||||
async function startActor(token, actorId, inputJson) {
|
||||
// Convert "author/actor" format to "author~actor" for API compatibility
|
||||
const apiActorId = actorId.replace('/', '~');
|
||||
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(inputJson);
|
||||
} catch (e) {
|
||||
console.error(`Error: Invalid JSON input: ${e.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': `${USER_AGENT}/start_actor`,
|
||||
},
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
|
||||
if (response.status === 404) {
|
||||
console.error(`Error: Actor '${actorId}' not found`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: API request failed (${response.status}): ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
return {
|
||||
runId: result.data.id,
|
||||
datasetId: result.data.defaultDatasetId,
|
||||
};
|
||||
}
|
||||
|
||||
// Poll run status until complete or timeout
|
||||
async function pollUntilComplete(token, runId, timeout, interval) {
|
||||
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
|
||||
const startTime = Date.now();
|
||||
let lastStatus = null;
|
||||
|
||||
while (true) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to get run status: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const result = await response.json();
|
||||
const status = result.data.status;
|
||||
|
||||
// Only print when status changes
|
||||
if (status !== lastStatus) {
|
||||
console.log(`Status: ${status}`);
|
||||
lastStatus = status;
|
||||
}
|
||||
|
||||
if (['SUCCEEDED', 'FAILED', 'ABORTED', 'TIMED-OUT'].includes(status)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000;
|
||||
if (elapsed > timeout) {
|
||||
console.error(`Warning: Timeout after ${timeout}s, actor still running`);
|
||||
return 'TIMED-OUT';
|
||||
}
|
||||
|
||||
await sleep(interval * 1000);
|
||||
}
|
||||
}
|
||||
|
||||
// Download dataset items
|
||||
async function downloadResults(token, datasetId, outputPath, format) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/download_${format}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
|
||||
if (format === 'json') {
|
||||
writeFileSync(outputPath, JSON.stringify(data, null, 2));
|
||||
} else {
|
||||
// CSV output
|
||||
if (data.length > 0) {
|
||||
const fieldnames = Object.keys(data[0]);
|
||||
const csvLines = [fieldnames.join(',')];
|
||||
|
||||
for (const row of data) {
|
||||
const values = fieldnames.map((key) => {
|
||||
let value = row[key];
|
||||
|
||||
// Truncate long text fields
|
||||
if (typeof value === 'string' && value.length > 200) {
|
||||
value = value.slice(0, 200) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
value = JSON.stringify(value) || '';
|
||||
}
|
||||
|
||||
// CSV escape: wrap in quotes if contains comma, quote, or newline
|
||||
if (value === null || value === undefined) {
|
||||
return '';
|
||||
}
|
||||
const strValue = String(value);
|
||||
if (strValue.includes(',') || strValue.includes('"') || strValue.includes('\n')) {
|
||||
return `"${strValue.replace(/"/g, '""')}"`;
|
||||
}
|
||||
return strValue;
|
||||
});
|
||||
csvLines.push(values.join(','));
|
||||
}
|
||||
|
||||
writeFileSync(outputPath, csvLines.join('\n'));
|
||||
} else {
|
||||
writeFileSync(outputPath, '');
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`Saved to: ${outputPath}`);
|
||||
}
|
||||
|
||||
// Display top 5 results in chat format
|
||||
async function displayQuickAnswer(token, datasetId) {
|
||||
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'User-Agent': `${USER_AGENT}/quick_answer`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
console.error(`Error: Failed to download results: ${text}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
const total = data.length;
|
||||
|
||||
if (total === 0) {
|
||||
console.log('\nNo results found.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Display top 5
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
console.log(`TOP 5 RESULTS (of ${total} total)`);
|
||||
console.log('='.repeat(60));
|
||||
|
||||
for (let i = 0; i < Math.min(5, data.length); i++) {
|
||||
const item = data[i];
|
||||
console.log(`\n--- Result ${i + 1} ---`);
|
||||
|
||||
for (const [key, value] of Object.entries(item)) {
|
||||
let displayValue = value;
|
||||
|
||||
// Truncate long values
|
||||
if (typeof value === 'string' && value.length > 100) {
|
||||
displayValue = value.slice(0, 100) + '...';
|
||||
} else if (Array.isArray(value) || (typeof value === 'object' && value !== null)) {
|
||||
const jsonStr = JSON.stringify(value);
|
||||
displayValue = jsonStr.length > 100 ? jsonStr.slice(0, 100) + '...' : jsonStr;
|
||||
}
|
||||
|
||||
console.log(` ${key}: ${displayValue}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n${'='.repeat(60)}`);
|
||||
if (total > 5) {
|
||||
console.log(`Showing 5 of ${total} results.`);
|
||||
}
|
||||
console.log(`Full data available at: https://console.apify.com/storage/datasets/${datasetId}`);
|
||||
console.log('='.repeat(60));
|
||||
}
|
||||
|
||||
// Report summary of downloaded data
|
||||
function reportSummary(outputPath, format) {
|
||||
const stats = statSync(outputPath);
|
||||
const size = stats.size;
|
||||
|
||||
let count;
|
||||
try {
|
||||
const content = require('fs').readFileSync(outputPath, 'utf-8');
|
||||
if (format === 'json') {
|
||||
const data = JSON.parse(content);
|
||||
count = Array.isArray(data) ? data.length : 1;
|
||||
} else {
|
||||
// CSV - count lines minus header
|
||||
const lines = content.split('\n').filter((line) => line.trim());
|
||||
count = Math.max(0, lines.length - 1);
|
||||
}
|
||||
} catch {
|
||||
count = 'unknown';
|
||||
}
|
||||
|
||||
console.log(`Records: ${count}`);
|
||||
console.log(`Size: ${size.toLocaleString()} bytes`);
|
||||
}
|
||||
|
||||
// Helper: sleep for ms
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// Main function
|
||||
async function main() {
|
||||
// Parse args first so --help works without token
|
||||
const args = parseCliArgs();
|
||||
|
||||
// Check for APIFY_TOKEN
|
||||
const token = process.env.APIFY_TOKEN;
|
||||
if (!token) {
|
||||
console.error('Error: APIFY_TOKEN not found in .env file');
|
||||
console.error('');
|
||||
console.error('Add your token to .env file:');
|
||||
console.error(' APIFY_TOKEN=your_token_here');
|
||||
console.error('');
|
||||
console.error('Get your token: https://console.apify.com/account/integrations');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Start the actor run
|
||||
console.log(`Starting actor: ${args.actor}`);
|
||||
const { runId, datasetId } = await startActor(token, args.actor, args.input);
|
||||
console.log(`Run ID: ${runId}`);
|
||||
console.log(`Dataset ID: ${datasetId}`);
|
||||
|
||||
// Poll for completion
|
||||
const status = await pollUntilComplete(token, runId, args.timeout, args.pollInterval);
|
||||
|
||||
if (status !== 'SUCCEEDED') {
|
||||
console.error(`Error: Actor run ${status}`);
|
||||
console.error(`Details: https://console.apify.com/actors/runs/${runId}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Determine output mode
|
||||
if (args.output) {
|
||||
// File output mode
|
||||
await downloadResults(token, datasetId, args.output, args.format);
|
||||
reportSummary(args.output, args.format);
|
||||
} else {
|
||||
// Quick answer mode - display in chat
|
||||
await displayQuickAnswer(token, datasetId);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(`Error: ${err.message}`);
|
||||
process.exit(1);
|
||||
});
|
||||
Reference in New Issue
Block a user