Built with Claude code

instagram-scraper-prompt.md

Instagram Creator Scraper for Nigerian Creators

Overview

Scrape Instagram to find Nigerian creators in the "Make Money Online" and "Educational Content" niches. Target: 500 creators with minimum 1,000 followers.

Google Sheet

• Spreadsheet ID: 19iH5Due87qCKMhBlHCsrBk1mE8liS9ycfqrssO2UZSs
• Columns: Name, Email, Platform, Profile Name, Profile Link, Subscriber Count, Country, Niche, Language, Status, Instagram, Telegram, Twitter, Website, TikTok, Facebook

Technical Approach

1. Session-Based Browser Automation

Uses Playwright with a persistent browser session to avoid repeated logins:

this.context = await chromium.launchPersistentContext('./instagram-session', {
  headless: false,
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...',
  viewport: { width: 1280, height: 800 },
  locale: 'en-US',
  timezoneId: 'Africa/Lagos'
});

2. Search Keyword Endpoint (Current Working Approach)

Uses Instagram's search keyword endpoint instead of hashtag pages:

https://www.instagram.com/explore/search/keyword/?q=%23{hashtag}&hl=en

This approach is more reliable than /explore/tags/{hashtag}/ which often returns empty results.

3. Modal-Based Username Extraction

Instead of navigating to each post page:

1. Click post thumbnail to open modal
2. Extract username from modal's article header
3. Press Escape to close modal
4. Visit profile to get follower count

// Click post to open modal
await postElement.click();
await this.page.waitForTimeout(2500);

// Extract username from modal
const username = await this.page.evaluate(() => {
  const usernameLink = document.querySelector('article header a[href^="/"]');
  if (usernameLink) {
    const href = usernameLink.getAttribute('href');
    const match = href.match(/^\/([a-zA-Z0-9._]+)\/?$/);
    if (match) return match[1];
  }
  return null;
});

// Close modal
await this.page.keyboard.press('Escape');

4. Rate Limiting Strategy

• Session limit: 50 profiles per session
• Random delays: 4-8 seconds between actions
• Longer delays: 10-15 seconds between hashtags
• Re-scroll after returning: Load more posts after visiting a profile

Configuration

const CONFIG = {
  maxCreators: 500,
  maxProfilesPerSession: 50,
  minFollowers: 1000,
  sessionDir: './instagram-session',
  outputFile: '/tmp/instagram_creators.json',
  progressFile: '/tmp/instagram_progress.json'
};

Hashtags Used

const HASHTAGS = {
  'make_money_online': [
    'makingmoneyonlinenigeria', 'makemoneyonlinenigeria', 'makemoneyonline',
    'onlinebusinessnigeria', 'passiveincomenig', 'passiveincome',
    'hustlenigeria', 'digitalincomenigeria', 'freelancingnigeria',
    'naijahustle', 'nigeriaentrepreneur', 'makemoneynaija',
    'onlineincomenigeria', 'sidehustlenigeria', 'financialfreedomnigeria',
    'wealthbuilding'
  ],
  'educational_content': [
    'educationnigeria', 'learnonlinenigeria', 'skillacquisitionnigeria',
    'techtutorialnigeria', 'personaldevelopmentnigeria', 'naijaeducation',
    'onlinelearningnigeria', 'codinginnigeria', 'nigeriatutorial',
    'selfimprovementnaija', 'financialliteracy', 'investmenttips', 'moneytips'
  ]
};

Running the Scraper

First-time Setup

1. Run login script to create persistent session:

node instagram_login.js

2. Manually log in to Instagram in the browser window
3. Session is saved to ./instagram-session/

Running a Scraping Session

# Reset session counter (keeps existing creators)
cat /tmp/instagram_progress.json | node -e "
const data = require('fs').readFileSync('/dev/stdin', 'utf8');
const progress = JSON.parse(data);
progress.stats.profilesVisited = 0;
require('fs').writeFileSync('/tmp/instagram_progress.json', JSON.stringify(progress, null, 2));
console.log('Reset session counter. Total creators:', progress.creators.length);
"

# Run scraper
node instagram_scraper.js

After Each Session

Extract new creators and append to Google Sheets:

// Get new creators (adjust slice index based on previous count)
const creators = require('/tmp/instagram_creators.json');
const newCreators = creators.slice(PREVIOUS_COUNT);

// Format for Google Sheets
const rows = newCreators.map(c => [
  c.name, c.email, c.platform, c.profileName, c.profileLink,
  c.subscriberCount, c.country, c.niche, c.language, c.status,
  c.instagram, c.telegram, c.twitter, c.website, c.tiktok, c.facebook
]);

Data Format

Each creator object:

{
  name: 'username or display name',
  email: '',
  platform: 'instagram',
  profileName: '@username',
  profileLink: 'https://www.instagram.com/username/',
  subscriberCount: 50000,  // numeric
  country: 'NG',
  niche: 'instagram make money online nigeria' | 'instagram educational content nigeria',
  language: '',
  status: 'Not Contacted',
  instagram: 'https://www.instagram.com/username/',
  telegram: '',
  twitter: '',
  website: '',
  tiktok: '',
  facebook: ''
}

Progress Tracking

The scraper maintains a progress file (/tmp/instagram_progress.json) with:

• creators: Array of all collected creators
• visited: Set of usernames already visited (prevents duplicates)
• stats: Session statistics
• lastUpdate: Timestamp

Anti-Detection Measures

1. Persistent session - Reuses logged-in state
2. Realistic user agent - Chrome on macOS
3. Random delays - Variable timing between actions
4. Human-like scrolling - Gradual page scrolling
5. WebDriver detection bypass:

await this.page.addInitScript(() => {
  Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
});

Current Progress

• Total Instagram creators: 265
• Target: 500
• Sessions completed: Multiple (each adds ~40-50 creators)
• Google Sheet rows: 1,666 (includes YouTube creators from rows 1-1538)

Notable Creators Found

• @layiwasabi - 2.4M followers
• @bammybestowed - 2.1M followers
• @odumodublvck - 1.9M followers
• @diaryofanaijagirl - 987K followers
• @tanimzamanofficial - 458K followers
• @derrickpwhitehead - 441K followers

Files

• instagram_scraper.js - Main scraper script
• instagram_login.js - Login helper script
• ./instagram-session/ - Browser session data
• /tmp/instagram_creators.json - All collected creators
• /tmp/instagram_progress.json - Progress tracking
• /tmp/instagram_new_rows.json - Formatted rows for current session