The starting point
A social media management agency reached out. They managed 45 brand accounts across beauty, food, lifestyle, and fitness verticals. Each brand had 5 to 15 direct competitors they wanted to track. That added up to 300+ competitor profiles, plus 150 branded hashtags and 80 industry hashtags.
The agency had been tracking competitors manually. One junior strategist per vertical, checking profiles every Monday, copying numbers into a spreadsheet, writing a summary for the account manager. It was taking 15+ hours per week across the team, the data was a week old before anyone saw it, and staff turnover meant the institutional knowledge of how to read the competitive landscape walked out the door every few months.
They needed automated, consistent, structured competitive intelligence that did not depend on any single person's process.
What the pipeline delivers
Per competitor profile, weekly:
- Follower count and growth (net change since last week)
- Posts published in the period
- Average engagement rate (likes + comments as a percentage of followers)
- Top-performing content format (reel vs carousel vs static)
- Average reel view count
- Posting cadence (posts per week)
- Top hashtags by engagement contribution
Per brand account, weekly:
- Competitive ranking within the defined competitor set
- Engagement rate delta vs the competitor set benchmark
- Content format distribution vs competitors
- Optimal posting window (based on engagement patterns)
- Trending hashtags in the competitive set
- New content themes appearing across competitors
Delivered as:
- Weekly PDF report per brand (designed for account managers)
- Shared Google Sheet per brand (raw data for analysts)
- Slack notification when a competitor's engagement spikes above 2x their average
What the agency does with it
The weekly reports changed three things in the agency's workflow:
Content strategy meetings have data. Instead of "I think Reels are doing well for our competitors," the report shows exactly which competitors shifted to Reels, what their engagement delta was, and which reel formats outperformed. Strategy decisions are evidence-based.
Client reviews show competitive context. When a client asks "how are we doing?", the account manager can show a ranking within the competitive set, not just absolute numbers. A 3% engagement rate means nothing in isolation. A 3% engagement rate that puts you second in a competitive set of 12 tells a story.
Format experimentation is faster. When the data shows that carousel posts outperform static posts for 8 out of 12 competitors in the beauty vertical, the agency can test that format for their client within the same week. The feedback loop from "competitor signal" to "client experiment" shortened from months to days.
What broke
The GraphQL rotation problem
Instagram rotates its internal API identifiers (doc_ids) every 2 to 4 weeks. When a rotation happens, the extraction endpoints that were working yesterday return errors. The pipeline monitors for this and I update the identifiers within 24 hours.
Over the course of a year, this happened roughly 15 times. The agency never noticed data gaps because the backfill runs before the next weekly report is generated.
The engagement rate baseline problem
In the first month, several brand account managers flagged "inaccurate" engagement rates for competitors. The issue was not accuracy. The issue was that the pipeline calculated engagement rate as (likes + comments) / followers, while some account managers were using (likes + comments) / reach. Since reach is not publicly visible for competitor profiles, the pipeline uses follower count as the denominator.
The fix was documenting the calculation method in the report header and adding a footnote explaining why reach-based engagement rates are not possible for competitor profiles. The confusion disappeared once the methodology was transparent.
The private profile problem
About 5% of the competitor profiles on the initial rosters were set to private. Private profiles cannot be tracked from public data. The pipeline flags these immediately and suggests alternatives (secondary accounts, brand regional pages, or replacement competitors).
In practice, brand accounts on Instagram are almost never private because visibility is their core function. The 5% were mostly personal accounts of founders or creative directors that account managers had added to rosters.
The 22% engagement lift
After 6 months of weekly competitive intelligence, the agency measured the impact across their 45 managed accounts. The average engagement rate increased 22% compared to the 6 months prior to the pipeline launch.
The agency attributed the lift to three factors:
- Format shifts. 28 of 45 accounts shifted their content format mix based on competitive data (most commonly increasing Reels percentage). Accounts that shifted saw a 31% engagement lift.
- Posting time optimization. 15 accounts adjusted their posting schedule based on the "optimal posting window" data. These saw a 14% lift.
- Hashtag strategy. The trending hashtag data surfaced category-specific hashtags that the accounts were not using. Adding these to the content strategy correlated with a 9% reach increase.
The 22% is an average across all 45 accounts. Some saw 40%+. Some saw 5%. The accounts that acted most aggressively on the data saw the largest gains.
What it costs to run this
The pipeline costs the agency $499/mo on the ongoing tracking tier (up to 100 profiles). At the agency intelligence tier (300+ profiles, multi-brand rollups), the engagement scopes based on the brand portfolio size.
For the agency, the math is simple. The competitive intelligence pipeline costs less than one junior strategist's monthly salary and delivers data that is more consistent, more structured, and does not take PTO.
When this approach fits
Instagram competitive intelligence as a managed pipeline works best for:
- Agencies managing 10+ brand accounts that each have a competitive set worth tracking
- In-house teams at brands with 20+ direct competitors in their category
- Any team that currently does competitive tracking manually and spends more than 5 hours per week on it
The Instagram data service page has the full output schema, pricing, and delivery details. For self-serve access, the ScrapeBase API at scrapebase.io has Instagram endpoints for profiles, posts, reels, and comments.