Source registry & import health

https://www.spr.gov.my

Method

static_html

Reliability

98%

Rate limit

30/h

Registry

Seed only

Last crawl

Never

Last success

Never

Official portal. Crawl gently; verify robots.txt before each run.

Public informational pages only. Robots-checked static fetch; anything interactive or blocked routes to manual import.

https://mysprsemak.spr.gov.my

Method

manual

Reliability

98%

Rate limit

—

Registry

Seed only

Last crawl

Never

Last success

Never

Unavailable — access restricted. Interactive lookup service; no bulk crawling. Manual verification workflow only.

Interactive lookup service (CAPTCHA-protected). Never scraped. Official data enters via the manual import templates in data/import/.

Manual review required

https://open.dosm.gov.my

Method

api

Reliability

97%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Official open-data API for demographic context.

data catalogue API at api.data.gov.my (no key required; gentle rate limits).

https://dashboard.spr.gov.my

Method

manual

Reliability

97%

Rate limit

12/h

Registry

Seed only

Last crawl

Never

Last success

Never

JavaScript-heavy dashboard. Automated extraction is off until its terms and robots policy are confirmed; use manual import in the meantime.

If automated access is confirmed permissible, a Playwright adapter may render public pages; otherwise stays manual-import only.

Manual review required

https://api.data.gov.my

Method

api

Reliability

97%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Official open-data catalogue API (shared infrastructure with OpenDOSM).

Used by the OpenDOSM adapter for population datasets.

https://electiondata.my

Method

api

Reliability

90%

Rate limit

120/h

Registry

Seed only

Last crawl

Never

Last success

Never

Machine-readable cross-check for historical results and candidates.

Bearer token via ELECTIONDATA_API_KEY. Endpoint paths are configured, not guessed — see the adapter for setup.

https://www.bernama.com

Method

rss

Reliability

92%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

No public feed

Feed status

Access pending

A working RSS exists at https://www.bernama.com/en/rssfeed.php (parses, 10 items), BUT robots.txt sets 'User-agent: * / Disallow: /' — respecting robots.txt, DO NOT crawl. Left unconfigured (no feedUrls) so ingestion skips it. Revisit only with an API or explicit written permission. Checked 2026-07-04.

https://www.astroawani.com

Method

rss

Reliability

88%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

No public feed

Feed status

Access pending

No RSS as of 2026-07-04: /rss and /feed return HTML, /?feed=rss2 is not WordPress. Only sitemap.xml exists (not RSS); sitemap ingestion would need new logic — deferred. No feed configured.

https://www.thestar.com.my

Method

rss

Reliability

88%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Pending verification

Feed status

Access pending

No usable RSS as of 2026-07-04: the configured path, all tested /rss/* paths, and the /rss index return HTTP 404. Pending a confirmed feed URL from the publisher; no scraping.

https://www.malaymail.com

Method

rss

Reliability

86%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Verified

Feed status

verified

Feed verified working 2026-07-02 (see data/reports/news-source-health.md); article pages robots-allowed.

https://www.malaysiakini.com

Method

rss

Reliability

87%

Rate limit

30/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

No public feed

Feed status

Access pending

Unavailable — access restricted. Most articles are behind a paywall; only openly published headlines/summaries may be used. No paywall bypass.

Paywalled — disabled. Only openly published headlines/summaries may ever be used; no paywall bypass.

Manual review required

https://www.sinarharian.com.my

Method

rss

Reliability

84%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

No public feed

Feed status

Access pending

No usable RSS found as of 2026-07-04: /feed, /rssFeed, /rss.xml and /?feed=rss2 return 404 or HTML. No feed configured.

https://www.bharian.com.my

Method

rss

Reliability

85%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Verified

Feed status

verified

Feed verified working 2026-07-04 (bot UA, robots.txt 'User-agent: * Allow: /', rss-parser 50 items). Named AI-training crawlers are blocked but * is permitted.

https://www.hmetro.com.my

Method

rss

Reliability

80%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Verified

Feed status

verified

Feed verified working 2026-07-04 (bot UA, robots.txt 'User-agent: * Allow: /', rss-parser 51 items). Named AI-training crawlers are blocked but * is permitted.

https://www.freemalaysiatoday.com

Method

rss

Reliability

84%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Verified

Feed status

verified

WordPress feed verified working 2026-07-02 (see data/reports/news-source-health.md); article pages robots-allowed.

https://www.nst.com.my

Method

rss

Reliability

86%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Verified

Feed status

verified

Feed verified working 2026-07-04 (bot UA, robots.txt 'User-agent: * Allow: /', rss-parser 50 items). Named AI-training crawlers are blocked but * is permitted.

https://www.utusan.com.my

Method

rss

Reliability

80%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Verified

Feed status

verified

WordPress feed verified working 2026-07-02 (see data/reports/news-source-health.md); article pages robots-allowed.

https://malaysiagazette.com

Method

rss

Reliability

78%

Rate limit

30/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Pending verification

Feed status

Access pending

No usable RSS as of 2026-07-04: /feed/ and /?feed=rss2 return the HTML homepage (WP feed disabled or Cloudflare-gated), not RSS. No working feed.

Manual review required

https://berita.rtm.gov.my

Method

rss

Reliability

90%

Rate limit

60/h

Registry

Seed only

Last crawl

Never

Last success

Never

Feed

Verified

Feed status

verified

Public broadcaster news portal. Feed verified working 2026-07-02 (see data/reports/news-source-health.md); article pages robots-allowed.

about:registry/public-social

Method

api

Reliability

55%

Rate limit

30/h

Registry

Seed only

Last crawl

Never

Last success

Never

Only via allowed APIs or legally accessible public embeds. No private groups, no login-only content, no personal-data collection.

Disabled until a legally reviewed public-API integration exists.

Manual review required