Automated processes that access FCC online resources

Updated 05/05/2024 6:40 AM EDT

This page is information for FCC Office of the Managing Director (OMD) staff and FCC IT contractors to advise of the operations of REC Networks that automatically access FCC public systems on a routine basis for information collection and delivery to the general public, pursuant to the OPEN Data Act and other federal laws.

We have done a complete refresh of this page to reflect the changes we have made to FCC resource access policies and to be more transparent on our automated activities to the FCC online resources.

3-14-24: Statement regarding ByteDance

For the best interests of both REC and FCC system integrity and to promote national security, over the past weekend, for all REC services that use our IP blocking table (including FCCdata.org), we have installed a hard block on all known Singapore based IP ranges for ByteDance due to constant incoming activity from their ByteSpider application.  IP addresses that attempt to access REC services that utilize our IP blocking table from such blocked IP addresses will encounter a HTTP 401 response code and denial of access to REC services.

 

Upcoming Special projects

Resumption of Operations mass run (5/4/2024 or 5/5/2024)

4-30-2024 4PM EDT - This is a first advsory that REC is planning a mass run of applications through eLMS.  This mass run will focus on resumption of operations for AM and the various FM classes of service.  There are over 8,000 2,600 applications that need to be processed in this mass run.  As with previous mass runs, we plan to run it overnight and with rate limits to control the number of requests made.  This run is necessary to bring these applications into the eLMS environment in a proper manner.  This will be the first time we have attempted to do a full eLMS run on these applications.  Going forward, these applications will go through the full eLMS process when they are first discovered by LMSlist.  Due to the volume, we may split this mass run over multiple days.  We will update this page when we have the dates we plan to execute this mass run.

5-3-2024 5:30PM EDT - We will perform this mass run during a period on Saturday May 4 or Sunday May 5, 2024.  We have reevaluated the requirements and are revising the volume of this mass run to 1,932 AM and and 722 FM applications.  This mass run will only make one call to FCC systems (LMS application image) per application.  We will rate limit at 6 applications per minute (one every 10 seconds).  The overall mass run will operate for approximately 7 hours and 20 minutes.  This mass run will use the standard eLMS user agent.  Regular eLMS services will continue to operate every 60 minutes during the mass run. 

5-4-2024 9:00AM EDT - Barring any issues that we may encounter with an unrelated network update at Riverton, we plan to commence the mass run today (Saturday 5-4).  It may start as soon as 10:30AM EDT but cound happen at any time during this weekend.  

5-4-2024 11:20AM EDT - The mass run has commenced with 723 FM and an estimated 1,932 AM applications to process.  There will be 10 seconds of sleep before each query, therefore there will be 6 calls per minute.  Thank you for your cooperation.  

5-5-2024 6:40AM EDT - The mass run has completed.  We are evaluating the results to determine if we need to re-run a subset of the original run. 

System40 Transition

The System40 Project has been cancelled. 

About these tools

REC Networks is a major provider of FCC database information to the general public.  In addition to scheduled tasks that retrieve raw data once a day from LMS, CDBS, IBFS and ULS, we also operate a "near real-time" interface to the public LMS system.  This system retrieves broadcast applications in HTML form, parses those applications and then saves them in our database and are then displayed on various websites including, but not limited to:

More details about many of these websites below. 

REC time standard (RECOT) used on our operations schedules

With our former roots in Arizona, REC operates systems based on REC Operations Time (or "RECOT" for short).  RECOT is based on a time standard of UTC -7, which matches the time in Phoenix.  This means that:

  • During Standard time, Eastern time is 2 hours ahead of RECOT.  (6:00AM RECOT = 8:00AM EST).
  • During Daylight time, Eastern time is 3 hours ahead of RECOT. (5:00AM RECOT = 8:00AM EDT).

Therefore, from a Washington DC perspective, our schedules shift twice per year.  Please keep that in mind when viewing these schedules.

RECOT 00 01 02 03 04 05 06 07 08 09 10 11
Eastern Daylight Time 3 AM 4 AM 5 AM 6 AM 7 AM 8 AM 9 AM 10 AM 11 AM 12 PM 1 PM 2 PM
Eastern Standard Time 2 AM 3 AM 4 AM 5 AM 6 AM 7 AM 8 AM 9 AM 10 AM 11 AM 12 PM 1 PM
Coordinated Universal Time (UTC) 07 08 09 10 11 12 13 14 15 16 17 18
Japan Standard Time (JST) 4 PM 5 PM 6 PM 7 PM 8 PM 9 PM 10 PM 11 PM 12 AM+ 1 AM+ 2 AM+ 3 AM+

 

RECOT 12 13 14 15 16 17 18 19 20 21 22 23
Eastern Daylight Time 3 PM 4 PM 5 PM 6 PM 7 PM 8 PM 9 PM 10 PM 11 PM 12 AM+ 1 AM+ 2 AM+
Eastern Standard Time 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 8 PM 9 PM 10 PM 11 PM 12 AM+ 1 AM+
Coordinated Universal Time (UTC) 19 20 21 22 23 00+ 01+ 02+ 03+ 04+ 05+ 06+
Japan Standard Time (JST) 4 AM+ 5 AM+ 6 AM+ 7 AM+ 8 AM+ 9 AM+ 10 AM+ 11 AM+ 12 PM+ 1 PM+ 2 PM+ 3 PM+

+ - next day.

User Agents used in REC FCC interfaces

Requests originating from REC Networks will come in on specific IP addresses (which are available to FCC staff on request) and in most cases, will include one of the following User-Agent strings in the request header (note: version numbers may vary based on development):

LMSlist Mozilla/5.0 (compatible; LMSlist/4.1; +https://recnet.com/eLMS-info) (Formerly known as LMSbot) LMSlist interfaces with the LMS list of applications and then writes the information to the application lists and to the table that provides information for display on FCC.today.
eLMS Mozilla/5.0 (compatible; eLMS/3.1; +https://recnet.com/eLMS-info) For applications determined by LMSlist to be qualified for real-time processing, eLMS will retrieve the HTML of the application from LMS and then parse the information in the REC databases and for display in FCCdata.org.  (Different version numbers of eLMS may appear)
FCC API Mozilla/5.0 (compatible; FCCcontours/1.0; +https://recnet.com/eLMS-info) Requests from this user agent are from various REC websites and tools, both external and internal to REC.  We use the FCC Contours API to provide free government resources to our users, such as contour mapping data and transmitter site engineering information.  While much of our mapping is cached and does not always go to the API, our more high volume public systems have been rate limited to reduce or eliminate traffic from bots and others who excessively use data. 
EDOCS Mozilla/5.0 (compatible; EDOCS/1.1; +https://recnet.com/eLMS-info) Requests from REC to the FCC Content API for the retrieval of data related to electronic documents (EDOCS).  This includes requests associated with news updates and broadcast applications public notice handling.
ECFS Mozilla/5.0 (compatible; ECFS/1.1; +https://recnet.com/eLMS-info) Requests from the REC website to view recent filings in the Electronic Comments Filing System.
OPIF Mozilla/5.0 (compatible; OPIF/1.1; +https://recnet.com/eLMS-info) Used to access the Online Public Inspection File API. Used primarily to verify the presence of a public file and the presence of a station logo.
PN Mozilla/5.0 (compatible; eLMSPN/1.0; +https://recnet.com/eLMS-info) Daily requests from REC to retrieve information related to the daily (Broadcast) 'Applications' public notice.

When REC is retrieving a PDF for an authorization document, a more default user agent that does not appear above may be used.

These are not search engine bots, nor are they spiders (beyond the extent of LMSlist commanding eLMS to pick up an application detected).  These tools only interface with the FCC and no other entity. 

Routine scheduled operations throughout the day

eLMS

LMSlist will first access the public LMS list of applications.  In the first transaction will access https://enterpriseefiling.fcc.gov/dataentry/public/tv/publicAppSearch.html where it will pick up a session cookie and other coding from the source, then it will send a POST transaction to  https://enterpriseefiling.fcc.gov/dataentry/public/tv/publicAppSearchRes... LMSlist will then send another POST tranaction to increase the display size to 100 records.  LMSlist will then page through all of the applications until it has gathered all of the application listings from the current business day and the previous business day.  This generates a JSON file which is made available to LMSlist, which resides on a different server and has no direct contact with FCC resources.   Therefore, on each scheduled run, LMSlist will send three calls to LMS to obtain information on the first 100 applications within the evaluation date range and then one additional call for each additional 100 applications within the evaluation date range (i.e. 300 applications = 5 calls).

LMSlist will then determine if we already have the data for the application on file, if we do not and the application is eligible for eLMS handling, then it will be queued for eLMS to gather the application details.  eLMS will then send a GET transaction to LMS to obtain the HTML of a broadcast application.  eLMS will then internally parse the HTML and create an application record in the REC database.  On granted applications, eLMS will make a second call to download a PDF of the authorization document so that document can be parsed and added to the database.  On applications involving engineering, eLMS will also send a GET transaction to the FCC Contours API in order to gather information on the application's service contour coordinates.   Therefore:

  • Non-engineering applications will get one call (LMS).
  • Engineering pending applications will get two calls (LMS, Contour API).
  • Engineering granted applications will get three calls (LMS, Contour API & LMS Authorization).

eLMS normally retrieves information on an application what it is filed, and when the status is changed (including to superceded in the event of an amendment).

This routine operates on the following schedule:

  LMSlist run (RECOT) eLMS processing (RECOT) Frequency
Monday-Sunday 04:00 Immediately after processing LMS data dump. One time per day
Monday-Friday 05:00~17:00 05:05~17:05 Every 15 minutes
Monday-Friday 18:00~21:00 18:05~21:05 Every 60 minutes
Saturday-Sunday 05:00~21:00 05:05~21:05 Every 60 minutes

During eLMS processing of applications, the eLMS system will go into a locked state until the processing is complete.  If the eLMS parser is called at a scheduled (or unscheduled) time and eLMS is still working from a previously scheduled time, eLMS will reject the subsequent request and that scheduled retrieval will be cancelled.  eLMS will try again at the next scheduled time.  This prevents the FCC LMS system from being hit with multiple simultaneous transactions initiated by different eLMS processing sessions.  

Monday-Friday schedules will also operate on federal holidays.

The number of transactions may vary based on the application activity.  If LMSlist does not detect any new or updated applications, no additional eLMS calls to the FCC LMS site will be made. 

During the routine operation, eLMS enforces rate limiting towards the FCC LMS system through a mandatory sleep time of 5 seconds following the completion of the eLMS routines related to a broadcast application.  This would mean that the average load from eLMS for application processing to the FCC LMS system is about 8 calls per minute.  

Broadcast Applications Public Notice

On Monday through Friday (including federal holidays), REC will send a tranaction to the FCC Content API that will check for the presence of the day's Applications Public Notice.  If the Applications Public Notice has been posted, the routine will make a second-call to the FCC Content API to get additional information on how to locate the specific document and then a third call that downloads the plain text version of the Notice.  REC systems will than parse that information in order to update eLMS to indicate that a broadcast application has been accepted for filing. 

This routine will start at 05:05 RECOT and will run each hour until either the Notice is released and the document has been downloaded and will run no later than 17:05 RECOT.

EDOCS

REC calls the FCC's Content API to obtain electronic document data (such as news releases, orders and notices).  EDOCS does not download the actual PDF notices, only the meta-data from the API.  This is done to build links to these items as they appear on FCC.today and other services.  REC EDOCS service will run on the following schedule:

  RECOT Frequency
Daily Prior to 05:10/05:30 No service
Monday-Friday 05:10~17:50 Every 20 minutes
Saturday-Sunday 05:30~17:30 Every 60 minutes
Daily After 17:50/17:30 No service

Filing window sanity checker

On 2-13-24, a new Sanity Checker process has been activated for the 2023 LPFM Filing Window. The Sanity Checker will run once per day during the morning routine to download LMS dump data.  This application trues up our overall application data with window related data.  If there is a discrepancy, we will do a call using the eLMS routine in order to refresh those applications.   As of 2-13-24, this is currently at three applications.  The sanity checker is designed to where there is more than 10 applications in this "lost" status, the automatic sanity checker will not run the eLMS routine but instead will provide notification to REC of an issue.  We may elect to manually run these through eLMS again or identify another issue that may be causing these discrepancies. 

Public Notice retrieval

This routine runs Monday through Friday, once per hour starting at 05:05 RECOT and no later than 17:05 RECOT. This routine will first use the EDOCS API to obtain the daily (broadcast) 'Applications' public notice and then once it is found, it will retrieve the plain text version of that document in order to parse it to update eLMS on applications that are accepted for filing.  Once the program is successful in picking up the public notice, it will stop running for the day. 

REC scheduled daily/weekly tasks to FCC resources

Days RECOT Description of task
Saturday 01:03 ASR and maritime calls data dump - Once per week, obtain three files from ULS (r_tower.zip, l_ship.zip, l_coast_zip).
Sunday 01:03 ULS weekly data dump - Once per week, obtain three license file dumps from ULS.  (l_LMpriv.zip, l_LMbcast.zip, l_micro.zip).
Monday~Sunday 01:12 Census Update - This service creates an internal JSON file that includes information obtained through our internal Census database to provide a profile of the station's coverage.  If the routine encounters an FM application that does not have full eLMS information, it will make calls to LMS and the Contours API (same as the eLMS related calls made during the day) to pick up the data to rebuild the record in eLMS.  These calls to FCC resources should be more of a rare exception and not common.  There is a built in 3 second pause per application to provide some rate limiting.
Monday 01:17 IBFS weekly data dump - Once per week, obtain one file (Bureaus/International/databases/IBFS.zip) from the FCC's FTP site.
Monday~Sunday 03:45 LMS Raw Data dump - Once per day, obtain the daily LMS dump file (Current_LMS_Dump.zip).  This processing of the data from this dump starts at 04:05 RECOT.  Immediately after this processing, an eLMS routine will be performed to obtain any applications filed between 17:00 the previous evening until 04:00 this morning.
Monday~Sunday 08:55 ULS Amateur Radio - Once per day, obtain the daily file dumps from ULS for Amateur Radio activity (2 files downloaded).  a_am_XXX.zip & l_am_XXX.zip (XXX = day of week).
Monday~Friday 18:35 (temporarily disabled) LMS dismissal letter check - Resend eLMS transactions to each broadcast application that was dismissed during the business day to check to see if any additional application attachments (marked as internal) were added by staff in connection with a Media Bureau action. 

Rate limiting and bot prevention

REC has implemented a global rate limit on publicly accessible resources that call out to FCC Contour API resources as past abuse by third party bots resulted in excessive calls made to FCC resources from the REC network of systems.  We have taken the following proactive steps to reduce calls to the FCC systems:

  • We are limiting access to certain publicly accessible resources to no more than 6 hits per minute based on the first 7 characters in the string of an IP address.  This pretty much puts the block at the Class B IP address level.  
  • On FCCdata, when a user agent has the string "bot/" or "Bytespider" in it, we are blocking access to the link to retrieve the REC HAAT tool.  This is based on the trends in activity that were discovered in the enhanced access logging functionality that we recently put into place. 
  • On FCCdata, more than 10 accesses within a one minute period will return an HTTP error 403.

Attempts to access resources where rate limiting is applied or a 403 error is returned will count against the system limits as if they were otherwise permitted accesses.  This means that any user or bot must maintain silence for at least one minute before they can re-access our resources.   Accesses to FCCdata does not automatically result in a call to FCC resources, only when specific mapping applications are also called or if the HAAT link is also clicked (which is why we disabled access to that link for bots).

Internal requests made manually by a human at REC Networks are not rate limited.   

Rate limits and operating schedules on the eLMS functionality that is automatically started by REC Networks are detailed in the information above.  

Development and Repairs

During all hours (including business hours), REC may send one-off requests through eLMS to obtain applications for the purpose of developing new eLMS functionality or to send a "manual reset" of an application in eLMS that may have been mis-parsed or had a bad contour map due to the failure of the FCC Contour API.  

Special Projects

Special projects involve the need to have eLMS obtain a large number of pre-identified applications.  This is done in order to address specific issues or to migrate new application types into eLMS handling.  

Normally, special projects will normally only be conducted between Friday at 6PM EDT and Monday at 7AM EDT as to use the systems during periods of predicted low demand.   In case of a major operational issue that cannot wait until the weekend, REC will limit special projects to between 6PM and 7AM on any day of the week or any time on the weekends.

Sometimes, special projects are intentionally timed for federal holiday weekends if it is necessary to obtain the applications we need.

During special projects, requests to the FCC's LMS system are normally rate limited to one request every 10 seconds (6 per minute) (on granted applications, one "request" may involve two calls to LMS to get the application and authorization document), but in no case shall special projects generate requests less than every 5 seconds apart (12 per minute).  We are cognizant of the needs of the LMS server. 

REC Networks public websites

REC Networks operates various websites that use data gathered from FCC resources.  Most of these services are free of charge for use.  The following information describes the website and how FCC resources are used.

FCCdata

https://fccdata.org

FCCdata is the most comprehensive website for inquiries into broadcast facilities, amateur radio and other services.  FCCdata also includes broadcast and in some cases, amateur radio information from Canada, Mexico, Australia, Japan, United Kingdom and Ireland.  FCCdata is unique from other broadcast query websites (including those at the FCC) because it can display full FM facility data, usually within 20 minutes of the filing of the application.  The same timeline exists for non-engineering applications from all LMS supported broadcast services for the display of application information.

FCC broadcast data is derived from the data gathered in our eLMS routines as well as from daily data dumps from LMS, ULS and IBFS.  Site usage may result in calls to the OPIF API and to the FCC Contours API.  Users are rate limited where if they make more than 6 accesses per minute, they will be restricted from automatically receiving FCC resources.  More than 10 accesses per minute will block them from the site. 

As of 2-14-2024, FCCdata averages about 4.7K page views per week.

fcc.today

https://fcc.today

FCC.today operates as a “dashboard” for application and action activity in the Audio and Video Divisions of the Media Bureau.  FCC.today uses data collected through the REC LMSlist routine to provide “near real time” updates on applications and actions that are processed through LMS.  FCC.today also displays information on recent FCC releases that involve the Media Bureau, FCC daily business items, emergency alerts received through IPAWS OPEN, information on broadcast pleadings as well as DC federal employee reporting status.  It is also the home of “fcc.today – the podcast”.

Public website usage on FCC.today does not create any direct calls to FCC resources.  All data shown on FCC.today comes from the various scheduled background processes that are described above.  This includes scheduled data gathering routines using our LMSlist and EDOCS processes and the LMS daily dump file as well as data received through the API at the Office of Personnel Management and through our memorandum of understanding with the Federal Emergency Management Agency.

As of 2-14-2024, FCC.today averages about 3.8K page views per week.

REC Main Website

https://recnet.com

REC’s main website is a comprehensive Drupal-based platform which is used for both public websites as well as walled garden applications for internal use only.  Data used can range across our entire universe of stored databases across multiple federal agencies.  Some tools replicate FCC tools that utilize JavaScript such as Curves and FM-Model. 

Some applications on the REC Main Website may generate user requested calls to FCC resources.  Most of these applications have been rate limited, mainly to 6 requests per minute. The most frequently used of such tools is:

HAAT/Elevation/ERP Tool

https://recnet.com/haat

This tool makes calls to the FCC Contours API to provide an improved display of height above average terrain, site elevations and calculate effective radiated power.  Also obtains data from the US Geological Survey and private sector data providers.

As of 2-14-2024, recnet.com gets an average of about 2.8K page views per week.  We have no specific measurement for the HAAT Tool.

myLPFM

https://myLPFM.com

myLPFM is a password protected system that is limited in access to broadcast stations in the Low Power FM Radio Service.  myLPFM allows LPFM stations to perform station management functions, manage their visibility on station directories provided by both REC and third party providers, provide services that will assist in filing modification applications, obtain emergency alert information, engineering information for HD Radio upgrades and assistance with interference from subsequently authorized FM translator stations as well as information on potential encroachment from any subsequently applied for facility.

Various features within myLPFM will make user-requested calls to the FCC Contours API to get various information necessary for the different sub-applications within myLPFM.  Some data received through the Contours API is cached for future access thus speeding up the user experience and eliminating unnecessary traffic to the FCC API.

myLPFM experiences a very low usage level.

Contact Information

In the event of issues with REC Networks' automated access to the FCC systems, please first contact Michelle Bradley, CBT via email at lpfm@recnet.com or at 202 621-2355.  

More information about eLMS available at https://recnet.com/eLMS

As stated.  REC's need to access these public FCC resources are for business, research and public dissemination purposes, consistent with 44 USC §3501 et seq. and therefore, is not malicious or with other ill-intent. 

REC does not have any automated resources that accesses FCC resources behind a password.  We have humans to do that at the request of our clients.

Thank you for your understanding.