Automated processes that access FCC online resources

Updated 07/04/2024 6:05 PM EDT

This page is information for FCC Office of the Managing Director (OMD) staff and FCC IT contractors to advise of the operations of REC Networks that automatically access FCC public systems on a routine basis for information collection and delivery to the general public, pursuant to the OPEN Data Act and other federal laws.

We have done a complete refresh of this page to reflect the changes we have made to FCC resource access policies and to be more transparent on our automated activities to the FCC online resources.

3-14-24: Statement regarding ByteDance

For the best interests of both REC and FCC system integrity and to promote national security, over the past weekend, for all REC services that use our IP blocking table (including FCCdata.org), we have installed a hard block on all known Singapore based IP ranges for ByteDance due to constant incoming activity from their ByteSpider application.  IP addresses that attempt to access REC services that utilize our IP blocking table from such blocked IP addresses will encounter a HTTP 401 response code and denial of access to REC services.

SPECIAL PROJECT COMPLETED

HAAT/ERP Mass Run Project (06/29/2024~07/04/2024) - SEE UPDATES

6-28-24 9:15AM ET - eLMS has added some new functionality to store ERP and HAAT on FM facilities at all 360 radials.  This data cacheing will reduce our need to have to call the API in the future for special projects and will eventually be incorporated into some of our products where it will result in a reduction in API calls to the FCC.  For each facility record, we will only make one call to the "coverage" endpoint on the FCC Contours API.  Our systems will do the rest.  This mass run will not call LMS. For voluntary rate limiting, we will set up for a sleep of 6 seconds between calls thus meaning that the API will receive no more than 10 calls per minute.  These project calls can be identified with the user agent "FCCHaatErp/1.0".  "Overnight" runs will normally start at or after 6PM ET except on weekends and holidays when they can start earlier.

6-28-24 2:15 PM ET - We are going to start the overnight process tonight.  

6-28-24 8:15 AM ET - Due to the issue with the coverage endpoint defaulting to 30 meters for all HAATs below 30, we need to make a second call to the HAAT API in order to get a more accurate HAAT along each radial.  We will need to rerun the LPFM task to accommodate those corrected HAAT and then go back into Class A.  The call to the HAAT endpoint will show in the server logs with user agent "FCCcontours/1.0".  

6-29-24 04:10 PM ET - We are updating the estimates based on 9 calls per minute instead of 10.  This is based on the time that it takes for processing the received data on our end.  We are not changing our sleep time, but providing more of a realistic or high-ball estimate of how long it is taking to get through a phase of the mass run.  This will be reflected when we start the Class A mass run on Saturday evening.

6-30-24 1:45 PM ET - We have made a change to the update schedule.  FM translators in Subclasses 6 and 7 will be moved to July 4.  This is to prevent a 12 hour overnight run that will need to start after 6PM.  (Tonight's 12 hour run should start before 6PM.)  Only Subclass 8 translators (2,974 records) will run overnight on Monday July 1.  

6-30-24 4:24 PM ET - We have started the largest of the mass run segments.  After this run, we will holding mass runs on Monday, Tuesday and Wednesday evenings after 6PM, outside of business hours.  These mass runs should take no more than 8 hours each and be finished by about 4AM the next morning.  We will have two mass runs scheduled for the July 4 federal holiday and one small cleanup round.  Once those rounds are done, that will finish the mass runs for this project.  Of course, if there's any questions or concerns from staff about these runs, please let me know.  Thanks again. =m

7-1-24 11:40 AM ET - We have had to add an additional overnight to the mass run.  This will take place on the night of Friday, July 5.  See the schedule below for details.

7-2-24 8:35 AM ET - We have shifted the schedule to make better utilization of the time outside of business hours.  Depending on the overall total of the cleanup run, we will determine how we will schedule that once we have completed all of the passes.  We note, that the "second pass" will only apply to LPFM, Class A FM, and D1~D5 translators.  All other service classes will be receiving the additional information needed that LPFM, A and lower translators have to conduct a second pass for.  If this amount stays very low, we may do it during July 5 business hours with a very slow rate limit.  Again, we will make that determination once all of the passes are done and we know how many strays we have to work with.

7-4-24 2:45 PM ET - We are about 3.4 hours away from finishing up the last major piece of this mass run project.  Once this last mass run finishes, we need to make another attempt for some "stray" application records that may have not been picked up the first time because the FCC API returned an error or somehing like that.  Before starting the last mass run piece, we had 162 identified applications.  We may get a few more after this last run.  We estimate that it will run for about an hour.  Once that is done, we do not forsee any additional mass run needs at this time.  We'll be back later. 

7-4-24 6:02 PM ET - This mass run project has been completed.  Next up, we will be doing a lot of  internal processing.  Our internal programs do have a mode that if for some reason we are unable to find the data, the system will go out and get the application information.  This is going to be extremely rare and sporadic.  But overall, the big run is now completed.  Thank you to OMD for your patience and if you have any comments about this mass run, please let me know.  Our next steps will be to develop code that will use this data as well as the new data that is now collected on routine eLMS ingestions which are intended to reduce the number of calls to the FCC Contours API overall.  This is going to take some time to develop, but this was the start of that project.  

Description Approximate number of facility records involved Estimated run time (@ average 7 calls per endpoint, per min) Project status
LPFM facilities 3,758   Completed.
Class A FM facilities 6,380   Completed
FM Translators in REC sub classes 1 and 2 (LPFM lower tier, up to 7.3km service contour) 3,403   Completed
FM Translators in REC sub classes 3, 4 and 5 (LPFM middle tier, 7.3~13.3 km service contour) 5,048   Completed
FM Translators in REC sub class 8 2,957   Completed
FM Translators in REC sub classes 6 & 7, FM Boosters, Class B, B1 and D FM stations 4,604   Completed
Class C3, C2, C1, C0 and C FM stations 6,526   Completed
LPFM, Class A and FM Translators in REC sub classes 1 through 5 with directional antennas (second pass) 4,066   Completed
"Cleanup" run to pick up facility records that could not be processed the first time. 148   Completed

 

About these tools

REC Networks is a major provider of FCC database information to the general public.  In addition to scheduled tasks that retrieve raw data once a day from LMS, CDBS, IBFS and ULS, we also operate a "near real-time" interface to the public LMS system.  This system retrieves broadcast applications in HTML form, parses those applications and then saves them in our database and are then displayed on various websites including, but not limited to:

More details about many of these websites below. 

REC time standard (RECOT) used on our operations schedules

With our former roots in Arizona, REC operates systems based on REC Operations Time (or "RECOT" for short).  RECOT is based on a time standard of UTC -7, which matches the time in Phoenix.  This means that:

  • During Standard time, Eastern time is 2 hours ahead of RECOT.  (6:00AM RECOT = 8:00AM EST).
  • During Daylight time, Eastern time is 3 hours ahead of RECOT. (5:00AM RECOT = 8:00AM EDT).

Therefore, from a Washington DC perspective, our schedules shift twice per year.  Please keep that in mind when viewing these schedules.

RECOT 00 01 02 03 04 05 06 07 08 09 10 11
Eastern Daylight Time 3 AM 4 AM 5 AM 6 AM 7 AM 8 AM 9 AM 10 AM 11 AM 12 PM 1 PM 2 PM
Eastern Standard Time 2 AM 3 AM 4 AM 5 AM 6 AM 7 AM 8 AM 9 AM 10 AM 11 AM 12 PM 1 PM
Coordinated Universal Time (UTC) 07 08 09 10 11 12 13 14 15 16 17 18
Japan Standard Time (JST) 4 PM 5 PM 6 PM 7 PM 8 PM 9 PM 10 PM 11 PM 12 AM+ 1 AM+ 2 AM+ 3 AM+

 

RECOT 12 13 14 15 16 17 18 19 20 21 22 23
Eastern Daylight Time 3 PM 4 PM 5 PM 6 PM 7 PM 8 PM 9 PM 10 PM 11 PM 12 AM+ 1 AM+ 2 AM+
Eastern Standard Time 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 8 PM 9 PM 10 PM 11 PM 12 AM+ 1 AM+
Coordinated Universal Time (UTC) 19 20 21 22 23 00+ 01+ 02+ 03+ 04+ 05+ 06+
Japan Standard Time (JST) 4 AM+ 5 AM+ 6 AM+ 7 AM+ 8 AM+ 9 AM+ 10 AM+ 11 AM+ 12 PM+ 1 PM+ 2 PM+ 3 PM+

+ - next day.

User Agents used in REC FCC interfaces

Requests originating from REC Networks will come in on specific IP addresses (which are available to FCC staff on request) and in most cases, will include one of the following User-Agent strings in the request header (note: version numbers may vary based on development):

LMSlist Mozilla/5.0 (compatible; LMSlist/4.1; +https://recnet.com/eLMS-info) (Formerly known as LMSbot) LMSlist interfaces with the LMS list of applications and then writes the information to the application lists and to the table that provides information for display on FCC.today.
eLMS Mozilla/5.0 (compatible; eLMS/3.1; +https://recnet.com/eLMS-info) For applications determined by LMSlist to be qualified for real-time processing, eLMS will retrieve the HTML of the application from LMS and then parse the information in the REC databases and for display in FCCdata.org.  (Different version numbers of eLMS may appear)
FCC API Mozilla/5.0 (compatible; FCCcontours/1.0; +https://recnet.com/eLMS-info) Requests from this user agent are from various REC websites and tools, both external and internal to REC.  We use the FCC Contours API to provide free government resources to our users, such as contour mapping data and transmitter site engineering information.  While much of our mapping is cached and does not always go to the API, our more high volume public systems have been rate limited to reduce or eliminate traffic from bots and others who excessively use data. 
EDOCS Mozilla/5.0 (compatible; EDOCS/1.1; +https://recnet.com/eLMS-info) Requests from REC to the FCC Content API for the retrieval of data related to electronic documents (EDOCS).  This includes requests associated with news updates and broadcast applications public notice handling.
ECFS Mozilla/5.0 (compatible; ECFS/1.1; +https://recnet.com/eLMS-info) Requests from the REC website to view recent filings in the Electronic Comments Filing System.
OPIF Mozilla/5.0 (compatible; OPIF/1.1; +https://recnet.com/eLMS-info) Used to access the Online Public Inspection File API. Used primarily to verify the presence of a public file and the presence of a station logo.
PN Mozilla/5.0 (compatible; eLMSPN/1.0; +https://recnet.com/eLMS-info) Daily requests from REC to retrieve information related to the daily (Broadcast) 'Applications' public notice.

When REC is retrieving a PDF for an authorization document, a more default user agent that does not appear above may be used.

These are not search engine bots, nor are they spiders (beyond the extent of LMSlist commanding eLMS to pick up an application detected).  These tools only interface with the FCC and no other entity. 

Routine scheduled operations throughout the day

eLMS

LMSlist will first access the public LMS list of applications.  In the first transaction will access https://enterpriseefiling.fcc.gov/dataentry/public/tv/publicAppSearch.html where it will pick up a session cookie and other coding from the source, then it will send a POST transaction to  https://enterpriseefiling.fcc.gov/dataentry/public/tv/publicAppSearchRes... LMSlist will then send another POST tranaction to increase the display size to 100 records.  LMSlist will then page through all of the applications until it has gathered all of the application listings from the current business day and the previous business day.  This generates a JSON file which is made available to LMSlist, which resides on a different server and has no direct contact with FCC resources.   Therefore, on each scheduled run, LMSlist will send three calls to LMS to obtain information on the first 100 applications within the evaluation date range and then one additional call for each additional 100 applications within the evaluation date range (i.e. 300 applications = 5 calls).

LMSlist will then determine if we already have the data for the application on file, if we do not and the application is eligible for eLMS handling, then it will be queued for eLMS to gather the application details.  eLMS will then send a GET transaction to LMS to obtain the HTML of a broadcast application.  eLMS will then internally parse the HTML and create an application record in the REC database.  On granted applications, eLMS will make a second call to download a PDF of the authorization document so that document can be parsed and added to the database.  On applications involving engineering, eLMS will also send a GET transaction to the FCC Contours API in order to gather information on the application's service contour coordinates.   Therefore:

  • Non-engineering applications will get one call (LMS).
  • Engineering pending applications will get two calls (LMS, Contour API).
  • Engineering granted applications will get three calls (LMS, Contour API & LMS Authorization).

NOTE: Engineering applications which have one or more radials returning from the Contour API Coverage endpoint with a HAAT of 30 meters will result in a single additional call to the Contour API HAAT endpoint in order to obtain the accurate HAAT along each radial.  This is a bug in the FCC Contour API that really needs to be fixed.

eLMS normally retrieves information on an application what it is filed, and when the status is changed (including to superceded in the event of an amendment).

This routine operates on the following schedule:

  LMSlist run (RECOT) eLMS processing (RECOT) Frequency
Monday-Sunday 04:00 Immediately after processing LMS data dump. One time per day
Monday-Friday 05:00~17:00 05:05~17:05 Every 15 minutes
Monday-Friday 18:00~21:00 18:05~21:05 Every 60 minutes
Saturday-Sunday 05:00~21:00 05:05~21:05 Every 60 minutes

During eLMS processing of applications, the eLMS system will go into a locked state until the processing is complete.  If the eLMS parser is called at a scheduled (or unscheduled) time and eLMS is still working from a previously scheduled time, eLMS will reject the subsequent request and that scheduled retrieval will be cancelled.  eLMS will try again at the next scheduled time.  This prevents the FCC LMS system from being hit with multiple simultaneous transactions initiated by different eLMS processing sessions.  

Monday-Friday schedules will also operate on federal holidays.

The number of transactions may vary based on the application activity.  If LMSlist does not detect any new or updated applications, no additional eLMS calls to the FCC LMS site will be made. 

During the routine operation, eLMS enforces rate limiting towards the FCC LMS system through a mandatory sleep time of 5 seconds following the completion of the eLMS routines related to a broadcast application.  This would mean that the average load from eLMS for application processing to the FCC LMS system is about 8 calls per minute.  

Broadcast Applications Public Notice

On Monday through Friday (including federal holidays), REC will send a tranaction to the FCC Content API that will check for the presence of the day's Applications Public Notice.  If the Applications Public Notice has been posted, the routine will make a second-call to the FCC Content API to get additional information on how to locate the specific document and then a third call that downloads the plain text version of the Notice.  REC systems will than parse that information in order to update eLMS to indicate that a broadcast application has been accepted for filing. 

This routine will start at 05:05 RECOT and will run each hour until either the Notice is released and the document has been downloaded and will run no later than 17:05 RECOT.

EDOCS

REC calls the FCC's Content API to obtain electronic document data (such as news releases, orders and notices).  EDOCS does not download the actual PDF notices, only the meta-data from the API.  This is done to build links to these items as they appear on FCC.today and other services.  REC EDOCS service will run on the following schedule:

  RECOT Frequency
Daily Prior to 05:10/05:30 No service
Monday-Friday 05:10~17:50 Every 20 minutes
Saturday-Sunday 05:30~17:30 Every 60 minutes
Daily After 17:50/17:30 No service

Filing window sanity checker

On 2-13-24, a new Sanity Checker process has been activated for the 2023 LPFM Filing Window. The Sanity Checker will run once per day during the morning routine to download LMS dump data.  This application trues up our overall application data with window related data.  If there is a discrepancy, we will do a call using the eLMS routine in order to refresh those applications.   As of 2-13-24, this is currently at three applications.  The sanity checker is designed to where there is more than 10 applications in this "lost" status, the automatic sanity checker will not run the eLMS routine but instead will provide notification to REC of an issue.  We may elect to manually run these through eLMS again or identify another issue that may be causing these discrepancies. 

Public Notice retrieval

This routine runs Monday through Friday, once per hour starting at 05:05 RECOT and no later than 17:05 RECOT. This routine will first use the EDOCS API to obtain the daily (broadcast) 'Applications' public notice and then once it is found, it will retrieve the plain text version of that document in order to parse it to update eLMS on applications that are accepted for filing.  Once the program is successful in picking up the public notice, it will stop running for the day. 

REC scheduled daily/weekly tasks to FCC resources

Days RECOT Description of task
Saturday 01:03 ASR and maritime calls data dump - Once per week, obtain three files from ULS (r_tower.zip, l_ship.zip, l_coast_zip).
Sunday 01:03 ULS weekly data dump - Once per week, obtain three license file dumps from ULS.  (l_LMpriv.zip, l_LMbcast.zip, l_micro.zip).
Monday~Sunday 01:12 Census Update - This service creates an internal JSON file that includes information obtained through our internal Census database to provide a profile of the station's coverage.  If the routine encounters an FM application that does not have full eLMS information, it will make calls to LMS and the Contours API (same as the eLMS related calls made during the day) to pick up the data to rebuild the record in eLMS.  These calls to FCC resources should be more of a rare exception and not common.  There is a built in 3 second pause per application to provide some rate limiting.
Monday 01:17 IBFS weekly data dump - Once per week, obtain one file (Bureaus/International/databases/IBFS.zip) from the FCC's FTP site.
Monday~Sunday 03:45 LMS Raw Data dump - Once per day, obtain the daily LMS dump file (Current_LMS_Dump.zip).  This processing of the data from this dump starts at 04:05 RECOT.  Immediately after this processing, an eLMS routine will be performed to obtain any applications filed between 17:00 the previous evening until 04:00 this morning.
Monday~Sunday 08:55 ULS Amateur Radio - Once per day, obtain the daily file dumps from ULS for Amateur Radio activity (2 files downloaded).  a_am_XXX.zip & l_am_XXX.zip (XXX = day of week).
Monday~Friday 18:35 (temporarily disabled) LMS dismissal letter check - Resend eLMS transactions to each broadcast application that was dismissed during the business day to check to see if any additional application attachments (marked as internal) were added by staff in connection with a Media Bureau action. 

Rate limiting and bot prevention

REC has implemented a global rate limit on publicly accessible resources that call out to FCC Contour API resources as past abuse by third party bots resulted in excessive calls made to FCC resources from the REC network of systems.  We have taken the following proactive steps to reduce calls to the FCC systems:

  • We are limiting access to certain publicly accessible resources to no more than 6 hits per minute based on the first 7 characters in the string of an IP address.  This pretty much puts the block at the Class B IP address level.  
  • On FCCdata, when a user agent has the string "bot/" or "Bytespider" in it, we are blocking access to the link to retrieve the REC HAAT tool.  This is based on the trends in activity that were discovered in the enhanced access logging functionality that we recently put into place. 
  • On FCCdata, more than 10 accesses within a one minute period will return an HTTP error 403.

Attempts to access resources where rate limiting is applied or a 403 error is returned will count against the system limits as if they were otherwise permitted accesses.  This means that any user or bot must maintain silence for at least one minute before they can re-access our resources.   Accesses to FCCdata does not automatically result in a call to FCC resources, only when specific mapping applications are also called or if the HAAT link is also clicked (which is why we disabled access to that link for bots).

Internal requests made manually by a human at REC Networks are not rate limited.   

Rate limits and operating schedules on the eLMS functionality that is automatically started by REC Networks are detailed in the information above.  

Development and Repairs

During all hours (including business hours), REC may send one-off requests through eLMS to obtain applications for the purpose of developing new eLMS functionality or to send a "manual reset" of an application in eLMS that may have been mis-parsed or had a bad contour map due to the failure of the FCC Contour API.  

Special Projects

Special projects involve the need to have eLMS obtain a large number of pre-identified applications.  This is done in order to address specific issues or to migrate new application types into eLMS handling.  

Normally, special projects will normally only be conducted between Friday at 6PM EDT and Monday at 7AM EDT as to use the systems during periods of predicted low demand.   In case of a major operational issue that cannot wait until the weekend, REC will limit special projects to between 6PM and 7AM on any day of the week or any time on the weekends.

Sometimes, special projects are intentionally timed for federal holiday weekends if it is necessary to obtain the applications we need.

During special projects, requests to the FCC's LMS system are normally rate limited to one request every 10 seconds (6 per minute) (on granted applications, one "request" may involve two calls to LMS to get the application and authorization document), but in no case shall special projects generate requests less than every 5 seconds apart (12 per minute).  We are cognizant of the needs of the LMS server. 

REC Networks public websites

REC Networks operates various websites that use data gathered from FCC resources.  Most of these services are free of charge for use.  The following information describes the website and how FCC resources are used.

FCCdata

https://fccdata.org

FCCdata is the most comprehensive website for inquiries into broadcast facilities, amateur radio and other services.  FCCdata also includes broadcast and in some cases, amateur radio information from Canada, Mexico, Australia, Japan, United Kingdom and Ireland.  FCCdata is unique from other broadcast query websites (including those at the FCC) because it can display full FM facility data, usually within 20 minutes of the filing of the application.  The same timeline exists for non-engineering applications from all LMS supported broadcast services for the display of application information.

FCC broadcast data is derived from the data gathered in our eLMS routines as well as from daily data dumps from LMS, ULS and IBFS.  Site usage may result in calls to the OPIF API and to the FCC Contours API.  Users are rate limited where if they make more than 6 accesses per minute, they will be restricted from automatically receiving FCC resources.  More than 10 accesses per minute will block them from the site. 

As of 2-14-2024, FCCdata averages about 4.7K page views per week.

fcc.today

https://fcc.today

FCC.today operates as a “dashboard” for application and action activity in the Audio and Video Divisions of the Media Bureau.  FCC.today uses data collected through the REC LMSlist routine to provide “near real time” updates on applications and actions that are processed through LMS.  FCC.today also displays information on recent FCC releases that involve the Media Bureau, FCC daily business items, emergency alerts received through IPAWS OPEN, information on broadcast pleadings as well as DC federal employee reporting status.  It is also the home of “fcc.today – the podcast”.

Public website usage on FCC.today does not create any direct calls to FCC resources.  All data shown on FCC.today comes from the various scheduled background processes that are described above.  This includes scheduled data gathering routines using our LMSlist and EDOCS processes and the LMS daily dump file as well as data received through the API at the Office of Personnel Management and through our memorandum of understanding with the Federal Emergency Management Agency.

As of 2-14-2024, FCC.today averages about 3.8K page views per week.

REC Main Website

https://recnet.com

REC’s main website is a comprehensive Drupal-based platform which is used for both public websites as well as walled garden applications for internal use only.  Data used can range across our entire universe of stored databases across multiple federal agencies.  Some tools replicate FCC tools that utilize JavaScript such as Curves and FM-Model. 

Some applications on the REC Main Website may generate user requested calls to FCC resources.  Most of these applications have been rate limited, mainly to 6 requests per minute. The most frequently used of such tools is:

HAAT/Elevation/ERP Tool

https://recnet.com/haat

This tool makes calls to the FCC Contours API to provide an improved display of height above average terrain, site elevations and calculate effective radiated power.  Also obtains data from the US Geological Survey and private sector data providers.

As of 2-14-2024, recnet.com gets an average of about 2.8K page views per week.  We have no specific measurement for the HAAT Tool.

myLPFM

https://myLPFM.com

myLPFM is a password protected system that is limited in access to broadcast stations in the Low Power FM Radio Service.  myLPFM allows LPFM stations to perform station management functions, manage their visibility on station directories provided by both REC and third party providers, provide services that will assist in filing modification applications, obtain emergency alert information, engineering information for HD Radio upgrades and assistance with interference from subsequently authorized FM translator stations as well as information on potential encroachment from any subsequently applied for facility.

Various features within myLPFM will make user-requested calls to the FCC Contours API to get various information necessary for the different sub-applications within myLPFM.  Some data received through the Contours API is cached for future access thus speeding up the user experience and eliminating unnecessary traffic to the FCC API.

myLPFM experiences a very low usage level.

Contact Information

In the event of issues with REC Networks' automated access to the FCC systems, please first contact Michelle Bradley, CBT via email at lpfm@recnet.com or at 202 621-2355.  

More information about eLMS available at https://recnet.com/eLMS

As stated.  REC's need to access these public FCC resources are for business, research and public dissemination purposes, consistent with 44 USC §3501 et seq. and therefore, is not malicious or with other ill-intent. 

REC does not have any automated resources that accesses FCC resources behind a password.  We have humans to do that at the request of our clients.

Thank you for your understanding.