MiMax-

White Paper: Automated Driving System (ADS)

Driving and Anti-Collision Rasters using Video Compression Macroblocks and Sprites Hardware Method ( DARVMS )

1. Foreword

Most “how-to” technology white papers do not require a “foreword” section, prior to the abstract. Automated Driving Systems (ADS) technology is experiencing a worldwide rage of attention for multifaceted reasons of actual need and benefit versus real safety issues. Can this ADS actually be done? Thus however this paper, on thus a subject as complex as ADS, discusses very briefly, the state of the industry, schools, open-source ADS code offerings, and government interests in ADS.

Despite potential problems of investment (good ones and bad ones) to finance industrial sized ADS projects and even politics distorting the ADS markets, the science of ADS progresses rapidly. As such, before even discussing GPU hardware raster methods, applied to ADS, a preview of the ADS issues, and at least mention of some of the parties by way of their publications in January 2022, can help bring to focus, where this paper discussing electronic circuits resides in the larger scheme of ADS research.

Many white papers and standards are being published from entities as large as the well-known auto-truck manufacturers to government agencies, down to master’s degree science/engineering students. Below is a very short list of ADS publicly available documents. If any of the hyperlinks below somehow bother the documents’ writer/owner/lessee (so on and so forth), please let us know and we will look into removing it from this paper.

It is assumed all of these are internet public viewable published papers, and web-pages (found by free search engines) are from entities that want to have other similar subject matter papers hyperlink-ed to them as recommended reading.

https://www.iso.org/standard/70939.html
ISO/PAS 21448:2019 Road Vehicles – Safety of the intended functionality (SOTIF)
“The absence of unreasonable risk due to hazards resulting from functional insufficiencies of the intended functionality or by reasonably foreseeable misuse by persons is referred to as the Safety of the Intended Functionality (SOTIF). This document provides guidance on the applicable design, verification and validation measures needed to achieve the SOTIF. This document does not apply to faults covered by the ISO 26262 series or to hazards directly caused by the system technology (e.g. eye damage from a laser sensor).”

https://www.scribd.com/document/415231287/Safety-First-for-Automated-Driving
This publication summarizes widely known safety by design and verification and validation (V&V) methods of SAE L3 and L4 automated driving.

(Comment on this scribd-dot-com hyperlink. As ADS lawyers statements go’ such as this example that is quoted and hyperlinked one below, may be an education in of itself for law schools. And it shows what a dicey, risk-averse subject that auto-driving-systems are in the views of attorneys.

However at some point, there needs to be open discussions of electronic circuits and software design, at least for the sake of education, that can be applied to ADS and to other similar subjects. As an example of another similar subject, it appears that ADS has some common technology aspects with automated electronic military targeting systems that seek to create collisions, which of course is opposite to collisions-avoidance.

The authors herein on this white paper are discussing circuit design concepts. Any systems ever tested around humans should have project dedicated, experienced, and competent risk/hazard-analysis/safety engineers involved. We make no claims or recommendations on this referenced legal statement example, but to direct readers to its hyperlink. Parts are redacted, so readers can seek out its authors, and most current version, where this white paper found it published. Please visit “scribd—dot-com” web-site to read in entirely.)

© Copyright 2019 by Aptiv Services US, LLC; AUDI AG; Bayrische Motoren Werke AG; Beijing Baidu Netcom Science Technology Co., Ltd; Continental Teves AG & Co oHG; Daimler AG; FCA US LLC; HERE Global B.V.; Infineon Technologies AG; Intel; Volkswagen AG. All r_____reserved.

The document and information contained herein is not a _________, either expressly or impliedly, to any intellectual property owned or controlled by any of the authors or developers of this publication, and _________ to this document and information should not be considered to be have been made available to parties receiving and/or reviewing this ____ and ____.

The ___________ contained herein is provided on an “AS IS” basis, and to the maximum extent permitted by applicable law, the authors and developers of this document hereby disclaim all other _______ and _______, either express, implied or statutory, including but not limited to, any (if any) implied _____, ____ or ____ of ____, of fitness for a particular purpose, of accuracy or completeness of responses, of _____, of ____, of lack of viruses, of lack of negligence.

THERE IS NO WARRANTY OR CONDITION OF ____, ____ ____, QUIET POSSESSION, OR NON-______

Below several more ADS Hyperlinks and quotes ADS Gov, EDU, OpenSource hyperlinks and quotes, that demonstrate the wide technology interest, and government attention

https://www.researchgate.net/publication/329131260_Safety_System_Architecture_for_the_Design_of_Dependable_and_Adaptable_Autonomous_Vehicles
Abstract —- Current systems lack common sense and have trouble behaving in a truly cautionary manner, which is why a fallback-ready user is expected to take over in the event of a performance-relevant system failure affecting the dynamic driving task. Yet it seems unwise to rely on human drivers to act as a safety net for the purpose of offsetting the lack of maturity of Automated Driving Systems

https://velodynelidar.com/wp-content/uploads/2019/08/ADAS-Whitepaperver.12.2018.pdf
“Similar to Level 2, Level 3 vehicles include features described as allowing drivers to remove their hands and feet from controlling the vehicle’s speed and direction. However, unlike Level 2, Level 3 places the responsibility of monitoring the surrounding environment on the vehicle’s sensor system, and instead makes the driver responsible primarily for any fallback actions that should be taken in case the automation fails.”

“In support of a safety-first perspective, we therefore propose that any vehicle rated below Level 4 should not be marketed as being “autonomous” or “self-driving.”

“The key advantages of LiDAR become evident when comparing the capabilities of vehicle systems with radars and stereo cameras with those that have all three sensors.”

“Lane Keep Assist (LKA) : LKA systems have similar functionality to Lane Departure Warning (LDW) systems with the added function of automating corrective action by steering the car away from the lane marker to re-center the vehicle in the driving lane.”

https://www.transportation.gov/av/grants
“On September 18, 2019, U.S. Transportation Secretary Elaine L. Chao announced $60 million in Federal grant funding to the selected award recipients for the Automated Driving Systems (ADS) Demonstration Grant Notice of Funding Opportunity. (Press Release)”

https://safetrec.berkeley.edu/sites/default/files/edwardkimcscrsfinalreport_2020.pdf
“Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World”

This white paper particularly thanks the open-source team of GitHub OpenPilot for publishing their ADS technical project details, so that many engineering and science teams and persons may better compare technologies for ADS. Microsoft appears to be committed to GitHub being a place where open source projects can thrive better, with the aid of MS’s support for the servers needed, and with the man-hours of labor to maintain that system.

https://nira.com/github-history/
“How GitHub Democratized Coding, Built a $2 Billion Business, and Found a New Home at Microsoft..

… As was typical of most open-source projects at that time, Rails’s codebase was managed by a small, tight-knit group of coders who managed contributions to the codebase manually. These coders served as de facto gatekeepers. Hyett and Wanstrath had to not only petition these gatekeepers to look at their code but also convince them that it was worth committing to the Rails project. Even if one of the project gatekeepers found code suggestions useful, actually merging patches wasn’t straightforward, either. Essentially, contributing to the Rails project was about who you knew, not what you knew…

…Git attempted to address some of these problems. Linus Torvalds’s version control system was as remarkable as the operating system he had built single-handedly just a few years before. Git allowed coders to collaborate without petitioning gatekeepers for access. Git was a crucial first step in the ultimate democratization of coding, particularly in the open-source community.”https://fortune.com/2019/01/07/microsoft-github-free-code-projects-small-teams-repositories/
“Microsoft-Owned GitHub Just Made It Free for Coders to Keep Projects Private in Small Teams…
http://github.com/commaai/openpilot#what-is-openpilot
http://github.com/commaai/openpilot/wiki/introduction-to-openpilot
“locationd: combines inputs from sensors, GPS, and model into real-time estimates for movement and position” … “THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT. YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS. NO WARRANTY EXPRESSED OR IMPLIED”

GitHub’s public open-source user group (as noted in links above) is a place to review and contribute to free software called “openpilot” that may possibly, safely auto-drive a research test car. And Github team recommends readers of openpilot, also read ISO 26262-1:2011 “Road vehicles — Functional safety”. It would be best to read all the safety standards concerning this subject of ADS for any persons or teams testing any of these methods.

This paper’s GPU-Phaselocked-Multi-Raster-Macroblock circuit concepts compares mostly only several of the OpenPilot code sections of “locationd”, “camerad” and “sensord”. These sections of OpenPilot may potentially lend themselves to running faster and with more reliability if run as hardware solutions, besides software. Or to be further validated by hardware circuits.

GPU’s and their inherent raster and blitter (block line transfer) engines and video compression techniques, combined with additional logic, for actual possible ADS test implementations can possibly increase safety, and lower costs, and decrease computational power consumption. And at a minimum, can be used to laboratory compare and validate the driving controls outputs of other methods systems.

This paper makes no claims of technology ownership or recommendations, but rather to assist in the open discussion of methods to test in laboratory and simulator environments. Additionally this paper would quote all the links above warnings about for test and research purposes only.

This paper is published as to add to the discussion of electronic circuit design. Any parties using the material in any manner have self-at-risk obligation to verify all concepts, methods, drawings etc. Many hyperlinks are added to assist the readers to find similar published webpages, white papers, university thesis, and commercial offerings of similar materials.

May we dedicate this paper to the brilliant designers of the 6502 processor family and Jay Miner designer of the Atari 800 and Amiga family’s and designers of Commodore family’s computers for their contributions to video graphics hardware sprites technology, and also to the R&D teams of the OpenPilot, Udacity, Tesla, and many other ADS projects.

Any reviewed driving systems software or hardware herein, are congratulated for their teams hard work, on this complex, and what may be an imperfect ADS science for many years to come.

2. Abstract

This paper titles this Automated Driving System (ADS) data processing method as:

“Driving and Anti-Collision Rasters using Video Compression Macroblocks & Sprites Method” or acronym DARVMS.

Real-time driving and collision avoidance data processing are needing to move away from classic general computing flow chart software designs, to more small parallel repetitious raster processes with local dedicated cache memory. These concepts are put forth for aids in research, and design considerations.

Generally hardware circuits can provide graphical or 2 or 3 dimensional solutions faster than software. So far in most white papers and non-profit engineering terms pursuing ADS, software is the preferred method. Hardware can offer server unique uses of GPU rasters and sprites when the multiple rasters are phase locked for faster and more coherent data transfers from raster to rater. Also Blitters (block-line-transfer acceleration hardware for pixel math and Boolean logic functions) are put to data processing function.

These hardware concepts may also lend themselves to an additional layer of safety when melded with the present software solutions, by an alternative method to prevent driving decisions that could result in a possible or probable collisions.

GitHub’s open-source software called “OpenPilot”, critical central core for multiple rapid changing objects management, can be considered to be their section named “locationd”. “locationd: combines processed-object-data, that was originally from inputs of sensors, GPS, and next then processed into real-time estimates for movement vectors and position differences between the center vehicle and objects, whereas objects are both of other moving vehicles and stationary.

This “DARVMS” GPU-Mufti-Raster method is similar in function to the GitHub section “location”, whereas the moving objects are all themselves moving in relation to the Center Vehicle (CV).

In some sense, this Center Vehicle can be considered an object that mimics artificial intelligence. Intelligent learning logic circuits and methods are not discussed here, but rather the fastest processing of the objects data, and with the lowest possible processing errors, or for a system to be over saturated by computational needs, and unable to make driving decisions, fast enough to avoid collisions.

The intelligent mimic behavior comes from the later stages of data processing of collision avoidance, similar in behavior of wanting to avoid self-harm by avoiding air-to-air collisions. And similarly that the bird also has a goal, to arrive at a new location. Graphics Processor Units (GPU’s) can process pixel data very fast, that represent pre-processed macroblocks of objects, of data that originally came from camera inputs and other sensor inputs.

Present map-plan software’s for CV’s to change location, are in day to day use by many millions of vehicle operators, for best travel-plan course, with real-time updates. The map plan software’s do not claim or attempt to compute real-time, approximate 100 Meter range, Geographic Localized Anti-Collision Control (GLACC).

This white paper discusses this 100 meter range GLACC real-time many objects data processing. Also reviewed is conversion camera data to create object data blocks. The emphasis in this paper, is on the time coordination of pre-processing of real-world physical object data macroblocks, in a manner like video compression, to use the advantage of efficiency and features of GPU’s multi-raster processing, resulting in driving decisions. Combinations of multiple GPU IC’s can be used to increase the number of rasters. Alternatively, classic CPU sensor data to driving decisions processing, suffers from the time-uncoordinated temporal relationship, of software stacks.

Pre-processing of objects data and final processing of all objects that relate to the CV, can make use Graphics Processing Units (GPU) multiple rasters and Boolean operations and their inherent ability or create data macroblocks that represent objects, much as macroblocks represent logical sub-sections of portions of moving images, that is core part of video compression techniques. Even in GPU’s all operations needed for collision avoidance cannot be simultaneously-instantly completed, but like CPU software, it is still in a multi-step process.

However if the data rasters periodic frame rates are phase temporally controlled, then the flow of data of pre-processing to final collision avoidance driving decisions, can achieve faster driving control outputs, from input sensor data stream changes. GPU multiple rasters that are temporally coherent, comprise deterministic multiple raster process compared to a more semi-deterministic CPU software interrupt process for anti-collusion data processing.

3. Other Examples of ADS: OpenPilot, Udacity, Tesla

GitHib’s OpenPilot Location-d, Camera-d, and Sensor-d Code sections, along with discussion of Udacity’s Lane-Finder and a NHTSA crash report are compared to raster-hardware ADS add-ons for additional layer of safety.

GitHib’s OpenPilot Locationd Section models Real-Time Estimates of Movement and Position. Rather than dissect the operation of OpenPilot’s code section “locationd”, this paper will quote a few text sections and images the Guthub’s team’s web-page.

https://github.com/commaai/openpilot/wiki/introduction-to-openpilot
“What is openpilot? openpilot is an open source driver assistance system. Currently, openpilot performs the functions of Adaptive Cruise Control (ACC), Automated Lane Centering (ALC), Forward Collision Warning (FCW) and Lane Departure Warning (LDW) for a growing variety of supported car makes, models and model years.

In addition, while openpilot is engaged, a camera based Driver Monitoring (DM) feature alerts distracted and asleep drivers. See more about the vehicle integration and limitations.”…

Figure 1

Above are eight photos of GitHub open source software team shooing OpenPilot in test, with the software running in a test vehicle.

Below is a block diagram of the OpenPilot software published by GitHub, that shows the functional subsections and their complex interaction. This white paper will compare GPU based multiple raster and spite hardware functions, to several sections of OpenPilot and Open-CV Lane-Find projects.

Figure 2

3.1 OpenPilot “locationd” Section Compares to MM-Final CV-Raster-Set

Below is a list of details quote from OpenPilot software from Github team.
…..”Safety and Testing

openpilot observes ISO26262 guidelines, see SAFETY.md for more details.
openpilot has software in the loop tests that run on every commit.
The code enforcing the safety model lives in panda and is written in C, see code rigor for more details.
panda has software in the loop safety tests.
Internally, we have a hardware in the loop Jenkins test suite that builds and unit tests the various processes.
panda has additional hardware in the loop tests.
We (GitHub) run the latest openpilot in a testing closet containing 10 comma devices continuously replaying routes.”

Below is a zoom of software section “locationd”, where this white paper discusses an alternate hardware cross checking (double checking for added safety) using GPU phase locked rasters and Boolean logic of macroblocks, that represent objects, for Real-Time Estimates of Movement and Position. Additionally video game graphics hardware controller IC and video memory methods for the use of “sprites” can also be applied in the section for “locationd”. See section on “Sprites”

Figure 3

3.2 OpenPilot “sensord” & “camerad” Sections Compares to MM-Object-Macroblock Create

Similar to GitHub’s OpenPilot software, pre-processing of object data is needed prior to locationd section, As quoted from GitHub OpenPilot

https://github.com/commaai/openpilot/wiki/introduction-to-openpilot
camerad: interfaces with available camera hardware to generate images, which are encoded as videos.
sensorsd: interfaces with available sensor hardware to generate data from the hardware’s IMU

Figure 4

OpenPilot’s sections “camerad” and “sensord” are a reasonably close functional equivalents of what this paper shows as the GPU-multi-raster conversion of raw camera data to vehicle object characteristic macroblocks, in a fashion not too dissimilar with video compression. Whereas the final object macro block is the data that is later to be moved into the locationd raster process.

The preferred method, hardware or software, can then be implemented while the other can be used as an error checking mechanism.

3.3 Open CV Udacity SDCND Lane Finding Projects

The approximate 100 Meter range, Geographic Localized Anti-Collision Control (GLACC) function via Driving and Anti-Collision Rasters using Video Compression Macroblocks & Sprites Method” (DARVMS) can act an additional sprite hardware method layer on top of an example software Lane-Finder method for added safety.

Considerable and impressive software code work has been done by teams in the area of lane-finding, such as in the Udacity example below.  The real-time software lane-find processes may or may not be easily improved on with multiple raster hardware designs.

Lane-Find function may be improved by real-time cross-checking the software Lane-Find against hardware sprites for lane limits (see section 7)

Below is a graphic of a published video of Udacity SDCNC project Lane Finder in an apparent test scenario.  Momentarily at second count 28 apparent drivable lane forward, drifts into oncoming lane (green color overlay zone).

Figure 5-a

Next is same video image, but only lines on white background for easier printing. North American example “RH” (right hand) driving arrow standard pointers added.

Figure 5-b

Potentially safety can be improved by an extra layer of safety via hardware circuits that use raster based sprites.

Whereas multiple sources of input data are used to define sprite lane limits, and can offer an added layer of safety cross check, and not slow down the software lane find process. (see sprites section 7).

An example video from one of the Github.io groups multiple lane-finding projects posts list video to on the internet.

Above image is of video from
“Udacity SDCND. Project: Advanced Lane Lines Detection. Video 3 “
943 views Feb 19, 2017 (at seconds 28 of 48)
https://www.youtube.com/watch?v=tgQwXVzhrBI

Additional helpful links from Udacity for lane find and ADS in general:

https://medium.com/udacity/how-self-driving-cars-work-f77c49dca47e
” localization is how we figure out where we are in the world….
……..In the path planning phase, we chart a trajectory through the world to get to we want to go….
then decide which maneuver we want to take in response to those vehicles. Finally we build a trajectory…”
https://labonneguigue.github.io/Udacity-Self-Driving-Car-Nanodegree/
https://labonneguigue.github.io/CarND-Advanced-Lane-Lines/

3.4 NHTSA Tesla Crash Investigation PE-16-007 Text and Conclusion

This NHTSA report (below) is probably not particularly special or out of the ordinary, in the course of ADS science and products progression over a period of years.

https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.pdf

This final part of the conclusion is:

“NHTSA’s examination did not identify any defects in design or performance of the AEB or Autopilot systems of the subject vehicles nor any incidents in which the systems did not perform as designed.”

Essentially, crashes will happen, even with ADS products that are deemed officially not to have defects. We would add that an additional layer of ADS hardware, on top of ADS software could be added, to further reduce accidents where ADS is involved.

And we would agree that:

“The Automatic Emergency Braking (AEB) or Autopilot systems may not function as designed, increasing the risk of a crash”

Below is a very shortened brief of copied snippets of the PE-16-007 report. It is recommend readers interested in ADS science and technology, to download and read the full NHTSA PE-16-007 document.

https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.pdf
PE-16-007 Date Closed 01/19/2017

.“ …..The Automatic Emergency Braking (AEB) or Autopilot systems may not function as
designed, increasing the risk of a crash……..

…..20 automobile manufacturers, representing 99 percent of the U.S. new-car market, to voluntarily make AEB “standard on virtually all light-duty cars and truck…..

….. The Autopilot system is an Advanced Driver Assistance System (ADAS) that requires the continual and full attention of the driver to monitor the traffic environment and be prepared to take action to avoid crashes……

…..Data obtained from the Model S indicated that: 1) the Tesla was being operated in Autopilot mode at the time of the collision; 2) the Automatic…..

…..Emergency Braking (AEB) system did not provide any warning or automated braking for the collision event; and 3) the driver took no braking, steering or other actions to avoid the collision……

…..AEB technologies. Automatic Emergency Braking includes the following crash avoidance technologies: Forward Collision Warning (FCW), Dynamic Brake Support (DBS), and Crash Imminent Braking (CIB). An FCW is presented to the driver if the system predicts a crash with an object in the vehicle’s forward path is imminent……

…..Tesla AEB system. The Tesla AEB system is a radar/camera fusion system that is functional when switched ON regardless of Autopilot status……

…..The driver can select the timing of FCW alerts with four options: Early, Medium, Late, or OFF…..

…..The camera system uses Mobileye’s EyeQ3 processing chip which uses a large dataset of the rear images of vehicles to make its target classification decisions. Complex or unusual vehicle shapes may delay or prevent the system from classifying certain vehicles as targets/threats……

…..7.0 CONCLUSION Advanced Driver Assistance Systems, such as Tesla’s Autopilot, require the continual and full attention of the driver to monitor the traffic environment and be prepared to take action to avoid crashes.

Automated Emergency Braking systems have been developed to aid in avoiding or mitigating rear-end collisions. The systems have limitations and may not always detect threats or provide warnings or automatic braking early enough to avoid collisions. Although perhaps not as specific as it could be, Tesla has provided information about system limitations in the owner’s manuals, user interface and associated warnings/alerts, as well as a driver monitoring system that is intended to aid the driver in remaining engaged in the driving task at all times. Drivers should read all instructions and warnings provided in owner’s manuals for ADAS technologies and be aware of system limitations.23 While ADAS technologies are continually improving in performance in larger percentages of crash types, a driver should never wait for automatic braking to occur when a collision threat is perceived.

NHTSA’s examination did not identify any defects in design or performance of the AEB or Autopilot systems of the subject vehicles nor any incidents in which the systems did not perform as designed. “

4. Attributes of Inter-Connected Phase Locked Multiple GPU Rasters

GPU rasters are applied to ADS systems into two ways, first to assist with pixel data conversion to macroblock compressions, such as pedestrians or other vehicles. And second to later track all the objects for risk, distance, angles and speed.

GPU rasters inherent pixel flows on a display line on the X axis of data, and the line to line relationship of the Y axis, inherently create a data format that lends itself to fast finding of hypotenuse distance to the Center Vehicle (CV). This would include vector angles as relating to a Center Vehicle (CV).

The distance and angles then can be preprocessed with known data frame rates allowing for deterministic and low latency outputs to the driving controls.

Some of this method is a re-purposing from the 1980’s video game hardware designs that used similar processes for entirely virtual environments. Whereas the human operator supplied data input to avoid collisions, while simultaneously another human player, or a computer program created objects for the CV-equivalent to avoid.

4.1 Example : MM-Pedestrian Macroblock Created with Rasters, Blitters/ FPGA Logic

Here the camera views are converted from more raw Pedestrian video data into Characteristic Compressed MacroBlock data blocks. Prior to this GPU further steps in the raster and macroblock process, the pixel data is transformed to a top-down
view.

Angular normalization of views is not covered here, as it is already a common process in automotive camera systems for parking assistance, and often processed by stretch/shrink Blitter function calls. “MM” is added to this method name, to keep it distinguished from software methods of other teams, for discussion in this paper.

Figure 6

Once the pedestrian raw image data, is first processed at the camera level, to include the zeroing of all extraneous pixels around the pedestrian, then the pedestrian pixel data proceeds to next steps of macroblock data compression processing.

The macroblock that is being created in real-time contains compressed “WC” worse case data, and “AV” average data of the object. At this same process stage, “worst case” values can be inserted that can later cause instant frame rate warnings to the main ADS program.

“Blitters” can move pixel data for bit, byte and words, or pixels copying and Boolean logic processing (aka operations) at raster pixel flow speeds.

Vehicle Object Macroblock formation, similar to video compression methods, is the lowering of resolution to keep only the most valued data, in this example of a 6 x 3, compression to a 2 x 1 data block in the next drawing.

Figure 7

Some basics of Blitters (block line transfer accelerator) can be further reviewed in these article links.

https://www.intel.com/content/dam/develop/external/us/en/documents/the-architecture-of-intel-processor-graphics-gen11-r1new.pdf
page 8 “Gen 11 architecture is split into: Global Assets which contains some fixed function blocks the interface to the rest of the SoC, the media FF, and 2D Blitter and the slice.”

https://st-news.com/issues/st-news-volume-2-issue-7/features/the-blitter-chip/
“The name ‘blitter’ is derived from ‘Bit-BLT’, which stands for ‘Bit Block Transfer’.

https://news.ycombinator.com/item?id=23512897
Fast 2D Rendering on GPU (raphlinus.github.io) multiple discussions of Bit-Blit (aka blitter function)

https://pro.arcgis.com/en/pro-app/latest/tool-reference/image-analyst/pixel-classification.htm
“Not every pixel in an image is used in the classification of training samples ….
..The goal is to semantically project the discriminative features (lower resolution)”

Field Programmable Logic Arrays (FPGA) synchronously timed on pixel-clock, are often the most affordable method to create the glue logic to complete Boolean Macroblock processing where particular GPU IC may not have exactly the set of Blitter Boolean math needed.

4.2 Example : MM Car-Truck Macroblock Created with Rasters, Blitters/FPGA Logic

“MM Car-Truck Macroblock” method is similar to the MM-Pedestrian Macroblock creation, a common two axle truck is converted from camera data. And has the “MM” letter added to distinguish from other software methods, this paper compares to.

It is a common process to use techniques of zero-ing pixels around the image data of a truck, data base’s of vehicles are compared to find solutions of most likely, or to be unidentified. This paper does not cover “learning software” (aka artificial inelegance) that can be engaged, such as when the driver arrives at destination, the system could ask the driver questions, to help better identify lesser understood vehicles, for on-going data base improvements.

Figure 8

For illustration herein, a camera input data side view of preprocessed and data base compared truck view is shown. Where possible additional cameras can optional add improved top-down views.

Once the truck imagery has been pre-processed at the camera level (such as dual rasters in feedback loop to remove all moving pixels, or reverse, remove all non moving pixels), to include the zero-ing of all extraneous pixels around the 2-axle-truck, then the truck pixel data proceeds to next steps of rasterization and then to macroblock processing.

This typical two axle truck is processed into a macroblock of compresses data of multiple types of object macroblocks, with each macroblock multiple pixel sub-types of bits for: risk levels, position-location, and relative and absolute speeds, and angles. A feedback loop process is required, as the distance from CV requires information of the CV and the other vehicle object.

More detail of the macroblock creation process, shows an example of a 1 pixel tall and 6 pixel wide memory zone (next drawing) of a portion of truck, pixel data raster zone, is processed as two 1 pixel tall and 3 pixel wide pixel zones are compressions of characteristic truck object data, into a 1 pixel tall, 2 pixels wide portion of a macroblock.

This Truck compressed macroblock, contains information such as “WC” worse case data, and “AV” average data of the object.

Figure 9

The characteristic data is vehicle object type, relative speed, relative direction vector, its relative acceleration, its absolute speed to the geography, risk rating, average YUU imagery, headlight spacing, vehicle orientation, macroblock quality rating (how reliable is the data), and finally bits remaining for enhancing visual raster viewing.

Figure 10

Higher resolution systems for these vehicle object macroblocks can double up pixels, such as to create 64 bit, rather than 32 bit pixels, to contain more detailed information, in a similar fashion as how the YUV system, often stores and transports a single fully defined pixel’s data as two pixels of Y, and one each of U and V.

4.3 Object Macroblock Pixel Motion Processing With Two Raster Feedback Loop

Both car/truck and pedestrian macroblock objects data quality is spatially improved by an optional step of pixel motion processing, to better separate a vehicle object from other extraneous background pixels.

Figure 11

This white paper briefly covers step for use of two phase locked rasters to create a video feedback loop that in turn creates a real-time data streaming scenario of present frame and old frame simultaneously.

And next then test for either (A) characteristically same pixels only in old and new frame (aka pixel matches), such as to remove moving backgrounds, behind a vehicle in a neighboring lane to our CV, moving in same direction. Or,

The opposite (B) characteristically difference pixels only in old and new frame,(changing pixels only), such as to grab only the changing pixels, as our CV remains still (motionless) and it is desired to grab pixel data only of vehicles in motion nearby.

This Raster Feedback loop method to process pixels for motion subtraction is covered in this noted these other two whitepapers, where the two rasters temporal timing lock to 1 pixel clock accuracy is needed. More can be researched on the two links below

First, for “Pixel Clock Subtraction” (free) to phase lock the two rasters, of the feedback loop.

Figure 12

https://mimax.com/optimizing-embedded-hardware-using-interleave-memory-phase-locking/

Then next, to set up the self-sustaining feedback video data steam where raster #1 output flows back into raster #2 input, then raster #1 and #2 data outputs are compared real-time. Whereas this two raster loop setup creates a new-frame and 1-frame-old (aka: one frame temporally old in time).

Figure 13

https://mimax.com/statistical-research-of-blockchain-data-with-gpu-hardware-acceleration/

Both the (A) and (B) feedback loop, of the camera data processing, for motion effects (a method used in other Mimax whitepapers (noted above) that cover phase locked rasters and video compression)

5. Top Raster of Center Vehicle, Fed Data from Objects Processing Rasters

A more holistic view can be made of data flow in multiple rasters that are in temporal (periodic timing) slave phase control, moving the macroblock data through the multiple raster head steps. This is where the incoming objects data, are in an X-Y plane, surrounding the Center Vehicle (CV).

The automated periodic raster frames with Boolean logic interactions, and hypotenuse distance and velocity calculations being compared against the Center Vehicle, compress a type of mimic or artificial intelligence, similar to that of insects, where many faceted eye’s act in a manner like a multi camera system, and the insect alters its movements for its own safety concerns.

5.1 Final CV Raster-Set Combined with Anti-Collision Sprite Processing

Phase Lock Value (PLV) concept can be applied to the raster-to-raster data stream flow, for best processing speed. Data and processing steps moves toward the Final CV Raster-Set, and the double check against sprite safety processing. (See section 7 for hardware Sprite Safety Process)

100 meter range GLACC, is of intensive rapid periodic review, of all objects of concern, such as pedestrians, other hikes, signs, traffic control lights, emergency vehicles, lane markings, over-road height limits, withe limits, all compared mealtime, against other issues such as weather conditions, day-night conditions that all effect many active changing varietals of risk ratings.

Figure 14

In this system, the raster period is typicality constant, and itself can be a test for proof of system function. The earlier processed macoblocks of example pedestrian type macroblocks and example vehicle type macroblocks, are the objects of most concern that surround, and in a sense, threaten, the Center Vehicle (CV)

Like with the neuroscience of human motor control and epilepsy, Phase Lock Value (PLV) of the rasters that feed other rasters data strongly effects efficiency of the whole process. Whereas PLV is generally a real number value of 0.00 to 1.00, the more phase locked, the system of period data rasters moves pixel-like data coherently in time, where the system designers and operators know at all times, how old in time, the data acted on, is.

GPU rasters in this concept, unlike CPU software, don’t suffer from large variations of delay in processing occur due to temporary data processing overload. The processing overload by CPU type processing is temporary and usually not very real-world applied, predicable buildup of software stacks, and worse yet temporary memory storage being used, due to stacking of processes. Or can be stated as coherent deterministic multiple raster process compared to a semi-deterministic CPU process.

See these white papers and public posted pages, for more discussion of Phase Lock Value, both in neuroscience and electronic data processing.

Below are Phase Locking Value (PLV) Hperlinks for further inquiry.

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0146443
[more accurate estimations of oscillatory synchronization across a wide range of different synchronization regimes]

https://pubmed.ncbi.nlm.nih.gov/22661936/
“Differential synchronization in default and task-specific networks of the human brain”

https://www.wiley.com/enus/Phase+Locking+in+High+Performance+Systems%3A+From+Devices+to+Architectures-p-9780471447276
“eighty-three key papers provide an eminently readable foundation in phase-locked systems. Analog and digital circuit designers will glean a wide range of practical information”

Conceptually, PLV will correlate to the statistical probability that most temporally coherent data will move through a group of serial data flowing from one periodic repetitious raster, to the next, via the path of out-data-port-Raster-A to the next raster’s in-data-port-Raster-B. Then what data flows out of Raster-B, will be acted on for decisions.

The effected final decisions of software or hardware logic acting on the data, will have statically better results, to the original real-world data that flows into the series of data rasters for data processing, for final decisions of steering, throttle and braking.

6. Ships Drone Defense System Similarities to Vehicle Object X,Y Plane Tracking

Similar to many objects moving near and around a self-driving wheeled vehicle, seagoing ships can have similar issues with a threat swarm-cloud of drones, would need to track and process the high volume of changing real-time data.

The ship, like an automobile Center Vehicle (CV), is shifted into the X,Y raster to best accommodate the ship’s movement geographic vector against the larger background raster, and the drone cloud’s average movement vector.

Figure 15

In this manner of offset-shifting the raster position of the “moving center ship” (MCS), the best uses of the territory of the raster pixels provide a lowered average latency reaction capability, for the distance and travel times of munitions in either direction, to or from the drones.

Drone macroblocks by this processing phase have their threat value-number continuously real-time updated on each frame period, based multiple factors, of the designer’s selection.

The flow of pixels in the video lines, provide X data stream, and line counts provide the Y data counts, for the continuous delta position and velocity-vector feedback updating to the drone macroblocks. Several links below of published articles discuss the growing drone threat.

https://www.defensenews.com/global/europe/2021/02/15/the-drone-defense-dilemma-how-unmanned-aircraft-are-redrawing-battle-lines/
“the balance of power between drones and air defense systems is shaping up to be a key to global wars in the near future.”

https://www.army.mil/article/242276/army_to_lead_new_dod_strategy_against_drone_attacks
“Defense Department unveiled a counter strategy during a media event Friday. The strategy calls for risk-based assessments and viewing counter-sUAS defense from a joint perspective to rapidly track, defend and defeat drone attacks.”

https://www.accipiterradar.com/products/safety/drone-uav-detection-tracking-alerting/
“tracking, post processing, and display coupled with on-the-fly and interactive analytics”

6.1 Multiple Phase Locked X-Y Rasters for Z-Dimension of 3D Drone Cloud Tracking

This use of GPU rasters would use 64 ARM variant processors each with four raster heads, to create a 256 Z-dimension of the same threat drone swarm cloud. Phase locking of the rasters would be required for the fasted overall all processing with all three planes X, Y, Z being in temporal coherence.

Figure 16

The processing of the three planes of changing data creates solutions, that can bac acted on. Phase locking of the rasters would be required for the fasted overall all processing with all three planes X, Y, Z being in temporal coherence. Purpose build raster IC’s and their SDRAM memories make a good choice for overall cost, wattage, small size of the electronics box. The raster method has the most predicable performance behavior competed to classic CPU-software solutions.

7. Many Graphic Sprites Overlay for Layers of Risk Zone Management

Modern GPU designs that are modified for vehicle driving, can also incorporate the 1970’s and 80’s video game systems for the sprite logic applied to virtual objects collision detection. The older systems were notoriously low on sprite counts, as large transistors sizes in IC’s allowed for the just the bare minimum for a functional video game.

Sprites of yesteryear were usually a small count of pixels on the X-Y coordinate system, such as 16 x 16, and of low pixel-bit depth, such as 4 bits. Several hyperlinks below review historical gaming sprite technology.

https://en.wikipedia.org/wiki/Sprite_(computer_graphics)
” In computer graphics, a sprite is a two-dimensional bitmap that is integrated into a larger scene, most often in a 2D video game. Originally, the term sprite referred to fixed-sized objects composited together, by hardware, with a background.[1] Use of the term has since become more general.

Systems with hardware sprites include arcade video games of the 1970s and 1980s; game consoles such as the Atari VCS (1977), ColecoVision (1982), Nintendo Entertainment System (1983), and Sega Genesis (1988); and home computers such as the Texas Instruments TI-99/4A (1979), Atari 8-bit family (1979), Commodore 64 (1982), MSX (1983), Amiga (1985), and X68000 (1987). Hardware varies in the number of sprites supported, the size and colors of each sprite, and special effects such as scaling or reporting pixel-precise overlap.

Hardware composition of sprites occurs as each scan line is prepared for the video output device, such as a CRT, without involvement of the main CPU and without the need for a full-screen frame buffer.[1] Sprites can be positioned or altered by setting attributes used during the hardware composition process. The number of sprites which can be displayed per scan line is often lower than the total number of sprites a system supports. For example, the Texas Instruments TMS9918 chip supports 32 sprites, but only 4 can appear on the same scan line.”

https://www.jstage.jst.go.jp/article/syntheng/6/2/6_94/_pdf
” [TTL video game limitations]
During this period, the hardware (TTL circuit) itself was responsible for realizing the game concept hardware engineers with an understanding of game concepts were few in number, and human resources were limited Initially, around 200 TTL circuits were sufficient for commercial video games, but the number of TTL circuits required increased as demand grew for more complex games with more on-screen objects, and hardware costs soon became prohibitive”

Now that these sprites can be updated in modern IC’s for large counts of Sprites, large sizes, and large pixel depths. This affords the sprites to be used not for virtual world games, but virtualized real world data processing, whereas the data resents actual physical objects.

7.1 Multiple Methods of Sprite Graphics Memory Function

Sprites work by several methods, of either raster pixels-in-line (X axis) counts and line counts (Y axis), that will real-time trigger on exact matches with the main background raster. Some sprite systems can also trigger with an additional logic layer of greater-than or less-than count comparisons to the background or to other sprites that are moving in the X-Y plane in reference to each other and the background.

Other sprite systems will real-time trigger on overlays of rasters or sub-raster-size zones of pixels, for each pixel value.

Figure 17

In this white paper method, multiple sprites are used, one on top of another, as to declare risk zones and act as a double check for vehicle safety control decisions at hardware processing speeds. As in other examples in this white paper, purpose built raster IC’s and SDRAMS can take on the task of large high speed refresh rate rasters, far better than a CPU software are able, for the repetitious periodic comparison of many sprites (that represent objects of any declared type).

In this case the concept method is to use sprite for moving and stationary objects. Typical non-moving objects are “road lane right side limit sprite” and road hazard height limit zone sprite, such as a low bridge, that “Center Vehicle” needs to pass under.

A large moving sprite can be a collision zone that surrounds the “Center Vehicle” on all sides, and is displaced, in the direction that “Center Vehicle” travels, as to provide more risk reaction time, in the forward direction. The spacing and size of such a closing safety zone is sized in pixels that represent needed reaction time. It could at the designed requirements take more factors into account.

The drawing here, shows one collision zone for the “Center Vehicle”. It could however be two zones or more, as smaller zones represent greater risk. The zones can be adjusted real-time, such as many human drivers, do, such as when they perceive another type of vehicle, such as a large truck, to have greater risk association. Another typical medication of the sprite zone would be the longitudinal (the long, forward travel axis of the road) gradient, whereas going downhill, required longer braking, and thus the collision safety zone would temporarily extend more pixels in that direction.

Overall, the “center vehicle” would typically be on the left side of the background pixel raster, as to balance out pixels for reaction time, in the entire scene of multiple objects of both stationary and moving types.

7.2 Multi Phase Locked Raster Overlay Color-Key Method for Risk Objects Processing

When any number of rasters can arbitrarily be added that are phase locked to timing accuracy of better than a single pixel, then GPU circuits that are lacking sprite features can alternatively uses color-key methods to sense geographic zones of risk that are portrayal to pixel in the X, Y 2D raster map. Real-time streaming pixels of a generic raster-A can be compared real-time to pixels of generic raster-B as detection logic.

In a somewhat similar fashion to sprites, 32 bit pixels can be visually display in test systems, or installed driving system, as 16 bit 6,5,5 RGB, leaving the reaming 16 bits of a pixel to resent additional data. In an example case, hypotenuse of X,Y raster positions would not need to be calculated for distance information , but rather only using concentric rings of risk pixels zones, that represent the mix of objects.

The center vehicle (CV) for instance could be in another raster, whereas the system designers can perform a multitude of new behaviors, actions and features on the CV, with real-time results for safety control reactions, and with real-time hardware low latency.

When the external objects (not the center vehicle) contact (occur in the same temporal pixel event period that is typical only a few nanoseconds) safety control outputs can be created. These can server to double check software anti-collision systems.

Figure 18

This next four hyperlink’s examples are of “HiSilicon” and “STmicroelectronics” brands video graphics IC’s that have overlay functions and clock gating functions that can operate in RGB-888 or RGB-565 pixel modes, or overlay modes with lock-to-an-external-video-source Hsync to provide a colorkey effect. Often these IC’s employ methods, rather than root-raster timing phase locking down to an accuracy of a pixel (typically a few nano seconds), rather operate as Asynchronous rasters, with large pixel FIFO’s that create pixel streams that can achieve frame, line and pixel locked timing.

https://www.96boards.org/documentation/consumer/hikey/hikey970/hardware-docs/files/hikey970-hi3670-v100-data-sheet.pdf

https://www.androidauthority.com/huawei-hisilicon-852231/
http://www.ic72.com/pdf_file/10pdf/7345.pdf

https://forum.videolan.org/viewtopic.php?t=51626

8. More PhaseLock Examples, Free IP, and other applicable IP

MiMax.com website has more white papers to assist with designs that can benefit from phase locking video rasters down to sub-pixel timing accuracy, of both custom GPU’s and COTS GPU’s. Pixel clock subtraction is Free.

8.1 Examples of Timing Locked Multi-Raster Applications, Free Pixel Clock Subtraction

Pixel clock subtraction for phase locking period devices is a free method further described in these links (that include digital logic timing diagrams)

https://mimax.com/optimizing-embedded-hardware-using-interleave-memory-phase-locking/

https://mimax.com/automotive-display-synchronization-with-phase-locking/

https://mimax.com/statistical-research-of-blockchain-data-with-gpu-hardware-acceleration/

https://mimax.com/wp-content/uploads/2021/12/US-Pat-6262695-Method-Appratus-for-Phase-Locking-Display-Divices-2001-July-17.pdf

Another common raster line and frame “lock” process often called “genlock” where a local oscillator creates a new (but rather unstable) pixel clock that obtains a corrective phase adjustment at the start of each new horizontal display line. Discussion of this method can be found in next hyperlink.

Don Lancaster’s Hardware Hacker Selected reprints — volume IV, Electronics Now series (January 1992 – January 1994)

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.445.3093&rep=rep1&type=pdf
“yes, that genlock. When you take any old pair of video sources and try to switch between them, you will get a horrible and useless glitch. The only way around this is to make sure that each of your video sources gets carefully locked to the exact same timing. By an exact lock, this means that all horizontal lines must start precisely at the same instant and last exactly the same time.”

8.2 Hyperlinks of Applicable Patents

These can be found at MiMax.com website. (some free/expired, filings & in-term)

Free (expired), US-patent 6,262,695 V-Lock (pixel clock subtraction).

Other IP of MiMax Inc is not free, however is very affordable to most non-profit organizations such as universities (at the present time, and this can potentially change, please contact Mimax inc). The other IP assists with video compression both pixel and macroblock, and realtime camera data difference detection.

US Pat No. 8,139,072 Network hardware graphics adapter compression (video feedback loop)
US Pat No. 8,441,493 Network hardware graphics adapter compression (multiple video feedback loops)
US Pat No. 10,499,072 Macrocell Display Compression Multi-Head-Raster-GPU
Patent applied-for of subject of Autodriving and Positive Train Control with use of PhaseLocked Rasters (in GPU’s)

Note “macroblocks” in this white paper, are treated the same as “macrocells” in some patents and papers. In the graphics chips industry there are other similar examples such as “video controller IC” and “graphics processor unit IC” being used to largely have the same meaning.

Internet hyperlinks of the MiMax patents and current filing(s) can be found at MiMax.com website. This applicable list is not exhaustive, as there may be patents or patent applications of other parties not known when this paper was written.

https://Mimax.com

9. Conclusion

This paper reviews alternative methods, for the purpose of research, that are more hardware oriented, with use of Graphics Processor Units IC’s inherent rasters when in phase locked period timing control, and Video Game Controllers in general their inherent “sprite” logic for an Automated Driving System (ADS) that the GitHub OpenPilot team refresh to as “locationd”.

These methods being very hardware oriented, and using Custom Off the Shelf (COTS) IC’s may reduce costs and when acting as a cross checking of the GitHub OpenPilot code, and can as an added layer of safety for collision avoidance. This paper proposes concepts of cross check verification GitHub OpenPilot code with hardware in sections, locationd, camerad, sensord.

Periodic rasters that were originally designed for video displays can scan memory faster than almost any IC’s types, and for the lowest cost and wattage. Further the phase locking of multiple rasters speeds up the raster to raster movement of temporally coherent data blocks. This is especially important to data representing moving objects: distances, and movement vectors, and real-time risk rating of objects, whereas driving safety is concerned.

Also, similar to video compression methods, rasters with Blitter Boolean functions and with FPGA added Boolean functions can make real time characteristic macroblock data from moving objects in the cameras.

Also, to review that same GPU Raster methods of this paper can also be conceptually applied to the reverse concept where collisions may be desired, such as a ship defending itself from a drone swarm.

Whenever pixel type processes of multiple rasters or sprites are employed in hardware, when the final comparisons of pixel data, from multiple rasters or multiple sprites, are performed with Boolean logic real-time, the pixel trimming is critical down to 1 pixel nanosecond accuracy or better.

Hardware sprites that are based on map data that are implemented in raster hardware systems may be useful as a safety cross check on lane-finding software that will not slow down the overall function of lane-find.

Logical numbered pixel (ie frame number, line number and pixel number) timing accuracy needs to be achieved either by a Fully Synchronous hardware raster/sprite system, or by Asynchronous large pixel FIFO’s, that can emulate a pixel, line and frame lock. This paper suggests pixel clock subtraction as the lowest cost method and most able to match pixel timing in a system of multiple blocks of rasters or sprites.