Google Patentsopen access

Embedding human labeler influences in machine learning interfaces in computing … — Intel Corporation (US11526713B2)

ABSTRACT

Abstract

A mechanism is described for facilitating embedding of human labeler influences in machine learning interfaces in computing environments, according to one embodiment. A method of embodiments, as described herein, includes detecting sensor data via one or more sensors of a computing device, and accessing human labeler data at one or more databases coupled to the computing device. The method may further include evaluating relevance between the sensor data and the human labeler data, where the relevance identifies meaning of the sensor data based on human behavior corresponding to the human labeler data, and associating, based on the relevance, human labeler data with the sensor data to classify the sensor data as labeled data. The method may further include training, based on the labeled data, a machine learning model to extract human influences from the labeled data, and embed one or more of the human influences in one or more environments representing one or more physical scenarios involving one or more humans.

Description

FIELD

Embodiments described herein relate generally to data processing and more particularly to facilitate embedding of human labeler influences in machine learning interfaces in computing environments.

BACKGROUND

In conventional techniques, human references are often used for naming objects in images, identifying sounds in audio clips, detecting activities in video clips, etc. However, such conventional techniques are severely limited in that they ignore a great deal of human behavior and other such variables, such as uncertainties of human conduct, preferences, biases, etc. Such deficiencies are carried over into conventional inference models, making today's machine learning models somewhat incomplete and inconclusive.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

FIG. 1 illustrates a computing device employing a human labeler mechanism according to one embodiment.

FIG. 2 illustrates a human labeler mechanism according to one embodiment.

FIG. 3 A illustrates a transaction sequence for embedding human labeler influence in machine learning models according to one embodiment.

FIG. 3 B illustrates a conventional early fusion scheme.

FIG. 3 C illustrates a conventional late fusion scheme.

FIG. 4 A illustrates an embodiment of a method for an early fusion-based embedding of human labeler influence in machine learning models and/or networks according to one embodiment.

FIG. 4 B illustrates an embodiment of a method for a late fusion-based embedding of human labeler influence in machine learning models and/or networks according to one embodiment.

FIG. 5 illustrates a computer device capable of supporting and implementing one or more embodiments according to one embodiment.

FIG. 6 illustrates an embodiment of a computing environment capable of supporting and implementing one or more embodiments according to one embodiment.

FIG. 7 illustrates a machine learning software stack according to one embodiment.

FIG. 8 A illustrates neural network layers according to one embodiment.

FIG. 8 B illustrates computation stages associated with neural network layers according to one embodiment.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth. However, embodiments, as described herein, may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Embodiments provide for a novel technique for embedding human labeler influences in machine learning interfaces in computing environments. In one embodiment, this novel technique is achieved through acquiring human labeler data (such as through sensors, historical data, categorical data associated with personal profile, etc.) and associating such human labeler data with sensor data for classifying human behaviors and variables and embedding such knowledge into machine learning models, deep learning neural networks, etc., to make such models and networks more intelligent so they may exact proper human behaviors, variables, etc. In one embodiment, human labeler data is used for detecting and considering human behavior and variables that occur at various times, such as before or during or after a final classification decision is made in training a neural network or other machine learning models.

It is contemplated that terms like “request”, “query”, “job”, “work”, “work item”, and “workload” may be referenced interchangeably throughout this document. Similarly, an “application” or “agent” may refer to or include a computer program, a software application, a game, a workstation application, etc., offered through an application programming interface (API), such as a free rendering API, such as Open Graphics Library (OpenGL®), DirectX® 11, DirectX® 12, etc., where “dispatch” may be interchangeably referred to as “work unit” or “draw” and similarly, “application” may be interchangeably referred to as “workflow” or simply “agent”. For example, a workload, such as that of a three-dimensional (3D) game, may include and issue any number and type of “frames” where each frame may represent an image (e.g., sailboat, human face). Further, each frame may include and offer any number and type of work units, where each work unit may represent a part (e.g., mast of sailboat, forehead of human face) of the image (e.g., sailboat, human face) represented by its corresponding frame. However, for the sake of consistency, each item may be referenced by a single term (e.g., “dispatch”, “agent”, etc.) throughout this document.

In some embodiments, terms like “display screen” and “display surface” may be used interchangeably referring to the visible portion of a display device while the rest of the display device may be embedded into a computing device, such as a smartphone, a wearable device, etc. It is contemplated and to be noted that embodiments are not limited to any particular computing device, software application, hardware component, display device, display screen or surface, protocol, standard, etc. For example, embodiments may be applied to and used with any number and type of real-time applications on any number and type of computers, such as desktops, laptops, tablet computers, smartphones, head-mounted displays and other wearable devices, and/or the like. Further, for example, rendering scenarios for efficient performance using this novel technique may range from simple scenarios, such as desktop compositing, to complex scenarios, such as 3D games, augmented reality applications, etc.

It is to be noted that terms or acronyms like convolutional neural network (CNN), CNN, neural network (NN), NN, deep neural network (DNN), DNN, recurrent neural network (RNN), RNN, and/or the like, may be interchangeably referenced throughout this document. Further, terms like “autonomous machine” or simply “machine”, “autonomous vehicle” or simply “vehicle”, “autonomous agent” or simply “agent”, “autonomous device” or “computing device”, “robot”, and/or the like, may be interchangeably referenced throughout this document.

FIG. 1 illustrates a computing device 100 employing a human labeler mechanism 110 according to one embodiment. Computing device 100 represents a communication and data processing device including or representing (without limitations) smart voice command devices, intelligent personal assistants, home/office automation system, home appliances (e.g., washing machines, television sets, etc.), mobile devices (e.g., smartphones, tablet computers, etc.), gaming devices, handheld devices, wearable devices (e.g., smartwatches, smart bracelets, etc.), virtual reality (VR) devices, head-mounted display (HMDs), Internet of Things (IoT) devices, laptop computers, desktop computers, server computers, set-top boxes (e.g., Internet-based cable television set-top boxes, etc.), global positioning system (GPS)-based devices, automotive infotainment devices, etc.

In some embodiments, computing device 100 includes or works with or is embedded in or facilitates any number and type of other smart devices, such as (without limitation) autonomous machines or artificially intelligent agents, such as a mechanical agents or machines, electronics agents or machines, virtual agents or machines, electro-mechanical agents or machines, etc. Examples of autonomous machines or artificially intelligent agents may include (without limitation) robots, autonomous vehicles (e.g., self-driving cars, self-flying planes, self-sailing boats, etc.), autonomous equipment (self-operating construction vehicles, self-operating medical equipment, etc.), and/or the like. Further, “autonomous vehicles” are not limed to automobiles but that they may include any number and type of autonomous machines, such as robots, autonomous equipment, household autonomous devices, and/or the like, and any one or more tasks or operations relating to such autonomous machines may be interchangeably referenced with autonomous driving.

Further, for example, computing device 100 may include a computer platform hosting an integrated circuit (“IC”), such as a system on a chip (“SoC” or “SOC”), integrating various hardware and/or software components of computing device 100 on a single chip.

As illustrated, in one embodiment, computing device 100 may include any number and type of hardware and/or software components, such as (without limitation) graphics processing unit (“GPU” or simply “graphics processor”) 114 , graphics driver (also referred to as “GPU driver”, “graphics driver logic”, “driver logic”, user-mode driver (UMD), UMD, user-mode driver framework (UMDF), UMDF, or simply “driver”) 116 , central processing unit (“CPU” or simply “application processor”) 112 , memory 108 , network devices, drivers, or the like, as well as input/output (I/O) sources 104 , such as touchscreens, touch panels, touch pads, virtual or regular keyboards, virtual or regular mice, ports, connectors, etc. Computing device 100 may include operating system (OS) 106 serving as an interface between hardware and/or physical resources of the computing device 100 and a user.

It is to be appreciated that a lesser or more equipped system than the example described above may be preferred for certain implementations. Therefore, the configuration of computing device 100 may vary from implementation to implementation depending upon numerous factors, such as price constraints, performance requirements, technological improvements, or other circumstances.

Embodiments may be implemented as any or a combination of: one or more microchips or integrated circuits interconnected using a parentboard, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, an application specific integrated circuit (ASIC), and/or a field programmable gate array (FPGA). The terms “logic”, “module”, “component”, “engine”, “circuitry”, “element”, and “mechanism” may include, by way of example, software, hardware and/or a combination thereof, such as firmware.

In one embodiment, as illustrated, human labeler mechanism 110 may be hosted by memory 108 in communication with I/O source(s) 104 , such as microphones, speakers, etc., of computing device 100 . In another embodiment, human labeler mechanism 110 may be part of or hosted by operating system 106 . In yet another embodiment, human labeler mechanism 110 may be hosted or facilitated by graphics driver 116 . In yet another embodiment, human labeler mechanism 110 may be hosted by or part of graphics processing unit (“GPU” or simply graphics processor”) 114 or firmware of graphics processor 114 . For example, human labeler mechanism 110 may be embedded in or implemented as part of the processing hardware of graphics processor 114 . Similarly, in yet another embodiment, human labeler mechanism 110 may be hosted by or part of central processing unit (“CPU” or simply “application processor”) 112 . For example, human labeler mechanism 110 may be embedded in or implemented as part of the processing hardware of application processor 112 .

In yet another embodiment, human labeler mechanism 110 may be hosted by or p

FIELD

Embodiments described herein relate generally to data processing and more particularly to facilitate embedding of human labeler influences in machine learning interfaces in computing environments.

BACKGROUND

In conventional techniques, human references are often used for naming objects in images, identifying sounds in audio clips, detecting activities in video clips, etc. However, such conventional techniques are severely limited in that they ignore a great deal of human behavior and other such variables, such as uncertainties of human conduct, preferences, biases, etc. Such deficiencies are carried over into conventional inference models, making today's machine learning models somewhat incomplete and inconclusive.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

FIG. 1 illustrates a computing device employing a human labeler mechanism according to one embodiment.

FIG. 2 illustrates a human labeler mechanism according to one embodiment.

FIG. 3 A illustrates a transaction sequence for embedding human labeler influence in machine learning models according to one embodiment.

FIG. 3 B illustrates a conventional early fusion scheme.

FIG. 3 C illustrates a conventional late fusion scheme.

FIG. 4 A illustrates an embodiment of a method for an early fusion-based embedding of human labeler influence in machine learning models and/or networks according to one embodiment.

FIG. 4 B illustrates an embodiment of a method for a late fusion-based embedding of human labeler influence in machine learning models and/or networks according to one embodiment.

FIG. 5 illustrates a computer device capable of supporting and implementing one or more embodiments according to one embodiment.

FIG. 6 illustrates an embodiment of a computing environment capable of supporting and implementing one or more embodiments according to one embodiment.

FIG. 7 illustrates a machine learning software stack according to one embodiment.

FIG. 8 A illustrates neural network layers according to one embodiment.

FIG. 8 B illustrates computation stages associated with neural network layers according to one embodiment.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth. However, embodiments, as described herein, may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Embodiments provide for a novel technique for embedding human labeler influences in machine learning interfaces in computing environments. In one embodiment, this novel technique is achieved through acquiring human labeler data (such as through sensors, historical data, categorical data associated with personal profile, etc.) and associating such human labeler data with sensor data for classifying human behaviors and variables and embedding such knowledge into machine learning models, deep learning neural networks, etc., to make such models and networks more intelligent so they may exact proper human behaviors, variables, etc. In one embodiment, human labeler data is used for detecting and considering human behavior and variables that occur at various times, such as before or during or after a final classification decision is made in training a neural network or other machine learning models.

It is contemplated that terms like “request”, “query”, “job”, “work”, “work item”, and “workload” may be referenced interchangeably throughout this document. Similarly, an “application” or “agent” may refer to or include a computer program, a software application, a game, a workstation application, etc., offered through an application programming interface (API), such as a free rendering API, such as Open Graphics Library (OpenGL®), DirectX® 11, DirectX® 12, etc., where “dispatch” may be interchangeably referred to as “work unit” or “draw” and similarly, “application” may be interchangeably referred to as “workflow” or simply “agent”. For example, a workload, such as that of a three-dimensional (3D) game, may include and issue any number and type of “frames” where each frame may represent an image (e.g., sailboat, human face). Further, each frame may include and offer any number and type of work units, where each work unit may represent a part (e.g., mast of sailboat, forehead of human face) of the image (e.g., sailboat, human face) represented by its corresponding frame. However, for the sake of consistency, each item may be referenced by a single term (e.g., “dispatch”, “agent”, etc.) throughout this document.

In some embodiments, terms like “display screen” and “display surface” may be used interchangeably referring to the visible portion of a display device while the rest of the display device may be embedded into a computing device, such as a smartphone, a wearable device, etc. It is contemplated and to be noted that embodiments are not limited to any particular computing device, software application, hardware component, display device, display screen or surface, protocol, standard, etc. For example, embodiments may be applied to and used with any number and type of real-time applications on any number and type of computers, such as desktops, laptops, tablet computers, smartphones, head-mounted displays and other wearable devices, and/or the like. Further, for example, rendering scenarios for efficient performance using this novel technique may range from simple scenarios, such as desktop compositing, to complex scenarios, such as 3D games, augmented reality applications, etc.

It is to be noted that terms or acronyms like convolutional neural network (CNN), CNN, neural network (NN), NN, deep neural network (DNN), DNN, recurrent neural network (RNN), RNN, and/or the like, may be interchangeably referenced throughout this document. Further, terms like “autonomous machine” or simply “machine”, “autonomous vehicle” or simply “vehicle”, “autonomous agent” or simply “agent”, “autonomous device” or “computing device”, “robot”, and/or the like, may be interchangeably referenced throughout this document.

FIG. 1 illustrates a computing device 100 employing a human labeler mechanism 110 according to one embodiment. Computing device 100 represents a communication and data processing device including or representing (without limitations) smart voice command devices, intelligent personal assistants, home/office automation system, home appliances (e.g., washing machines, television sets, etc.), mobile devices (e.g., smartphones, tablet computers, etc.), gaming devices, handheld devices, wearable devices (e.g., smartwatches, smart bracelets, etc.), virtual reality (VR) devices, head-mounted display (HMDs), Internet of Things (IoT) devices, laptop computers, desktop computers, server computers, set-top boxes (e.g., Internet-based cable television set-top boxes, etc.), global positioning system (GPS)-based devices, automotive infotainment devices, etc.

In some embodiments, computing device 100 includes or works with or is embedded in or facilitates any number and type of other smart devices, such as (without limitation) autonomous machines or artificially intelligent agents, such as a mechanical agents or machines, electronics agents or machines, virtual agents or machines, electro-mechanical agents or machines, etc. Examples of autonomous machines or artificially intelligent agents may include (without limitation) robots, autonomous vehicles (e.g., self-driving cars, self-flying planes, self-sailing boats, etc.), autonomous equipment (self-operating construction vehicles, self-operating medical equipment, etc.), and/or the like. Further, “autonomous vehicles” are not limed to automobiles but that they may include any number and type of autonomous machines, such as robots, autonomous equipment, household autonomous devices, and/or the like, and any one or more tasks or operations relating to such autonomous machines may be interchangeably referenced with autonomous driving.

Further, for example, computing device 100 may include a computer platform hosting an integrated circuit (“IC”), such as a system on a chip (“SoC” or “SOC”), integrating various hardware and/or software components of computing device 100 on a single chip.

As illustrated, in one embodiment, computing device 100 may include any number and type of hardware and/or software components, such as (without limitation) graphics processing unit (“GPU” or simply “graphics processor”) 114 , graphics driver (also referred to as “GPU driver”, “graphics driver logic”, “driver logic”, user-mode driver (UMD), UMD, user-mode driver framework (UMDF), UMDF, or simply “driver”) 116 , central processing unit (“CPU” or simply “application processor”) 112 , memory 108 , network devices, drivers, or the like, as well as input/output (I/O) sources 104 , such as touchscreens, touch panels, touch pads, virtual or regular keyboards, virtual or regular mice, ports, connectors, etc. Computing device 100 may include operating system (OS) 106 serving as an interface between hardware and/or physical resources of the computing device 100 and a user.

It is to be appreciated that a lesser or more equipped system than the example described above may be preferred for certain implementations. Therefore, the configuration of computing device 100 may vary from implementation to implementation depending upon numerous factors, such as price constraints, performance requirements, technological improvements, or other circumstances.

Embodiments may be implemented as any or a combination of: one or more microchips or integrated circuits interconnected using a parentboard, hardwired logic, software stored by a memory device and executed by a microprocessor, firmware, an application specific integrated circuit (ASIC), and/or a field programmable gate array (FPGA). The terms “logic”, “module”, “component”, “engine”, “circuitry”, “element”, and “mechanism” may include, by way of example, software, hardware and/or a combination thereof, such as firmware.

In one embodiment, as illustrated, human labeler mechanism 110 may be hosted by memory 108 in communication with I/O source(s) 104 , such as microphones, speakers, etc., of computing device 100 . In another embodiment, human labeler mechanism 110 may be part of or hosted by operating system 106 . In yet another embodiment, human labeler mechanism 110 may be hosted or facilitated by graphics driver 116 . In yet another embodiment, human labeler mechanism 110 may be hosted by or part of graphics processing unit (“GPU” or simply graphics processor”) 114 or firmware of graphics processor 114 . For example, human labeler mechanism 110 may be embedded in or implemented as part of the processing hardware of graphics processor 114 . Similarly, in yet another embodiment, human labeler mechanism 110 may be hosted by or part of central processing unit (“CPU” or simply “application processor”) 112 . For example, human labeler mechanism 110 may be embedded in or implemented as part of the processing hardware of application processor 112 .

In yet another embodiment, human labeler mechanism 110 may be hosted by or part of any number and type of components of computing device 100 , such as a portion of human labeler mechanism 110 may be hosted by or part of operating system 116 , another portion may be hosted by or part of graphics processor 114 , another portion may be hosted by or part of application processor 112 , while one or more portions of human labeler mechanism 110 may be hosted by or part of operating system 116 and/or any number and type of devices of computing device 100 . It is contemplated that embodiments are not limited to certain implementation or hosting of human labeler mechanism 110 and that one or more portions or components of human labeler mechanism 110 may be employed or implemented as hardware, software, or any combination thereof, such as firmware.

Computing device 100 may host network interface device(s) to provide access to a network, such as a LAN, a wide area network (WAN), a metropolitan area network (MAN), a personal area network (PAN), Bluetooth, a cloud network, a mobile network (e.g., 3 rd Generation (3G), 4 th Generation (4G), etc.), an intranet, the Internet, etc. Network interface(s) may include, for example, a wireless network interface having antenna, which may represent one or more antenna(e). Network interface(s) may also include, for example, a wired network interface to communicate with remote devices via network cable, which may be, for example, an Ethernet cable, a coaxial cable, a fiber optic cable, a serial cable, or a parallel cable.

Embodiments may be provided, for example, as a computer program product which may include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines such as a computer, network of computers, or other electronic devices, may result in the one or more machines carrying out operations in accordance with embodiments described herein. A machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs (Compact Disc-Read Only Memories), and magneto-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable Read Only Memories), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, or other type of media/machine-readable medium suitable for storing machine-executable instructions.

Moreover, embodiments may be downloaded as a computer program product, wherein the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of one or more data signals embodied in and/or modulated by a carrier wave or other propagation medium via a communication link (e.g., a modem and/or network connection).

Throughout the document, term “user” may be interchangeably referred to as “viewer”, “observer”, “speaker”, “person”, “individual”, “end-user”, and/or the like. It is to be noted that throughout this document, terms like “graphics domain” may be referenced interchangeably with “graphics processing unit”, “graphics processor”, or simply “GPU” and similarly, “CPU domain” or “host domain” may be referenced interchangeably with “computer processing unit”, “application processor”, or simply “CPU”.

It is to be noted that terms like “node”, “computing node”, “server”, “server device”, “cloud computer”, “cloud server”, “cloud server computer”, “machine”, “host machine”, “device”, “computing device”, “computer”, “computing system”, and the like, may be used interchangeably throughout this document. It is to be further noted that terms like “application”, “software application”, “program”, “software program”, “package”, “software package”, and the like, may be used interchangeably throughout this document. Also, terms like “job”, “input”, “request”, “message”, and the like, may be used interchangeably throughout this document.

FIG. 2 illustrates human labeler mechanism 110 of FIG. 1 according to one embodiment. For brevity, many of the details already discussed with reference to FIG. 1 are not repeated or discussed hereafter. In one embodiment, human labeler mechanism 110 may include any number and type of components, such as (without limitations): detection and monitoring logic 201 ; classification logic 203 ; evaluation and filtering logic 205 ; model creation and training logic 207 ; communication/ compatibility logic 209 ; and scoring logic 211 .

Computing device 100 is further shown to include user interface 219 (e.g., graphical user interface (GUI)-based user interface, Web browser, cloud-based platform user interface, software application-based user interface, other user or application programming interfaces (APIs), etc.). Computing device 100 may further include I/O source(s) 108 having input component(s) 231 , such as camera(s) 242 (e.g., Intel® RealSense™ camera), sensors, microphone(s) 241 , etc., and output component(s) 233 , such as display device(s) or simply display(s) 244 (e.g., integral displays, tensor displays, projection screens, display screens, etc.), speaker devices(s) or simply speaker(s), etc.

Computing device 100 is further illustrated as having access to and/or being in communication with one or more database(s) 225 and/or one or more of other computing devices over one or more communication medium(s) 230 (e.g., networks such as a proximity network, a cloud network, the Internet, etc.).

In some embodiments, database(s) 225 may include one or more of storage mediums or devices, repositories, data sources, etc., having any amount and type of information, such as data, metadata, etc., relating to any number and type of applications, such as data and/or metadata relating to one or more users, physical locations or areas, applicable laws, policies and/or regulations, user preferences and/or profiles, security and/or authentication data, historical and/or preferred details, and/or the like.

As aforementioned, computing device 100 may host I/ O sources 108 including input component(s) 231 and output component(s) 233 . In one embodiment, input component(s) 231 may include a sensor array including, but not limited to, microphone(s) 241 (e.g., ultrasound microphones), camera(s) 242 (e.g., two-dimensional (2D) cameras, three-dimensional (3D) cameras, infrared (IR) cameras, depth-sensing cameras, etc.), capacitors, radio components, radar components, scanners, and/or accelerometers, etc. Similarly, output component(s) 233 may include any number and type of display device(s) 244 , projectors, light-emitting diodes (LEDs), speaker(s) 243 , and/or vibration motors, etc.

As aforementioned, terms like “logic”, “module”, “component”, “engine”, “circuitry”, “element”, and “mechanism” may include, by way of example, software or hardware and/or a combination thereof, such as firmware. For example, logic may itself be or include or be associated with circuitry at one or more devices, such as application processor 112 and/or graphics processor 114 of FIG. 1 , to facilitate or execute the corresponding logic to perform certain tasks.

For example, as illustrated, input component(s) 231 may include any number and type of microphones(s) 241 , such as multiple microphones or a microphone array, such as ultrasound microphones, dynamic microphones, fiber optic microphones, laser microphones, etc. It is contemplated that one or more of microphone(s) 241 serve as one or more input devices for accepting or receiving audio inputs (such as human voice) into computing device 100 and converting this audio or sound into electrical signals. Similarly, it is contemplated that one or more of camera(s) 242 serve as one or more input devices for detecting and capturing of image and/or videos of scenes, objects, etc., and provide the captured data as video inputs into computing device 100 .

As previously described, human references may be used for labeling data for training neural networks, such as to name an object in an image, identify a sound in an audio clip, detect an activity in a video clip, etc. In some cases, mechanical turk is used as a service for such labeling. For example, after human labeling is gathered, heuristics may be used to select the data in training, such as in some cases, a minimum percent of agreement across labelers may be desired. Nevertheless, conventional techniques ignore or disregard a great deal of human behavior (and other variables), such as what if a labeler takes longer than normal time to decide on a classification that has potential meaning in the inference model that is created. It is contemplated that Decision Field Theory and other models of decision-making have long demonstrated and modelled how uncertainty leads to longer decision-making time, where this uncertainty in labelling may be used in creation of inference models, such as observations like lack of diversity in race or gender may be used to impact an inference model.

A well-known example of Frogger has demonstrated that humans think aloud, such as while playing Frogger, to create a corpus of utterances for an explainable artificial intelligence (AI) interface to use to indicate what a person may say when making the same decision and/or choice as the AI that the person was trained to play. For example, any utterances and training of AI to play may be done separately, such as a player not using the epochs that created the machine learning to play the game. However, the Frogger example does not include any true evaluation or judgement from the labelers in the inference model, but just an associated utterance.

Another known technique is eye-tracking, which has been used to annotate object locations within an image to create bounding boxes, such as how long a fixation lasts may also signify its importance. Based on derivatives of any fixation points (with humans giving instructions to look for an object type), objects are delineated to establish ground through in an automated way. The idea of monitoring human reactions to label data can be used and applied in several situations, such as monitoring emotional reactions to train a robot.

Further, semantic and categorical data may be fused with sensor data (e.g., video streams, images, sounds, etc., of scenes), such as data captured through camera(s) 242 , microphone(s) 241 , accelerometers, and other such sensors and detectors, in creating machine learning interference model, where, for example, semantic data fusion may be divided into multiple categories of methods, such as: multi-view learning-based method, similarity-based method, probabilistic dependency-based method, and transfer learning-based method. Such methods may be used for knowledge fusion as opposed to schema mapping and data merging, which distinguishes between cross-domain data fusion and traditional data fusion. For example, if a CNN is trained based on images that contain an object of interest, a second stage of fusion (e.g., weighting with uncertainty data, using probabilistic dependency (or one of the other fusion types) may produce the final model and thus integrating the human behavior into the model.

For deep learning, multimodal fusion is becoming common. As will be further illustrated and described with reference to FIGS. 4 A- 4 B , a solution for utterances of the labeler may be standard audio/video fusion approaches with CNN and RNN combined models being fused early or late. Further, image content and textual description-based detection fusion may be used, such as relying on vector concatenation for both early and late fusion schemes to obtain a multimodal representation, while, in some cases, for late fusion, a probabilistic outcome score may be concatenated after visual analysis as facilitated by scoring logic 211 . This probabilistic score may result from textual analysis and kept for future use when analyzing the same or similar human behaviors and/or variables for certain machine learning models and neural networks.

Embodiments provide for a novel technique for detecting and determining human behaviors and variables around conscious data labeling tasks, such as through human labeler data obtained through sensors, historical data, categorical data, personal profile data, etc., as facilitated by human labeler mechanism 110 . For example, distinct from eye tracking or emotional reactions, embodiments provide for a novel technique for showing a labeling decision through latency, such as repeatedly playing a sample before classifying, or other similar specific behaviors as facilitated by human labeler mechanism 110 . Alternatively, for example, embodiments may deal with tackling various human behaviors (or personal variables like gender) that precede a classification decision, where human labeler behaviors and/or variables or their influences are added actual machine learning model, neural networks, etc., as facilitated by human labeler mechanism 110 .

In one embodiment, detection and monitoring logic 201 may be used to detect, observe, and/or monitor various human labeler data associated with human behaviors and/or variables that can occur before, during, and/or after a final classification decision in training neural networks and/or other machine learning models. Similarly, detection and monitoring logic 201 is used to detect, observe, and/or monitor sensor data by facilitating one or more sensor and/or detectors, such as camera(s) 242 , microphone(s) 241 , touch sensors (e.g., touch pads, touch panels, etc.), capacitors, radio components, radar components, scanners, and accelerometers, etc. to capture objects (e.g., living beings (e.g., persons, animals, plants, etc.), non-living things (vehicles, furniture, rocks, etc.)) within a scene, whether be indoors or outdoors, where such capture includes video streams, images, sounds, etc. For example, sensor data may include information obtained through direct observation of a person, such as capturing images through camera(s) 242 , voices or sounds through microphone(s) 241 , movements through other sensors, etc. In one embodiment, as will be further described later in this document, sensor data may be classified based on human labeler data, where this human labeler data is used as a reference to define and associate various human behaviors and variables corresponding to various portions of the sensor data.

For example, once human labeler data is detected and/or monitored by detection and monitoring logic 201 , classification logic 203 may then be triggered to classify any sensor data based on the human labeler data to for the various portions of the sensor data to be meaningful in terms of creating and/or training machine learning models, neural networks, etc., so the models/networks may intelligently incorporate and apply human behaviors and/or variables in their inferences and outputs.

In one embodiment, upon detecting, obtaining, and/or monitoring of human labeler data by detection and monitoring logic 201 , the sensor data is then classified based on the human labeler data to reflect the various human behaviors and other variables. For example, the following are some of the examples of potentially meaningful human labeler data that reflect human behaviors or other observable variables: 1) time to decide on a label may indicate visual difficulty for a person, which may have relevance, such as red green blue (RGB)-based autonomous recognition; 2) tilting of head, squinting of eyes, closing of eyes, repeatedly listening when classifying an audio sample by indicating more difficult decisions, etc.; 3) “extraneous” verbalizations (e.g., “ummm”, “huh”, etc.) during visual classification may indicate a certain feeling or reservation, such as hesitation in performing an act or being surprised at seeing something, etc.

Similarly, other examples may include: 4) human labelers being instructed to “think aloud” while classifying data to increase the available verbal data (e.g., “this looks iffy”, “maybe this is a cat”, etc.). For example, unlike the Frogger, in one embodiment, nature of utterances are evaluated during actual labeling to then be used in an inference model training; 5) rate of movement of a person when using touch input or delineated objects; 6) number of sniffs taken by a person before classifying a smell; 7) other variables associated with a person, such as race, gender, nationality, location, country, etc.

It is contemplated that observable variables like race, gender, nationality, ethnicity, religious affiliation, political leaning, sexual orientation, etc., may also involve other considerations or classifications, such as bias, discrimination, and other similar ethical or moral AI issues. As will be further described in this document, evaluation and filtering logic 205 may be triggered to deal with such variables that can trigger discrimination or bias. For example, evaluation and filtering logic 205 may, in one embodiment, filter out such variables, such as if they are considered minor or irrelevant or nuisance, or instead, in another embodiment, incorporate and apply such variables for proper creation and/or training of machine learning models, neural networks. For example, model creation and training logic 207 may provide the pertinent training to ensure such variables are intelligently inferred and applied by machine learning models, neural networks, etc., to provide realistic outcomes that are not tainted by unintentional bias.

In one embodiment, upon classification of sensor data based on human labeler data, evaluation and filtering logic 205 may then be triggered to evaluate and determine the value or usefulness of the classified sensor data and the associated human labeler data in terms of their associated and classified human behaviors and/or variables for applying them to machine learning models, deep learning neural networks. For example, evaluation and filtering logic 205 may determine that less conclusive human labeler data samples (such as ones where the labeler data shows more doubt, hesitation, etc.) may have or be assigned lesser weight when creating an inference to be performed in a simplistic application. Similarly, if an inference is performed in complex situations, a machine learning model may be created with more challenging samples (such as where labelers took longer to decide) to be desirable and conclusive. For an individual inference model, the use or non-use of human labeler data may be tested with sample data.

Further, as described above, evaluation and filtering logic 205 may be used to filter out one or more of the human variables associated with one or more of inaccuracies, unintended consequences, and biases, such as variables based on one or more of age, gender, race, ethnicity, national origin, religion, religious affiliation, political leanings, sexual orientation, and/or the like. The filtered out human variables may further include accidental acts, coincidental items, etc., or when a co-variate is treated as a nuisance variable to be filtered out.

For example, since certain extraneous human behaviors or variables associated with human labeler data may potentially harm the training of a machine learning model, neural network, etc., such as if someone leaves a cup of coffee for 5 minutes, that 5-minute wait may be regarded as “hesitation”, in one embodiment, evaluation and filtering logic 205 may evaluate and choose to filter out this hesitation so it does not unnecessarily influence the model or neural network.

However, it is contemplated that such variables may be filtered out if they seem minor or irrelevant and that in some embodiment, these variables are considered, evaluated, and applied for correspondingly intelligent results. For example, evaluation and filtering logic 205 evaluates if there are labelers that are all of Race A, where they may tend to bias their labeling against persons of Race B, even if unintentionally. For example, if a labeler's purpose is to circle a shoplifter in an image, using the race of the labeler (e.g., Race A) and that of the labeled (e.g., Race B) may be tainted with societal or unintentional bias, such as the observation of Race A may bias the observation of the person of Race B in the image.

Accordingly, in one embodiment, evaluation and filtering logic 205 evaluates and adjusts this co-variate to have some correspondence and sense between the two variables associated with the labeler and the labeled and communicates this adjustment (such as to take the potential bias into consideration) to model creation and training logic 207 so that adjustment is incorporated and applied in machine learning models, neural networks, etc., for intelligent and realistic inference analysis and resulting outcomes.

Similarly, in one embodiment, evaluation and filtering logic 205 may then be used to determine whether any of the labeled and classified sensor data may be used to add predictivity to machine learning models, neural networks, etc., by regarding this as another source of data. For example, evaluation and filtering logic 205 possess intelligence to vary the usefulness of any labeled and classified data based on the its application to and output from machine learning models and/or neural networks. For example, a more explainable AI (XAI) may be enabled and used by allowing such machine learning models, neural networks, etc., with different labeler demographics or other potential biases to be compared for different uses and on different data sets. This may be regarded as an ongoing way of enabling a form of XAI, such as without having to change or fix the entire XAI challenge.

Further, in distinguishing from the conventional eye tracking techniques, embodiments provide for determining latency in showing a decision, repeatedly playing a sample before classifying, or other specific behaviors and variables as facilitated by evaluation and filtering logic 205 . Alternatively, embodiments provide for a novel technique for determining human labeler data-based behaviors and/or variables that precede classification decisions (as opposed to behaviors like eye movements) and/or uncertainly in behavior and use of an uncertainty estimate as facilitated by evaluation and filtering logic 205 .

In one embodiment, an uncertainly estimate refers to a level of uncertainty associated with each labeler, which is measured by evaluation and filtering logic 205 in any number of ways, depending on which way is regarded as most suitable for an application, such as (but not limited to): 1) response time in making a classification for an image or a sound; 2) presence or absence of a degree of eye squint (or head tilt, movement of head toward screen, etc.) while making a classification of an image or a sound; 3) number of extraneous verbalizations during visual classifications; 4) touch input time to delineate an object; 5) number of sniffs to classify a smell; and 6) count of evaluative words, like “similar”, “difficult”, “cannot tell”, “same”, etc.

Further, in one embodiment, uncertainty may not be the only variable of interest for human labeler tracking based on human labeler data, as other examples may include (but not limited to): 1) emotional response as detected by facial expression; 2) heart rate, heart rate variability, galvanic skin response, etc., to indicate excitement level; and 3) variables about the labeler, such as race, age, gender, nationality, country of origin, and other similar aspects that may cause or be associated with bias in labelling.

For example, evaluation and filtering logic 205 may evaluate labeled data based on researched and/or considered well-known human behaviors and/or variables, such as a mean response time per image (such as the time from the onset of the image to the pressing of the button) may be 889 ms for humans, etc., while drawing a bounding box has been shown to take 26 seconds, etc. Stated differently, there can be a wide range in labeler response time (RT) depending on the task. While human RT follows slightly skewed distributions (though close to normal), trimming data with standard deviation may be easily done. For example, any times that are 2 STDV above the mean RT may be filtered out and as for verbalizations, gestures, or other explicit behaviors that indicate hesitation, behaviors of interest may be defined for that specific task. For repeated plays of audio or video contents, simple frequencies may be tracked, with extreme repetitions above an appropriate threshold being filtered out. Further, in cases where human labeler data is absent (e.g., no facial expression change), some fusion approaches are regarded as more tolerant of sparse data.

In one embodiment, model creation and training logic 207 may then be triggered to apply the findings and evaluations of labeled/classified sensor data, such as based on labeled data-based behaviors and/or variables, to generate and/or train machine learning models/neural networks to allow for valuable and conclusive and reliable machine learning models, deep learning neural networks, etc. For example, in one embodiment, to create a new machine learning model, model creation and training logic 207 may be used to simply create multiple machine learning inference models using labeled sensor data from the specific measures of uncertainty, such as using human labeler “quick decision” data.

In some embodiments, such as in case of late fusion, after any models are trained separately for and using sensor data and labeler data being stored and maintained at database(s) 225 , scoring logic 211 may then be triggered to compute a score for each portion or element of human labeler data based on averaging the outcomes of each machine learning model or neural network based on the human labeler data. For example, such scores may be kept and maintained, such as at one or more database(s) 225 , to then be used in the future, such as when training a machine learning model, when similar or the same human labeler data or behavior/variable is encountered to add efficiency and preservation of system resources to the process.

Further, in one embodiment, semantic data may be used with sensor data to build machine learning models, neural networks, based on heterogenous data, such as after training a model or a neural network on features, another stage using semantic data may be used to create a machine learning model with semantic variables, as facilitated by model creation and training logic 207 . Categories of semantic fusion may include: multi-view-based, similarity-based, probabilistic dependency-based, and transfer-learning-based methods. Further, in another embodiment, a fused model may be generated by model creation and training logic 207 using early fusion and late fusion approaches, where human labeler data is treated as just another source of sensor data (e.g., audio, visual, audio-visual, etc.).

Considering an example, sensor data may be detected using one or more sensors, such as camera(s) 242 , microphone(s) 241 , of computing device 100 , while human labeler data is accessed at database(s) 225 , as facilitated by detection and monitoring logic 201 . In one embodiment, evaluation and filtering logic 205 may be triggered to evaluate any relevance between the sensor data and the human labeler data such that the relevance identifies the meaning of the sensor data based on human behavior corresponding to the human labeler data. In one embodiment, classification logic 203 may then be triggered to associate, based on the relevance, the human labeler data with the sensor data to classify the sensor data as labeled data, while model creation and training logic 207 is then uses the labeled data to facilitate training of a machine learning model to extract human influences from the labeled data, and embed one or more of the human influences in one or more environments representing one or more physical scenarios involving one or more humans.

Continuing with the example, training may be used to facilitate the machine learning model to interpret, based on the labeled data, the human influences according to multiple environments prior to embedding the one or more human influences in the one or more environments, where, for example, the interpretation of the human influences is based on acceptances of the human behavior and exceptions to the human behavior as derived from the labeled data and based on the relevance.

For example, the acceptances of the human behavior are based on verified data obtained from one or more of personal profiles, cultural traits, historical norms, societal preferences, personal prejudices, societal biases, habits, etc. For example, the exceptions to the human behavior are based on unverified data obtained from one or more of coincidences, accidents, inaccuracies, flukes, unintended consequences, etc., such that one or more of the human behaviors may be filtered out based on one or more of the exceptions to avoid associating inaccuracies to the human influences.

For example, the relevance is further based on a human-v

CLAIMS

Claims ( 20 )

What is claimed is:

1. At least one machine-readable medium comprising instructions which, when executed by a computing device, cause the computing device to perform operations comprising:

classifying, by one or more processors of the computing device, sensor data with human labeler data, where the sensor data is obtained through one or more sensors communicably coupled to the one or more processors; and

creating and training, by the one or more processors, a unified machine learning model based on features associated with the classified sensor data based on the human labeler data, wherein the features comprise human labeler influences as obtained from the human labeler data associated with the sensor data;

wherein training is further to facilitate the unified machine learning model to interpret, based on the human labeler data, the human labeler influences according to one or more environments prior to embedding the one or more human labeler influences in the one or more environments, wherein the interpretation of the human influences is based on acceptances of human behavior and exceptions to the human behavior as derived from the human labeler data and based on a relevance between the sensor data and the human labeler data, wherein the relevance identifies meaning of the sensor data based on human behavior corresponding to the human labeler data, and wherein the acceptances of the human behavior are based on verified data obtained from one or more of personal profiles, cultural traits, historical norms, societal preferences, personal prejudices, societal biases, or habits.

2. The machine-readable medium of claim 1 , wherein the operations further comprise:

detecting the sensor data through the one or more sensors including one or more of a camera, a microphone, a touch sensor, a capacitor, a radio component, a radar component, a scanner, and an accelerometer; and

monitoring the human labeler data to determine one or more of human behaviors and human variables, wherein the human labeler data is obtained through multiple sources including one or more of the one or more sensors, historical data, categorical data, and personal profiles.

3. The machine-readable medium of claim 2 , wherein the operations further comprise prior to classifying the sensor data, evaluate the human behaviors and human variables and their association with the sensor data.

4. The machine-readable medium of claim 3 , wherein the operations further comprise:

filtering out one or more of the human variables associated with one or more of inaccuracies, unintended consequences, and biases, wherein the filtered out one or more human variables include one or more of age, gender, race, ethnicity, national origin, religion, and sexual orientation, and wherein the filtered out one or more human variables further include one or more of accidental acts and coincidental items; and

recognizing and considering one or more variances between first human variables and second human variables in application of the first and second human variables in training the machine learning model.

5. The machine-readable medium of claim 1 , wherein the unified machine learning model is created and trained in an early fusion machine learning environment.

6. The machine-readable medium of claim 1 , wherein the operations further comprise creating and training multiple machine learning models such that each of the multiple machine learning models is based on first features of the features associated with the sensor data or second features of the features associated with human labeler data.

7. The machine-readable medium of claim 6 , wherein the operations further comprise computing scores based on average outcomes obtained from the multiple machine learning models associated with the sensor data and the human labeler data, wherein the scores are maintained in one or more databases to be used with creation and training of future machine learning models, wherein the computing device includes one or more processors comprising one or more of a graphics processor and an application processor, wherein the graphics processor and the application processor are co-located on a common semiconductor package.

8. A method comprising:

detecting sensor data via one or more sensors of a computing device;

accessing human labeler data at one or more databases coupled to the computing device;

evaluating relevance between the sensor data and the human labeler data, wherein the relevance identifies meaning of the sensor data based on human behavior corresponding to the human labeler data;

associating, based on the relevance, human labeler data with the sensor data to classify the sensor data as labeled data; and

training, based on the labeled data, a machine learning model to extract human influences from the human labeler data, and embed one or more of the human influences in one or more environments representing one or more physical scenarios involving one or more humans;

wherein training is further to facilitate the machine learning model to interpret, based on the human labeler data, the human influences according to multiple environments prior to embedding the one or more human influences in the one or more environments, wherein the interpretation of the human influences is based on acceptances of the human behavior and exceptions to the human behavior as derived from the human labeler data and based on the relevance, and wherein the acceptances of the human behavior are based on verified data obtained from one or more of personal profiles, cultural traits, historical norms, societal preferences, personal prejudices, societal biases, or habits.

9. The method of claim 8 , wherein the exceptions to the human behavior are based on unverified data obtained from one or more of coincidences, accidents, inaccuracies, flukes, and unintended consequences; and

wherein one or more of the human behaviors are filtered out based on one or more of the exceptions to avoid associating inaccuracies to the human influences.

10. The method of claim 8 , wherein the relevance is further based on a human-variables portion of the human behavior, wherein the human-variables portion is based on human variables that incite personal prejudices or societal biases, wherein the human variables include one or more of age, gender, race, ethnicity, national origin, political affiliation, religious association, and sexual orientation.

11. The method of claim 8 , wherein the machine learning model includes a unified machine learning model based on the sensor data and the human labeler data, wherein the unified machine learning model is employed during an early fusion scheme of a multimodal machine learning environment, wherein the early fusion scheme represents early fusing of the sensor data and the human labeler data.

12. The method of claim 8 , wherein the machine learning model includes separate machine learning models, wherein a first machine learning model of the separate machine learning models is based on the sensor data and not the human labeler data, wherein a second machine learning model of the separate machine learning models is based on the human labeler data and not the sensor data, and wherein the separate machine learning models are employed during a late fusion scheme of a multimodal machine learning environment, wherein the late fusion scheme represents late fusing of the sensor data and the human labeler data.

13. The method of claim 12 , further comprising:

obtaining a first score from the first machine learning model associated with the sensor data;

obtaining a second score from the second machine learning model associated with the human labeler data;

averaging the first and second scores; and

maintaining the averaged first and second scores at the one or more databases to be applied to subsequent trainings of the separate machine learning models.

14. The method of claim 13 , wherein the method is facilitated by one or more processors comprising one or more of a graphics processor and an application processor, wherein the graphics processor and the application processor are co-located on a common semiconductor package, and wherein the one or more sensors include one or more of a camera, a microphone, a touch sensor, a capacitor, a radio component, a radar component, a scanner, and an accelerometer.

15. An apparatus comprising:

one or more processors to:

classify sensor data with human labeler data, where the sensor data is obtained through one or more sensors communicably coupled to the one or more processors; and

create and train a unified machine learning model based on features associated with the classified sensor data based on the human labeler data, wherein the features comprise human labeler influences as obtained from the human labeler data associated with the sensor data;

wherein training is further to facilitate the unified machine learning model to interpret, based on the human labeler data, the human labeler influences according to one or more environments prior to embedding the one or more human labeler influences in the one or more environments, wherein the interpretation of the human influences is based on acceptances of human behavior and exceptions to the human behavior as derived from the human labeler data and based on a relevance between the sensor data and the human labeler data, wherein the relevance identifies meaning of the sensor data based on human behavior corresponding to the human labeler data, and wherein the acceptances of the human behavior are based on verified data obtained from one or more of personal profiles, cultural traits, historical norms, societal preferences, personal prejudices, societal biases, or habits.

16. The apparatus of claim 15 , wherein the one or more processors are further to:

detect the sensor data through the one or more sensors including one or more of a camera, a microphone, a touch sensor, a capacitor, a radio component, a radar component, a scanner, and an accelerometer; and

monitor the human labeler data to determine one or more of human behaviors and human variables, wherein the human labeler data is obtained through multiple sources including one or more of the one or more sensors, historical data, categorical data, and personal profiles.

17. The apparatus of claim 16 , wherein the one or more processors are further to prior to classifying the sensor data, evaluate the human behaviors and human variables and their association with the sensor data.

18. The apparatus of claim 17 , wherein the one or more processors are further to filter out one or more of the human variables associated with one or more of inaccuracies, unintended consequences, and biases, wherein the filtered out one or more human variables include one or more of age, gender, race, ethnicity, national origin, religion, and sexual orientation, and wherein the filtered out one or more human variables further include one or more of accidental acts and coincidental items, wherein the one or more processors are further to recognize and consider one or more variances between first human variables and second human variables in application of the first and second human variables in training the machine learning model.

19. The apparatus of claim 15 , wherein the unified machine learning model is created and trained in an early fusion machine learning environment.

20. The apparatus of claim 15 , wherein the one or more processors are further to:

create and train multiple machine learning models such that each of the multiple machine learning models is based on first features of the features associated with the sensor data or second features of the features associated with human labeler data; and

compute scores based on average outcomes obtained from the multiple machine learning models associated with the sensor data and the human labeler data, wherein the scores are maintained in one or more databases to be used with creation and training of future machine learning models, wherein the one or more processors comprise one or more of a graphics processor and an application processor, wherein the graphics processor and the application processor are co-located on a common semiconductor package.

US16/145,601

2018-09-28

2018-09-28

Embedding human labeler influences in machine learning interfaces in computing environments

Active

2041-08-07

US11526713B2

( en )

Priority Applications (3)

Application Number

Priority Date

Filing Date

Title

US16/145,601

US11526713B2

( en )

2018-09-28

2018-09-28

Embedding human labeler influences in machine learning interfaces in computing environments

EP19182743.5A

EP3629243A1

( en )

2018-09-28

2019-06-26

Embedding human labeler influences in machine learning interfaces in computing environments

CN201910796206.2A

CN110969257A

( en )

2018-09-28

2019-08-27

Embedding Human Annotator Influence in Machine Learning Interfaces in Computational Environments

Applications Claiming Priority (1)

Application Number

Priority Date

Filing Date

Title

US16/145,601

US11526713B2

( en )

2018-09-28

2018-09-28

Embedding human labeler influences in machine learning interfaces in computing environments

Publications (2)

Publication Number

Publication Date

US20190042894A1

US20190042894A1 ( en )

2019-02-07

US11526713B2

true

US11526713B2 ( en )

2022-12-13

Family

ID=65230322

Family Applications (1)

Application Number

Title

Priority Date

Filing Date

US16/145,601

Active

2041-08-07

US11526713B2

( en )

2018-09-28

2018-09-28

Embedding human labeler influences in machine learning interfaces in computing environments

Country Status (3)

Country

Link

US

( 1 )

US11526713B2

( en )

EP

( 1 )

EP3629243A1

( en )

CN

( 1 )

CN110969257A

( en )

Cited By (1)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20240078792A1

( en )

*

2022-09-02

2024-03-07

Lemon Inc.

Systems for multi-task joint training of neural networks using multi-label datasets

Families Citing this family (56)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

WO2018125928A1

( en )

2016-12-29

2018-07-05

DeepScale, Inc.

Multi-channel sensor simulation for autonomous control systems

WO2018176000A1

( en )

2017-03-23

2018-09-27

DeepScale, Inc.

Data synthesis for autonomous control systems

US11893393B2

( en )

2017-07-24

2024-02-06

Tesla, Inc.

Computational array microprocessor system with hardware arbiter managing memory requests

US11157441B2

( en )

2017-07-24

2021-10-26

Tesla, Inc.

Computational array microprocessor system using non-consecutive data formatting

US10671349B2

( en )

2017-07-24

2020-06-02

Tesla, Inc.

Accelerated mathematical engine

US11409692B2

( en )

2017-07-24

2022-08-09

Tesla, Inc.

Vector computational unit

US12307350B2

( en )

2018-01-04

2025-05-20

Tesla, Inc.

Systems and methods for hardware-based pooling

US11561791B2

( en )

2018-02-01

2023-01-24

Tesla, Inc.

Vector computational unit receiving data elements in parallel from a last row of a computational array

US11215999B2

( en )

2018-06-20

2022-01-04

Tesla, Inc.

Data pipeline and deep learning system for autonomous driving

US11361457B2

( en )

2018-07-20

2022-06-14

Tesla, Inc.

Annotation cross-labeling for autonomous control systems

US11636333B2

( en )

2018-07-26

2023-04-25

Tesla, Inc.

Optimizing neural network structures for embedded systems

US11562231B2

( en )

2018-09-03

2023-01-24

Tesla, Inc.

Neural networks for embedded devices

US11526713B2

( en )

2018-09-28

2022-12-13

Intel Corporation

Embedding human labeler influences in machine learning interfaces in computing environments

KR20250040098A

( en )

2018-09-28

2025-03-21

스냅 인코포레이티드

Neural network system for gesture, wear, activity, or carry detection on a wearable or mobile device

ES3063986T3

( en )

2018-10-11

2026-04-21

Tesla Inc

Systems and methods for training machine models with augmented data

US11196678B2

( en )

2018-10-25

2021-12-07

Tesla, Inc.

QOS manager for system on a chip communications

US20200134398A1

( en )

*

2018-10-29

2020-04-30

Sri International

Determining intent from multimodal content embedded in a common geometric space

US11094219B2

( en )

*

2018-11-28

2021-08-17

International Business Machines Corporation

Portable computing device having a color detection mode and a game mode for learning colors

US10878173B2

( en )

*

2018-11-29

2020-12-29

Adobe Inc.

Object recognition and tagging based on fusion deep learning models

US11816585B2

( en )

2018-12-03

2023-11-14

Tesla, Inc.

Machine learning models operating at different frequencies for autonomous vehicles

US11537811B2

( en )

2018-12-04

2022-12-27

Tesla, Inc.

Enhanced object detection for autonomous vehicles based on field view

US11610117B2

( en )

2018-12-27

2023-03-21

Tesla, Inc.

System and method for adapting a neural network model on a hardware platform

US11790239B2

( en )

*

2018-12-29

2023-10-17

International Business Machines Corporation

Deep learning testing

US11150664B2

( en )

2019-02-01

2021-10-19

Tesla, Inc.

Predicting three-dimensional features for autonomous driving

US10997461B2

( en )

2019-02-01

2021-05-04

Tesla, Inc.

Generating ground truth for machine learning from time series elements

US11567514B2

( en )

2019-02-11

2023-01-31

Tesla, Inc.

Autonomous and user controlled vehicle summon to a target

US10956755B2

( en )

2019-02-19

2021-03-23

Tesla, Inc.

Estimating object properties using visual image data

CN112102423B

( en )

*

2019-06-17

2026-02-17

通用电气精准医疗有限责任公司

Medical imaging method and system

JP7582195B2

( en )

*

2019-08-21

2024-11-13

ソニーグループ株式会社

Information processing device, information processing method, and program

US12120384B2

( en )

*

2019-09-27

2024-10-15

Mcafee, Llc

Methods and apparatus to improve deepfake detection with explainability

KR102203786B1

( en )

*

2019-11-14

2021-01-15

오로라월드 주식회사

Method and System for Providing Interaction Service Using Smart Toy

CN111046926B

( en )

*

2019-11-26

2023-09-19

山东浪潮科学研究院有限公司

An integrated learning method for computer vision image classification

CA3160259A1

( en )

*

2019-12-19

2021-06-24

Ryan Michael McKay

Self-optimizing labeling platform

US11687778B2

( en )

2020-01-06

2023-06-27

The Research Foundation For The State University Of New York

Fakecatcher: detection of synthetic portrait videos using biological signals

CN115702323A

( en )

*

2020-04-27

2023-02-14

具象有限公司

Method for semi-supervised data collection and machine learning using distributed computing devices

US12118297B2

( en )

*

2020-06-15

2024-10-15

Canon Kabushiki Kaisha

Information processing apparatus

SG10202006357UA

( en )

*

2020-07-01

2020-09-29

Alipay Labs Singapore Pte Ltd

A Document Identification Method and System

WO2022081175A1

( en )

*

2020-10-16

2022-04-21

Purdue Research Foundation

Enhanced images

CN112101327B

( en )

*

2020-11-18

2021-01-29

北京达佳互联信息技术有限公司

Training method of motion correction model, motion correction method and device

CN112418437B

( en )

*

2020-11-19

2022-06-21

华南师范大学

Multi-person decision-making-based ethical simulation virtual experiment method and robot

CN112819174B

( en )

*

2020-11-19

2023-04-28

华南师范大学

Ethical virtual simulation experiment method and robot based on artificial intelligence algorithm improvement

US12013874B2

( en )

2020-12-14

2024-06-18

International Business Machines Corporation

Bias detection

JP2024525119A

( en )

*

2021-05-18

2024-07-10

アトゥーン・メディア・ラブズ・パブリック・ベネフィット・コーポレイション

System and method for automatic generation of interactive synchronized discrete avatars in real time

US11893087B2

( en )

*

2021-06-16

2024-02-06

Robert Bosch Gmbh

Defending multimodal fusion models against single-source adversaries

US12148417B1

( en )

*

2021-06-22

2024-11-19

Amazon Technologies, Inc.

Label confidence scoring

CN113420878B

( en )

*

2021-07-09

2022-10-14

浙江工业大学

Deep learning model depolarization method based on bidirectional feature embedding

KR20240048533A

( en )

2021-08-19

2024-04-15

테슬라, 인크.

Training vision-based systems using simulated content

US12462575B2

( en )

2021-08-19

2025-11-04

Tesla, Inc.

Vision-based machine learning model for autonomous driving with adjustable virtual camera

US11803575B2

( en )

*

2022-01-24

2023-10-31

My Job Matcher, Inc.

Apparatus, system, and method for classifying and neutralizing bias in an application

US12623691B2

( en )

2022-05-20

2026-05-12

Tesla, Inc.

Fail-safe corrective actions based on vision information for autonomous vehicles

NL1044409B1

( en )

*

2022-08-30

2024-03-15

Zander Laboratories B V

A method performed by an information processing device, a program product, and a system for acquiring artificial intelligence from processing context related operational, human bio-signal and human conduct data.

FR3140191B1

( en )

*

2022-09-27

2025-03-28

Thales Sa

Method for automatically generating an algorithm to assist decision-making; Computer program product and associated computer device.

US12033042B2

( en )

2022-10-04

2024-07-09

Gmeci, Llc

Apparatus for bias eliminated performance determination

US12032618B2

( en )

2022-10-20

2024-07-09

Toyota Research Institute, Inc.

System and method to infer thoughts and a process through which a human generated labels for a coding scheme

US12619691B2

( en )

2024-01-10

2026-05-05

Cisco Technology, Inc.

Extending EAP for supporting generative AI challenges

CN117912373B

( en )

*

2024-03-20

2024-05-31

内江广播电视台

An offline mobile news media intelligent presentation method

Citations (9)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20110276396A1

( en )

*

2005-07-22

2011-11-10

Yogesh Chunilal Rathod

System and method for dynamically monitoring, recording, processing, attaching dynamic, contextual and accessible active links and presenting of physical or digital activities, actions, locations, logs, life stream, behavior and status

US8226572B2

( en )

*

2008-08-27

2012-07-24

Lifeline Biotechnologies, Inc.

Methods for collecting and analyzing thermal data based on breast surface temperature to determine suspect conditions

US20120269441A1

( en )

*

2011-04-19

2012-10-25

Xerox Corporation

Image quality assessment

US20150332087A1

( en )

*

2014-05-15

2015-11-19

Fuji Xerox Co., Ltd.

Systems and Methods for Identifying a User's Demographic Characteristics Based on the User's Social Media Photographs

US20170160813A1

( en )

*

2015-12-07

2017-06-08

Sri International

Vpa with integrated object recognition and facial expression recognition

US20190102656A1

( en )

*

2017-09-29

2019-04-04

Here Global B.V.

Method, apparatus, and system for providing quality assurance for training a feature prediction model

US20190197778A1

( en )

*

2017-12-21

2019-06-27

Luminar Technologies, Inc.

Object identification and labeling tool for training autonomous vehicle controllers

EP3629243A1

( en )

2018-09-28

2020-04-01

Intel Corporation

Embedding human labeler influences in machine learning interfaces in computing environments

US10762440B1

( en )

*

2015-09-24

2020-09-01

Apple Inc.

Sensor fusion and deep learning

2018

2018-09-28

US

US16/145,601

patent/US11526713B2/en

active

Active

2019

2019-06-26

EP

EP19182743.5A

patent/EP3629243A1/en

not_active

Withdrawn

2019-08-27

CN

CN201910796206.2A

patent/CN110969257A/en

active

Pending

Patent Citations (10)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20110276396A1

( en )

*

2005-07-22

2011-11-10

Yogesh Chunilal Rathod

System and method for dynamically monitoring, recording, processing, attaching dynamic, contextual and accessible active links and presenting of physical or digital activities, actions, locations, logs, life stream, behavior and status

US8226572B2

( en )

*

2008-08-27

2012-07-24

Lifeline Biotechnologies, Inc.

Methods for collecting and analyzing thermal data based on breast surface temperature to determine suspect conditions

US20120269441A1

( en )

*

2011-04-19

2012-10-25

Xerox Corporation

Image quality assessment

US20150332087A1

( en )

*

2014-05-15

2015-11-19

Fuji Xerox Co., Ltd.

Systems and Methods for Identifying a User's Demographic Characteristics Based on the User's Social Media Photographs

US10762440B1

( en )

*

2015-09-24

2020-09-01

Apple Inc.

Sensor fusion and deep learning

US20170160813A1

( en )

*

2015-12-07

2017-06-08

Sri International

Vpa with integrated object recognition and facial expression recognition

US20190102656A1

( en )

*

2017-09-29

2019-04-04

Here Global B.V.

Method, apparatus, and system for providing quality assurance for training a feature prediction model

US20190197778A1

( en )

*

2017-12-21

2019-06-27

Luminar Technologies, Inc.

Object identification and labeling tool for training autonomous vehicle controllers

EP3629243A1

( en )

2018-09-28

2020-04-01

Intel Corporation

Embedding human labeler influences in machine learning interfaces in computing environments

CN110969257A

( en )

2018-09-28

2020-04-07

英特尔公司

Embedding Human Annotator Influence in Machine Learning Interfaces in Computational Environments

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party

Title

Communication Pursuant to Article 94(3) EPC in EP Application No. 19182743.5, dated Dec. 2, 2021, 5 pages.

Extended European Search Report EP Application No. 19182743.5, dated Jan. 2, 2020, 8 pages.

RADIM TYLECEK, ROBERT FISHER: " Consistent Semantic Annotation of Outdoor Datasets via 2D/3D Label Transfer ", SENSORS, vol. 18, no. 7, pages 2249, XP055653466, DOI: 10.3390/s18072249

Sogaard, Anders, et al., " Selection Bias, Label Bias, and Bias in Ground Truth ", Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Tutorial Abstracts, Dublin, Ireland, Aug. 29, 2014 (Aug. 29, 2014), pp. 11-13, XP55653476.

Tylecek, Radim et al., " Consistent Semantic Annotation of Outdoor Datasets via 2D/3D Label Transfer ", School of Informatics, SENSORS, vol. 18, No. 7, Jul. 12, 2018 (Jul. 12, 2018), pp. 1-20, XP55653466.

Cited By (2)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US20240078792A1

( en )

*

2022-09-02

2024-03-07

Lemon Inc.

Systems for multi-task joint training of neural networks using multi-label datasets

US12243292B2

( en )

*

2022-09-02

2025-03-04

Lemon Inc.

Systems for multi-task joint training of neural networks using multi-label datasets

Also Published As

Publication number

Publication date

US20190042894A1

( en )

2019-02-07

CN110969257A

( en )

2020-04-07

EP3629243A1

( en )

2020-04-01

Similar Documents

Publication

Publication Date

Title

EP3629243A1

( en )

2020-04-01

Embedding human labeler influences in machine learning interfaces in computing environments

US11798271B2

( en )

2023-10-24

Depth and motion estimations in machine learning environments

US12596960B2

( en )

2026-04-07

Misuse index for explainable artificial intelligence in computing environments

US20240289930A1

( en )

2024-08-29

Deep learning-based real-time detection and correction of compromised sensors in autonomous machines

KR102222642B1

( en )

2021-03-05

Neural network for object detection in images

US11031005B2

( en )

2021-06-08

Continuous topic detection and adaption in audio environments

US9875445B2

( en )

2018-01-23

Dynamic hybrid models for multimodal analysis

US20190108447A1

( en )

2019-04-11

Multifunction perceptrons in machine learning environments

EP3899696B1

( en )

2023-08-16

Voice command execution from auxiliary input

US10728616B2

( en )

2020-07-28

User interest-based enhancement of media quality

US10438588B2

( en )

2019-10-08

Simultaneous multi-user audio signal recognition and processing for far field audio

CN113015984A

( en )

2021-06-22

Error correction in convolutional neural networks

US20160086088A1

( en )

2016-03-24

Facilitating dynamic affect-based adaptive representation and reasoning of user behavior on computing devices

US12217474B2

( en )

2025-02-04

Detection of moment of perception

US20240104744A1

( en )

2024-03-28

Real-time multi-view detection of objects in multi-camera environments

US20190045169A1

( en )

2019-02-07

Maximizing efficiency of flight optical depth sensors in computing environments

US20190096073A1

( en )

2019-03-28

Histogram and entropy-based texture detection

US20260073640A1

( en )

2026-03-12

Spacial organizer for multimedia results in extended reality

Nafee et al.

2024

Tiny-ML based person identification in dynamic motion

Legal Events

Date

Code

Title

Description

2018-09-28

AS

Assignment

Owner name : INTEL CORPORATION, CALIFORNIA

Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ANDERSON, GLEN J.;REEL/FRAME:047004/0665

Effective date : 20180928

2018-09-28

FEPP

Fee payment procedure

Free format text : ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

2018-11-13

STPP

Information on status: patent application and granting procedure in general

Free format text : APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED

2019-06-12

STPP

Information on status: patent application and granting procedure in general

Free format text : DOCKETED NEW CASE - READY FOR EXAMINATION

2022-04-11

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION MAILED

2022-07-17

STPP

Information on status: patent application and granting procedure in general

Free format text : RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

<time itemprop="date" datetime="2022-08

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964