Conceptio › Archive › Google Patents

Google Patentsopen access

… neural networks such as a single-shot-detector convolutional neural network — Inventive Government Solutions, Llc (US20200054306A1)

Inventive Government Solutions, Llc · Google Patents

Google Patents · Patents · License: Open Access

patent, google patents, intellectual property, US20200054306A1, Inventive Government Solutions, Llc, Courosh Mehanian, en, 2020

ABSTRACT

Abstract

In an embodiment, an intelligent system includes an electronic circuit configured to execute a neural network, to detect at least one feature in an image of a body portion while executing the neural network, and to determine a respective position and a respective class of each of the detected at least one feature while executing the neural network. For example, such a system can execute a neural network to detect at least one feature in an image of a lung, to determine a respective position within the image of each detected feature, and to classify each of the detected features as one of the following: A-line, B-line, pleural line, consolidation, and pleural effusion.

Description

CROSS-RELATED APPLICATION(S)

This application claims benefit of priority to U.S. Provisional Patent Application Ser. No. 62/719,429, titled âAutomated Ultrasound Video Interpretation Of A Body Part, Such As A Lung, With One Or More Convolutional Neural Networks Such As A Single-Shot-Detector Convolutional Neural Network,â which was filed 17 Aug. 2018, and which is incorporated by reference.

SUMMARY

Today, ailments, such as pulmonary (i.e., lung-related) ailments, are diagnosed by a doctor, such as a pulmonologist, or other medical professional, after he/she considers medical images, such as one or more medical images of one or both lungs taken by, e.g., x-ray, CAT, PET, and ultrasound.

Although the below discussion and embodiments are directed to ultrasound images, both still images and a time sequence/stream of multiple images (e.g., a video or a video stream), it is understood that the below discussion, described embodiments, and described techniques can be used for medical images other than ultrasound images.

To make his/her diagnosis regarding a pathology of a subject's lungs, a pulmonologist looks for features in one or more ultrasound images, such features including pleural line and absence of lung sliding along the pleural line, A-lines, B-lines, pleural effusion, consolidation, and merged B-lines.

Referring to FIG. 1 , the pleural line 10 corresponds to the tissue structure formed by the visceral and parietal pleurae, and the pleural cavity 12 between these pleurae. The visceral pleura 14 is the âinnerâ pleura, which is the serous membrane that covers the surface of a lung 16 and that dips into the tissues between the lobes of the lung. The parietal pleura 18 is the âouterâ serous membrane that attaches the lung 16 to the inner surface 20 of the thoracic cavity. In the pleural cavity 12 between the visceral 14 and parietal 18 pleurae is a âslipperyâ liquid that allows the pleurae to slide back and forth past one another as the diaphragm (the âbreathing muscle,â not shown in FIG. 1 ) contracts and relaxes to expand and compress the lung 16 , and, therefore, to cause the inhalation and the expiration phases of respiration.

A lack of sliding of the visceral pleura 14 relative to the parietal pleura 18 can indicate to the pulmonologist (not shown in FIG. 1 ) a problem such as pneumothorax (collapsed lung). Pneumothorax is a condition where air is trapped in the pleural cavity 12 between visceral pleura 14 and the parietal pleura 18 of the lung. Pneumothorax is a serious condition and intervention is usually indicated. When a patient with a normal lung respires, the visceral pleura 14 and the parietal pleura 18 slide across each other. This relative motion of the two pleura

14 and 18 is observable in a lung ultrasound video. A lung ultrasound video of a subject with pneumothorax does not exhibit lung sliding because the high-impedance contrast of the trapped air prevents the observation of relative pleural movement.

Still referring to FIG. 1 , A-lines 22 are an echo artifact of ultrasound that, by themselves, indicate the presence of air in the lung. Echoes are multiple discrete reflections (e.g., âbouncingâ between two impedance boundaries like the transducer head and liquid in tissue, where the delay of a bounced signal shows up as a âghostâ object at an integer multiple of the distance between the two impedance boundaries), whereas reverberation is a continuous signal caused by, for example, a signal âbouncing aroundâ inside a water droplet and showing multiple closely spaced âghostâ objects (e.g., B-lines, see FIG. 5 ) due to the reverb delay, which is typically, although not necessarily, shorter than an echo delay. The presence of A-lines 22 typically indicates a high confidence that the tissue in which the A-lines appear is normal. But the absence of A-lines 22 , by itself, does not indicate a pathology.

Still referring to FIG. 1 , A-lines 22 typically manifest, in an ultrasound image, as artifacts/lines that are effectively parallel to the transducer (not shown in FIG. 1 ) face, i.e., that are parallel (for a flat, hereinafter linear, transducer), or circumferential (for a round, hereinafter curvilinear, transducer), lines. When an ultrasound wavefront is incident on the pleura

14 or 18 , the pleura redirects a portion of the ultrasound wave back to the ultrasound transducer. The transducer receives a portion of this first redirected wave, and, in response, generates, in the ultrasound image, a bright line, which is the pleural line 10 . But the transducer redirects another portion of the first redirected wave back into the body due to the impedance mismatch at the skin/transducer boundary. When this second redirected wave is incident on the pleura

14 or 18 , the pleura redirects a portion of this second redirected wave back to the transducer. The ultrasound machine (not shown in FIG. 1 ) is âdumbâ in that it merely measures a time delay from the generation of a wave front to the receiving of a redirected wave front, translates the time delay into a distance, and displays a bright spot/line at the distance in the ultrasound image. That is, the ultrasound machine cannot distinguish an echo from a true first redirection of the wavefront. Therefore, because the delay between generation of the wavefront and receiving the redirected portion of the second redirected wave is twice the delay between generation of the wave front and receiving the redirected portion of the first redirected wave (from the pleura 14 or 18 ), the ultrasound machine generates a second bright line at twice the distance of the first line, i.e., at twice the distance from the true pleura distance. This redirection phenomena continues such that the ultrasound image has a series of A lines 22 , which are separated by the distance between the transducer and the pleura

14 or 18 . So, the pulmonologist (not shown in FIG. 1 ) knows that the first line is the true pleural line 10 , and the other lines at integer multiples of the pleura

14 or 18 distance are A-lines 22 , which are really image artifacts, but which do indicate the presence of air in the pleural cavity 10 (abnormal), or just behind the pleura 14 (normal). Although the lack of, or dimmer, A-lines 22 may indicate that there is little or no air in or just behind the pleura 14 , associating the lack of, or dimmer, A-lines with an abnormality concerning the lung 16 is not commonly accepted in the medical literature or in the medical community. That is, it is not commonly acceptable to diagnose a pathology of the lung 16 based solely on the lack of, or dim, A-lines 22 .

Referring to FIG. 2 and to FIG. 14 , B- lines 24 are an artifact of ultrasound reverberation caused by liquid/air interfaces inside of the lung, and the presence of B-lines may indicate a lung condition (e.g., pneumonia, acute respiratory distress syndrome (ARDS)) characterized by excess liquid in a portion of the lung. A small number, or a total absence, of B- lines 24 is typically indicative of normal levels of liquid in the lung tissue. B- lines 24 show up in an ultrasound image as artifacts/lines that are perpendicular to the transducer face (not shown in FIGS. 2 and 14 ) if the transducer is a linear transducer, and as lines that extend radially outward from a curvilinear transducer (not shown in FIGS. 5 and 14 ).

Referring to FIGS. 3-5 and FIG. 17 , pleural effusion 30 is characterized by an accumulation of excess fluid within the pleural cavity 12 (i.e., the space between the visceral pleura 14 and the parietal pleura 18 ). In severe cases, a pulmonologist, or other doctor or clinician, may need to perform an intervention, such as therapeutic aspiration to drain the fluid. In an ultrasound image, pleural effusion 30 shows up as dark regions along, or adjacent to, the pleural line 10 , and can be graded as, e.g., low severity, medium severity, and high severity.

Referring to FIG. 6 , consolidation 60 is characterized by a grouping of bright spots in the lung 16 , beyond the pleural line 10 , and can indicate liquid in the lung (ultrasound wave reflects from the air/fluid interfaces). The size of the consolidation 60 is related to the severity of the potential liquid buildup in the lung tissue.

CROSS-RELATED APPLICATION(S)

SUMMARY

14 or 18 . So, the pulmonologist (not shown in FIG. 1 ) knows that the first line is the true pleural line 10 , and the other lines at integer multiples of the pleura

And referring to FIG. 7 , a phenomenon similar to consolidation 60 of FIG. 6 is a merger 62 of B- lines 24 , in which multiple, closely spaced B-lines are found in the region below the pleural line 10 . A merger 62 of B- lines 24 can indicate liquid buildup more serious than that indicated by a single B-line. The number of B- lines 24 in a merger 62 of B-lines, and the separation between adjacent ones of the B-lines, are related to the severity of the potential liquid buildup.

Referring to FIGS. 1-7 , a problem with needing a pulmonologist to interpret ultrasound images is that in many regions of the world, such as in low-resource nations and in other economically underdeveloped regions, skilled pulmonologists are often scarce or unavailable.

Furthermore, even where a skilled pulmonologist is available, it may be desirable to provide an intelligent system to assist the pulmonologist in making a diagnosis, and to increase the chances that the pulmonologist's diagnosis is correct.

In an embodiment, such an intelligent system includes an electronic circuit configured to execute a neural network, to detect at least one feature in an image of a body portion while executing the neural network, and to determine a respective position and a respective class of each of the detected at least one feature while executing the neural network.

For example, such a system can execute a neural network to detect at least one feature in an image of a lung, to determine a respective position within the image of each detected feature, and to classify each of the detected features as one of the following: A-line, B-line, pleural line, consolidation, and pleural effusion.

In another embodiment, such an intelligent system includes an electronic circuit configured to execute a classifier neural network, to receive an image of a body portion, and to determine, while executing the classifier neural network, a probability that the image indicates a state of a function of the body portion, the function belonging to a particular class.

For example, such a system can receive a sequences of images of a lung, such as a video stream of images of a lung or a conventional M-mode image of a lung, and execute a classifier neural network to determine a probability that the image indicates that the lung exhibits, or does not exhibit, lung sliding.

In yet another embodiment, such an intelligent system includes an electronic circuit configured to execute a neural network having input channels, and configured, while executing the neural network, to receive each of an image of a body portion and at least one modified version of the image with a respective input channel, to detect at least one feature in the image in response to the image and the at least one modified version of the image, and to determine a respective position and a respective class of each of the detected at least one feature in response to the image and the at least one modified version of the image.

For example, the at least one modified version of the image can be a filtered version of the image to enhance the electronic circuit's ability to detect one or more features in the image.

In still another embodiment, such an intelligent system includes an electronic circuit configured to execute a neural network and configured to receive an image of a body portion, the image including at least one feature belonging to a class, and, while executing the neural network, to detect at least one feature in the image, and to determine, for each of the detected at least one feature, a respective position and a respective confidence level that the respective one of the detected at least one feature belongs to the class.

And an embodiment of a system for training a neural network includes an electronic circuit configured to generate, from each of at least one first training image, at least one second training image, and to train the neural network by executing the neural network to determine a respective probability that each of at least one feature in at least one of the at least one first training image and the at least one second training image belongs to a feature class, by determining, for each of the at least one feature, a probability difference between the determined respective probability and a corresponding annotated probability, and by changing a respective weighting of each of at least one synapse of the neural network in response to the probability difference.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an ultrasound image of a lung, the image including features corresponding to a pleural line of the lung and to A-lines.

FIG. 2 is an ultrasound image of a lung, the image including features corresponding to a pleural line of the lung and to a B-line.

FIGS. 3-5 are respective ultrasound images of a lung, each of the images including features corresponding to a respective pleural effusion.

FIG. 6 is an ultrasound image of a lung, the image including features corresponding to a consolidation.

FIG. 7 is an ultrasound image of a lung, the image including features corresponding to a merger of B-lines.

FIG. 8 is a workflow of an ultrasound-image-of-a-lung acquisition, of an ultrasound-image-of-a-lung analysis, and of a lung diagnosis, according to an embodiment.

FIG. 9 is a more detailed diagram of the workflow of the ultrasound-image-of-a-lung acquisition, of the ultrasound-image-of-a-lung analysis, and of the lung diagnosis, of FIG. 8 , according to an embodiment.

FIG. 10 is an ultrasound image of a lung, the image produced by a curvilinear ultrasound transducer and including features corresponding to B-lines.

FIG. 11 is a diagram of a method for enhancing an ultrasound image of a lung by filtering the image, according to an embodiment.

FIG. 12 is an M-mode ultrasound image of a lung, and three of the normal-mode ultrasound images of the lung in response to which the M-mode ultrasound image is generated, according to an embodiment.

FIG. 13 is a plot of B-lines and B-line clusters in an ultrasound image of a lung, according to an embodiment.

FIG. 14 is the plot of FIG. 13 compressed into one dimension along the horizontal axis of the plot, according to an embodiment.

FIG. 15 includes M-mode images that indicate lung sliding along the pleural line (the image on the left) and no lung sliding (the image on the right).

FIG. 16 is a diagram of a workflow for multi-class feature/object detection and identification/classification, according to an embodiment.

FIG. 17 is a plot of instances of pleural effusion by size (bounding-box height) along the abscissa and by the ratio of the number of dark pixels to the number of total pixels along the ordinate, according to an embodiment.

FIG. 18 is a plot generated by performing a non-maximum-suppression algorithm, according to an embodiment.

FIG. 19A is a workflow for training one or more detector-model-based neural networks, according to an embodiment.

FIG. 19B is a workflow for training one or more classifier-model-based neural networks, according to an embodiment.

FIG. 20 is a plot of training loss and validation loss (ordinate) versus the number of different final training images (abscissa) used to train an SSD CNN, according to an embodiment.

FIG. 21 is a general diagram of a CNN, such as an SSD CNN, according to an embodiment.

FIG. 22 is a diagram of a single multi-class SSD CNN, according to an embodiment.

FIG. 23 is a diagram of a feature map, according to an embodiment.

FIG. 24 is a diagram of an ultrasound system, such as the ultrasound system of FIG. 8 , according to an embodiment.

DETAILED DESCRIPTION

In general, an embodiment described herein has at least the following three aspects, in which feature/object identification and classification can be the same operation or separate operations:

1) Using one or more convolutional neural networks (CNNs), or one or more other neural networks (NNs), to detect, to identify/classify features/objects in an ultrasound image (or in a series of ultrasound images) of a lung, such features including, or corresponding to, pleural line and absence of lung sliding along the pleural line (hereinafter âabsence of lung sliding,â âpresence of lung sliding,â without the phrase âalong the pleural lineâ), A-lines, B-lines, pleural effusion, consolidation, and merger of B-lines (hereinafter merged B-lines)âas described below, determining whether or not a series, or video, of ultrasound images indicates the absence or the presence of lung sliding typically is a classification problem only, and does not entail feature detection; 2) Training the one or more CNNs, or one or more other NNs, to perform aspect (1); and 3) Rendering a diagnosis based on the features/objects detected, and identified/classified in aspect (1).

Other embodiments include improving M-mode image generation from ultrasound video images of the lung and selecting the trained NN model(s) to use for aspect (1).

FIG. 8 is a workflow 100 of an acquisition of an ultrasound image of a lung, an analysis of the acquired ultrasound image of the lung, and a lung diagnosis based on the analysis, and includes a diagram of an ultrasound system 130 , according to an embodiment. In FIG. 8 , rectangular boxes are processing steps; and diamonds are determined and provided (e.g., displayed) results.

There are two processing steps in FIG. 8 :

(1) Lung-feature- detection processing step 150 . The output of the processing step 150 include those of the following feature/objects that the ultrasound system 130 detects in an analyzed image: A-Line, pleural line, B-line, merged B-lines, absences of lung sliding, consolidation, and pleural effusion. (2) Lung-pathology- diagnosis processing step 170 . The output of the processing step 170 is the lung diagnosis 180 , e.g., pneumothorax, pneumonia, pleural effusion, or ARDS (Acute Respiratory Distress Syndrome).

The ultrasound system 130 includes an ultrasound transducer 120 coupled to an ultrasound machine 125 ; the ultrasound system is described in more detail below in conjunction with FIG. 24 .

An ultrasound technician, also called a sonographer (not shown in FIG. 8 ), manipulates the ultrasound transducer head 120 over the chest of a subject 110 to generate a time sequence of ultrasound images, i.e., to generate ultrasound video.

The ultrasound machine 125 , which can be, or which can include, a conventional computer system, and which can execute software, or which can be configured with firmware, that causes the computer system to process the ultrasound images as follows.

First, the machine 125 is configured to use conventional techniques, including image-filtering and other image-processing techniques, to render enhanced ultrasound video 140 (see FIG. 13 ), such as B-mode ultrasound video.

Then, referring to a step 150 , the ultrasound machine 125 executes a CNN, such as a Single Shot Detection (SSD) CNN, that detects and identifies features/objects in one or more frames of the ultrasound video 140 , such features/objects including, or representing, pleural line, A-lines, B-lines, pleural effusion, consolidation, and merged B-lines. Furthermore, because lung sliding is a classification problem only (lung sliding is either present in an M-mode image (see below) or not present in the M-mode image), an SSD CNN is not used to detect lung sliding in an M-mode image; instead, a classification algorithm, such as a classic CNN, is used to identify/classify lung sliding as being present in, or absent from, an M-mode image. Furthermore, hereinafter âfeaturesâ and âobjectsâ in an image, such as an ultrasound image, are considered to be equivalent terms, and, therefore, are used interchangeably. Moreover, hereinafter âidentifyâ and âclassifyâ features in an image such as an ultrasound image, are considered to be equivalent terms, and, therefore, are used interchangeably.

Next, at a step 170 , the ultrasound machine 125 executes a diagnosis algorithm that evaluates the classified lung features to render a lung- pathology diagnosis 180 . For example, such a pathology diagnosis is in response to the determined likelihood of the presence, and the respective severities, of conditions such as less-than-normal, or absence of, lung sliding, A-line, B-line, pleural effusion, consolidation, and merged B-line.

Examples of the lung diagnosis 180 include pneumonia, collapsed lung, and ARDS.

FIG. 9 is a workflow 200 with additional steps and results, and more detail of steps and

results

150 , 160 , 170 , and 180 of FIG. 8 , according to an embodiment that is referred to as single-class embodiment for reasons explained below. In the embodiment shown in FIG. 9 , the ultrasound machine 125 ( FIG. 8 ) executes a classification CNN to classify lung sliding in M- mode images 210 . The ultrasound machine 125 executes five single-class SSD CNNs to detect and to classify each of another five features A-line, B-line, pleural line, consolidation, and pleural effusion, respectively (see the column labeled âInferenceâ in FIG. 9 ); therefore, the workflow 200 can be referred to as a single-class SSD CNN embodiment, or, more generally, as a single-class CNN embodiment. And details of how a CNN, particularly an SSD CNN, operates are described further below.

At a step 201 , the ultrasound machine 125 ( FIG. 8 ) pre-processes ultrasound video images 140 by transforming and enhancing the images. The ultrasound machine 125 ( a ) extracts a plurality of image frames 211 from the ultrasound video 140 , (b) optionally applies one or more geometric transforms to the image frames, (c) optionally enhances the image frames, and (d) generates one or more reconstructed M- mode images 210 from the pre-processed video. The extraction of the image frames 211 from the ultrasound video 140 is conventional and, therefore, is not described further.

Two types of ultrasound transducers 120 ( FIG. 8 ) are generally used for pulmonary applications. Curvilinear transducers are typically used for adults, and linear transducers for children. Curvilinear transducers produce fan-shaped output images as shown, for example, in FIGS. 10 and 12 , whereas linear transducers produce rectangular output images (not shown). A fan-shaped output image may be described by a polar coordinate system, and a rectangular output image may be described by ordinary Cartesian coordinates. In an embodiment, the ultrasound machine 125 ( FIG. 8 ), at the step 201 , converts polar output from a curvilinear transducer to a rectangular output, and in another embodiment, converts rectangular output from a linear transducer to a polar output. Enhancement of images is described below.

For example, referring to FIGS. 9-10 , if the ultrasound transducer 120 ( FIG. 8 ) is a curvilinear transducer, then, at the step 201 , the ultrasound machine 125 ( FIG. 8 ) may translate the images 140 from polar coordinates to linear (Cartesian) coordinates. Furthermore, referring to FIG. 11 , the ultrasound machine 125 may filter, or apply other image-processing techniques to, the video images 140 to enhance features such as B-lines to make them easier for the CNN to detect and to classify.

At a step 210 , the ultrasound machine 125 reconstructs M-mode images, and effectively provides these images to the lung-sliding classifier 220 to allow the classifier to classify lung sliding. FIG. 15 includes reconstructed ultrasound M-mode images showing a pleural line, the left image 301 indicating lung sliding (fuzziness in the image indicates motion; such lung sliding in an M-mode image is informally called a âseashore patternâ), the right image 302 indicating no lung sliding (smooth, sharper pattern in the image indicates absence of motion; the lack of lung sliding in an M-mode image is informally called a âbar-code patternâ). Therefore, the difference between lung sliding and no lung sliding in a B-mode video is rather subtle and may be hard to discern. Consequently, many commercial ultrasound systems, such as the ultrasound system 130 , produce M-mode images, where the distinction between lung sliding and no lung sliding is more obvious. In an M-mode image, the vertical dimension represents depth and the horizontal dimension represents time. An M-mode image can be likened to a video camera that captures only a vertical slice of the field of view, one pixel wide and where these vertical slices are stacked horizontally to form an image. Like a conventional ultrasound device, the ultrasound system 130 ( FIG. 8 ) produces an M-mode image by recording the returned ultrasound signal at one particular position (e.g., an angular position for a curvilinear transducer, a lateral position for linear transducer) as a function of time. If an M-mode image is needed at another position of the lung, the ultrasound machine 125 conducts a separate M-mode-image capture session at the other position.

Referring to FIG. 12 , an M-mode image is described in more detail, according to an embodiment.

âM-mode imageâ stands for âmotion image,â which is a time sequence of a single column of pixels of in larger images, where the single column of pixels in each image represents the same location of the tissue (e.g., lung) being imaged.

FIG. 12 includes images showing how M-mode images are constructed. A pixel column of an image is selected and identified with straight vertical line 411 in image 401 . In each of subsequent (in time) images

402 and 403 , the same pixel column is identified with straight vertical lines

412 and 413 , respectively. That is, the image 420 is a time sequence of the same pixel column (actually, of the same region of the tissue being imaged), with the

vertical lines

411 , 412 , and 413 representing the same pixel column at different times (for example, the images 401 - 403 have undergone a polar-coordinate-to-rectangular-coordinate transformation, or the pixel columns are extracted, and put into a rectangular coordinate system, using radial column extraction). Gray-level differences in the pixel column over time are indicative of lung sliding. Uniformity in the pixel column over time is indicative of reduced or no lung sliding. The ultrasound machine 125 ( FIG. 8 ) is configured to generate and to analyze more than the one pixel column shown in FIG. 12 . By analyzing multiple pixel columns, accuracy of lung-sliding detection can be increased at least due to the averaging effect.

For example, still referring to FIG. 12 and to step 210 of the workflow 200 of FIG. 9 , the

straight lines

411 , 412 , and 413 each represent a single-column of pixels. In a conventional ultrasound system, a sonographer selects a column for M-mode treatment (e.g., by using a pointing device on the image display), and the ultrasound system thereafter shows M-mode images only of that column. But in an embodiment, the ultrasound system 130 ( FIG. 8 ) can extract, from a sequence of ultrasound images, e.g., from a video stream of ultrasound images, one or more M-mode columns for analysis. That is, instead of displaying only a single column in M-mode as conventional ultrasound machines do, an embodiment of the ultrasound system 130 can generate any number of sequences of M-mode images, up to one respective sequence per pixel column in the ultrasound video images 140 . This allows an embodiment of the ultrasound system 130 to perform, efficiently and simultaneously, lung-sliding detection at multiple locations of the pleural line. Where the ultrasound transducer 120 ( FIG. 8 ) is curvilinear so as to generate a âradialâ pixel column (the straight lines in the

images

401 , 402 , and 403 of FIG. 12 ), one way to generate an M- mode image 420 is to convert one or more corresponding ultrasound images, such as

ultrasound images

401 , 402 , and 403 , from polar coordinates to rectangular coordinates, such that each slanted pixel column is converted to a vertically

straight pixel column

411 , 412 , and 413 , respectively, from which the ultrasound system 130 can generate an M-mode image of the time-sequence at one or more of the straight pixel columns. Another way to generate an M-mode image is to compute the pixel intensities along a radial line (e.g., the slanted lines in

images

401 , 402 , and 403 of FIG. 12 ) to extract a pixel column from multiple time-sequenced images to form an M-mode image. That is, the ultrasound system 130 uses a radial-pixel-intensity algorithm in place of a polar-coordinate-to-rectangular-coordinate transformation of the ultrasound images to extract of the pixel column from the sequence of coordinate-transformed images.

<div id="p

CLAIMS

Claims ( 37 )

1 . A method, comprising:

receiving an image of a body portion; detecting, with a neural network, at least one feature in the image; and determining, with the neural network, a respective position and a respective class of each of the detected at least one feature.

2 . The method of claim 1 wherein the image of the body portion includes an image of a lung.

3 .- 4 . (canceled)

5 . The method of claim 1 wherein determining a respective position of each of the detected at least one feature includes determining a respective container that bounds the detected feature.

6 . The method of claim 1 wherein determining a respective position of each of the detected at least one feature includes determining a respective bounding box in which the feature is disposed.

7 . The method of claim 1 wherein determining a respective position of each of the detected at least one feature includes determining a coordinate of a respective bounding box in which the feature is disposed.

8 . The method of claim 1 wherein determining a respective position of each of the detected at least one feature includes determining a size of a respective bounding box in which the feature is disposed.

9 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining a respective probability that the feature belongs to the respective class.

10 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining a respective confidence level that the feature belongs to the respective class.

11 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes:

determining a respective probability that the feature belongs to the respective class; and

determining that the feature belongs to the respective class in response to the respective probability being greater than a threshold for the respective class.

12 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes:

determining probabilities that the detected at least one feature belongs to respective classes; and

determining that the feature belongs to the one of the respective classes corresponding to the highest one of the probabilities.

13 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining that at least one of the detected at least one feature includes an A-line.

14 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining that at least one of the detected at least one feature includes a pleural line.

15 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining that at least one of the detected at least one feature includes a pleural effusion.

16 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining that at least one of the detected at least one feature includes a B-line.

17 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining that at least one of the detected at least one feature includes merged B-lines.

18 . The method of claim 1 , further comprising:

detecting, with the neural network, at least one respective feature in each of multiple ones of the image and at least one other image of the body portion; determining, with the neural network, that each of the detected at least one respective feature is a respective detected B-line, and a respective position of each of the detected B-lines; grouping the detected B-lines in at least one cluster in response to the respective positions of the detected B-lines, each cluster corresponding to a respective actual B-line; and determining, with the neural network, a respective position of each actual B-line in response to a corresponding one of the at least one cluster.

19 . The method of claim 1 , further comprising:

detecting, with the neural network, at least one respective feature in each of multiple ones of the image and at least one other image of the body portion; determining, with the neural network, that each of the detected at least one respective feature belongs to a same class, and a respective position of each of the detected at least one of the respective feature; grouping the detected features in at least one cluster in response to the respective positions of the detected features, each cluster corresponding to a respective actual feature; and determining, with the neural network, a respective position of each actual feature in response to a corresponding one of the at least one cluster.

20 . The method of claim 1 wherein determining a respective class of each of the detected at least one feature includes determining that at least one of the detected at least one feature includes a consolidation.

21 . The method of claim 1 , further comprising:

wherein determining a respective class of each of the detected at least one feature includes determining that at least one of the detected at least one feature includes a pleural effusion; and determining a severity of the pleural effusion.

22 . The method of claim 1 , further comprising:

wherein the image of the body portion includes an image of a lung; and diagnosing a pathology of the lung in response to the respective determined class of each of the detected at least one feature.

23 . The method of claim 1 , further comprising:

wherein the image of the body portion includes an image of a lung; and diagnosing a pathology of the lung in response to the respective position and to the respective determined class of each of the detected at least one feature.

24 .- 64 . (canceled)

65 . A system, comprising:

an electronic circuit configured

to execute a neural network;

to detect at least one feature in an image of a body portion while executing the neural network; and

to determine a respective position and a respective class of each of the detected at least one feature while executing the neural network.

66 . The system of claim 65 wherein the neural network includes a convolutional neural network.

67 . The system of claim 65 wherein the neural network includes a single-shot-detector convolutional neural network.

68 . The system of claim 65 , further comprising an ultrasound transducer coupled to the electronic circuit and configured to acquire the image.

69 . The system of claim 65 wherein the electronic circuit, while executing the neural network, is configured to detect at least one feature in an ultrasound image of a lung.

70 .- 87 . (canceled)

88 . The system of claim 65 wherein the electronic circuit includes a control circuit.

89 .- 93 . (canceled)

94 . The system of claim 65 wherein the image includes an M-mode image.

95 . (canceled)

96 . The system of claim 65 wherein the body portion includes a lung and the function is lung sliding.

97 .- 145 . (canceled)

146 . A tangible, non-transitory computer-readable medium storing instructions that, when executed by a computing circuit, cause the computing circuit, or another circuit under control of the computing circuit, to execute a neural network:

to detect at least one feature in an image of a body portion; and to determine a respective position and a respective class of each of the detected at least one feature.

147 .- 150 . (canceled)

US16/540,759

2018-08-17

2019-08-14

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

Active

2040-05-28

US11446008B2

( en )

Priority Applications (6)

Application Number

Priority Date

Filing Date

Title

US16/540,759

US11446008B2

( en )

2018-08-17

2019-08-14

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

PCT/US2019/046917

WO2020037266A1

( en )

2018-08-17

2019-08-16

Automated ultrasound video interpretation of a body part, such as a lung, with one or more convolutional neural networks such as a single-shot-detector convolutional neural network

EP19849345.4A

EP3837695A4

( en )

2018-08-17

2019-08-16

AUTOMATED ULTRASOUND VIDEO INTERPRETATION OF A PART OF THE BODY, SUCH AS A LUNG, WITH ONE OR MORE CONVOLUTIVE NERVE NETWORKS SUCH AS A SINGLE-ATTEMPT SENSOR CONVOLUTIVE NERVE NETWORK

CA3109818A

CA3109818A1

( en )

2018-08-17

2019-08-16

Automated ultrasound video interpretation of a body part, such as a lung, with one or more convolutional neural networks such as a single-shot-detector convolutional neural network

CN201980068415.3A

CN113261066A

( en )

2018-08-17

2019-08-16

Automated ultrasound video interpretation of a body part such as the lung with one or more convolutional neural networks such as a single-step detector convolutional neural network

US17/820,072

US12310783B2

( en )

2018-08-17

2022-08-16

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

Applications Claiming Priority (2)

Application Number

Priority Date

Filing Date

Title

US201862719429P

2018-08-17

US16/540,759

US11446008B2

( en )

2018-08-17

2019-08-14

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

Related Child Applications (1)

Application Number

Title

Priority Date

Filing Date

US17/820,072

Continuation

US12310783B2

( en )

2018-08-17

2022-08-16

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

Publications (2)

Publication Number

Publication Date

US20200054306A1

true

US20200054306A1 ( en )

2020-02-20

US11446008B2

US11446008B2 ( en )

2022-09-20

Family

ID=69524194

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US16/540,759

Active

2040-05-28

US11446008B2

( en )

2018-08-17

2019-08-14

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

US17/820,072

Active

US12310783B2

( en )

2018-08-17

2022-08-16

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

Family Applications After (1)

Application Number

Title

Priority Date

Filing Date

US17/820,072

Active

US12310783B2

( en )

2018-08-17

2022-08-16

Automated ultrasound video interpretation of a body part with one or more convolutional neural networks

Country Status (5)

Country

Link

( 2 )

US11446008B2

( en )

( 1 )

EP3837695A4

( en )

( 1 )

CN113261066A

( en )

( 1 )

CA3109818A1

( en )

( 1 )

WO2020037266A1

( en )

Cited By (69)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN111419283A

( en )

2020-04-08

2020-07-17

ä¸æµ·é¿å¾å»é¢

An ultrasound analysis system for assessing the severity of COVID-19 patients

US10937156B2

( en )

2019-01-31

2021-03-02

Bay Labs, Inc.

Saliency mapping of imagery during artificially intelligent image classification

CN112819771A

( en )

2021-01-27

2021-05-18

ä¸åæä¸å¤§å¦

Wood defect detection method based on improved YOLOv3 model

US11043297B2

( en )

2018-12-13

2021-06-22

Rutgers, The State University Of New Jersey

Neural network-based object detection in visual input

US11068718B2

( en )

2019-01-09

2021-07-20

International Business Machines Corporation

Attribute classifiers for image classification

CN113496158A

( en )

2020-03-20

2021-10-12

ä¸ç§»(ä¸æµ·)ä¿¡æ¯éä¿¡ç§ææéå¬å¸

Object detection model optimization method, device, equipment and storage medium

CN113592066A

( en )

2021-07-08

2021-11-02

æ·±å³å¸ææèªå¨é©¾é©¶ææ¯æéå¬å¸

Hardware acceleration method, apparatus, device, computer program product and storage medium

IT202000009667A1

( en )

2020-05-05

2021-11-05

Imedicals S R L

DEVICE AND METHOD FOR THE DIAGNOSIS OF A COVID-19 TYPE PNEUMONIA BY ANALYSIS OF ULTRASOUND IMAGES

US20210345992A1

( en )

2020-05-11

2021-11-11

EchoNous, Inc.

Automatically identifying anatomical structures in medical images in a manner that is sensitive to the particular view in which each image is captured

US20210345986A1

( en )

2020-05-11

2021-11-11

EchoNous, Inc.

Automatic evaluation of ultrasound protocol trees

CN113838026A

( en )

2021-09-22

2021-12-24

ä¸åå¤§å¦

Non-small cell lung cancer detection method, non-small cell lung cancer detection device, computer equipment and storage medium

US11232604B2

( en )

2020-05-06

2022-01-25

Ebm Technologies Incorporated

Device for marking image data

WO2022016262A1

( en )

2020-07-20

2022-01-27

12188848 Canada Limited

Lung ultrasound processing systems and methods

US20220061819A1

( en )

2020-09-02

2022-03-03

China Medical University

Ultrasound Image Reading Method and System Thereof

US20220087645A1

( en )

2020-09-23

2022-03-24

GE Precision Healthcare LLC

Guided lung coverage and automated detection using ultrasound devices

US20220114772A1

( en )

2019-02-07

2022-04-14

Hamamatsu Photonics K.K.

Image processing device and image processing method

WO2022098859A1

( en )

2020-11-06

2022-05-12

EchoNous, Inc.

Robust segmentation through high-level image understanding

CN114521912A

( en )

2020-11-23

2022-05-24

éç¨çµæ°ç²¾åå»çæéè´£ä»»å¬å¸

Method and system for enhancing visualization of pleural lines

EP4016540A1

( en )

2020-12-18

2022-06-22

Optellum Limited

Method and arrangement for processing a signal

US20220215962A1

( en )

2019-09-25

2022-07-07

Fujifilm Corporation

Image diagnosis support device, operation method of image diagnosis support device, and operation program of image diagnosis support device

CN114764869A

( en )

2020-12-30

2022-07-19

è¾çº³æ®èåæ¯å¬å¸

Multi-object detection with single detection per object

US11436720B2

( en )

2018-12-28

2022-09-06

Shanghai United Imaging Intelligence Co., Ltd.

Systems and methods for generating image metric

US11436429B2

( en )

2019-03-21

2022-09-06

Illumina, Inc.

Artificial intelligence-based sequencing

WO2022187661A1

( en )

2021-03-05

2022-09-09

Brainscanology Inc.

Systems and methods for organ shape analysis for disease diagnosis and risk assessment

US20220301258A1

( en )

2020-07-10

2022-09-22

China Institute Of Water Resources And Hydropower Research

Rotated Rectangular Bounding Box Annotation Method

US20220334972A1

( en )

2021-04-20

2022-10-20

Meta Platforms, Inc.

Systems and methods for pre-processing and post-processing coherent host-managed device memory

US11488382B2

( en )

2020-09-10

2022-11-01

Verb Surgical Inc.

User presence/absence recognition during robotic surgeries using deep learning

WO2022236160A1

( en )

2021-05-07

2022-11-10

Carnegie Mellon University

System, method, and computer program product for extracting features from imaging biomarkers with machine-learning models

WO2022246314A1

( en )

2021-05-21

2022-11-24

The General Hospital Corporation

Methods and systems for conversion of one data type to another

US20220384042A1

( en )

2019-11-07

2022-12-01

Google Llc

Deep Learning System for Diagnosis of Chest Conditions from Chest Radiograph

US11527319B1

( en )

2019-09-13

2022-12-13

PathAI, Inc.

Systems and methods for frame-based validation

US11532084B2

( en )

2020-05-11

2022-12-20

EchoNous, Inc.

Gating machine learning predictions on medical ultrasound images via risk and uncertainty quantification

US20230021568A1

( en )

2020-01-10

2023-01-26

Carestream Health, Inc.

Method and system to predict prognosis for critically ill patients

US11593649B2

( en )

2019-05-16

2023-02-28

Illumina, Inc.

Base calling using convolutions

US20230075797A1

( en )

2020-04-30

2023-03-09

Electronic Arts Inc.

Extending knowledge data in machine vision

US20230090858A1

( en )

2021-09-23

2023-03-23

The Trustees Of The University Of Pennsylvania

Analysis of pleural lines for the diagnosis of lung conditions

US11627941B2

( en )

2020-08-27

2023-04-18

GE Precision Healthcare LLC

Methods and systems for detecting pleural irregularities in medical images

CN115990105A

( en )

2021-10-18

2023-04-21

å¤§ä¼æ±½è½¦è¡ä»½å¬å¸

Determination of chest plate position

EP4202950A1

( en )

2021-12-23

2023-06-28

Koninklijke Philips N.V.

Methods and systems for clinical scoring a lung ultrasound

WO2023117828A1

( en )

2021-12-23

2023-06-29

Koninklijke Philips N.V.

Methods and systems for clinical scoring a lung ultrasound

US11749380B2

( en )

2020-02-20

2023-09-05

Illumina, Inc.

Artificial intelligence-based many-to-many base calling

US20230346337A1

( en )

2022-05-02

2023-11-02

Fujifilm Sonosite, Inc.

Automated detection of lung slide to aid in diagnosis of pneumothorax

EP4287203A1

( en )

2022-06-01

2023-12-06

Koninklijke Philips N.V.

Methods and systems for analysis of lung ultrasound

WO2023232456A1

( en )

2022-06-01

2023-12-07

Koninklijke Philips N.V.

Methods and systems for analysis of lung ultrasound

WO2023244413A1

( en )

2022-06-16

2023-12-21

Bfly Operations, Inc.

Method and system for managing ultrasound operations using machine learning and/or non-gui interactions

US20240053470A1

( en )

2020-12-18

2024-02-15

Koninklijke Philips N.V.

Ultrasound imaging with anatomy-based acoustic settings

US11908548B2

( en )

2019-03-21

2024-02-20

Illumina, Inc.

Training data generation for artificial intelligence-based sequencing

WO2024046807A1

( en )

2022-08-30

2024-03-07

Koninklijke Philips N.V.

Ultrasound video feature detection using learning from unlabeled data

US20240148366A1

( en )

2022-11-04

2024-05-09

Alveolai Datafuel Inc.

System and method for detecting pneumothorax

WO2024097623A1

( en )

2022-10-31

2024-05-10

Drexel University

Sparse coding and extraction of ultrasound knowledge for explainable point-of-care ultrasound artificial intelligence

WO2024146823A1

( en )

2023-01-05

2024-07-11

Koninklijke Philips N.V.

Multi-frame ultrasound video with video-level feature classification based on frame-level detection

US12106828B2

( en )

2019-05-16

2024-10-01

Illumina, Inc.

Systems and devices for signal corrections in pixel-based sequencing

US20240331870A1

( en )

2023-03-29

2024-10-03

Acer Medical Inc.

Bmd model training method, bmd abnormality risk prediction method, bmd abnormality risk learning system, and bmd abnormality risk prediction system

US20240374241A1

( en )

2020-05-05

2024-11-14

iMedicals S.r.l.

Device and method for the diagnosis of a pneumonia by frequency analysis of ultrasound signals

US12144686B2

( en )

2021-10-25

2024-11-19

EchoNous, Inc.

Automatic depth selection for ultrasound imaging

US20250017570A1

( en )

2023-07-12

2025-01-16

Fujifilm Healthcare Corporation

Ultrasound diagnostic apparatus and model operation verification method

US12217829B2

( en )

2021-04-15

2025-02-04

Illumina, Inc.

Artificial intelligence-based analysis of protein three-dimensional (3D) structures

US12213840B2

( en )

2022-03-14

2025-02-04

EchoNous, Inc.

Automatically establishing measurement location controls for doppler ultrasound

US12249138B2

( en )

2019-11-15

2025-03-11

Qualcomm Technologies, Inc.

Context-driven learning of human-object interactions

WO2025049401A3

( en )

2023-08-25

2025-04-24

Schneider Electric USA, Inc.

Building object detection and image processing applications

WO2025114068A1

( en )

2023-11-29

2025-06-05

Koninklijke Philips N.V.

Systems and methods of quantifying b-lines and merged b-lines in lung ultrasound images

US12354008B2

( en )

2020-02-20

2025-07-08

Illumina, Inc.

Knowledge distillation and gradient pruning-based compression of artificial intelligence-based base caller

US12444482B2

( en )

2021-04-15

2025-10-14

Illumina, Inc.

Multi-channel protein voxelization to predict variant pathogenicity using deep convolutional neural networks

US12443849B2

( en )

2020-02-20

2025-10-14

Illumina, Inc.

Bus network for artificial intelligence-based base caller

WO2025230724A1

( en )

2024-05-01

2025-11-06

University Of Pittsburgh-Of The Commonwealth System Of Higher Education

System and method for oct-based tissue screening

US12524498B2

( en )

2020-12-30

2026-01-13

Synaptics Incorporated

Multi-object detection with single detection per object

US12525320B2

( en )

2021-03-16

2026-01-13

Illumina, Inc.

Neural network parameter quantization for base calling

US12530882B2

( en )

2021-07-01

2026-01-20

Illumina, Inc.

Efficient artificial intelligence-based base calling of index sequences

US12592298B2

( en )

2020-02-20

2026-03-31

Illumina, Inc.

Hardware execution and acceleration of artificial intelligence-based base caller

Families Citing this family (11)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

CN111031927B

( en )

2017-08-21

2023-07-07

Detection, Presentation and Reporting of B-lines in Lung Ultrasound

CN110490851B

( en )

2019-02-15

2021-05-11

è¾è®¯ç§æï¼æ·±å³ï¼æéå¬å¸

Breast image segmentation method, device and system based on artificial intelligence

KR102113172B1

( en )

2019-12-20

2020-05-20

ìë í¸ì¤í ì£¼ìíì¬

Method for inspecting a labeling for a bounding box using deep learning model and apparatus using the same

CN111297396B

( en )

2020-02-24

2021-05-07

åä¸å¸èå¤§å¦

An automatic identification and positioning method of ultrasonic pleural line based on Radon transform

CN112364931B

( en )

2020-11-20

2024-03-19

A few-sample target detection method and network system based on meta-features and weight adjustment

CN113763353B

( en )

2021-09-06

2024-12-10

æå·ç±»èç§ææéå¬å¸

A lung ultrasound image detection system

US12446860B2

( en )

2022-05-02

2025-10-21

Fujifilm Sonosite, Inc.

Automated detection of lung slide to aid in diagnosis of pneumothorax

CN116453091A

( en )

2023-04-14

2023-07-18

éåºé®çµå¤§å¦

Lightweight traffic sign detection method, storage medium and system

CN121620331A

( en )

2023-08-01

2026-03-06

Ultrasound imaging of the lungs

EP4501242A1

( en )

2023-08-01

2025-02-05

Koninklijke Philips N.V.

A-line detection in lung ultrasound

WO2025137841A1

( en )

2023-12-25

2025-07-03

Nvidia Corporation

Neural networks to identify objects in modified images

Family Cites Families (11)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US7912263B2

( en )

2007-08-06

2011-03-22

Carestream Health, Inc.

Method for detecting clipped anatomy in medical images

AU2012275114A1

( en )

2011-06-29

2014-01-16

The Regents Of The University Of Michigan

Analysis of temporal changes in registered tomographic images

US8914097B2

( en )

2012-01-30

2014-12-16

The Johns Hopkins University

Automated pneumothorax detection

US10217213B2

( en )

2013-09-30

2019-02-26

The United States Of America As Represented By The Secretary Of The Army

Automatic focused assessment with sonography for trauma exams

US20170086790A1

( en )

2015-09-29

2017-03-30

General Electric Company

Method and system for enhanced visualization and selection of a representative ultrasound image by automatically detecting b lines and scoring images of an ultrasound scan

US10650512B2

( en )

2016-06-14

2020-05-12

The Regents Of The University Of Michigan

Systems and methods for topographical characterization of medical image data

US10346982B2

( en )

2016-08-22

2019-07-09

Koios Medical, Inc.

Method and system of computer-aided detection using multiple images from different views of a region of interest to improve detection accuracy

CA3040518C

( en )

2016-10-21

2023-05-23

Nantomics, Llc

Digital histopathology and microdissection

EP3554380B1

( en )

2016-12-13

2022-11-02

Koninklijke Philips N.V.

Target probe placement for lung ultrasound

EP3602568B1

( en )

2017-03-28

2025-12-24

Koninklijke Philips N.V.

Ultrasound clinical feature detection and associated devices, systems, and methods

CN108198179A

( en )

2018-01-03

2018-06-22

ååçå·¥å¤§å¦

A kind of CT medical image pulmonary nodule detection methods for generating confrontation network improvement

2019

2019-08-14

US16/540,759

patent/US11446008B2/en

active

Active

2019-08-16

CA3109818A

patent/CA3109818A1/en

active

Pending

2019-08-16

CN201980068415.3A

patent/CN113261066A/en

active

Pending

2019-08-16

EP19849345.4A

patent/EP3837695A4/en

active

Pending

2019-08-16

PCT/US2019/046917

patent/WO2020037266A1/en

not_active

Ceased

2022

2022-08-16

US17/820,072

patent/US12310783B2/en

active

Active

Cited By (97)

* Cited by examiner, â Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

US11043297B2

( en )

2018-12-13

2021-06-22

Rutgers, The State University Of New Jersey

Neural network-based object detection in visual input

US11436720B2

( en )

2018-12-28

2022-09-06

Shanghai United Imaging Intelligence Co., Ltd.

Systems and methods for generating image metric

US11068718B2

( en )

2019-01-09

2021-07-20

International Business Machines Corporation

Attribute classifiers for image classification

US11281912B2

( en )

2019-01-09

2022-03-22

International Business Machines Corporation

Attribute classifiers for image classification

US10937156B2

( en )

2019-01-31

2021-03-02

Bay Labs, Inc.

Saliency mapping of imagery during artificially intelligent image classification

US11893660B2

( en )

2019-02-07

2024-02-06

Hamamatsu Photonics K.K.

Image processing device and image processing method

US20220114772A1

( en )

2019-02-07

2022-04-14

Hamamatsu Photonics K.K.

Image processing device and image processing method

US11676685B2

( en )

2019-03-21

2023-06-13

Illumina, Inc.

Artificial intelligence-based quality scoring

US11783917B2

( en )

2019-03-21

2023-10-10

Illumina, Inc.

Artificial intelligence-based base calling

… neural networks such as a single-shot-detector convolutional neural network — Inventive Government Solutions, Llc (US20200054306A1)

Related documents