Google Patentsopen access

Systems and methods for acquiring and inspecting lens images of opthalmic … — Coopervision International Limited (US20250022121A1)

ABSTRACT

Abstract

Systems and methods for acquiring and inspecting lens images of ophthalmic lenses using one or more cameras to acquire the images of the lenses in a dry state or a wet state. The images are preprocessed and then inputted into an artificial intelligence network, such as a convolutional neural network (CNN), to analyze and characterize for type of lens defects. The artificial intelligence network identifies defect regions on the images and output defect categories or classifications for each of the images based in part on the defect regions.

Description

This application is a continuation of U.S. patent application Ser. No. 17/713,978, filed Apr. 5, 2022, which claims the benefit of U.S. Provisional Patent Application Ser. No. 63/226,419, filed Jul. 28, 2021, the contents of which are hereby expressly incorporated herein by reference in their entirety.

FIELD OF ART

The present disclosure is generally related to lens inspection systems and more particularly to lens inspection systems that use artificial intelligence to evaluate images of ophthalmic lenses and classify the images according to different lens defect categories or classes.

BACKGROUND

Artificial intelligence (AI), such as machine learning (ML), has demonstrated significant success in improving inspection accuracy, speed of image characterization or classification, and image interpretation for a wide range of tasks and applications. Machine learning is being used in almost every type of industry. It is helping people to minimize their workload as machines are capable of executing most of the human tasks with high performance. Machines can do predictive analysis such as classification & regression (predicting numerical values) and tasks like driving car which require intelligence and dynamic interpretation.

Machine learning involves providing data to the machine so that it can learn patterns from the data, and it can then predict solutions for similar future problems. Computer vision is a field of Artificial Intelligence which focuses on tasks related to images. Deep learning combined with computer vision is capable of performing complex operations ranging from classifying images to solving scientific problems of astronomy and building self-driving cars.

However, many deep learning networks are unable to process images with sufficient accuracy or trained with proper parameters to warrant being relied upon in a high-speed large-scale setting. In still other settings, deep learning routines may not be sensitive enough to distinguish between regions in an image or adapted with proper parameters to implement in a particular large scale manufacturing operation.

SUMMARY

Aspects of the invention comprise systems and methods for lens inspection and outputting defect classes that are representative of the different defects found on one or more images, using artificial intelligence (AI) models. In exemplary embodiments, systems and methods are provided in which a lens edge image and a lens surface image for each lens or a plurality of ophthalmic or contact lenses to be inspected by the lens inspection system of the present invention are separated into edge image datasets and surface image datasets. The two different datasets are processed by two different AI models to predict defects, if any, that are captured on the images and then outputting the defects based on each defect's class or type.

Exemplary embodiments include using convolutional neural networks (CNNs) as the AI models to analyze and classify contact lens images. Preferred CNN models include the VGG16 Net and VGG19 Net models, which can be re-trained to analyze and classify lens defect classes based on images of the lenses. While using an “edge” AI model to analyze and classify lens edge images and using a “surface” AI model to analyze and classify lens surface images are preferred, aspects of the invention contemplate training the same AI model to analyze and predict defects for both the lens edge and the lens surface on the same image. For example, an imaging system can have a large depth of field, and a large aperture or large f-number, and can capture an image with both the lens edge and lens surface in focus, thus allowing for a single AI model to analyze and predict lens edge defects, lens surface defects, or both defect types within a same image.

A further aspect of the invention includes the preparation of input ready images based on image templates. If a dataset contains images taken from eight cameras, as an example, eight image templates can be created to account for variations in the distance of the camera and the focal length of each camera. Training images and production images can be normalized, such as resized, based on the templates used to locate the lens region inside the raw image. In some examples, the training and the production images can be resized without the use of templates.

In yet other aspects of the invention, intermediate activations of the different convolutional layers are visualized, such as outputted as images, known as class activation maps (CAMs). A CAM image shows different regions of the image that have been used by the model to influence or contribute to the final output of the model. During training, CAMs can be used by the trainer to evaluate the performance of the model and, if necessary, to make corrections to the model, such as to lock and unlock additional layers, use additional training images, etc. Images generated can be provided as heatmaps and can be annotated with bounding boxes to more readily convey the regions of interest.

Other aspects of the invention include a method for inspecting ophthalmic lenses and assigning classification to images of the ophthalmic lenses. The method can comprise: a) accessing a first image with a computer system, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) identifying a region on the first image to analyze for at least one lens defect by processing the first image with an artificial intelligence (AI) network implemented with a hardware processor and a memory of the computer system; c) generating a first intermediate activation image based on the first image and outputting the first intermediate activation image with a defect region; d) labelling the defect region on the first intermediate activation image with at least one of a heatmap and a bounding box to define a labeled intermediate activation image; and e) generating and outputting a classification for the first image with the AI network to produce a first classified image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

The defect region of an image of a lens can be represented by a region on the image with distinct pixel intensities from the remaining regions of the image. The defect region can be labeled with a heat map and/or a bounding box.

The AI network or model can be a convolutional neural network (CNN) for image classification, and specifically a VGG Net with other deep neural networks for image processing contemplated, such as LeNet, AlexNet, GoogLeNet/Inception, and ResNet, ZFNet.

The first image of the ophthalmic lens can be acquired when the ophthalmic lens is located on a support in a dry state. In an example, the support can be made from a sapphire material.

The first image of the ophthalmic lens can alternatively be acquired when the ophthalmic lens is in a liquid bath, such as in a transfer tray or a blister pack in a wet state. The liquid bath can be a saline solution.

The plurality of lens surface defect classes can comprise at least two classes, at least three classes, or more than three classes. In other examples, the classes can comprise at least eight classes, which can include a class for a good lens.

The at least two classes for the surface model can comprise a Good Lens class and a Bad Lens. The at least three classes can comprise a Good Lens class, a Bubble class, and a Scratch class. Where eight classes are utilized to classify the lens defects, they can include “Brains”, “Bubble”, “Debris”, “Dirty sapphire”, “Double lens”, “Good lens”, “Scratch”, and “Void”. However, the surface models can be trained on more or fewer than eight surface defect types. For example, the surface models can be trained with images with no lens, with a pinched lens, with a lens that is too small, with a lens that is too large, and/or with a lens that is too eccentric. The number of classes can include any number of combinations of the foregoing listed classes.

The classification outputted by the AI network can be a text, a number, or an alpha-numeric identifier.

The first image can be acquired by a first camera and the first image having a height pixel value and a width pixel value and wherein the height and width pixel values are sized based on a template image acquired by the first camera.

The first image can have a second set of pixel intensities that has been inverted from a first set of pixel intensities.

The first image can have two additional identical images, and the first image and the two additional identical images define a three-channel image.

The first image can have a lens center and wherein the lens center can be defined relative to an upper left corner of the first image.

The first ophthalmic lens represented in the first image can be represented in a polar coordinate system or a cartesian coordinate system.

The polar coordinate system can be converted from the cartesian coordinate system.

The first image can be created by rotating an original image.

The first image can be created by flipping an original image, by zooming in or out of the original image by a small value, by adjusting the light intensity of the original image, or combinations thereof. The flipping can be performed in 90-degrees increment or by a different angular rotation.

The method can further comprise retraining or finetuning the AI network based on information provided by a labeled intermediate activation image.

The first image can be a lens edge image and the method can further comprise accessing a second image with the computer system, the second image being a lens surface image of the first ophthalmic lens.

The second image can be acquired by a second camera and the second image can have a height pixel value and a width pixel value and wherein the height and width pixel values of the second image are sized based on a template image acquired by the second camera.

The second image can have two additional identical images, and the second image and the two additional identical images define a three-channel image.

The second image can have a second set of pixel intensities that has been inverted from a first set of pixel intensities.

The method can further comprise generating a second intermediate activation image based on the second image and outputting the second intermediate activation image with a defect region.

The method can further comprise labelling the defect region on the second intermediate activation image with at least one of a heatmap and a bounding box to define a second labeled intermediate activation image. In some examples, the defect region on an original image, for each of a plurality of images to be analyzed or trained, is labeled, such as with a bounding box or by highlighting the contour of the defect region. The labelled original image may be used for training, for displaying, for marketing, etc.

The second labeled intermediate activation image can be used to re-train or finetune the AI network.

The method can further comprise generating and outputting a classification for the second image with the AI network, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes.

In some embodiments, a plurality of lens edge defect classes can comprise at least two classes, at least three classes, or more than three classes. In other examples, the classes can comprise at least eight classes, which can include a class for a good lens.

The method can further comprise comparing the classification for the first classified image against a preprocessed image or an original image of the first image that has been manually examined and identified as ground truth and from which the first classified image is generated. The manual examination of the preprocessed image or original image of the first image classified with one of the plurality of lens surface defect classes, one of the plurality of lens edge defect classes, or both the lens surface defect and lens edge defect classes.

The at least two classes for the edge model can comprise a Good Lens class and a Bad Lens. The at least three classes for the edge model can comprise a Good Lens class, a Bubble class, and a Scratch class. Additional classes or where more than three edge classes are practiced, the edge models can be trained with images with no lens, with an edge split, with a pinched lens, with an edge chip, with a lens that is too small, with a lens that is too large, and/or with a lens that is too eccentric. The number of classes can include any number of combinations of the foregoing listed classes.

A still further aspect of the invention includes a system for classifying lens images of ophthalmic lenses. The system for classifying lens images of ophthalmic lenses can comprise: at least one hardware processor; a memory having stored thereon instructions that when executed by the at least one hardware processor cause the at least one hardware processor to perform steps comprising: a) accessing a first image from the memory, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) accessing a trained convolutional neural network (CNN) from the memory, the trained CNN having been trained on lens images of ophthalmic lenses in which each of the ophthalmic lenses is either a good lens or has at least one lens defect; c) generating an intermediate activation image based on the first image and outputting the intermediate activation image with a defect region; d) labelling the defect region on the intermediate activation image with at least one of a heatmap and a bounding box to define a labeled intermediate activation image; and e) generating and outputting a classification for the first image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

The intermediate activation image, or CAM, can be taken from an output of the last convolutional layer of the CNN model. The intermediate activation image can be superimposed on the first image prior to labelling the defection region of the intermediate activation image. In an example, the intermediate activation image is the last intermediate image created as the output of the last convolutional layer, which may be referred to as a class activation ma

This application is a continuation of U.S. patent application Ser. No. 17/713,978, filed Apr. 5, 2022, which claims the benefit of U.S. Provisional Patent Application Ser. No. 63/226,419, filed Jul. 28, 2021, the contents of which are hereby expressly incorporated herein by reference in their entirety.

FIELD OF ART

The present disclosure is generally related to lens inspection systems and more particularly to lens inspection systems that use artificial intelligence to evaluate images of ophthalmic lenses and classify the images according to different lens defect categories or classes.

BACKGROUND

Artificial intelligence (AI), such as machine learning (ML), has demonstrated significant success in improving inspection accuracy, speed of image characterization or classification, and image interpretation for a wide range of tasks and applications. Machine learning is being used in almost every type of industry. It is helping people to minimize their workload as machines are capable of executing most of the human tasks with high performance. Machines can do predictive analysis such as classification & regression (predicting numerical values) and tasks like driving car which require intelligence and dynamic interpretation.

Machine learning involves providing data to the machine so that it can learn patterns from the data, and it can then predict solutions for similar future problems. Computer vision is a field of Artificial Intelligence which focuses on tasks related to images. Deep learning combined with computer vision is capable of performing complex operations ranging from classifying images to solving scientific problems of astronomy and building self-driving cars.

However, many deep learning networks are unable to process images with sufficient accuracy or trained with proper parameters to warrant being relied upon in a high-speed large-scale setting. In still other settings, deep learning routines may not be sensitive enough to distinguish between regions in an image or adapted with proper parameters to implement in a particular large scale manufacturing operation.

SUMMARY

Aspects of the invention comprise systems and methods for lens inspection and outputting defect classes that are representative of the different defects found on one or more images, using artificial intelligence (AI) models. In exemplary embodiments, systems and methods are provided in which a lens edge image and a lens surface image for each lens or a plurality of ophthalmic or contact lenses to be inspected by the lens inspection system of the present invention are separated into edge image datasets and surface image datasets. The two different datasets are processed by two different AI models to predict defects, if any, that are captured on the images and then outputting the defects based on each defect's class or type.

Exemplary embodiments include using convolutional neural networks (CNNs) as the AI models to analyze and classify contact lens images. Preferred CNN models include the VGG16 Net and VGG19 Net models, which can be re-trained to analyze and classify lens defect classes based on images of the lenses. While using an “edge” AI model to analyze and classify lens edge images and using a “surface” AI model to analyze and classify lens surface images are preferred, aspects of the invention contemplate training the same AI model to analyze and predict defects for both the lens edge and the lens surface on the same image. For example, an imaging system can have a large depth of field, and a large aperture or large f-number, and can capture an image with both the lens edge and lens surface in focus, thus allowing for a single AI model to analyze and predict lens edge defects, lens surface defects, or both defect types within a same image.

A further aspect of the invention includes the preparation of input ready images based on image templates. If a dataset contains images taken from eight cameras, as an example, eight image templates can be created to account for variations in the distance of the camera and the focal length of each camera. Training images and production images can be normalized, such as resized, based on the templates used to locate the lens region inside the raw image. In some examples, the training and the production images can be resized without the use of templates.

In yet other aspects of the invention, intermediate activations of the different convolutional layers are visualized, such as outputted as images, known as class activation maps (CAMs). A CAM image shows different regions of the image that have been used by the model to influence or contribute to the final output of the model. During training, CAMs can be used by the trainer to evaluate the performance of the model and, if necessary, to make corrections to the model, such as to lock and unlock additional layers, use additional training images, etc. Images generated can be provided as heatmaps and can be annotated with bounding boxes to more readily convey the regions of interest.

Other aspects of the invention include a method for inspecting ophthalmic lenses and assigning classification to images of the ophthalmic lenses. The method can comprise: a) accessing a first image with a computer system, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) identifying a region on the first image to analyze for at least one lens defect by processing the first image with an artificial intelligence (AI) network implemented with a hardware processor and a memory of the computer system; c) generating a first intermediate activation image based on the first image and outputting the first intermediate activation image with a defect region; d) labelling the defect region on the first intermediate activation image with at least one of a heatmap and a bounding box to define a labeled intermediate activation image; and e) generating and outputting a classification for the first image with the AI network to produce a first classified image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

The defect region of an image of a lens can be represented by a region on the image with distinct pixel intensities from the remaining regions of the image. The defect region can be labeled with a heat map and/or a bounding box.

The AI network or model can be a convolutional neural network (CNN) for image classification, and specifically a VGG Net with other deep neural networks for image processing contemplated, such as LeNet, AlexNet, GoogLeNet/Inception, and ResNet, ZFNet.

The first image of the ophthalmic lens can be acquired when the ophthalmic lens is located on a support in a dry state. In an example, the support can be made from a sapphire material.

The first image of the ophthalmic lens can alternatively be acquired when the ophthalmic lens is in a liquid bath, such as in a transfer tray or a blister pack in a wet state. The liquid bath can be a saline solution.

The plurality of lens surface defect classes can comprise at least two classes, at least three classes, or more than three classes. In other examples, the classes can comprise at least eight classes, which can include a class for a good lens.

The at least two classes for the surface model can comprise a Good Lens class and a Bad Lens. The at least three classes can comprise a Good Lens class, a Bubble class, and a Scratch class. Where eight classes are utilized to classify the lens defects, they can include “Brains”, “Bubble”, “Debris”, “Dirty sapphire”, “Double lens”, “Good lens”, “Scratch”, and “Void”. However, the surface models can be trained on more or fewer than eight surface defect types. For example, the surface models can be trained with images with no lens, with a pinched lens, with a lens that is too small, with a lens that is too large, and/or with a lens that is too eccentric. The number of classes can include any number of combinations of the foregoing listed classes.

The classification outputted by the AI network can be a text, a number, or an alpha-numeric identifier.

The first image can be acquired by a first camera and the first image having a height pixel value and a width pixel value and wherein the height and width pixel values are sized based on a template image acquired by the first camera.

The first image can have a second set of pixel intensities that has been inverted from a first set of pixel intensities.

The first image can have two additional identical images, and the first image and the two additional identical images define a three-channel image.

The first image can have a lens center and wherein the lens center can be defined relative to an upper left corner of the first image.

The first ophthalmic lens represented in the first image can be represented in a polar coordinate system or a cartesian coordinate system.

The polar coordinate system can be converted from the cartesian coordinate system.

The first image can be created by rotating an original image.

The first image can be created by flipping an original image, by zooming in or out of the original image by a small value, by adjusting the light intensity of the original image, or combinations thereof. The flipping can be performed in 90-degrees increment or by a different angular rotation.

The method can further comprise retraining or finetuning the AI network based on information provided by a labeled intermediate activation image.

The first image can be a lens edge image and the method can further comprise accessing a second image with the computer system, the second image being a lens surface image of the first ophthalmic lens.

The second image can be acquired by a second camera and the second image can have a height pixel value and a width pixel value and wherein the height and width pixel values of the second image are sized based on a template image acquired by the second camera.

The second image can have two additional identical images, and the second image and the two additional identical images define a three-channel image.

The second image can have a second set of pixel intensities that has been inverted from a first set of pixel intensities.

The method can further comprise generating a second intermediate activation image based on the second image and outputting the second intermediate activation image with a defect region.

The method can further comprise labelling the defect region on the second intermediate activation image with at least one of a heatmap and a bounding box to define a second labeled intermediate activation image. In some examples, the defect region on an original image, for each of a plurality of images to be analyzed or trained, is labeled, such as with a bounding box or by highlighting the contour of the defect region. The labelled original image may be used for training, for displaying, for marketing, etc.

The second labeled intermediate activation image can be used to re-train or finetune the AI network.

The method can further comprise generating and outputting a classification for the second image with the AI network, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes.

In some embodiments, a plurality of lens edge defect classes can comprise at least two classes, at least three classes, or more than three classes. In other examples, the classes can comprise at least eight classes, which can include a class for a good lens.

The method can further comprise comparing the classification for the first classified image against a preprocessed image or an original image of the first image that has been manually examined and identified as ground truth and from which the first classified image is generated. The manual examination of the preprocessed image or original image of the first image classified with one of the plurality of lens surface defect classes, one of the plurality of lens edge defect classes, or both the lens surface defect and lens edge defect classes.

The at least two classes for the edge model can comprise a Good Lens class and a Bad Lens. The at least three classes for the edge model can comprise a Good Lens class, a Bubble class, and a Scratch class. Additional classes or where more than three edge classes are practiced, the edge models can be trained with images with no lens, with an edge split, with a pinched lens, with an edge chip, with a lens that is too small, with a lens that is too large, and/or with a lens that is too eccentric. The number of classes can include any number of combinations of the foregoing listed classes.

A still further aspect of the invention includes a system for classifying lens images of ophthalmic lenses. The system for classifying lens images of ophthalmic lenses can comprise: at least one hardware processor; a memory having stored thereon instructions that when executed by the at least one hardware processor cause the at least one hardware processor to perform steps comprising: a) accessing a first image from the memory, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) accessing a trained convolutional neural network (CNN) from the memory, the trained CNN having been trained on lens images of ophthalmic lenses in which each of the ophthalmic lenses is either a good lens or has at least one lens defect; c) generating an intermediate activation image based on the first image and outputting the intermediate activation image with a defect region; d) labelling the defect region on the intermediate activation image with at least one of a heatmap and a bounding box to define a labeled intermediate activation image; and e) generating and outputting a classification for the first image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

The intermediate activation image, or CAM, can be taken from an output of the last convolutional layer of the CNN model. The intermediate activation image can be superimposed on the first image prior to labelling the defection region of the intermediate activation image. In an example, the intermediate activation image is the last intermediate image created as the output of the last convolutional layer, which may be referred to as a class activation map or CAM.

A further aspect of the invention includes a method for inspecting ophthalmic lenses and assigning classification to images of the ophthalmic lenses comprising: a) accessing a first image with a computer system, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) identifying a region on the first image to analyze for lens defect by processing the first image with an artificial intelligence (AI) network implemented with a hardware processor and a memory of the computer system; c) generating a class activation map (CAM) based on the first image and outputting the CAM with a defect region; d) labelling the defect region on the CAM with at least one of a heatmap and a bounding box to define a labeled CAM; and e) generating and outputting a classification for the first image with the AI network to produce a first classified image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

The first image of the ophthalmic lens can be acquired when the ophthalmic lens is located on a support in a dry state.

The first image of the ophthalmic lens can be acquired when the ophthalmic lens is in a liquid bath in a wet state.

The plurality of lens surface defect classes can comprise at least two classes.

The plurality of lens surface defect classes can comprise at least three classes and the at least three classes comprise a Good Lens class, a Bubble class, and a Scratch class.

The classification outputted by the AI network can comprise a text, a number, or an alpha-numeric identifier.

The first image can be acquired by a first camera and the first image can have a height pixel value and a width pixel value and wherein the height and width pixel values can be sized based on a template image acquired by the first camera.

The first image can have a second set of pixel intensities that can be inverted from a first set of pixel intensities.

The first image can have two additional identical images, and the first image and the two additional identical images can define a three-channel image.

The first image can a lens center and wherein the lens center can be defined relative to an upper left corner of the first image.

The first ophthalmic lens represented in the first image can be represented in a polar coordinate system.

The polar coordinate system can be converted from a cartesian coordinate system.

The first image can have a second set of pixel intensities that has been inverted from a first set of pixel intensities.

The first image can be rotated from an original image.

The first image can be flipped from an original image.

The method can further comprise the steps of retraining or finetuning the AI network based on information provided by the labeled CAM.

The method can further comprise the step of retraining or finetuning the AI network by performing at least one of the following steps: (1) removing fully connected nodes at an end of the AI network where actual class label predictions are made; (2) replacing fully connected nodes with freshly initialized ones; (3) freezing earlier or top convolutional layers in the AI network to ensure that any previous robust features learned by the AI model are not overwritten or discarded; ( 4 ) training only fully connected layers with a certain learning rate; and ( 5 ) unfreezing some or all convolutional layers in the AI network and performing additional training with same or new datasets with a relatively smaller learning rate.

The first image can be a lens edge image of a first ophthalmic lens and a second image accessed by the computer system can be a lens surface image of the first ophthalmic lens.

The second image can be acquired by a second camera and the second image having a height pixel value and a width pixel value and wherein the height and width pixel values of the second image are sized based on a template image acquired by the second camera.

The second image can have two additional identical images, and the second image and the two additional identical images can define a three-channel image.

The second image can have a second set of pixel intensities that has been inverted from a first set of pixel intensities.

The CAM can be a first CAM and can further comprise generating a second CAM based on the second image and outputting the second CAM with a defect region.

The method can further comprise labelling the defect region on the second CAM with at least one of a heatmap and a bounding box to define a second labeled CAM.

The second labeled CAM can be used to re-train or finetune the AI network.

The method can further comprise generating and outputting a classification for the second image with the AI network, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes.

The plurality of lens surface defect classes can comprise at least three classes.

The at least three classes can comprise a Good Lens class, a Bubble class, and a Scratch class.

The method can further comprise a step of classifying lens surface defects, or lens edge defects, or both the lens surface defects and lens edge defects to generate the lens surface defect classes, the lens edge defect classes, or both.

The step of classifying the lens can be performed before the accessing step.

The step of classifying the lens can be performed manually.

The first image can be labeled with a bounding box around a region of interest, wherein the bounding box around the region of interest on the first image can be based on the labeled intermediate activation image.

The CAM can be computed based on the output of the last convolutional layer.

The first image can be a preprocessed image and wherein the CAM is extrapolated and superimposed over the preprocessed first image.

A further aspect of the invention is a system for classifying lens images of ophthalmic lenses comprising: at least one hardware processor; a memory having stored thereon instructions that when executed by the at least one hardware processor cause the at least one hardware processor to perform steps comprising: a) accessing a first image from the memory, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) accessing a trained convolutional neural network (CNN) from the memory, the trained CNN having been trained on lens images of ophthalmic lenses in which each of the ophthalmic lenses is either a good lens or has at least one lens defect; c) generating class activation map (CAM) based on the first image and outputting the CAM with a defect region; d) labelling the defect region on the CAM with at least one of a heatmap and a bounding box to define a labeled CAM; and e) generating and outputting a classification for the first image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

Yet another aspect of the invention is a method for inspecting ophthalmic lenses and assigning classification to images of the ophthalmic lenses comprising: a) identifying a region on a first image to analyze for lens defect by processing the first image with an artificial intelligence (AI) network implemented with a hardware processor; b) generating a CAM based on the first image and outputting the CAM with a defect region; c) labelling the defect region on the CAM with at least one of a heatmap and a bounding box to define a labeled CAM; d) generating and outputting a classification for the first image with the AI network, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes; e) identifying a region on a second image to analyze for lens defect by processing the second image with the artificial intelligence (AI) network implemented with the hardware processor.

The AI network can reside on the Cloud, on a computer system having a storage memory with the first image and the second image stored thereon, or on a computer system having a storage memory not having the first image and the second image stored thereon.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the present devices, systems, and methods will become appreciated as the same becomes better understood with reference to the specification, claims and appended drawings wherein:

FIG. 1 is a schematic for one configuration of a lens inspection system in accordance with aspects of the invention.

FIG. 2 is another schematic for one configuration of the present disclosure using a computer system.

FIG. 3 is a flowchart depicting one configuration of the disclosure for identifying defects in lenses and outputting class of defects and optionally labeled intermediate activation images.

FIG. 4 A shows an exemplary image augmentation by vertically flipping an original image to generate two images; and FIG. 4 B shows an exemplary image augmentation by horizontally flipping the original image to generate a third image from the original image.

FIG. 5 is a flowchart depicting one configuration of the disclosure for generating a template from which training datasets, validating datasets, and production datasets can be created.

FIG. 6 is a flowchart depicting one configuration of preprocessing a lens surface image to generate an input ready image for the computer system.

FIG. 7 is a flowchart depicting one configuration of preprocessing a lens edge image to generate an input ready image for the computer system.

FIG. 8 shows an image depicting the lens in a cartesian coordinate system and converting the image to a polar coordinate system.

FIG. 9 shows an image in the polar coordinate system being cropped to obtain an image to the inside and outside of the lens edge.

FIG. 10 shows the cropped image of FIG. 9 being inverted from a first set of image intensities to a second set of image intensities.

FIG. 11 is a schematic of an AI network, and more particularly a convolutional neural network (CNN).

FIG. 12 shows conceptual blocks of a “surface” model of the CNN along with the input and output shapes of each layer.

FIG. 13 shows conceptual blocks of an “edge” model along with the input and output shapes of each layer.

FIG. 14 shows exemplary settings used for transfer training of a pre-trained CNN model.

FIG. 14 A is a flowchart depicting an exemplary transfer training protocol of CNN models of the invention.

FIG. 15 shows how accuracy changes over range of epochs for both training datasets and validation datasets.

FIG. 16 shows graphs depicting the output of the loss function over the training epochs for training and validation datasets.

FIGS. 17 - 24 are images representative of eight different classes or categories of lens surface defects, which broadly cover physical defects as well as good lens.

FIGS. 25 - 27 are images representative of three different classes or categories of lens edge defects, which broadly cover physical defects as well as good lens.

FIG. 28 shows tabulations in a table format for prediction versus ground truth on the training dataset.

FIG. 29 shows data tabulated in a table format to show performance metrics of the surface model of the present invention on the training dataset.

FIG. 30 shows tabulations in a table format for predictions versus ground truth on the training dataset for the “edge” model of the present invention.

FIG. 31 shows data tabulated in a table format to show performance metrics of the edge model of the present invention on the training dataset.

FIG. 32 shows conceptual blocks of an “edge” model along with the output shapes of each layer using an alternative CNN and FIG. 33 shows a dropout layer added to the model.

FIG. 34 shows an additional convolutional layer added to the model of FIG. 12 without a dropout layer.

FIG. 35 shows three images as examples of the output of the lens region cropping step of the pre-processing step.

FIGS. 36 A, 36 B, and 36 C show three different sets of 64 channel-output of the first convolution operation of the first convolutional block of the CNN model of the invention for the three input images of FIG. 35 .

FIG. 37 shows enlarged images of one of the 64 channels for each of the three output examples of FIGS. 36 A- 36 C .

FIG. 38 shows a CAM of the final prediction class superimposed on a preprocessed image for a “Bubble” surface defect image.

FIG. 39 shows a CAM of the final prediction class superimposed on a preprocessed image for a “Scratch” surface defect image.

FIGS. 40 A and 40 B show CAM images for the “Scratch” category for the same image, but under different training protocols.

FIG. 41 (A) shows a preprocessed image having a bounding box and a CAM of the final prediction class superimposed on the preprocessed image with a bounding box, and FIG. 41 (B) shows a preprocessed image having two bounding boxes and a CAM of the final prediction class superimposed on the preprocessed image with bounding boxes.

FIG. 42 is an example of a lens surface image showing lens with a “Debris”.

FIG. 43 is an exemplary output of the lens inspection system in a table format showing probabilities of different defect classes for the lens surface image of FIG. 42 .

DETAILED DESCRIPTION

The detailed description set forth below in connection with the appended drawings is intended as a description of the presently preferred embodiments of lens inspection systems provided in accordance with aspects of the present devices, systems, and methods and is not intended to represent the only forms in which the present devices, systems, and methods may be constructed or utilized. The description sets forth the features and the steps for constructing and using the embodiments of the present devices, systems, and methods in connection with the illustrated embodiments. It is to be understood, however, that the same or equivalent functions and structures may be accomplished by different embodiments that are also intended to be encompassed within the spirit and scope of the present disclosure. As denoted elsewhere herein, like element numbers are intended to indicate like or similar elements or features.

With reference now to FIG. 1 , a schematic diagram illustrating a lens inspection system 100 is shown, which can be used to automatically inspect lenses, such as contact lenses or other ophthalmic lenses, by inspecting images of the lenses and outputting classes or categories of defects using artificial intelligence (AI) and machine learning, such as convolutional neural networks (CNNs). The lens inspection system can be referred to as a system for classifying lens images of ophthalmic lenses. The lens inspection system 100 is structured and equipped with hardware and software to inspect lenses in their dry, or non-hydrated, state, and outputting classes or categories of defects from images acquired on the lenses in their dry state. For example, a lens can be separated from a mold having a male mold part and a female mold part and then inspected in its dry state prior to cleaning and/or hydrating the lens. In other examples, as further discussed below, the lens can be inspected while placed in a blister package in a solution, such as when the lens is in a wet state.

In the illustrated embodiment of the invention, the lens inspection system 100 inspects each lens after the lens has been removed from a lens mold and placed onto a support, such as onto a sapphire crystal support structure. Lenses passed through the lens inspection system 100 are each inspected by taking images of the lens, such as by imaging both the lens edge and the lens surface of each lens, and processing the images using CNNs loaded on computers of the lens inspection system. The lens inspection system uses CNNs to characterize defect characteristics of the lens, if any, to aid manufacturers in understanding failure modes of the lens, such as defect due to the presence of bubbles, scratches, debris, etc., as opposed to only pass/fail, which allows the manufacture to improve manufacturing processes to obtain higher yield. Contact lens that can be imaged and analyzed using the lens inspection system of the present invention include both soft silicone hydrogel contact lenses and conventional hydrogel contact lens

The lens inspection system 100 generally comprises an image acquisition subsystem 102 and an image analysis subsystem 104 that operates the CNN software, as further discussed below. The image acquisition subsystem 102 comprises one or more inspection heads 108 , each with a plurality of image acquisition devices 106 , which may be any number of high-resolution digital monochrome cameras, such as the Basler Scout GigE scA1390-17gm digital camera with other commercially available digital cameras with sufficient resolution and processing speed being usable. Each camera 106 may be used with a fixed focal length camera lens to acquire a desired field of view from a mounted or fixed working distance with reference to a focal plane. An exemplary camera lens can include the Linos 35 mm/1.6 or 35 mm/1.8 lens with other commercially available fixed or variable lenses with sufficient focal lengths and f-stops being usable. In some examples, the lens can be a liquid lens that contains small cells containing optical ray liquid that changes shape when a voltage is applied, thus allowing fast electronic focusing that can change the focal lengths of the lens and therefore the depth of field and the focus point of the lens, thus enabling the same camera to capture a lens edge image and a lens surface image. Generally speaking, any lens and camera combination that produces a high-quality image of the lens edge, the lens surface, or both the lens edge and lens surface can be adopted with the system of the present invention.

In the illustrated lens inspection system 100 of the invention, the image acquisition subsystem 102 comprises four groups of inspection assemblies or inspection heads 108 . Each inspection head 108 can have one or more illumination devices 110 or light sources, one or more image acquisition devices or cameras 106 , one or more supports 114 , one for each contact lens 116 to be inspected, and at least one operating computer system 118 . In an embodiment, each inspection head 108 of the lens inspection system comprises an illumination device 110 and four image acquisition devices 106 , which can be a camera. The illumination device 110 can comprise a single housing with a plurality of LEDs sized and structured to emit working light with the one or more cameras 106 . Alternatively, the illumination device 110 can embody separate housings each with a sufficient number of LEDs to provide sufficient light for the camera 106 paired with the separated housing. Each combination of illumination device 110 and image acquisition device 106 is arranged such that the camera is in optical alignment with the paired light source. In other words, light emitted from a particular light source projects to a lens of a paired camera.

The LEDs of the illumination device 110 are operably connected to a light controller 122 so that the LEDs receive a signal, such as a current pulse, from the light controller to activate when a lens is to be imaged. In an example, the LEDs may emit NIR light at a peak wavelength of approximately 880 nm with other frequencies contemplated. In the example shown, one light controller can be programmed to operatively control illumination devices 110 of two inspection heads 108 . Optionally, each illumination device can be paired with its own controller for controlling the functions of the illumination device.

The contact lens support 114 is structured to hold one or more lenses 116 to be inspected by the lens inspection system 100 . In an example, a demolding table (not shown) is provided with a surface that rotates about an axis and the supports 114 are located on the table and the supports are rotatable by the table for imaging and cycling the lenses through the lens inspection system. As the demolding table rotates, each of the lenses 116 to be inspected and supported by the supports 114 passes between a camera 106 and a light source 110 to permit the camera to obtain an image of the lens to be inspected. In alternative embodiments, the lenses are inspected while each is still attached to a male mold part or a female mold part, before delensing. Thus, as the demolding table rotates so that a lens passes under a camera, the camera can be configured to capture an image of the lens while the lens is still attached to a male mold part or a female mold part.

In an example, each of the four inspection heads 108 has four cameras 106 with other number of cameras and corresponding light sources contemplated. The four inspection heads 108 can be designated as camera unit 1 (CU1), camera unit 2 (CU2), camera unit 3 (CU3), and camera unit 4 (CU4). The four cameras of each camera unit can be designated as camera 1 (C1), camera 2 (C2), camera 3 (C3), and camera 4 (C4). The cameras 112 can be staggered to focus on different field of views so that a camera within an inspection head, such as C1 of CU1, can focus on the lens edge of a lens while another camera, such as C2 of CU1, can focus on the lens surface of a second lens.

In an example, camera units CU1 and CU2 can be arranged with cameras to image similar sequence of lens profile and camera units CU3 and CU4 can be arranged with cameras to image similar sequence of lens profile so that the lens inspection system 100 can capture two sets of eight images (i.e., CU1 and CU2 represents a first set of eight images and CU3 and CU4 represents a second set of eight images) per imaging cycle.

As shown, each of camera units CU1 and CU2 is arranged to capture a lens edge image with camera C1, a lens surface image with camera C2, a lens edge image with camera C3, and a lens surface image with camera C4 of four different lenses. Then after the eight lenses are imaged by camera units CU1 and CU2, the lenses located on the supports are indexed to camera units CU3 and CU4, which have cameras 106 that are staggered to take the other one of the lens edge image or lens surface image of the same eight lenses captured by the camera units CU1 and CU2. Thus, each of camera unit CU3 and CU4 is arranged to capture a lens

CLAIMS

Claims ( 20 )

What is claimed is:

1 . A method for inspecting ophthalmic lenses and assigning classification to images of the ophthalmic lenses comprising:

a) accessing a first image with a computer system, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) identifying a region on the first image to analyze for lens defect by processing the first image with an artificial intelligence (AI) network implemented with a hardware processor and a memory of the computer system; c) generating a class activation map (CAM) based on the first image and outputting the CAM with a defect region; d) labelling the defect region on the CAM with at least one of a heatmap and a bounding box to define a labeled CAM; and e) generating and outputting a classification for the first image with the AI network to produce a first classified image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

2 . The method of clause 1 , wherein the first image of the ophthalmic lens is acquired when the ophthalmic lens is located on a support in a dry state or when the ophthalmic lens is in a liquid bath in a wet state.

3 . The method of clause 1 or clause 2 , wherein the plurality of lens surface defect classes comprise at least two classes.

4 . The method of clause 3 , wherein the plurality of lens surface defect classes comprise at least three classes and the at least three classes comprise a Good Lens class, a Bubble class, and a Scratch class.

5 . The method of any preceding clause, wherein the first image is acquired by a first camera and the first image having a height pixel value and a width pixel value and wherein the height and width pixel values are sized based on a template image acquired by the first camera.

6 . The method of clause 5 , wherein the first ophthalmic lens represented in the first image is represented in a polar coordinate system.

7 . The method of clause 6 , wherein the polar coordinate system has been converted from a cartesian coordinate system.

8 . The method of clause 6 , wherein the first image has a second set of pixel intensities that has been inverted from a first set of pixel intensities.

9 . The method of any preceding clause, further comprising retraining or finetuning the AI network based on information provided by the labeled CAM.

10 . The method of clause 9 , further comprising retraining or finetuning the AI network by performing at least one of the following steps: (1) removing fully connected nodes at an end of the AI network where actual class label predictions are made; (2) replacing fully connected nodes with freshly initialized ones; (3) freezing earlier or top convolutional layers in the AI network to ensure that any previous robust features learned by the AI model are not overwritten or discarded; (4) training only fully connected layers with a certain learning rate; and (5) unfreezing some or all convolutional layers in the AI network and performing additional training with same or new datasets with a relatively smaller learning rate.

11 . The method of any preceding clause, wherein the first image is a lens edge image and further comprising accessing a second image with the computer system, the second image being a lens surface image of the first ophthalmic lens.

12 . The method of clause 11 , wherein the second image is acquired by a second camera and the second image having a height pixel value and a width pixel value and wherein the height and width pixel values of the second image are sized based on a template image acquired by the second camera.

13 . The method of any preceding clause, further comprising a step of classifying lens surface defects, or lens edge defects, or both the lens surface defects and lens edge defects to generate the lens surface defect classes, the lens edge defect classes, or both.

14 . The method of clause 13 , wherein the step of classifying the lens is performed before the accessing step.

15 . The method of any preceding clause, wherein the CAM is computed based on the output of the last convolutional layer.

16 . The method of clause 15 , wherein the first image is a preprocessed image and wherein the CAM is extrapolated and superimposed over the preprocessed first image.

17 . A system for classifying lens images of ophthalmic lenses comprising:

at least one hardware processor; a memory having stored thereon instructions that when executed by the at least one hardware processor cause the at least one hardware processor to perform steps comprising: a) accessing a first image from the memory, the first image comprising a lens edge image or a lens surface image of a first ophthalmic lens; b) accessing a trained convolutional neural network (CNN) from the memory, the trained CNN having been trained on lens images of ophthalmic lenses in which each of the ophthalmic lenses is either a good lens or has at least one lens defect; c) generating class activation map (CAM) based on the first image and outputting the CAM with a defect region; d) labelling the defect region on the CAM with at least one of a heatmap and a bounding box to define a labeled CAM; and e) generating and outputting a classification for the first image, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes.

18 . The system of clause 17 , wherein the first image is labeled with a bounding box around a region of interest, wherein the bounding box around the region of interest on the first image is based on the labeled CAM.

19 . A method for inspecting ophthalmic lenses and assigning classification to images of the ophthalmic lenses comprising:

a) identifying a region on a first image to analyze for lens defect by processing the first image with an artificial intelligence (AI) network implemented with a hardware processor; b) generating a CAM based on the first image and outputting the CAM with a defect region; c) labelling the defect region on the CAM with at least one of a heatmap and a bounding box to define a labeled CAM; d) generating and outputting a classification for the first image with the AI network, the classification based at least in part on the defect region, the classification being one of a plurality of lens surface defect classes or one of a plurality of lens edge defect classes; e) identifying a region on a second image to analyze for lens defect by processing the second image with the artificial intelligence (AI) network implemented with the hardware processor.

20 . The method of clause 19 , wherein the AI network resides on the Cloud, on a computer system having a storage memory with the first image and the second image stored thereon, or on a computer system having a storage memory not having the first image and the second image stored thereon.

US18/742,540

2021-07-28

2024-06-13

Systems and methods for acquiring and inspecting lens images of opthalmic lenses

Abandoned

US20250022121A1

( en )

Priority Applications (1)

Application Number

Priority Date

Filing Date

Title

US18/742,540

US20250022121A1

( en )

2021-07-28

2024-06-13

Systems and methods for acquiring and inspecting lens images of opthalmic lenses

Applications Claiming Priority (3)

Application Number

Priority Date

Filing Date

Title

US202163226419P

2021-07-28

2021-07-28

US17/713,978

US12051191B2

( en )

2021-07-28

2022-04-05

Systems and methods for acquiring and inspecting lens images of ophthalmic lenses

US18/742,540

US20250022121A1

( en )

2021-07-28

2024-06-13

Systems and methods for acquiring and inspecting lens images of opthalmic lenses

Related Parent Applications (1)

Application Number

Title

Priority Date

Filing Date

US17/713,978

Continuation

US12051191B2

( en )

2021-07-28

2022-04-05

Systems and methods for acquiring and inspecting lens images of ophthalmic lenses

Publications (1)

Publication Number

Publication Date

US20250022121A1

true

US20250022121A1 ( en )

2025-01-16

Family

ID=81392713

Family Applications (2)

Application Number

Title

Priority Date

Filing Date

US17/713,978

Active

2043-02-18

US12051191B2

( en )

2021-07-28

2022-04-05

Systems and methods for acquiring and inspecting lens images of ophthalmic lenses

US18/742,540

Abandoned

US20250022121A1

( en )

2021-07-28

2024-06-13

Systems and methods for acquiring and inspecting lens images of opthalmic lenses

Family Applications Before (1)

Application Number

Title

Priority Date

Filing Date

US17/713,978

Active

2043-02-18

US12051191B2

( en )

2021-07-28

2022-04-05

Systems and methods for acquiring and inspecting lens images of ophthalmic lenses

Country Status (12)

Country

Link

US

( 2 )

US12051191B2

( en )

EP

( 1 )

EP4352699B1

( en )

JP

( 1 )

JP7602093B2

( en )

KR

( 1 )

KR102676508B1

( en )

CN

( 1 )

CN117677986A

( en )

AU

( 2 )

AU2022318275B2

( en )

CA

( 1 )

CA3226780A1

( en )

GB

( 2 )

GB2629055B

( en )

HU

( 1 )

HUE069743T2

( en )

MX

( 1 )

MX2024001307A

( en )

MY

( 1 )

MY206934A

( en )

WO

( 1 )

WO2023007110A1

( en )

Families Citing this family (15)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

KR20230131795A

( en )

*

2022-03-07

2023-09-14

이미지 에이아이 피티이 리미티드

System and method for detecting defects using ir wavelength for dry ophthalmic lenses

US20230400714A1

( en )

*

2022-06-08

2023-12-14

Johnson & Johnson Vision Care, Inc.

Methods for quality control of contact lenses

KR20240062454A

( en )

*

2022-11-01

2024-05-09

삼성전자주식회사

Method and apparatus for extracting the region of interest

US20240256894A1

( en )

*

2023-02-01

2024-08-01

International Business Machines Corporation

Reprogrammable federated learning

CN116381934B

( en )

*

2023-03-24

2025-09-05

华中科技大学

Liquid lens performance optimization method and system

DE102023111989A1

( en )

*

2023-05-08

2024-11-14

Isra Vision Gmbh

inspection device for a surface coating

CN116993682B

( en )

*

2023-07-10

2024-02-23

欧几里德(苏州)医疗科技有限公司

Cornea shaping mirror flaw area extraction method based on image data analysis

CN117197550A

( en )

*

2023-08-29

2023-12-08

合肥市商巨智能装备有限公司

VR lens defect detection method and system based on image cube and deep learning

CN117197101B

( en )

*

2023-09-18

2024-10-18

湖南大学

A defect detection method, computing device and storage medium

DE102023126786B3

( en )

2023-10-02

2025-03-06

Schneider Gmbh & Co. Kg

Method and system for quality control of ophthalmic lenses

KR102758059B1

( en )

*

2024-03-12

2025-02-03

주식회사 인터엑스

Product inspection system capable of artificial intelligence self learning

CN118379525B

( en )

*

2024-06-25

2024-09-03

深圳市微晶光学科技有限公司

Intelligent image processing-based lens defect detection method

WO2026038733A1

( en )

*

2024-08-12

2026-02-19

삼성전자 주식회사

Wearable electronic device equipped with vision correction lens and display correction method using same

CN119540151B

( en )

*

2024-10-28

2025-08-29

无锡鑫巨宏智能科技股份有限公司

A method and system for detecting surface defects of plastic optical lenses

CN119125008B

( en )

*

2024-11-13

2025-02-14

吉林瑞尔康光学科技有限公司

An artificial intelligence-based method for detecting defects in contact lens production

Family Cites Families (28)

* Cited by examiner, † Cited by third party

Publication number

Priority date

Publication date

Assignee

Title

JP3734512B2

( en )

1993-12-27

2006-01-11

株式会社メニコン

Contact lens appearance inspection method and appearance inspection apparatus

US6765661B2

( en )

2001-03-09

2004-07-20

Novartis Ag

Lens inspection

US7256881B2

( en )

*

2002-02-15

2007-08-14

Coopervision, Inc.

Systems and methods for inspection of ophthalmic lenses

US7330579B2

( en )

*

2002-11-13

2008-02-12

Johnson & Johnson Vision Care, Inc.

Automated inspection of tinted ophthalmic parts

GB0229577D0

( en )

2002-12-19

2003-01-22

Bausch & Lomb

Contact lens manufacture

US7477366B2

( en )

2006-12-07

2009-01-13

Coopervision International Holding Company, Lp

Contact lens blister packages and methods for automated inspection of hydrated contact lenses

JP5009663B2

( en )

*

2007-03-30

2012-08-22

富士フイルム株式会社

Appearance inspection system

US7990531B2

( en )

2008-06-05

2011-08-02

Coopervision International Holding Company, Lp

Multi-imaging automated inspection methods and systems for wet ophthalmic lenses

SG164298A1

( en )

2009-02-24

2010-09-29

Visionxtreme Pte Ltd

Object inspection system

SG173233A1

( en )

2010-01-28

2011-08-29

Visionxtreme Pte Ltd

Inspection of defects in a contact lens

SG10201801793YA

( en )

2013-09-11

2018-04-27

Novartis Ag

Contact lens inspection system and method

GB2532056B

( en )

2014-11-07

2019-04-24

Shelton Machines Ltd

Apparatus and method for inspecting contact lenses

SG10201509497VA

( en )

2015-11-18

2017-06-29

Emage Vision Pte Ltd

Contact lens defect inspection using uv illumination

CN107153072A

( en )

*

2017-06-21

2017-09-12

苏州卡睿知光电科技有限公司

A kind of eyeglass flaw inspection method and device

CN109685756A

( en )

2017-10-16

2019-04-26

乐达创意科技有限公司

Image feature automatic identifier, system and method

JP7135504B2

( en )

*

2018-07-02

2022-09-13

カシオ計算機株式会社

Image identification device, image identification method and program

KR102148817B1

( en )

2018-12-14

2020-08-28

한국광기술원

System and method for detecting bad contact lens

JP7137487B2

( en )

2019-01-22

2022-09-14

株式会社日立ハイテク

Image evaluation device and method

JP7370759B2

( en )

*

2019-08-08

2023-10-30

キヤノン株式会社

Image processing device, image processing method and program

CN110648310B

( en )

2019-08-30

2023-08-04

上海理工大学

Weakly Supervised Casting Defect Recognition Method Based on Attention Mechanism

CN111062961A

( en )

*

2019-12-30

2020-04-24

天津大学

Contact lens edge defect detection method based on deep learning

CN111292228B

( en )

*

2020-01-16

2023-08-11

宁波舜宇仪器有限公司

Lens defect detection method

SG10202001656VA

( en )

*

2020-02-25

2021-09-29

Emage Ai Pte Ltd

A computer implemented process to enhance edge defect detection and other defects in ophthalmic lenses

CN111239162A

( en )

*

2020-03-12

2020-06-05

深圳市吉升龙电子有限公司

A fully automatic spectacle lens testing equipment

CN111583187B

( en )

2020-04-14

2023-11-07

佛山市南海区广工大数控装备协同创新研究院

A PCB circuit board defect detection method based on CNN visualization

CN111507960A

( en )

2020-04-16

2020-08-07

重庆三峡学院

Camera module lens contamination detection method based on neural network visualization

CN111667455B

( en )

*

2020-04-28

2023-10-24

广东三三智能科技有限公司

An AI detection method for various defects of brushes

CN112767365A

( en )

2021-01-23

2021-05-07

道合永盛(北京)科技有限公司

Flaw detection method

2022

2022-04-05

US

US17/713,978

patent/US12051191B2/en

active

Active

2022-04-13

WO

PCT/GB2022/050932

patent/WO2023007110A1/en

not_active

Ceased

2022-04-13

KR

KR1020247000620A

patent/KR102676508B1/en

active

Active

2022-04-13

HU

HUE22719610A

patent/HUE069743T2/en

unknown

2022-04-13

EP

EP22719610.2A

patent/EP4352699B1/en

active

Active

2022-04-13

MY

MYPI2023007559A

patent/MY206934A/en

unknown

2022-04-13

GB

GB2403337.5A

patent/GB2629055B/en

active

Active

2022-04-13

AU

AU2022318275A

patent/AU2022318275B2/en

active

Active

2022-04-13

CA

CA3226780A

patent/CA3226780A1/en

active

Pending

2022-04-13

GB

GB2205494.4A

patent/GB2609699B/en

active

Active

2022-04-13

MX

MX2024001307A

patent/MX2024001307A/en

unknown

2022-04-13

CN

CN202280049826.XA

patent/CN117677986A/en

active

Pending

2022-04-13

JP

JP2024504241A

patent/JP7602093B2/en

active

Active

2024

2024-04-30

AU

AU2024202801A

patent/AU2024202801B2/en

active

Active

2024-06-13

US

US18/742,540

patent/US20250022121A1/en

not_active

Abandoned

Also Published As

Publication number

Publication date

MX2024001307A

( en )

2024-02-14

US12051191B2

( en )

2024-07-30

GB202205494D0

( en )

2022-05-25

GB2609699A

( en )

2023-02-15

GB2609699B

( en )

2024-04-24

US20230033187A1

( en )

2023-02-02

AU2022318275B2

( en )

2024-04-18

GB2629055B

( en )

2025-04-02

AU2024202801B2

( en )

2025-10-09

KR102676508B1

( en )

2024-06-24

KR20240014575A

( en )

2024-02-01

CA3226780A1

( en )

2023-02-02

EP4352699A1

( en )

2024-04-17

MY206934A

( en )

2025-01-20

AU2022318275A1

( en )

2024-01-04

JP2024530432A

( en )

2024-08-21

EP4352699B1

( en )

2024-10-16

JP7602093B2

( en )

2024-12-17

CN117677986A

( en )

2024-03-08

GB2629055A

( en )

2024-10-16

AU2024202801A1

( en )

2024-05-16

HUE069743T2

( en )

2025-04-28

WO2023007110A1

( en )

2023-02-02

GB202403337D0

( en )

2024-04-24

Similar Documents

Publication

Publication Date

Title

AU2024202801B2

( en )

2025-10-09

Acquiring and inspecting images of ophthalmic lenses

JP7760165B2

( en )

2025-10-27

Method and system for performing non-invasive genetic testing using artificial intelligence (AI) models

JP7730152B2

( en )

2025-08-27

Method and system for selecting embryos

Deitsch et al.

2019

Automatic classification of defective photovoltaic module cells in electroluminescence images

US11676365B2

( en )

2023-06-13

Explainable artificial intelligence (AI) based image analytic, automatic damage detection and estimation system

Banus et al.

2021

A deep-learning based solution to automatically control closure and seal of pizza packages

CN117557836A

( en )

2024-02-13

A plant disease early diagnosis method, device, electronic equipment and storage medium

CN120235872B

( en )

2025-08-05

A method and system for intelligent detection of photovoltaic module damage based on drone images

US20240338826A1

( en )

2024-10-10

Method for analyzing digital images

Neelagar et al.

2024

Convolutional neural network based glaucoma detection and classification using optic disc localization

HK40102595A

( en )

2024-06-07

Acquiring and inspecting images of ophthalmic lenses

Srilakshmi et al.

2025

Data augmentation-based diabetic retinopathy classification and grading with the dynamic weighted optimization approach

Sadhana et al.

2024

Swarm-Optimized Deep Convolutional Networks for Precise Detection and Grading of Diabetic Retinopathy

Kumar et al.

2025

Detection of Brain Tumor Using a Deep Learning Approach

Das et al.

2025

Identification of dental diseases using convolutional and regional convolutional neural networks

UmaRani et al.

2025

Efficient Chromosome Abnormal Detection using Yolo-V8 model

Ünal et al.

2025

Surface and Edge Defect Inspections on Metals Using Deep Learning and Optical Camera Data

Shanthini et al.

2024

Automated Detection and Prediction of Brain Tumor using ML

Annamalai et al.

2025

Hybrid CNN Approach for Automated Glaucoma Detection and Severity Grading from Fundus Image

Marques et al.

2025

3D-Aware Multi-View Learning for Defect Classification in the Ceramic Industry: A CBAM-Inspired Architecture

CN119688333A

( en )

2025-03-25

System for detecting and classifying anomalies in rubber products

Rathod-Jadhav et al.

2025

A Deep Learning Approach for Diabetic Retinopathy Classification Using Retinal Images

Musam et al.

2026

Robotics in Modern Manufacturing

Niccolai et al.

2021

Artificial Intelligence for Image Classification and Anomaly Detection in the Food Sorting Industry A Comparative Study

FR3153447A3

( en )

2025-03-28

System for detecting and classifying anomalies in rubber products

Legal Events

Date

Code

Title

Description

2024-11-11

AS

Assignment

Owner name : COOPERVISION INTERNATIONAL LIMITED, UNITED KINGDOM

Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:COOPERVISION MANUFACTURING LIMITED;REEL/FRAME:069197/0682

Effective date : 20220621

Owner name : COOPERVISION MANUFACTURING LIMITED, UNITED KINGDOM

Free format text : ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YOSHIDA, ALEXANDRA-FLORENTINA;MCCULLOCH, KIRSTY;SIGNING DATES FROM 20220421 TO 20220425;REEL/FRAME:069197/0569

2024-12-31

STPP

Information on status: patent application and granting procedure in general

Free format text : NON FINAL ACTION MAILED

2025-07-09

STCB

Information on status: application discontinuation

Free format text : ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

Related documents

• Generating augmented reality experiences utilizing physical objects to … — Meta Platforms Technologies, Llc (US20260037214A1)#653960
• Virtual reality system using multiple force arrays for a solver — DeepMotion, Inc. (US10657696B2)#653961
• Tracking position of device inside-out for virtual reality interactivity — Sony Interactive Entertainment America Llc (US10535153B2)#653962
• Augmented reality system with spatialized audio tied to user manipulated … — Magic Leap, Inc. (US11231770B2)#653963
• Big data-based autonomous flight drone system and autonomous flight method … — Rgblab Co., Ltd. (US20210390867A1)#653964